Advancing Interpretability of CLIP Representations with Concept Surrogate Model
Authors: Nhat Hoang-Xuan
Xiyuan Wei
Wanli Xing
Tianbao Yang
My T. Thai
Venue: NeurIPS 2025 poster ()
Keywords: Surrogate Model
Concept-based explanations
Representation Learning
Interpretability
CLIP
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Contrastive Language-Image Pre-training (CLIP) generates versatile multimodal embeddings for diverse applications, yet the specific information captured within these representations is not fully understood. Current explainability techniques often target specific tasks, overlooking the rich, general semantics inherent in the representations. Our objective is to reveal the concepts encoded in CLIP embeddings by learning a surrogate representation, which is expressed as a linear combination of human-understandable concepts evident in the image. Our method, which we term EXPLAIN-R, introduces a novel approach that leverages CLIP's learned instance-instance similarity to train a surrogate model that faithfully mimics CLIP's behavior. From the trained surrogate, we derive concept scores for each input image; these scores quantify the contribution of each concept and act as the explanation for the representation. Quantitative evaluations on multiple datasets demonstrate our method's superior faithfulness over the baseline. Moreover, a user study confirms that our explanations are perceived as more relevant, complete, and useful. Our work provides a novel approach for interpreting CLIP image representations, enhancing the user interpretability of representations and fostering more trustworthy AI systems.

Advancing Interpretability of CLIP Representations with Concept Surrogate Model
Nhat Hoang-Xuan University of Florida
Xiyuan Wei Texas A&M University
Wanli Xing University of Florida
Tianbao Yang
My T. Thai University of Florida
Texas A&M University
Abstract
Contrastive Language-Image Pre-training (CLIP) generates versatile multimodal embeddings for diverse applications, yet the specific information captured within these representations not fully understood. Current explainability techniques often target specific tasks, overlooking the rich, general semantics inherent in the representations. Our objective is to reveal the concepts encoded in CLIP embeddings by learning surrogate representation, which is expressed as a linear combination of undirssaandtt concepts evident in the image. Our method, which term EXPLAIN-R, introduces novel approach that leverages CLIP's learned instance-instance similarity to train surrogate model that faithfully mimics CLIP's behavior. From the trained surrogate, derive concept scores for each input image; these scores quantify the contribution of each concept and act as the explanation for the representation. Quantitative evaluations on multiple datasets demonstrate our method's 's superior faithfulness over the baseline. Moreover, a user study confirms that our explanations are perceived as more relevant, complete, and useful. Our work provides a novel approach for interpreting CLIP image representations, enhancing the user interpretability of representations and fostering more trustworthy AI systems.
Introduction
The CLIP model [1] exemplifies the success of representation learning, which aims to create general- purpose embeddings applicable to a multitude of downstream tasks. These representations have become integral numerous applications, including text-to-image generation (e.g., Stable Diffu- sion (2)), Large Multimodal Models 45), and open-set object detection 7). The widespread adoption and diverse applications of learned representations like CLIP emphasize the need un- derstand their underlying semantics. Therefore, effective explanation methods are required to characterize the information encoded within these representations. As these learned embeddings are inherently task-agnostic and depend only on the input data, their explanations should therefore describe the information they hold, regardless any specific downstream application.
Explaining general-pune purpose representations presents a significant challenge for traditional explainable AI (XAI) techniques. Methods such as GradCAM 3 LIME 19, and Integrated Gradients (10) are fundamentally designed to explain model predictions for specific tasks or classes. For instance, Grad- CAM utilizes class-specific gradients to generate activation maps, while LIME approximates local decision boundaries. Critically, this inherent focus on task-specific predictions prevents their general- ization to explaining the underlying representations. A more recent line of work [T 12 approaches
"ooresponding author. Email: mythai @cisc.ufl.edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

this challenge using matrix-fiid factorization techniques, such as Von-negative Matrix Factorization (NMF), to find low-rank decompositions of embeddings. While such methods can produce bases that effectively reconstruct the original embeddings, the resulting basis is not inherently interpretable and requires further analysis to discern its meaning.
To address these limitations, we propose EXPLAIN-R SAAPLAAN-Reerreeenattiii designed to generate interpretable conceptual explanations for CLIP image representations, indepen- dent of downstream tasks. By design, EXPLAIN-R discovers buman-understandable concepts from each input image. Our method utilizes these discovered concepts to construct surrogate representa- tion, which is then trained to mimic the behavior of the original CLIP embeddings. Post-traing Post-training, our method computes concept scores that quantify the influence of each discovered concept on an input's representation. Our contributions are as follows:
novel method
We introduce EXPLAIN-E R,, a novel method for learning faithful surrogate representation of CLIP image embeddings, which is formed by linearly combining interpretable concepts, and propose a theoretically
algorithm for its training.
Extensive quantitative experiments across multiple datasets validate the faithfulness of EXPLAIN-R, demonstrating that the surrogate representation accurately preserves the predictive behavior of the original CLIP model.
We establish via user study that EXPLAIN-R -R produces explanations considered relevant to the input image, sufficiently complete to explain the model's capabilities, and useful for overall model comprehension.
2 Related Work
Interpreting CLIP vision encoder. Existing methodologies for explaining CLIP models can be broadly categorized into several distinct lines of work, each adopting a unique approach. Pixel attribution techniques [13 [4] [3] [6] identify input regions influencing model outputs via heatmaps. Being visually accessible, they can explain the "where" but fall short in explaining the "what" (0] For instance, a visual explanation highlighting the poodle in an image does not clarify if the model recognizes "poodle" specifically merely "dog". Mechanistic interpretability methods [17 pursue a different strategy, associating human concepts with internal model components (e.g., neurons, attention heads). Concept Bottleneck Model (20 21 22 23 aims to substitute the opaque representation with an explicit concept layer. However, CBMs typically focus on task-specific predictions rather than explaining the versatile representation that is applicable multiple tasks. In contrast, our method focuses on generating interpretable concept-based explanations for these abstract image representations directly. Furthermore, method can be extended to provide concept explanations for zero-shot tasks without additional training.
18 19
Concept-based representation explanations. One specific line of work aims to find concept basis spanning the representation space using matrix decomposition [12 24 25 These methods typically factorize dataset embeddings via SVD NMF into smaller basis and corresponding coefficient matrices. Although capable of achieving low reconstruction error, the resulting basis vectors are not inherently interpretable by design. Their semantic meaning needs be inferred through subsequent analysis, like visualizing top activating inputs [I]. SpLiCE (26), the most relevant prior work, decomposes CLIP image features into sums of text features using the CLIP text encoder. However, imperfections in the text encoder can lead to spurious or unrelated concepts (as observed in Section 4.2), potentially causing confusion for the user. Our method distinguishes itself from these approaches through its design for inherent interpretability: the discovered concepts are, by construction, explicitly linked to the input image, and do not rely on the text encoder. Furthermore, our proposed
training strategy yields superior faithfulness while preserving the surrogate model's simplicity and interpretability.
3 Methodology
This section details EXPLAIN-R, a novel method for interpreting CLIP image representations through a learned concept surrogate model. EXPLAIN-R constructs a surrogate embedding from a linear combination of interpretable concepts grounded in the image. The process encompasses three key

---

CLIP image
umbeddding
Surrogate embedding
Projection based Emccptt scaring
Image
(/9)
((e)
ex0g
0.3
Claatsi aviite
Captions o
mer bm mns
eylks)
Thp conoppe
saca dyy vepaekk
bvd
.20
6((
crane
bil
Concapts
CUI tuat
Step 1: Cancapt identification
Step 2: Surrogate Learning
Stop 3: Explanation
Figure 1: EXPLAIN-R methodology overview. (1) Concept Identification: Image captions processed into a concept vector C;- (2) Surrogate Learning: The surrogate f. with parameters W.b, is trained to mimic the CLIP image embedding by matching image-to-texx (P..P.) .P.") and text-to-image (not visualized) similarity distributions using KL divergence loss. Symbol X denotes cosine similarity followed by softmax. (3) Explanation: Explanatory concept scores for the representation of the image x; are obtained by projecting the rows of W onto f(c,) and weighing by ct.
steps, visualized in Fig. [ (1) discovery of interpretable concepts from the image dataset build vocabulary C; (2) learning surrogate model that transforms these image-specific concepts into an embedding that maintains similarity with text embeddings; and (3) computation of concept scores to quantify each concept's contribution, thus providing an explanation for the image's CLIP representation.
Notations. Let [(x;; |= 1 an image-text dataset. CLIP's image encoder er(-) and text encoder er(-) yield Lg-normalized representations in Rd For each image x,, we obtain the concept vector C E RCl and form the augmented dataset defined as f(;; W, b) (c, W + b). Here W E IR [Cxx is matrix where rows are the (learned) concept basis vectors, b Id is bias, and (-) is the L2 normalization operator. The surrogate function linearly transforms the concepts into the embedding space, which is then normalized.
{(x,, c₁ t:)]. The surrogate model f R - Id is
3.1 Captions-based concept identification
Explaining generic, task-agnostic image representations with open-ended concepts necessitates concept identification methods that are scalable, task-inde dusk-independent, and yield intuitive, human-centric concepts. Our work is directed towards developing such an approach.
However, existing concept identification paradigms are largely incompatible with achieving these combined objectives. For instance, (i) supervised concept detectors [27
28 29 require labeled data for predefined concepts, limiting their scalability for open-ended concept discovery. (ii) L..d-based concept generation [30 [22] 21 is typically designed to yield task-specifi concepts (e.g.. for class discrimination), which lacks the inkeeeddeeeeeecc required for general representation understanding. (iii) Furthermore, directly employing CLIP's alignment scores B1 23 can produce unintuitive concept associations, as CLIP's training objective aligns images with full sentences or captions rather than prioritizing the descriptive accuracy of individual words concepts in human-lise manner.
To obtain explanations that are intuitive, broadly applicable, and thus more aligned with our objectives. we advocate sourcing concepts from data reflecting typical human image descriptions. Image captions are ideal as they tend to describe diverse range of image features (such as objects, attributes, and actions). This descriptive characteristic means that concepts derived from captions are largely neutral making them well-suited for explaining task-agnoste representations. This caption- based approach also offers benefits such as transparency into how concept scores derived. While human-anan -anootated captions are preferred, those from advanced models
4 (trained to emulate human styles via metrics correlated with human judgment [32 33) offer scalable alternative. Based on these ideas, we formulate the concept representation for image as vector c; E 1IC1 where each component C; quantifies the prominence of the k-th concept from global vocabulary within human-like descriptions of x,, empirically measuring its descriptive likelihood.
3

---

The construction of the vocabulary C' and the concept vectors proceeds systematically. First, we compile an initial, comprehensive vocabulary by extracting all nouns, verbs, and adjectives from the image captions across the dataset. This raw vocabulary is then pruned to create C by removing overly frequent terms, as these are often less discriminative, guided by an adjustable threshold to control the desired concept sparsity (i.e., the average number of active concepts per image). Subsequently, for each image x,, its concept vector component Ci,6 is computed as z/m, where m is the number of captions associated with and is the frequency of the k-th concept (from C*) within those m captions. The resulting concept vector, cú reflect the qualities of an image that are emphasized in typical human descriptions.
3.2 Learning interpretable surrogate representation
Previous works 26 on finding concept- based explanations focus on high-fidelity numerical reconstruction of the embeddings. For example, the baseline SpLiCE attempts to maximize the cosine similarity between the reconstructed and the original embedding. While numerically identical reconstruction would entail perfect faithfulness, achieving this ideal is often infeasible in the presence of interpretability constraints like sparsity. Moreover, given that CLIP trained to capture bimodal relationships, unimodal similarities can behave unintuitively, pointed out in [34]. This makes optimizing for the similarity between an original embedding and its reconstruction a less reliable method to capture CLIP's bimodal behavior.
Our proposed approach alternatively focuses on what is learned with CLIP's contrastive objective. We posit that CLIP's contrastive training, which requires distinguishing an image's 's paired caption from numerous alternatives, leads to a rich similarity distribution over texts, and vice versa. This pattern of similarity to other inputs, we argue, offers insights into CLIP's behavior that previous methods overlooked. Consequently, our surrogate model, f. trained to reproduce this distribution of similarities, without specific downstream task supervision.
We now formally state the surrogate learning problem. The distribution of image-to-text similarities of the i-th image for the original model is defined as:
(xple((xi)Ter(tt)))) (T Σ. exp(er(x,) (t(t))/T)
where the temperature. The similarity distribution for the surrogate is analogously given as:
exp( p/ E. exp( eT((t)/7)
(2)
Likewise, the text-to-image similarity distribution Q5 and Of are obtained by modifying the normal- izing factor in the denominator of (T) and ( to sum over image indices. Faithfulness in a surrogate model necessitates that its output similarity distributions, pf and Q/ closely mirror those of the target model, P and Q5, as these distributions determine predicted probabilities and outputs. To promote such faithfulness, we propose minimizing the Bellbackk-eibler (KL) divergence between these corresponding distribution pairs. This makes the distributions more similar by reducing the information available to distinguish between them [35]. In this case, the KL divergence also has an intuitive interpretation: the term (KI(PE P.),, ), for instance, is the average difference of log probability between P. and P/ weighted by PT. Taking the expectation of the divergence over the dataset, we obtain the final loss function:
E p() + Σ DKz (10) ==1
L(W, b)
2n
We note that this loss function have been used in conjunction with various other mechanisms (weak/strong augmentation, momentum encoders, etc.) for self-supervised learning from scratch [36]. However, our motivation (encouraging faithfulness) and use case (XAI) is completely different, and to the best our ability, the idea of training surrogate by distilling instance-ing instance-instance instance similarity has not

---

Top conoate
Top comppte
Top amsoppie
bad
.25
mieswon (t.)
40
burouatee
.15
cry (t.)
13
Rign
4B
change (n)
tall (s.)
miNnAAN (v.)
.05
didl
vegettation
(n.)
Figure 2: Example images from ImageNet and their top concepts, ranked by CLIP representation concept score et- In of ambiguity, the concepts are annotated with their part of speech. The images are labeled as crane, microwave, and barometer respectively. We visualize the top seven concepts for brevity; surrogate representations contain 15-20 concepts.
been explored for XAI. Finally, note that our choice of a linear surrogate promotes interpretability by allowing computation of mojection-based concept scores in Section 3.4 It is also consistent with assumptions in existing works [12 [Л]. including the baseline [26].
3.3 Algorithm for Surrogate Learning
In this section, we present our algorithm for optimizing 33). As will show, naively using mini-batch optimization for (3) leads to biased gradient. This is common problem for contrastive losses and their variants, which existing works circumvent by either using larger batch size (for example, OpenAl uses batch size of 32,768 on 256 GPUs to train the CLIP model []), using a queue of past samples [37 [3., or using non-contrastive losses [38 (39 Instead, our solution to correct this bias does not require large batch size and it is theoretically motivated.
We will focus only on (P, || P!)) since the procedure for optimizing DKL (Q; ll 2/) can be derived analogously. From the definition of KL divergence, the first part of (3) can be equivalently written as (c.f. Appendix for for more detailed derivation):
ΣΣ
iES jes gle,, er; x,, log e,,
S
1 Exes exp [re.)_er(ts)-[ep er(t,) [8]
where ot, = then given by:
The gradient w.r.t. f is
ΣΣ iES jES gler,,erxx,,e er; er; t,, t, ty,
S
At each iteration, only have access mini-batch of triplets B {x,, C,2 tt)) of batch size B. The obtained mini-batch gradient estimator is simply replacing S with the above equation. However, due the non-linearity of the reciprocal function 1/,, the expectation over B is not equal to the true gradient. Thus the mini-hatch estimator is a biased estimator of the true gradient. To solve this problem, we two moving average sequences u and approximate g(er, er; x,, S) and gff, eT; Ci2 t,. S) respectively:
(x((x))
(e(x)))er(tt) (e((te))
We+1. (1 7) Jiu 71
exp
(5)
B.
KEB.
[(xi))eT(taa) (T(()))
01111 = (1 -22)0.5 +
exp
(6)
KEB.
where 71 12 E (0. are two hyperparameters. Then we can approximate g(el, er; xs. t, S) and g(f,er; t,. using u exp (e) (x;) ez (t,)/r) and 0.41.// exp (f(c,) (iat)Ter(tt (r(tt)/T) respec- tively. The benefit of using the moving average technique is that now we can guarantee that the distance between the estimators and the true values (g(er, (g(er,eT; eT;x,, S) and eT, c,, S)) di- minishes to expectation, instead of remaining at a constant level (11BB) the moving average technique is not used [40 41. We present the pscudocode and full derivation in Appendix B.2
5

---

Concepts 'mixing day
bow]
Concept for "barmmttee
Coroepts for 'sall clock
bunouneee
J830
clock
pottery
.04
clock
.077
barometer
042
wheel
/22
meter
J125
meter
0220
potter
gatage
dhange (n.)
spin
dial
.130
016
hand
chonge (n..
.016
.06
mechumical
mechonical
(a) Image of class "potter's wheel" misclas- sified as "mixing bowl" by CLIP. Note the significant contribution of the bowl con- cept, which is main predictor "mixing bowl" class, as seen in Fig. 6
(b) Class concept scores of the same image for different classes. The contribution of concepts varies in strength depending on the target class.
the
Figure 3: Class concept attribution score for ImageNet images.
3.4 Interpreting representations with projection-basd -aased concept scores
This section introduces a method for attributing the CLIP image representations to their underlying concepts. Standard attribution methods, such SHAP 42 typically rely on the notion of prediction to operate. In contrast, our approach leverages the linearity of the explainer f to efficiently computes concept attribution scores 2 (k)) Building upon this, we further develop cross-modal attribution score Vi (A)) to quantify how concept modulates the similarity between image i and text j.
We begin by expanding the functional form of the surrogate f(c;) a (We; + b), which yields:
f(c)) = ECa.s Wx + bb) b). KEC
where Wa is the column of W corresponding to the embedding of the k-th concept. Equation 7 shows that each concept contributes term Ci We the pre-normalized representation. A term is considered to have a greater influence if it aligns with the direction of the final representation f(c)). We quantify this alignment with the scalar projection of C, W. onto f(c)), which we detine as the initial score: (k) C W f(.)). Note that summing the projection recovers the premnonmalization magnitude, i.e., Exec (k( + b f(c,) = ||We bll. To obtain scores reflecting the relative contribution of each concept while ignoring the bias term, we normalize them by >; (I[We; bll - bT f(.))) -1 We define the final attribution score (k)) as:
v. (k) (,o:(k) A.c., f(e.).
This score (k) quantifies the normalized contribution of the k-th concept, based on the linear relationship between the k-th concept embedding and the final surrogate representation f(c;), such that Σkec (k)
Cross-model concept scores. We adapt this projection-based approach to attribute the cross- modal similarity between the i-th image and the j-th text to individual concepts k. Recall that the text embedding is er (ty), and the similarity is computed as $i, = f(c;) (ty). We aim to understand how each concept & influences the similarity 8ij. Analogous before, we project the contribution vector onto the direction of the text embedding er et(t,) to obtain the unnormalized score (k.)(k) = c, WT (t,)). The final cross-modal attribution score is then:
f(r,))er(tty))
(1,s(k) = Ar, W.
(9)
f(as) T(or))rr(te er(ts) makes the scores 0e (k) sum to sty all
in which the normalization factor At,5 concepts.

---

Ouns concepts:
SpLICE
top concepts:
SpLiCE top concepts:
bied
28
grent blu
microvave (n.)
401
micronae
.06
ctone
countet
08
Counnng machine
107
day (o.)
12
pusaa button
.04
Lxll
stamless
06
Carsiral mmsic
04
bill
wüdiile
microware (v.)
billnguua
X00
peayy
05
Landee]
00
sounds
segetattio
plessonn
untal
02
Ours
SpLiCE sop conoeptss
Ours tog conropptcc
Spl.C top comoeggaac
.44
maunffectuers
18
sand
.12
castle
building
.06
boit
48
sine ally
06
beach
.16
archanocooiiall
.0G
Implant
goology
.03
surdace
.07
belts
.04
sandy
.05
interneting
.02
Ieceoe (v.)
JNG
dhanap key
tay
415
small gu group
.02
2033
Eepair Lit
.04
bullying
Figure 4: Comparison between our concepts and the baseline. Images are from the SUN397 (lower right) and ImageNet (remaining) datasets. The top-left image depicts a crane (bird), while great blue and pheasant are other species of birds. We show only the top seven concepts for each image. Concept scores from method sum one for each image; SpLiCE's scores do not have this property.
4 Experiments
We conduct experiments using the CLIP ViT-B/32 1 model to demonstrate that explanations generated by our method are faithful, interpretable, and useful for users.
Datasets. We use the COCO 2017 43 validation set, Flickr30k 44 SUN397 45
test set, and ImageNet validation set [46] to study the CLIP model's behavior. These datasets cover a variety of image themes, including objects in context, internet images, scene understanding, and object recognition. Details and experiments on more datasets can be found in the Appendix.
Setup. For datasets that contain captions (COCO and Flickr30k), use the captions the text t;- For image-only datasets, we use the BLIP-2 B OPT 2.7B model captions per image. Then, we perform concept identification (Section 3.1 to obtain 15-20 concepts per image. We train the surrogate with A Algorithm| for 150 epochs, with batch size 1024, default temperature r 0.1, and n = 72 0.9. The optimizer used is AdamW with learning rate 10-3 All experiments are performed on a single A100 GPU. For the baseline SpLiCE (26), we use the official implementation linked in the paper. We obtain similar sparsity (average number of concepts per image) to our method by setting the L penalty for each dataset to facilitate fair comparison. The vocabulary used for SpLiCE is based on LAION and has a size of 15,000.
on COCO to obtain 10
4.1 Explanation faithfulness
Emphasizing our focus on post-hoc explanation, we assess the surrogate's faithfulness quantitatively by comparing the predictions made using the surrogate representation against those of the original image embedding. Faithfulness is measured using conventional metrics (e.g., accuracy), with the target model's prediction treated as the ground truth. We focus on zero-shot tasks as opposed to linear probing, since the former relies directly on the image representations and does not depend particular trained probe's behavior. For zero-shot classification, we report the accuracy. For zero-shot retrieval tasks, we report the rsum metrics 47 48 (defined as R@ R @@5 R@ 10) as a concise summary. instead of individual Recall @ (R @K) values. Detailed retrieval metrics and performance metrics the original tasks provided in Appendix A. and A.2
Table 1: Faithfulness of our method and the baseline on zero-shot retrieval and zero-shot classification. Higher values are better. The means and 2o intervals are computed over five runs. We note that the official implementation of SpLiCE is deterministie.
Method
COCO val
Flickr30k
SUN397
ImageNet val
SpLiCE 26
393.18
381.41
59.78
51.33 53.38 + 0.21
Ours
482.94 + 4.08
476.47 + 3.21
62.77 = 0.22
7

---

'Analog clock' Representation Concepts
"Digital clock" Representation Conrepts
dock
dock
66
alarm
0.03
alarm
0.11
0.02
digital
0.09
Sower
electooni
0.05
adove
2.01
radio
0.02
Concgp
Concept
Figure Average representation concept scores for images in ImageNet "Analog clock" and "Digital clock" classes.
As shown in Table our surrogate model consistently outperforms the baseline in faithfulness across diverse datasets and tasks, indicating its effectiveness in replicating the target model's behavior. This success demonstrates that faithful surrogate concept representations can achieved without solely relying on the minimization of reconstruction mean-square error, strategy prevalent in the baseline and other methods 20 [2].
4.2 Qualitative assessment of representation concepts
We visualize the representation concepts generated by our methods and the baseline to evaluate their relevance to the image content. We show several images from ImageNet and SUN397 and their corresponding explanations Fig. 4 with more details and examples in Appendix AA.3 Our method predominantly discovers concepts highly pertinent to the image content. Our explanations suggest that the CLIP image representation attends to both the primary subject and its surrounding context; the top-ranked concept often identifies the main object with high score, while others capture secondary objects or specific attributes of the primary subject. Conversely, the baseline (SpLiCE) sometimes produces concepts that, while potentially related, are not depicted (e.g.. "great blue," "cleaning machine," "geolooyy"), or are entirely unrelated (e.g., "classical music," "cheap key").
Representation Concept score Histograms. Per-image concept scores (exemplified in Fig. can be aggregated to offer insights into how the CLIP model represents classes of images in a dataset. To illustrate, Figure presents presents the aggregated top seven concept scores for two related classes: "analog clock" and "digital clock". The analysis reveals distinct patterns: "analog clock" image embeddings are predominantly characterized by the "clock" concept, whereas "digital clock" images show more uniform concept distribution. Notably, while several top concepts like "alarm", "time", and "watch" are common both, the concepts "digital" and "electronic" effectively distinguish the two classes.
4.3 Class prediction analysis
This section illustrates how class concept scores (vt,)) can be applied to understand model predictions for a specific class, enabling misprediction analysis. To reiterate for clarity in this context, these scores, Uj.j. quantify how each concept from an image contributes to its similarity 8L, with class j. and are defined such that their sum equals st.- This class-specifie nature contrasts with task-agnostic
"polter's 's wheel! Class Concept Altribution
'mixing bowl Class Concept Attribution
pottery
boad (n.)
potter
0.00
0.08
0.0
bord (v.)
0.00
zaale
0.04
plate
0.00
design
0.000
0.025
0.050
0.075
0.000
0.125
0.150
0.175
0.50
0.05
0.10
0.15
0.20
0.25
0.50
0.35
Class
Concept Sccce
Class Concept Scove
Figure 6: Average class concept scores for ImageNet classes "potter's wheel" and "mixing bowl".

---

representation concept scores (t²), which describe the overall image content. Figure 3b analyzes the class concept scores (Us,z) for a specific image; the task-agnostic representation concepts (v;) for this same image are depicted in the rightmost panel of Figure For this image, the "barometer" class prediction is primarily driven by the "barometer" concept (rank 1) followed by "clock" (rank 2), while the "wall clock" class prediction prioritizes "clock" (rank 1) over "barometer" (rank 2). This shows that the contribution individual concepts to Vi varies significantly with the class.
Misprediction analysis. Analysis of class concept scores for mispredicted images can reveal concepts responsible for incorrect classifications. For instance, Figure 3a presents an ImageNet image of class "potter's wheel" that both CLIP and the surrogate misclassified as "mixing bowl" (a bowl used for cooking). The concepts contributing to this prediction are shown alongside the image. To determine the typical concepts used for each class, aggregated the class concept scores V..j for both "potter's 's wheel" and "mixing bowl", visualizing the results in Figure 6 This combined information suggests that the "bowl" concept is the primary driver of the misclassification. Specifically, the aggregation indicates that "bowl" is, on average, the dominant concept for predicting "mixing bowl", while it is notably absent from the top concepts associated with "potter's wheel".
4.4 User study
We performed a small scale user study to evaluate the interpretability of our explanations (results in Fig. 7). largely following the protocol of (26) but with modified criteria. Users were shown 20 random ImageNet images and the top ten concepts from EXPLAIN-R and SpLiCE. The evaluation centered on three criteria: (I) Relevance: the degree to which concepts pertain to the input image; (2) Completeness: the extent to which explanations reflect the semantic richness and task-agnostic characteristics of the image representations; and (3) Utility: the perceived usefulness of each method for understanding the model's 's behavior. Responses were captured using a 5-point Likert scale. User study findings reveal significant preference for our explanations across all evaluated criteria. The statistical significance of these results was established via one-sided t-test, employing a significance level of p 0.05. Additional details of the user study protocol are available in the Appendix.
User Evaluation EXPLAIN-R vs. SpLiCE
Relevance
Woengl Freter KAPLAUN Trode A2---
Completeness
Prrke RyUCCE
Uniltt:
Percentage
(%)
Figure 7: User study results comparing representation explanations generated by EXPLAIN-R and SpLiCE. Overall, users indicated clear preference for EXPLAIN-R's explanations across all three evaluated criteria.
5 Discussion
In this work, we show that the instance similarities learned by CLIP can be utilized generate concept explanations for CLIP image embeddings. Quantitative experiments demonstrated EXPLAIN-PR superior faithfulness over baselines in preserving original model predictions, while a user study confirmed its explanations are more relevant, complete, and useful for model understanding. These results suggest that our fimilarity-matchiie approach offers a promising direction for developing more faithful and human-aligned explanations for general-purposs purpose representations.
Limitations Our approach, like prior studies 226 I 12 assumes that concepts interact linearly
in the embedding space. While EXPLAIN-R significantly improves faithfulness over the baseline, perfect fidelity remains challenging. This gap may stem from several factors: the target model might learn concepts that are not easily captured by concise textual descriptions, there are some

---

non-linear concept interactions. A further consideration is our use of a captioning model instead of human captions. While this makes the approach more feasible and scalable, it also risks the captioner hallucinating concepts not present in the image, which can negatively affect the relevance of the explanations. Selecting a properly evaluated captioning model that is suitable for the target image domain can help minimize this issue.
Acknowledgments and Disclosure of Funding
We express appreciation for the volunteers in our user study. We thank the anonymous reviewers for their constructive feedback. This work is partially supported by National Science Foundation (NSF) grants SCH-2123809 and SCH-2306572, Learning Engineering Virtual Institute grant G-23- 2137070 and University of Florida Presidential Strategic Funding Award. Any opinions, findings, and conclusions or recommendations expressed in this paper, however, are those of the authors and do not necessarily reflect the views of the funding agency.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-88 0766.873..13 PMLR, 2021.
[2] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 10684-10695, June 2022.
[3] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of ICML''2, 23,, pages 19730-19742, Honolulu, Hawaii, USA, July 2023. JMLR.org.
[4] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning, October 2023, arXiv:2310.03744 [es].
[5] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In The twelfth international conference learning representations, 2024.
[6] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-Vocabulary DETR with Conditional Matching. In Computer Vision ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 106-122, Berlin, Heidelberg, October 2022. Springer- Springer-Verlag.
[7] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple Vocabulayy Object Detection. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X, pages 728 788-755, Berlin, Heidelberg, October 2022. Springer- Springer-Verlag.
[8] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient- Based Localization. International Journal of Computer Vision, 128(2):336-35,, February 2020.
[9] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explain- ing the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '1, pages 1135-1144, New York, NY, USA, August 2016. Association for Computing Machinery.
10

---

[10] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pages 1319-3328, Sydney, NSW, Australia, August 2017. 2017.JML..00 JMI.R.rg.
[11] Thomas Fel, Agustin Picard, Louis Béthune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. CRAFT: Concept Recursive Activation FacTorization for Explainability, pages 2711-2721, 2023.
[12] Mara Graziani, Laura Mahony, An-phi Nguyen, Henning Müller, and Vincent Andrearczyk. Uncovering Unique Concept Vectors through Latent Space Decomposition. Transactions Machine Learning Research, August 2023.
[13] Ying Wang. Tim G. Rudner, and Andrew Gordon Wilson. Visual Explanations of Image- Text Representations via Multi-Modal Information Bottleneck Attribution. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[14] Zhiyu Zhu, Zhibo Jin, Jiayu Zhang, NAN YANG, Jiahao Huang, Jianlong Zhou, and Fang Chen. Narrowing Information Bottleneck Theory for Multimodal Image-Tex Representations Interpretability. In The Thirteenth International Conference Learning Representations, 2025.
[15] Peijie Chen, Qi Li, Saad Biaz, Trung Bui, and Anh Nguyen. gScoreCAM: What objects is CLIP looking at? In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 1959-1975, December 2022.
[16] Chenyang Zhao, Kun Wang, Xingyu Zeng, Rui Zhao, and Antoni B. Chan. Gradient-based Vi- sual Explanation for Transformer-based CLIP. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61072-61091. PMLR, July 2024.
[17] Sriram Balasubramanian, Samyadeep Basu, and Soheil Feizi. Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[18] Yossi Gandelsman, Alexci A. Efros, and Jacob Steinhardt. Interpreting CLIP's Image Represen- tation via Text-Based Decomposition. October 2023.
[19] Jayneel Parekh, Pegah KHAYATAN, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. A Cansept-Based Explainability Framework for Large Multimodal Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[20] Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover-then-Name Name:: Task- Agnostic Concept Bottlenecks via Automated Concept Discovery. In Computer Vision ECCV 2024: 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXVII, pages 444-461, Berlin, Heidelberg, October 2024. Springer-Verlag Verlag.
[21] Yuan Zang, Tian Yun, Hao Tan, Trung Bui, and Chen Sun. Pre-trained Vision-Language Models Learn Discoverable Visual Concepts. Transactions on Machine Learning Research, 2025.
[22] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Buree and Mark Yatskar, Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19187-19197, June 2023.
[23] Konstantinos P. Panousis, Dino Ienco, and Diego Marcos. Coarse-to- -ine Concept Bottleneck Models. November 2024.
[24] Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees. Transactions on Machine Learning Research, 2023.
[25] Ruihan Zhang, Prashan Madumal, Tim Miller, Krista Ehinger, and Benjamin Rubinstein. Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors. Proceedings of the AAAI Conference on Artificial Intelligence, 35:1 1690, May
2021.
11

---

[26] Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE). November 2024.
[27] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantifying Interpretability of Deep Visual Representations. pages 6541-6549, 2017.
(28) Xinyue Xu, Yi Qin, Lu Mi, Hao Wang, and Xiaomeng Li. Energy- Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations. In The Twelfth International Conference on Learning Representations, 2024.
[29] Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JogR. 2023, event-place: Honolulu, Hawaii, USA.
[30] Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-War Weng. Label-free Concept Bottleneck Models. September 2022.
[31] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc Concept Bottleneck Models. September 2022.
[32] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattem recognition (CVPR), June 2015.
[33] Alon Lavie and Abbaya Agarwal. Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT '07, pages 228-231, USA, June 2007. Association for Computational Linguistics.
[34] Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan A. Rossi, Vishwa Vinay, and Aditya Grover. CyCLIP: Cyclic Contrastive Language-Image Pretraining. October 2022.
[35] S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, March 1951. Publisher: Institute of Mathematical Statistics.
[36] Mingkai Zheng, Shan You, Fei Wang. Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. ReSSL: Relational self-Supervised Learning with Weak Augmentation. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[37] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, June 2020. ISSN: 2575-7075
[38] lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap Your Own Latent- A New Approach to Self-Supervee Learning. In Advances in Neural Information Processing Systems, volume 33, pages 21271-21111 212244. Curran Associates, Inc., 2020.
[39] Jure Zbontar, Li Jing. Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow Twins: Self- Supervised Learning via Redundancy Reduction. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12310-12320. PMLR, 2021.
[40] Bokun Wang and Tianbao Yang. Finite-sum coupled compositional stochastic optimization: Theory and applications. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23292- 23317. PMLR, 17-23 Jul 2022.
12

---

[41] Zhuoning Yuan, Yuexin Wu, Zi-Hao Qiu, Xianzhi Du Lijun Zhang, Denny Zhou, and Tianbao Yang. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages PMLR, 17-23 Jul 2022.
[42] Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[43] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014 13th European Conference, Zurich, Swirzerland, September 6-12, 2014, Pro- ceedings, Part volume 8693 of Lecture Notes in Computer Science, pages 740-755, Springer, 2014.
[44] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[45] Jianxiong Xiao, James Hays. Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pages 3485-3499 June 2010. ISSN: 1063-6919.
[46] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, June 2009. ISSN: 1063-6919.
[47] Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Ji Liu, and Jungong Han. IMRAM: Iterative Matching With Recurrent Attention Memory Cross-Modal Image-Text Retrieval. 2020 IEEF/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 12652- 12660, June 2020. ISSN: 2575-7075.
[48] Yan Huang, Wei Wang, and Liang Wang. Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM. In 2017 IEEE Conference Computer Vision and Pattern Recognitiion (CVPR), pages 7254-7262, July 2017. ISSN: 1063-6919.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide quantitative, qualitative experiments, a human evaluation, and a case study to support claims in Section 4 Additional metries and visualizations
provided the Appendix (Sect. A.I A.2 and A.3).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We provide limitation section in the discussion section (Sect. 55.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--peeiiiaiinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not introduce theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide pseudocode for our Algorithm We describe the dataset used, metrics, hyper-parameters in the Experiment section (Sect. 4) and provide further experi- mental details Appendix B.3 We provide source code and all of the used datasets are publicly accessible.
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code along with instructions to reproduce our results in the supplemental materials. The datasets used are publicly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ptiips.cc//uulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the experiments in the Experiments section provide additional details in Appendix B.3 and provide code in supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We repeat experiments 5 times and report 2o error intervals in Table 1 in the main paper. We state the confidence level and p-value for the user study Appendix B.4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in the experiments section of the paper.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: Our work adheres NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss the broader impacts in Appendix B.5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
18

---

Our paper explains an existing model. We do not release any data nor train new model; therefore our work poses no such risk.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification:
We cite and credit all used asset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi www.wethhde...m caree.comdaee com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code comes with documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
19

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We describe the protocol for our small-scale user study Section 4.4 and Appendix [B.4
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our study pose more than minimal risk to participant and is ruled exempt by the instituion's IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
LLMs are not involved in the core method development of our paper.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/nneurips.cm http://arrrpp.ccoooom///e/////// for what should or should be described.
20

---

Appendix Contents
Additional results
22
A.I
Retrieval faithfulness metrics
22
A.2
Task performance metrics
22
More visualizations
4
More datasets
22
Method details)
24
Concept identification details
24
B
2
Algorithm for Surrogate Learning
24
B.3
Experimental details
25
B.4
User study details
25
B.5
Broader impacts
26
21

---

A
Additional results
A.1
Retrieval faithfulness metries
Table 2: COCO retrieval faithfulness metrics. For method, report the mean over 5 runs.
Method
COCO I-+T
COCO T-+1
R 1
R
R@ 10
R@1
R@5
R@ 10
SpLiCE (26 Ours
38.12 12
72.40
86.00
38.26
72.74
85.66
61.99
89.31
95.51
55.03
86.64
94.46
Table 3: Flickr30k retrieval faithfulness metrics. For our method, report the mean over 5 runs.
Method
Flickr30k I-xT
Flickr30KT
R@1 1
R@5
R@ 10
R@1
R@5
@ 10
SpLiCE [26]
40.25
70.41
80.65
40.50
69.56
80.04
Ours
66.37
86.32
91.26
60.31
82.79
89.42
2and Table T shows the detailed R@ K. metrics for COOC and Flickr30k of our method and SpLiCE, which is supplementary to Table the main paper.
A.2 Task performance metrics
Table 4: Performance metrics. For our method, we report the mean over runs.
Method
COCO I-,T
COCO T~x
SUN397
ImageNet
R R@I
R@5
R 10
R@1 |
R@5
R@ 10
SpLiCE 226
32.26
62.94
76.44
29.92
59.22
73.00
52.41
42.91
CLIP ViT-B/32
51.90
81.26
90.24
47.48
77.10
87.08
60.71
61.91
Ours
58.03
85.68
93.19
56.30
84.98
93.09
57.98
52.37
Table 4 5 presents a comparison of performance metrics (evaluated against dataset ground truth) for method, SpLiCE, and the original CLIP model. The baseline is setup similar the experiment in Tab. [ As can be seen in Table our method consistently outperforms the baseline SpLiCE across tasks given the same sparsity.
Furthermore, although designed for post-hoc explanation, our surrogate representation exhibits the ability to sometimes outperform the CLIP model it explains on the zero-shot tasks tested, despite never having access to the dataset labels and only being trained on the similarities produced by the CLIP model. We hypothesize that performance improvement (when they exists) stems from the increased robustness of concept-based inputs, which may be less susceptible to common image degradations such occlusion, blurriness, general noise, compared to image inputs.
A.3
More visualizations
In Figure we provide a more comprehensive list concepts of the images visualized in Fig
A.4
More datasets
In Table we provide more zero-shot classification results more datasets (Flowers 102, Food101). Our method continues to yield consistent improvements on these fine-graines classification tasks. In Figure 9 we visualize some results from these datasets.
22

---

Table 5: Performance metrics (continued), For our method, we report the mean over runs.
Method
Flickr30k I-+
Flickr30k T-
R@1
R@5
@@
R@D
R@5
R 10
SpLiCE [26
36.43
65.03
75.36
36.83
63.50
74.20
CLIP ViT-B/32
67.17
88,90
93.80
63.60
86.78
92.26
Ours
74.53
92.58
96.24
73.60
92.67
96.30
Table 6: Faithfulness metrics for more datasets.
Method SpLiCE
Flowers 102
Food101
26.12
51.80
Ours
37.73
63.62
Curs tap creepii
Spritt top conoeiac
Our top cansaple:
SpLiCE LIC tag conceptic
bind
20
gran the
11
microwae (n)
microvaee
CE
crane
13
.07
countor
charning madhine
individia
201
buttee
tall
vogazation
phozont
fcraping
cleas
culinery
featar
forup sund
JLL
mas (h-)
blank gn
marning
fore ny
intrrctionn
plaper
80
fazer
Horciing
01
dapand
Manding
marah
101
stlaee
clarred
floo
Spi
Par
fedeeer
bult
staiiiess
[vc]
A77
battz
scrowod
Zic ally
Imp wre
.4
J6
.55
/0
beach
cealle
sand
kedt
Ours conrepts
2ard
Spt concepte
archoo log
men ldingg
1bb
05
33
cullying
nedie
Tanily
play
Lse
chi
02
ad12
20th shower 00
Mugge
trimal
01
Clps
01
siting
-2.0
ridal
Figure 8: More comprehensive depiction of concept list from our method and the baseline.
SpluCBiicp aneeste
Otes top commpta:
Syd comtupecc
pooddho
bosutid flwor
ooaage (ed)
bousthly Dowar
.55
had
-26
toogiaa lnoes
yellow (v.)
.11
crange Blwer
00
cakeig
.06
toopial Bol
J55
seaeae (s))
J0K
jual
J03
yallbwe
JE7
wile fower
J54
penie [t.]
JF7
screaan yollow
JI1
troptic
.0C
costa too
X3
daky
mocro
Sabee
cate kerta
Jub
/C2
coler
fora
blouniig
calln Bowe
J02
gandes
gindes
Miodiweddy
laak
Iomaniifl
cuanppts
SpLICE top Coamptts:
top commptts
SyC commptor
arpp
apperber
medds
seceal
matilo
Itllao Dodd
pork
oveiet fud
beocodi
pook
0000
100
scodle soup
60
mxds
/03
rob
J00
Jepanse food
.03
Teal
105
made
beeth
J02
star
sbolee
notortast
delilien
Servico
bottle
Kowae
-012
Inch
Seeal
Figure 9: Visualizations for samples from the Flowers 102 dataset (upper) and Food101 dataset (lower).
23

---

B Method details
B.1 Concept identification details
We redescribe our concept identificatin process step-by-ster with more details:
1. (If not already present) obtain the captions for each image by using a captioning model
2. Extract nouns, verbs, and adjectives concepts via part-of-speech tagging with the nltk library
3. Filter out words not presenting in WordNet, infrequent and overly frequent concepts to obtain the vocabulary C
4. Estimate concept prevalence score for each image-concept pair obtain the concept vectors C₁
B.2 Algorithm for Surrogate Learning
Below is the pscudocode for the algorithm described in Section 3.3
Algorithm 1: Algorithm for Surrogate Learning
CLIP encoders e,, er, surrogate model J, dataset S, concepts C, temperature batch size B, number of iterations T for t .,..., do
Input:
Sample mini-batch of triplets B, { (x,, C₁ tt)] from the dataset
Update Me+1, Ut41, using (5) and (6) for i E B;
Set U141, = Up,i, 01+1, = Ve.; for & I.
Compute gradient estimator w.r.t. fe:
(x.) (t((,))
exp (e((t))
exp
er; B.)
1B.12
U++1,
U2+1,
LEB
JEB.
Update fe+1 from fa using an optimizer
We now present the full derivation of Algorithm We will focus only on ( || P.') since the procedure for optimizing D (Q; || Q can be derived analogously. From the definition of KI. divergence, we can write the first part of (3) as
(P
|| P!)
PT() log P( + 2n
PE log PFO).
2n
2n
1=1
=1
Note that is fixed and we want optimize only f. we will discard the second term on the right hand side since it does not involve f. Plugging and (2) into the above equation, we get
n
n
log P! (i)
2n
(x,(er(x,) T et(t))/T) log exp(f(c,) Terfl,)))rr(t// er(t,)/7) exp(er((x))
2n
(r(())/7) (erp(t))))Ter((t)
E. exp(
(x;))) T (T - ex(xi) eT((t))
=
Σ exp
(10)
2n
[(.i) ((ci))er(tx) - f(c,) Ter(t))
log
exp
km
24

---

Recall that S denotes the whole dataset, and
f(ci) [(ca))er(tt))) er(tz) f(c)) er((s)
(er(ts)
exp
k=1
[((c))er((u)) e exx (fli)Ter(ta)
exp
k=1
Then 110 can be equivalently written as
lo
(1I)
S
er: eT; ()
iES jES
The gradient w.r.t. f is given by
1
ΣΣ
SSP
gler, er; Xx, S) gI,, el, S)
iES
Since we only have access to one mini-batch of data B at each iteration, the obtained mini-batch gradient estimator is
1
ΣΣ
1 g(e,, er: t,. glf, er: B) Volt.er;a,, iEB jEB
|BI
However, due to the non- -ineeiearityy of the reciprocal function, the expectation over B is not equal the true gradient. Thus the mini-batch estimator is biased. To address this problem, we use two moving average sequences u and & to approximate g(er, er; x,, S). and g(f, eT: c, Sy. respectively:
(x,( er((t))
1e+1, (1 1-7) 7/, + 71 B
exp
REB
[(xx))er(tt) en(tt)
U+1, = - (2-72))r.4
72
exp
Be
T
kEB,
where n. 12 E (0, 1] are two hyperparameters. Then we can approximate g(e,, err Xi, tj, S) and ef, f 8) using
((c:)]er(ts)))
4411// exp
and u/41.// exp
T
T
B.3 Experimental details
Datasets. For the Flickr30k dataset, we explain on the full dataset. For the SUN397 dataset, we use the first official testing split. For the COCO 2017 and ImageNet dataset, the validation split. Following (26). for computational efficiency, the retrieval metries are computed in batches of size 1000 and averaged over the full dataset.
Training. We use the Amsgrad variant of the Adam W optimizer with learning rate 10-3 3 and weight decay 10-6, During training, we distill image-text similarities within the batch following Algorithm We perform augmentations on both modalities: selecting a random caption as text augmentation, and random center crop and horizontal flip as image augmentation.
B.4 User study details
The user study involved 10 volunteers who did not receive monetary compensation. The interface is shown in Figure 10 Notably, we observed that SpLiCE's weights are often lower than that of us, since they do not sum to one. To reduce the chance the user differentiating the two methods based on the weights, scaled SpLiCE's weights for each individual image so that their sum equals ours.
The study was ruled exempt by our institution's 's IRB, as no more than minimal risk posed to the participants. No identifying information collected.
25

---

Criteria
CI
p-value
Relevance
1.61 + 0.54
x 10
Completeness
1.69 + 0.72
1 10
Utility
1.66 + 0.62
x 10-7
Table 7: The p-values and confidence intervals (CT) for hypothesis testing in our user study. value of 1 denotes strong preference for EXPLAIN-R, averaged across the 20 shown samples. The hypothesis tested is whether the population mean is less than 3, which denotes neurality (no preference for SpLiCE or EXPLAIN-R).
B.5 Broader impacts
Our work addresses the problem of interpreting CLIP image representations in task independent manner. EXPLAIN-R provides tool for users and researchers to inspect the semantic content of the representation and provide a simple, intuitive summarization of the learned concepts for each image. For the users, this will enhance transparency and trustworthiness in multimodal representations, which traditionally relies on downstream evaluation. Researchers can use EXPLAIN-E R to inspect individual embeddings and model predictions, as well as aggregate them over the dataset to obtain more holistic view the concepts learned by the model.
26

---

Explanation A:
Explanation
grumpy
bid
.7
cute bins
branch
22
mid adult
perch
вperrow
Which explanation is more relevant the image?*
Strongly prefer explanation A
Strongly prefer explanation B
Which explanation captures concepts that sufficient explain the CLIP model's capabilities?
Strongly prefer explanstion
Strongly prefer explanation
Which explanation is more helpful for understanding what information encoded in the representation?
2
O
Strongly prefer explanation A
Strongly prefer explanation
Figure 10: The interface for our user study. For each input image, show the top 10 concepts from both methods, along with the weights. We scale SpLiCE's weights so they have the same mean as ours.
27


Reviews:
[{"review_id": "WWTCTl1BjD", "reviewer": "Reviewer_iH42", "summary": "In this work, the authors propose EXPLAIN-R, an interpretability for CLIP that attempts to faithfully reconstruct the behavior of\nthe vision encoder while operating instead over concepts. In doing so, they provide a concept-based explanation of the vision\nencoder while also remaining faithful to the downstream performance of the CLIP model.", "questions": "* Do you think image captions contain all the relevant concepts in an image? For example, in the image example in Fig. 1,\nyour captions cover “A tall bird with a visible bill” and “dry vegetation”. However, I could imagine a relevant concept being “sunset” or “sunny weather” despite annotators generally describing the foreground and background over other visually-represented concepts such as the time of day or the weather.\n * I would be curious to also compare performance with the ground truth labels instead of CLIP — you optimize for faithfulness so it is expected that you do well on this metric.\n * It appears to me that this paper is more focused on explaining the behavior of CLIP’s image encoder over concepts than the concept extraction from images itself — your method for concept extraction is only a small section in the paper. I think this might be important to explain in the framing of your work.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n * Training a model to be faithful to the vision encoder is a novel idea and helps develop off of previous concept-based\nexplanations of CLIP . The ability to get cross-modal concept scores and use them in class-specific concept decompositions for\nmisprediction analysis was interesting as well.\n * The experiments validate the faithfulness of the method and the qualitative examples also seem to validate the method.\n\n## Weaknesses\n * Your description of how to compute concept vectors in 3.1 is very clear but I would appreciate an equation, figure, or\nalgorithm to diagram out the computation of each concept. In figure 1 the specific computation of the weights of the\nconcept vector are abstracted away .\n * To some extent this is a concept bottleneck model - you extract concepts and then train a model on these concepts for a\ntask (in this case, replicating CLIP’s image embedding). You may want to address this connection in more detail.", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your encouraging comments! We will address the formatting issues in the final version. Below we address your concerns:\n\n**Q1: Do you think image captions contain all relevant concepts in an image?**\n\nA: Extending our reasoning in the third paragraph of Section 3.1, when we use captions, we want them to be a) as similar to human captions as possible, and b) varied enough to capture human description variety. \n\nTo achieve a), we believe that using models trained on existing annotated data is the best. To achieve b), we sample multiple captions for variety. To answer your specific inquiry, we believe that a more sophisticated sampling strategy that penalizes repetitions should lead to more varied concepts, including more non-visual ones. In general, concept extraction is not the key part of our work, but better concept extraction would definitely benefit the explanation.\n\n**Q2: On performance with ground truth labels**\n\nA: We reported the task performance (i.e., metrics computed with the dataset ground truth instead of CLIP outputs) in the supplementary material Appendix A.2. We report results for 3 datasets: COCO, SUN397, and ImageNet. In some cases, our surrogate outperforms the original CLIP model. In all cases, our surrogate is more performant than the baseline.\n\n**Q3: The focus of the paper**\n\nA: Indeed, our method is focused on explaining the behavior of the CLIP image encoder. Our idea is that each image has a set of concepts that humans care about. This set is independent of the model used. Each model would then “weigh” this set differently depending on how they are trained. This explains our framework of model-agnostic concept extraction, followed by learning the weight for each concept.\n\n**Q4: On computation of concept vectors**\n\nA: We provided details on how to compute the coefficients for each concept in the last paragraph of Section 3.1 of the main paper and Appendix B.1 in the supp. matt. We will rewrite it as an algorithm in the final version to make it clearer and more concise."}, {"review_id": "nO0dduzIXo", "reviewer": "Reviewer_hVGW", "summary": "The authors propose EXPLAIN-R, a new method to tackle interpretability when using CLIP representations, independantly of the downstream taskl. The method consists in extracting concepts from captions, generated by a captioning model, that are used as input of a surrogate model that aims at reproducing the distribution of image-text similarities. They conducted extensive experiments on several datasets.", "questions": "1. Could you please clarify how different captions were generated using a single captioning model? Specifically, was the goal of generating 10 captions to eliminate noise, or perhaps to capture a wider range of concepts? Additionally, were these captions distinct from one another in terms of content and phrasing?\n\n2. Did you consider using multiple models to generate diverse captions?\n\n3. Could you please give more motivation behind the task independence?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:** The method, EXPLAIN-R, introduced in this work to tackle the interpretability of CLIP representations, which is a key challenge given the widespread use of the CLIP model. The authors provide a clear and intuitive explanation of the methodology, making it easy to follow. Regarding the experiment section, the inclusion of a user study strengthens the experimental evaluation. \n\n**Weaknesses:** As the authors themselves acknowledge, relying on captions generated by the same captioning model for concept extraction can be a limitation (see Question 2). Furthermore, I am uncertain about how different captions were produced using a single model and whether these captions were truly distinct from one another (see Question 1). Additionally, I remain unconvinced about the necessity of task-independent interpretability. I would think that the relevance of explanatory concepts likely depends on the specific task. For instance, distinguishing birds from planes may require different concepts than differentiating between two bird species (see Question 3).", "was_revised": true, "final_justification": "Based on the authors’ responses to my questions and the assessments provided by the other reviewers, I have decided to maintain my original rating.", "rebuttal": "We thank you for your comments and support! Below we address your concerns:\n\n**Q1: Details and goals of generating multiple captions**\n\nA: Our captioning model is a VLM (vision-language model), so it works very similar to an LLM. Specifically, we increase the temperature to encourage diversity and use top-p sampling to generate different captions given the same input. \n\nTo illustrate our point, we show the first 5 captions generated for an ImageNet image:\n\n\"a photo of an older gentleman holding a large fish in his hands\",\n\n \"a photo of a man in khaki pants is holding a big fish\",\n \n \"a photo of a man posing with a big fish that he caught\",\n \n \"a photo of the man is about to give away a huge fish he caught\",\n \n \"a photo of man holds large fish next to trees and stream\",\n \nThe different captions both introduced unique concepts (old man, khaki, trees, …) and reinforce concepts that certainly exists (big fish, the action of the man holding the fish, …)\n\nThe goal of using 10 captions is to capture more accurately the distribution of concepts that humans would use to describe the image. In that sense, it both reduces noise and encourages rarer concepts.\n\n**Q2: On using multiple models to generate captions**\n\nA: This is a practical idea that should give even more diverse captions. However, due to batching, using 1 model to generate 10 captions will be much quicker than, for example, using 10 models to generate 1 caption each. We experimented and found that our procedure, described in the answer to the previous question, hits a good balance between efficiency and diversity. We selected the BLIP2 model due to its good performance in captioning ([1]).\n\n**Q3: Motivation behind task independence**\n\nA: The CLIP image encoder generates a general-purpose embedding for each image that can be used for various downstream tasks. We draw parallel to humans. When given an image, a human can memorize certain features about that image that they can use to distinguish from other images. If asked to perform a certain task (e.g., find out which species of bird), then the human will pay more attention to certain features (e.g., fur color, size, environment, etc.). \n\nWe posit that the CLIP image encoder works in the same way, i.e., it encodes a set of concepts independent of task in the representation. Then, for each task, a certain subset of concepts will be more important than others. We realize this idea in the paper. We define the notion of the contribution of a concept to an embedding in Eq. (8) in the paper, and this score is what you see in Fig. 2 and 4. This approach allows converting embeddings into a interpretable semantic basis, which enable analysis that task-specific methods cannot. In Fig. 5 we show an example of this kind of analysis: we find that the notion of “analog clock” is mostly dependent on the notion of “clock” itself, while the notion of “digital clock” involves the coexistence of the notion of “clock” and other notions such as “alarm” or “digital”.\n\nFinally, we provide a mathematically meaningful way to convert embedding contribution to class-dependent contribution in Eq. (9), which aligns with the intuition in the first paragraph.\n\n[1] Sarto, S.; Cornia, M.; and Cucchiara, R. 2025. Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-25)."}, {"review_id": "eTSnWMkQfG", "reviewer": "Reviewer_3CxX", "summary": "This paper introduces EXPLAIN-R, a novel method for generating concept-based explanations for general-purpose CLIP image representations. The core idea is to train a simple, interpretable surrogate model to mimic CLIP's behavior. Instead of traditional reconstruction loss, the surrogate is trained to match the instance-instance similarity distributions of the original CLIP model, measured by KL divergence. While the paper presents an interesting new learning objective and shows promising results in a user study, I recommend rejection due to its insufficient experimental validation, strong underlying assumptions, and failure to acknowledge closely related prior work.", "questions": "see Strengths And Weaknesses", "limitations": "see Strengths And Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nThe paper addresses the important and challenging problem of explaining task-agnostic representations. The core idea of training a surrogate model by matching similarity distributions, rather than by direct embedding reconstruction, is novel and theoretically well-motivated. It aligns more closely with the contrastive nature of CLIP itself.\n\nWeaknesses\n\nDespite these strengths, the paper has several major weaknesses that prevent me from recommending acceptance.\n\n1. Insufficient experimental validation: This is the most significant flaw. The empirical evidence provided is not comprehensive enough to fully support the claims of a new, general-purpose explanation method: \n\na. Limited datasets: The experiments are confined to general-domain datasets (COCO, ImageNet, etc.). The method's effectiveness is not tested on more challenging fine-grained recognition tasks (e.g., CUB-200 for birds, Oxford Flowers), where concept-based explanations are arguably most needed and most difficult to generate. The proposed concept discovery method, which relies on statistical extraction from general captions, is unlikely to capture the subtle, domain-specific concepts required for these tasks.\n\nb. Single baseline comparison: The quantitative evaluation relies on a comparison against a single baseline, SpLiCE. To robustly demonstrate superiority, a new method should be compared against a wider range of relevant approaches, such as prototype-based models (e.g., ProtoPNet) or concept bottleneck models (CBMs), even if they require adaptation.\n\nc. Lack of ablation studies: The method's performance is heavily dependent on the quality of its concept vocabulary, which in turn depends on an external captioning model (BLIP-2) and heuristic filtering. The paper lacks a rigorous ablation study on how the quality of captions (e.g., human-authored vs. machine-generated) or the choice of filtering heuristics impacts the final explanation quality.\n\nd. Failure to cite and differentiate from highly related work: The paper fails to acknowledge or discuss highly relevant and concurrent work, which significantly undermines its claim to originality.\nThe work Align2Concept (ACM MM 2024)[1] shares a remarkably similar goal: providing language-guided, concept-level explanations for visual recognition by bridging visual and textual modalities through CLIP's multimodal space. Both methods aim to align visual parts with textual concepts to create interpretable models.\nWhile the technical approaches differ (surrogate learning vs. manifold alignment with prototypes), the high-level motivation and conceptual framework are nearly identical. The omission of such a closely related work is a major oversight and prevents a fair assessment of this paper's true contribution to the field.\n\ne. Unjustified strong assumption of linearity: The entire framework is built on the strong assumption that concepts in CLIP's embedding space interact linearly. This is a significant simplification of a complex, high-dimensional space where non-linear interactions are highly probable. The paper acknowledges this as a limitation but fails to discuss its potential impact, which could lead to misleading or overly simplistic explanations that do not capture the true reasoning of the model.\n\n[1] Align2Concept: Language Guided Interpretable Image Recognition by Visual Prototype and Textual Concept Alignment, ACM MM2024", "was_revised": true, "final_justification": "See my discussion notes.", "rebuttal": "We thank you for your comments. Below we address your concerns:\n\n**Q1: Limited evaluation datasets and lack of “challenging” datasets**\n\nA: First, we clarify that we evaluated and reported results on fine-grained recognition tasks. These results are in Appendix A.4 of the supplementary material. They include the Flowers102 (Oxford flowers) and Food101 datasets. In both datasets we outperform the baseline. We included visualizations which show images and their concept in Fig. 9 in the supp. matt. \n\nWe emphasize that “popular” datasets are not necessarily “easy”. In the main paper, we report results on 4 datasets (COCO, Flickr30k, SUN397, ImageNet). The SUN397 dataset contains many similar scene categories (e.g., abbey, basilica, cathedral, church) and can be considered fine-grained ([1], [2], [3]). The ImageNet also contains multiple breeds of the same animal (e.g., 120 dog breeds [4]), which also makes it fine-grained [5]. \n\n**Q2: Single baseline comparison and not citing Align2Concept**\n\nA: We would like to make a key distinction here: our work positions itself as an explainer, not a replacement to the CLIP model with better performance/interpretability. Our goal is to understand the CLIP image encoder, not to replace it. It is due to this positioning that we value faithfulness more than task performance throughout our paper. \t\n\nMany related works fall in the category of creating inherently interpretable methods that aim to replace opaque methods. While they might share similarities to us (concept extraction, concept alignment, …) the ultimate goals are completely different. For example, a replacement model is good if it performs better on downstream tasks, or it is more interpretable than the base CLIP model. It does not need to have the same predictions as the base model (i.e., it does not need to be faithful). A surrogate explainer is only good if it is faithful to the base model, which gives insight into how the base model actually works. The two different philosophies renders comparisons pointless.\n\nFurthermore, our work is motivated by the need to explain general-purpose CLIP representation (discussed in answer to Q2 of reviewer hVGW). Traditional task-specific methods such as CBMs cannot be simply “adapted” to work with this setting.\n\nThe above reasons explain why we compare our method with the most similar baseline SpLiCE, which also priorities explaining over replacement.\n\nRegarding the work Align2Concept, its abstract states: “Our experiments show that the ProCoNet provides higher accuracy and better interpretability compared to the single-modality interpretable model.” This confirms that this method belongs to the category of inherently interpretable methods, as discussed earlier. While it is suitable to discuss this work in our paper, the differing motivation and does not impact our paper’s novelty. \n\nThe other suggested “relevant approach” ProtoPNet also has the same issue: it is a whole new architecture that is interpretable, and it is not an explainer for existing models.\n\n**Q3: Linear representation assumption/hypothesis**\n\nA: We note that there is evidence for this hypothesis in the literature, including the baseline SpLiCE and other works ([6], [7], [8]). \n\nDuring the development of the paper, we also experimented with a non-linear surrogate. This resulted in greatly increased training time, without significant improvement in faithfulness.\n\nThe literature evidence coupled with our developmental result suggest that the hypothesis is not entirely unreasonable. While it might be a simplification, it allowed us to gain insights into the representation. Studying the exact geometry of hidden representation is an active research direction and is out of scope of our work.\n\n**Q4: Concept extraction ablation study**\n\nA: Our proposed concept extraction scheme is simple, computationally efficient, and effective. In the same spirit as our replies to Q1 of reviewer iH42 and Q2 of reviewer hVGW, we acknowledge the concept extraction can be improved, but it is not the key/innovative part of our method.\n\nIn Section 3.1 of the paper, we stated that human captions are ideal since they contain the concepts that humans are likely to use when describing the image and using an image captioning models train on human data achieves our goal. Even with BLIP2-generated concepts, our method remains effective, demonstrated via downstream faithfulness (Table 1 of main paper) and user preference (Figure 7 of main paper).\n\n[1] M. Zanella and I. B. Ayed, \"On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do we Really need Prompt Learning?,\" 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 23783-23793.\n\n[2] Maniparambil, Mayug, et al. \"Enhancing clip with gpt-4: Harnessing visual descriptions as prompts.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[3] Y. Liang, Y. Pan, H. Lai, W. Liu and J. Yin, \"Deep Listwise Triplet Hashing for Fine-Grained Image Retrieval,\" in IEEE Transactions on Image Processing, vol. 31, pp. 949-961, 2022.\n\n[4] Russakovsky, O., Deng, J., Su, H. et al. ImageNet Large Scale Visual Recognition Challenge. Int J Comput Vis 115, 211–252 (2015).\n\n[5] X. -S. Wei et al., \"Fine-Grained Image Analysis With Deep Learning: A Survey,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 8927-8948.\n\n[6] Trager, M., Perera, P., Zancato, L., Achille, A., Bhatia, P., & Soatto, S. (2023, October). Linear spaces of meanings: Compositional structures in vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 15395–15404).\n\n[7] Fel, T., Picard, A., Béthune, L., Boissin, T., Vigouroux, D., Colin, J., Cadène, R., & Serre, T. (2023, June). CRAFT: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2711–2721).\n\n[8] Papadimitriou, Isabel & Su, Huangyuan & Fel, Thomas & Saphra, Naomi & Kakade, Sham & Gil, Stephanie. (2025). Interpreting the Linear Structure of Vision-language Model Embedding Spaces. 10.48550/arXiv.2504.11695."}]

Author rebuttal:

Meta-review: The paper deals with the problem of interpretability of Contrastive Language-Image Pre-training models (CLIP models), which is addressed by using a surrogate representation of concepts present in an image. 

The paper was considered by three reviewers. The final ratings were: BA - A - BA

The reviewers liked the idea of training a model to assess the vision encoded and highlight the good experimental evaluation. Most issues were addressed in the rebuttal. The AC follows the consensus of the reviewer voting and recommends accepting the paper. The AC would encourage the authors to integrate the findings of the rebuttal in the CR version of the paper.
Decision: Accept (poster)