Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)
Authors: Usha Bhalla
Alex Oesterling
Suraj Srinivas
Flavio Calmon
Himabindu Lakkaraju
Venue: NeurIPS 2024 poster (2024)
Keywords: Interpretable Machine Learning
Dictionary Learning
Representation Learning
Multimodal Models
Interpretability
CLIP
Primary area: interpretability_and_explainability
TL;DR: We use dictionary learning to interpret CLIP embeddings by representing them as sparse combinations of semantic concepts, resulting in interpretability while maintaining high performance and unlocking novel use cases.

Abstract: CLIP embeddings have demonstrated remarkable performance across a wide range of multimodal applications. However, these high-dimensional, dense vector representations are not easily interpretable,  limiting our understanding of the rich structure of CLIP and its use in downstream applications that require transparency. 
In this work, we show that the semantic structure of CLIP's latent space can be leveraged to provide interpretability, allowing for the decomposition of representations into semantic concepts. 
We formulate this problem as one of sparse recovery and propose a novel method, Sparse Linear Concept Embeddings (SpLiCE), for transforming CLIP representations into sparse linear combinations of human-interpretable concepts. Distinct from previous work, \method is task-agnostic and can be used, without training, 
to explain and even replace traditional dense CLIP representations, maintaining high downstream performance while significantly improving their interpretability. We also demonstrate significant use cases of \method representations including detecting spurious correlations and model editing. Code is provided at https://github.com/AI4LIFE-GROUP/SpLiCE.

Interpreting CLIP with Sparse Linear
Concept Embeddings (SpLiCE 8)
Usha Bhalla* Harvard University 0.b
Alex Oesterling' Harvard University
Suraj Srinivas Harvard University
Flavio P. Calmon' Harvard University
Himabindu Lakkaraju' Harvard University b.c
Abstract
CLIP embeddings have demonstrated remarkable performance across wide range of multimodal applications. However, these high-dimensional, dense vector rep- resentations are not easily interpretable, limiting our understanding of the rich structure of CLIP and its use in downstream applications that require transparency. In this work, we show that the semantic structure of CLIP's latent space can be leveraged to provide interpretability, allowing for the decomposition of representa- tions semantic concepts. We formulate this problem as one of sparse recovery and propose a novel method, Sparse Linear Concept Embeddings (SpLiCE d)), for transforming CLIP representations into sparse linear combinations of human- interpretable concepts. Distinct from previous work, SpLiCE is task-agnostic and can be used, without training, to explain and even replace traditional dense CLIP representations, maintaining high downstream performance while signifi- cantly improving their interpretability. We also demonstrate significant use cases of SpLiCE representations including detecting spurious correlations and model editing. Code is provided at settps://githin com/ AI4LIFE ATT.AAAAAGGGOOOSSSS......
Introduction
Natural images include complex semantic information, such as the objects they contain, the scenes they depict, the actions being performed, and the relationships between them. Machine learning models trained on visual data aim to encode this semantic information in their representations to perform wide variety of downstream tasks, such as object classification, scene recognition, segmentation, or action prediction. However, it is often difficult enforce explicit encoding of these semantics within model representations, and it is even harder to interpret these semantics post hoc to better understand what models may have learnt and how they leverage this information. Further, model representations can be brittle, encoding idiosyncratic patterns specific to individual datasets and modalities instead of general
interartatterretballe semantic information. Multimodal models have been proposed a potential solution to this issue, and methods such as CLIP [I] have empirically been found to provide highly performant, semantically rich representations of image data. The richness of these representations is evident from their high performance such as zero-shot classification and image retrieval M.. image captioning (2]. and image generation 3. However, despite their performance, it remains unclear how quantify the semantic content contained in their dense representations. In this work, we answer the question: can we decompose
variety of tasks,
Equal contribution, order by coin flip. Equul contrihution, alphabetic order. Kempner Institute for Study Natural & Artificial Intelligence
b School of Engineering and Applied Sciences
Harvard Business School
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

E.00
cares
CLIP
interpretable
SpLiCE wh a sparse nomoogalivv inne selume
decomposilios,
eays
0.77
ovencomplate concept
Eannd cree
CLIP
graee.
ocean candy)
Figure I: Visualization of SpLiCE, which converts dense, uninterpretable CLIP representations (z) into sparse semantic decompositions (w) by solving for sparse nonnegative linear combination an overcomplete concept set (C).
CLIP embeddingg into numan-interpretable representations of the semantic concepts they encode? This provide insight into the types of tasks CLIP can solve, the biases it may contain, and the manner through which downstream predictions are made.
Existing literature in areas such as concept bottleneck models 4 disentangled representation learning [5) and mechanistic intepretability 6 have proposed various approaches to understanding the semantics encoded by representations. However, these methods generally require predefined sets of concepts 77.. data with concept labels [8) or rely on qualitative visualizations, which can be unreliable [9] Similar to these lines of work, aim to recover representations that reflect the underlying semantics the inputs. However, distinct from these works, we propose to do this in task-agnostic manner and without concept datasets, training, or qualitative analysis of visualizations.
Our method, SpLiCE, leverages the highly structured and multimodal nature of CLIP embeddings for interpretability, and decomposes CLIP representations via a semantic basis to yield a sparse, human- interpretable representation. Remarkably, these interpretable SpLiCE embeddings have favorable eecuracyinteeppreaabll tradeoffs when compared to black-box CLIP representations on metrics such zero-shot accuracy. Our overall contributions are:
Sections and 4 we formalize the sufficient conditions under sparse decomposition of CLIP is feasible, and introduce SpLiCE, novel method that decomposes dense CLIP embeddings into sparse, human-interpretablee concept embeddings.
Our extensive experiments in Section 5 reveal that SpLiCE recovers highly sparse | in- terpretable representations with high performance downstream tasks, while accurately capturing the semantics the underlying inputs.
In Section we present two case studies for applying SpLiCE: spurious correlation detection, and model editing. Using SpLiCE, dataset, where we the "woman" concept and the "swimwear" concept to be correlated owing to the prevalence of women in swimwear in CIFAR100.
uncover spurious correlation in the CIFAR 100
2
Related Work
Linear Representation Hypothesis. In language modeling, the linear representation hypothesis suggests that many semantic concepts are approximately linear functions model representations
[1 B [4]. Recent work has also shown that multimodal models encode concepts additively, behaving like bags-of- words representations [15]. Relatedly. [16] [7] show that there exists linear mapping between image and text embeddings in arbitrary models. Our work makes use of these distinct but related observations to convert dense CLIP representations to sparse semantic ones.
Concept Bottlenecks and Attribute Learning. Concept Bottleneck Models (CBMs) (18), and attribute- based models [9 20 21 learn intermediate representations of scores over concepts or image
we recommend and sparsity levels of ~ 10-30 in practice

---

wedding sign
0.38
0.06
0.04
snowboard
0.07
shower
0.06
shop logo
sking
0.07
tabby cat
0.08
slopes
0.06
scratching
0.08
sweetered
0.03
sports
0.34
shampoo
0.08
swing
0.02
ski resort
0.03
sink
0.00
bold hea aomeo pod ros in
buttouu bathroom with Itile""
sign on graae"
vitamin
0.10
red umbrelie
0.07
wild horses
0.10
oranges
umbrelia
blackandwhiite
fresh orange
0.08
bwr
depreesion
0.04
busieeseman
0.00
mane
unhappy
autumn winter
0.01
beach sand
0.03
background
horee
0.02
"Aloi stabiing Stange with
shot
0.08
"Ad,i in srange unbrella
background
bou on buach kiy to
reating
pavad
0.01
kissing
Figure 2: Example images from MSCOCO shown with their captions below and their concept decompositions the right. We display the top seven concepts for visualization purposes, but images in the figure had decompositions with 7-20 concepts.
attributes for with final linear classification head, creating interpretable concept representations. However, these require expert-labeled concept or attribute datasets to train, which is expensive. Recent work concept-botllenekss for multimodal models avoids needing such labeled datasets, but still requires concept labels for specific tasks, which obtained by querying large language models (LLMs) 22
2 24 making these methods task-specific and heavily reliant the domain knowledge and subject to the biases of LLMs. On the other hand, SpLiCE uses a large-scale and overcomplete concept dictionary, avoiding dependence on training, specific domain knowledge, or a downstream task. Consequently, it can even be applied to understand unstructured, unsupervised image datasets in label-free manner.
Mechanistic Interpretability and Disentanglement. Mechanistic interpretability explains represen- tations through model activations. by labeling circuits and neurons in networks with feature visualiza-
tion 6 25 or by measuring concept activations and directions in latent space [26] 27 7 28 29
30).
Recent work [1 combines these methods, using dictionary learning extract visual concept activations, whose semantics can be identified via feature visualization. Work in disentangled rep- resentation learning has developed architectures that capture independent factors of variation in 35). allowing for manual probing of disentangled representations for human- interpretable concepts. In both mechanistic interpretability and disentangled representation learning, methods typically rely on labeled concept sets, manual labeling of visualizations, or computationally intensive searches over data and latent representations or neurons to identify concepts. However, associating human -amman-iterrpettable semantics with arbitrary neurons or latent directions is challenging, leading to the unreliability ( 336] exhibited by such methods. Our approach side-steps this issue by decomposing CLIP representations into a predetermined set of concepts.
data 33
CLIP Interpretability. Many recent works leverage the semantic structure of CLIP and its text encoder to interpret representations. For example, (37], [38. and [39 construct concept similarity scores of image embeddings for use by downstream CBMs or probes, but these representations are interpretable due to their lack of sparsity and the presence of negative concepts. Chen et al. 400 create custom vision-language architecture with sparse latent dictionary, but it requires training from scratch and cannot be used post-hoe to explain existing models. Gandelsman et al. 41 also leverage the text encoder of CLIP to explain components of the image embedding, but are limited to ViT architectures and take a mechanistic interpretabillty-sty. approach requiring a labeled text dataset. Chattopadhyay al. (22) build concept bottlenecks for specific classification tasks by expressing CLIP image representations as sparse linear combination of task-specific concept vectors. However, their decomposition includes negative concepts, reducing interpretability, and uses task-specific concept dictionaries. Grootendorst 42 generate textual topics of datasets through multimodal topic modeling, which cannot provide explanations of individual representations. Distinet from these works, SpLiCE is more interpretable due to its sparsity, overcompleteness, and non-negativity. and is task-agnostic, aiming to serve as a drop-in replacement for black-box CLIP representations without requiring training.
3

---

Table 1: Sanity checking the linearity of CLIP Embeddings.
We
Wb
COSINE(2; 2)
IMAGENET
0.48 + 0.09
0.45 + 0.09
0.76 + 0.05
CIFARIO0
0.45 + 0.08
0.42 + 0.08
0.75 = 0.03
MIT STATES
0.48 + 0.09
0.45 + 0.09
0.76 = 0.05
COCO TEXT 0.59 + 0.12
0.47 + 0.12
0.88 + 0.04
3 When do Sparse Decompositions Exist?
In this section, we aim to answer the question: under what conditions can CLIP representations be decomposed into sparse semantic representations? To do so, we must reason about both the properties CLIP as well the properties of the underlying data.
Notation, Let x iing E R x E R4 be image and text data, respectively. Given the CLIP image encoder / Rd Rd and text encoder g Rd
R. we define CLIP representations in Rd as zimg [(x and zxxt = (x Our method uses dictionary learning to approximate zimg with a concept decomposition w E R2 over a fixed concept vocabulary E id We define the resulting reconstruction of i imgg from and w as 2im
The goal of our method is to approximate f(( Cw", such that w is non-negative and sparse, and in this section we formalize when this is possible. We begin by considering data- generating process for coupled image and text samples. Specifically, we model the generative process parameterized by a k-dimennional latent concept vector w E TR and random noise vector c E R as
x img
h im (w. c),
x (w, <),
P.
where p is a prior distribution over semantic concepts, Φ is a prior distribution over nonsemantic concepts (such as camera orientation and lighting for images or arbitrary choices between synonyms for text), and hmm * + R and hMd R + R represent the real-world data-genedt generating process from latent variables (w, e) to images and text respectively. Here, each coordinate w4 R, encodes the degree of prevalence of the 7hh concept in the underlying data. We now list a set of sufficient conditions for our data-generating process and CLIP that admit a sparse decomposition of images into concepts.
Sufficient Conditions for Sparse Decomposition.
1. Images and text are sparse in concept space: for some a & k2 we have ||w|lo S a, V w
2. CLIP captures semantic concepts w and not e: Ve, f o (w,, = f a h (imm@(s c') and similarly for A
3. CLIP is linear in concept space: g h and o hi are linear in w.
4. CLIP image and text encoders are aligned: for a given w, f h hi (w, e) g h h" e).
We emphasize that the goal of enumerating a set of sufficient conditions for sparse decomposition not to claim that these exactly hold in practice, but rather to reason about when sparse decompositions- as done in this work-are appropriate. In the Appendix (Section A. Prop. [T] we formalize and prove this claim, but in the interest of simplicity we keep the discussion here informal. We note that many of these are natural; Assumption reflects how real-world images and text are simple and rarely contain complex semantic content, and the CLIP training process optimizes for Assumption 2 and 4 Of these, the most critical one is Assumption 3, which closely relates to the linear representation hypothesis [II, which investigate below.
Sanity Checking CLIP's Linearity. We provide evidence for the third assumption, the linearity of CLIP, in toy setting. We begin by asking the following question confirm the general linearity of CLIP embeddings: "if two inputs are concatenated, does their joint embedding equal the average
'In practice we find that CLIP's image and text encoders are not fully aligned, so we apply a preprocessing step (.1c

---

of their two individual embeddings?". For the image domain, we combine two images, za Lb, to form their composition aa by placing Fa in the top left quarter and 2, in the bottom right quarter of a blank image. For the text domain, we simply append text to text Da to form Lab- We then embed za, Tb, Jab with CLIIP to get za, 2b, Rab- Solving the equation wa Za + Wh 2b = Pab for scalar weights wa: w, then allows us to assess the linearity of 2.; 2b. Zub. We report Way W/ and the cosine similarity between Eas- = [Za, _b] [Wa, wi! and Rab in Table
In general, we find that the composition of two inputs results in an embedding that is approximately equal to the average of the two input components, with wa: Wh, being very close to 0.5 across all datasets and for both modalities, providing preliminary evidence for the linearity of CLIP embeddings for both image and language.
4 Method
In this section, we introduce SpLiCE, a method for expressing CLIP's image representations as sparse, nonnegative, linear combinations of concept dictionary elements. We begin by framing this problem as one of sparse recovery. We then discuss our design choices, including how we choose the concept dictionary and how to address the modality gap between CLIP's images and text representations. Finally, we formalize the optimization problem used in this work.
4.1 Sparse Nonnegative Concept Decomposition
Our goal is to construct decompositions of dense CLIP representations that are human- iimaaninterpretablle useful, and faithful. To do so, we formulate decomposition as sparse recovery problem with three main desiderata. First, for the decompositions to be interpretable to humans they must be comprised of human interpretable atoms. We argue that language is a naturally interpretable interface for humans, and construct our concept vocabulary C out of 1- and 2-word atoms, such "coffee", "silver", and "birthday party". Second, our decompositions must be simple and concise, which can be formulated as sparsity constraint on the recovery. A large body of work in computational linguistics have demonstrated that a human-ale aeman-aligned semantic model should be sparse in representation. Furthermore, [48] found that users
[43 [44 [5 [14,, neuroscience [46, (4), and interpretability [48] 49 30
can best understand explanations with fewer than 32 concepts while in linguistics, [50] 5 52] find
participants describe concepts and objects with up to 20 semantic properties, motivating our desiderata of sparsity. Third, our decompositions must be constructive, i.e., we must decompose representations in terms of their constituent concepts. For this reason, we require the weights of decompositions to be strictly nonnegative, to avoid having "negative" concept weights which do not always carry semantic meaning. Furthermore, prior work by Zhou et al. (30] has argued that "negations of concepts are not as interpretable as poxitive concepts." More specifically, while a small set of concepts have well-defined antonyms which may be viewed as their negative counterparts (''day" +> "night"") negative concepts do not carry semantic meaning in general ('tigerr" <> ??). Furthermore, we find that even when antonyms exist, they are not negatives of each other in CLIP latent space (see Appendix IO To avoid dependence on negative weights and ensure that all concepts are captured, we construct an overcomplete dictionary containing a wide range of concepts, including antonyms. We build on top of this literature and provide semantic decomposition satisfying these properties suitable for multimodal models like CLIP.
Concept Vocabulary. Natural language is an intuitive, interpretable, and compact medium for communicating semantic information. Thus, we choose represent the semantic content contained in CLIP embeddings as combinations of natural language semantic concepts, where we define concepts as semantic units that can be expressed concisely, by one- or two-word phrases. Given that CLIP is used in a wide variety of downstream applications and is trained without a specific task in mind, we want our concept dictionary to be task-agnostic and span all possible concepts CLIP could have learnt. To construct this vocabulary, we consider the most frequent one- and two-word bigrams in the text captions of the LAION-400m dataset [53], the dataset that most CLIP variants are trained on. We filter the captions to remove any NSFW samples and prune our concept set such that no two concept embeddings have a cosine similarity greater than 0.9. We also remove bigrams highly similar (> 0.9 cosine similarity) to the average of their individual words. We finally choose the top 10, 000 most common single-word concepts and the top 5000 most common two-word concepts as our concept vocabulary. We note that this vocabulary offers distinct advantages over those used in prior works.
5

---

CIFAR100 Cosine Reconstruction
MIT States Coshhe Reconstruction
Imagenet Cosine Reconstruction
0.
0.
0.8
0.7
Centre Siiberdy
0.8
0.5
0.5
05
044
0.4
20
40
60
AD
100
40
62
80
100
20
40
62
80
100
12
Dpoomposiion 10 Nom
Decompositon 12 Nomm
Decompositon 10 Nom
CIFAR100 Shot Accuracy
MIT States Accuracy
Imagenet Zero Shot Accuracy
0.45
0.5
0.40
0.36
0.5
0.30
Accunny
0.28
0.20
0.
0.18
02
0.06
20 Decompoallio Nums
60
Decompooiion 12 Nor
Decomposiian 12 Nurt
Ialon
kared
images)
leamed (teet)
random
(baseline
CLIP
Ision
imagenat
clasdes
Figure 3: Performance of SpLiCE decomposition representations on zero-shot classification tasks (bottom row) and cosine similarity between CLIP embeddings and SpLiCE embeddings (top row). Our proposed semantic dictionary (yellow) closely approximates CLIP on zero-shot classification accuracy, but not the cosine similarity. This indicates that SpLiCE captures the semantic information in CLIP, but not its non-semantic components, explaining both the high zero-shot accuracy and low cosine similarity. See $5.2 for discussion.
In particular, it is task-agnostic, meaning that the efficacy of the decomposition is (in principle) independent of individual datasets. Furthermore, this dataset imposes minimal priors from outside curators, such as human experts or LLMs (22 23 24). This allows to interpret data through the lens of CLIP, to understand the information encoded, including potential biases and mistakes.
Modality Alignment. In order to decompose images into nonnegative combinations of text concepts, we must ensure that our concept set spans the space of possible image embeddings. However, [54] show the existence of modality gap in CLIP, where image and text embeddings can lie in non-identical spaces on the unit sphere. We empirically find that CLIP image and text embeddings exist on two cones, as the distribution of pairwise cosine similarities between pairs of MSCOCO images and pairs of MSCOCO text captions concentrate at positive values, whereas the distribution of pairwise cosine similarities across modalities concentrates closer to zero. (See Appendix Fig. (7). Not only does this prevent nonnegative decomposition, it also violates Assumption from Section B To rectify this, we mean-center CLIP images with the image
mean, estimated over MSCOCO
(Himg), ) and compute decompositions over the mean-centered concept vocabulary (Pcon). Note that the embeddings need be re-normalized after centering to ensure they lie on the unit sphere. To convert our decompositions back into dense representations (Zim)), we uncenter the normalized dense embeddings zime by adding the image mean back in and normalizing once again, ensure they lie on the same cone as the original CLIP embeddings mm).
Our optimization problem is formulated follows. Let x x/||x /||x||2 be the normalization operation. Given set of semantic concepts d'dog" "tabby cat", "cloudy", ... 1. we construct centered vocabulary C (x(")) ,. ... (g(x"")) - Mcco)), where we recall that g(-) is the CLIP text encoder. Now, given the dictionary C and a centered CLIP embedding z oz (img) ) we seek to find the sparsest solution that gives us a cosine similarity score of at least - c for some small c:
Optimization Problem. (x)
min ||w/oo s.t. (z, (Cw)) 2 1-
(1)
WERs

---

As is standard practice, we relax the to constraint and reformulate this as a minimization of MSE with an e, penalty, construct the following convex relaxation D of Eq. T
min ||Cw - = +
(2)
W WER
Given the solution to the above problem w", our recontturcnt embedding is: g!me = o (C"++/imm Himg))
5
Experiments
In this section, we evaluate our method to ensure that SpLiCE decompositions are interpretable, performant, and accurately reflect the semantic content of representations.
5.1
Setup
Models. All experiments shown in the main paper done with the OpenCLIP ViT-B/32 model [55] with results for an additional model in Appendix B. For all zero-shot classification tasks, we use the prompt template "A photo of (F" Datasets. We use CIFAR 100 [56), MIT States [57). CelebA [58], MSCOCO [59]. and ImageNetVal [60] for our experiments with results for additional datasets in the Appendix (Section B.4
Decomposition. For all experiments involving concept decomposition, we use sklearn's 61 Lasso solver with a non-negativity flag and an l penalty that results in solutions with lo norms of 5-20 (around 0.2-0.3 for most datasets). We use concept vocabulary chosen from a subset of LAION tokens as described in Section 4.1. Both image embeddings and dictionary concepts are centered and normalized as mentioned in Section[4.] with the image mean used for centering computed over the MSCOCO train set and the concept mean computed over our chosen vocabulary.
5.2 Sparsity-Performance Tradeofl's
We assess the performance of SpLiCE decompositions by evaluating the reconstruction error in terms of cosine similarity between SpLiCE representations and CLIP embeddings, the zero-shot performance of SpLiCE decompositions, and the retrieval performance of SpLiCE embeddings. We compare the performance of decompositions generated from our semantic concept vocabulary to decompositions over random vocabulary and learned dictionary vocabulary baselines. All vocab- ularies are of size 15,000 concepts. The random vocabulary sampled from a 512-dimensional normalized Gaussian distribution. The learned vocabularies are generated by using the Fast Iterative Algorithm (FISTA) 62 to learn optimal dictionaries given our sparse recov- ery problem (optimizing Equation (I) for both C and w). Note that we learn separate dictionaries Cimg and Cwst to reconstruct MSCOCO image and text embeddings respectively. In Figure 3 we plot the cosine reconstruction and zero-shot accuracy of image decompositions with the various dictionaries. We evaluate probing performance (Tables and text-to-image and image-to-text retrieval in the Appendix (Figure B.3).
These results overall show SpLiCE efficiently navigates the interpeeabiilty-accura Pareto frontier and retains much of the performance of black-box CLIP representations with the semantic, human- interpretable LAION dictionary, significantly outperforming other dictionaries on semantic tasks such zero-shot classification, probing, and retrieval. At the same time, find that our semantic LAION dictionary does not result in accurate cosine similarity reconstructions of the original CLIP, often being on par with using random dictionaries. We believe this is because CLIP encodes both semantics of the underlying image and non-semantic "noise", which violates Assumption #2 in Section 3 Given that our SpLiCE decompositions only aim encode semanties, they are unable to encode non-semantic aspects in the underlying representation, thus causing poor alignment in the cosine similarity sense, while simultaneously exhibiting excellent alignment on semantic tasks such as zero-shot accuracy. For ImageNet, we find that many classes are animal species that cannot easily be described by 1-2 words (e.g. tred-breasted merganser', 'Americ ian f frafhorddsire terrier'). ) Adding these class labels to our concept dictionary increases performance significantly, as shown by the dotted yellow line in Figure 3
For more discussion the relationship between Eq. 1 and Eq. Appendix, Sec. A.2
7

---

"African Elephant Cless
"Curly-coated Retriever Class
CIFAR100 'Swimwear' Concept
elephann
bnby eepoor
no0 0og
Woman
sorhh stfic
tradd
grasing
00g collar
32ened
10"
femalies
3etiover
Monaarch Butterfly' Class
"Digital Clock Class
10"
bytoff
Ster
countiow
mecro shot
3od dapleyy
nettor
acoraboerd
10"
0,000
0.755
0.175
Concept Weight
Figure 4: Left: SpLiCE decompositions of ImageNet 'African Elephant', 'Curly-coated Retriever', 'Monarch Butterfly'. 'Digital Clock' classes. "Swimwear" concept in 'Woman' and 'Man' classes of CIFARI00.
Right: Distribution of
CIFAR100 Zero Shol Accuracy
CIFAR100 Cosine Reconstruction
MSCOCO Sementic Relevancce
0.76
Sunards Ralerarca
3.03
0.00
0.98
50 120
140
22
40
50
100
10
20
30
42
Decompasiton 10 Norm
Decompasition Norm
Docompositin 10 Nor
SpLICE - GPT-generated vocab
Negative concept weights - No modality alignment --. CLIP
Figure 5: Ablation study evaluating the efficacy of SpLiCE design choices across three metrics: Zero-shot accuracy, cosine reconstruction, and semantic relevance of recovered tags. We find that all of our design choices, namely non-negativity, modality alignment, and usage of large task-agnostic dictionary are essential to performance. See $5.3 for discussion.
5.3 Ablation Studies
We perform ablation studies to evaluate the effectiveness of the design decisions of SpLiCE, including the choice of vocabulary, the nonnegativity of the decompositions, and the modality alignment by ablating each choice and observing the effect three metrics: zero-shot accuracy on CIFAR 100, cosine similarity between the reconstructed and original embeddings of CIFAR 100, and semantic relevance MSCOCO. The first two metrics are the same those presented in Figure 3 semantic relevance by tokenizing and filtering stop-words from the MSCOCO numan-generann captions and embedding each token with CLIP. Then, we take all non-zero concepts output by SpLiCE and compute the Hausdorff distance between the sets of SpLiCE concepts and caption token embeddings. This essentially measures how aligned decompositions observe that replacing our dictionary with the L-M--enerated concept dictionary used by [2 24 23 significantly worsens the decomposition in terms of zero shot accuracy and cosine reconstruction. While allowing for negative concept weights improves cosine reconstruction marginally, decreases the semantic relevance of the decompositions, negative concepts frequently correspond concepts not present in images, and as such, are unlikely to be represented by human captions. Finally, we see that modality alignment is necessary across all three metrics. Overall, these ablation studies show that each aspect of SpLiCE is necessary for creating numan-interpretabbe semantically relevant and highly performant decompositions.
We compute
with human captions. We
5.4 Qualitative Assessment of Decompositions
Concept Decompositions for Images. We visualize SpLiCE decompositions to qualitatively assess the semantic content the images they represent. In Figure provide six sample decompositions from MSCOCO with their corresponding captions. We display the top seven concepts for each

---

image and find that they generally well describe the semantics of the images. We also find that these qualitative examples yield interesting and unexpected insights into both CLIP and the data, In the top left image, we see that the decomposition includes the text present on the sign in the image, revealing that CLIP prioritizes text images over objects. For the bottom left image, the decomposition correctly includes the concept "macro shot", revealing that CLIP encodes information regarding geometric perspective. The bottom right decomposition similarly features the concept "blackandwhite bw", indicating that CLIP encodes not only the objects present in images but also information about the lighting and color. Overall, these results suggest that SpLiCE may also be used zero-shot image tagging method to understand images.
Concept Histograms for Datasets. Beyond concept- based explanations of individual images, we propose that SpLiCE can be used to better understand and summarize collections of images, such as entire datasets. To compute concept decompositions of sets of images, we decompose each individual image and aggregate the results, which we use to generate concept histograms of the dataset. We visualize four concept histograms for the ImageNet classes 'African Elephant', "Curey-coated Retriever', 'Monarch Butterfly', and 'Digital Clock', in Figure 4 These decompositions provide information about the distribution of the data and how CLIP represents it. For example, digital clocks are differentiated from analog clocks through the concepts "lcd display" and "countdomm". Monarch butterflies are highly correlated with the concept "lilac flowers" in ImageNet, which we validated through manual inspection (nearly half of the monarch butterfly images in the validation set feature purple flowers). Interestingly, Carry-coatedd retrievers are represented combinations of "poodle". "retriever dog". and "black dog", which perfectly describe the main characteristics of them: black retrievers with poodle-textured fur.
6 Case Studies and Applications of SpLiCE
In this section, we present two example case studies using SpLiCE: (1) spurious correlation and bias detection in datasets and (2) debiasing classification models. We present additional case studies for (1) and (2), as well as (3) monitoring distribution shift in Appendix [36 B.7 present results from user study to evaluate the human interpretability of SpLiCE in Appendix B. where we find that users prefer explanations generated by SpLiCE over existing Concept Bottleneck Model-based methods.
B B.9 We also
Discovering Spurious Correlations in CIFAR100. Existing methods to detect spurious correla- tions in datasets generally require subgroup and attribute labels rely on manual human inspection of images (see [63] for an overview), making it hard to scale to large datasets. SpLiCE, on the other hand, allows for fast automatic detection of such biases, without any labels, training, even a task. To illustrate this, we study two classes of CIFARI00: 'man' and 'woman', in Figure 4 Upon decomposing these classes, we found that ("bra", "swimwear" were two of the top ten most common concepts in the 'woman' class. On the other hand, the only clothing- related concepts that appear in the top 50 most activated concepts for 'man' are l"uniform", "tuxedo", "apparel") We visualize a histogram of the concept weights on swimwear- and undergarment-rellin ("swimvear", "bra", "trunks", "underwear") across both the train and test sets, and find that these concepts are much more likely to be activated for women than men. Manual inspection of CIFAR 100 verifies the trend highlighted by SpLiCE, where at least 70 of the 600 images in the 'woman' class feature women in bikinis, underclothes, or even partially undressed, revealing stereotype bias in this popular dataset. We provide a similar study of the concept "desert" with respect the 'camel' and 'kangaroo" classes in CIFARI00 in Appendix B.6
concepts
Model Editing on CelebA Attribute Classifiers. Concept-based representations unlock a key application: being able to intervene on and edit models. This edit can be performed in two equivalent ways: either on the concept representations themselves, where we can zero out a concept or on linear probes built upon the decompositions, where we can edit the weight matrix between concepts and class labels (similar to concept bottleneck models [88). Here, we evaluate the efficacy of SpLiCE for these forms of model editing. Specifically, we consider two tasks on CelebA, classifying gender and whether the subject wearing glasses. To test representation editing, we remove the concept of "eyeweer"" or "glasses" from CelebA image representations by zeroing out any weight placed on these concepts in our SpLiCE decompositions and evaluate classifier performance. We report the performance of sero-shot classification and lincar probes over our SpLiCE representation in Table

---

Table 2: Evaluation intervention on the concept 'Glasses' for the CelebA dataset. SpLiCE allows for surgical removal of information related to whether or not someone is wearing glasses, without impacting other features such as gender. (ZS = Zero Shot Accuracy)
GENDER
GLASSES
7. CLIP SPLICE
0,98
0.91
0.97
0.88
ZS INTERVENTION SPLICE
0,96
0.69
LINEAR PROBE INTERVENTION PROBE
0.89
0.88
0.85
0.59
In both cases, find that surgically target and remove information pertaining to glasses and reduce classifier performance while preserving information relevant to gender classification. We perform similar experiment the Waterbirds dataset [64] remove spurious background signals in B.7
7 Discussion
In this work, we show that the information contained in CLIP embeddings can be approximated by sparse, linear combination of simple semantic concepts, allowing us to interpret representations via sparse recovery. We propose SpLiCE, a method to transform the dense, uninterpretable embeddings of CLIP into human-interpretabbe sparse concept decompositions.
We empirically demonstrate that SpLiCE allows for an adjustable tradeoff on the interpretability- accuracy Pareto frontier, enabling users to decide the loss in performance they are willing to incur for interpretability. Furthermore, we find that the improved interpretability of SpLiCE allows for users diagnose and fix model mistakes, ideally increasing the effectiveness and performance of the overall system using VLM. We then provide concrete use cases for SpLiCE: spurious correlation detection and model intervention and editing, showcasing the benefits of using interpretable embeddings with known semantic content. We highlight that SpLiCE embeddings can serve as post-hoc interpretations of CLIP embeddings and can even replace them to ensure full transparency.
Broader Impact. Similar to many works in the field of interpretability, our work provides greater understanding of the behavior of models, including but not limited to the broader implicit biases they perpetuate as well as mistakes made on individual samples. We believe this particularly salient for CLIP, which is used in a variety of applications that are widely used in practice at this moment. We hope that insights gained from such interpretability allow users make more informed decisions regarding how they interact with and use CLIP, regardless of their familiarity with machine learning or domain expertise in the task they are using CLIP for. We also highlight that SpLiCE can be used as visualization- tool for exploring and summarizing datasets scale, allowing for easier auditing of spurious correlations and biases in both datasets and models.
Limitations. In this work, we large, overcomplete dictionary of one- and two-word concepts, however future work may wish to expand this dictionary or learn a dictionary over tokens (in discrete language space), to capture concepts with more than two words. This may also reduce the size of the dictionary and improve computation time. We note that this dictionary was constructed by looking at token frequency in the LAION-5B dataset, which has its all the salient concepts that CLIP encodes. Despite this, we find that SpLiCE performs well on a variety of tasks while outperforming state-of-ther of-the--tttttttt-- art concept dictionaries (Fig. 5 Appendix Fig. [3] and thus we believe LAION is good dataset to generate a concept vocabulary from. We also note be easily modified by practitioners consider additional concepts needed
biases and may not correctly capture
that this vocabulary for specific use cases. Finally, SpL.CE also uses an & penalty as the relaxation for &o regularization, but future work may consider alternative relaxations or even binary concept weights.
10

---

Acknowledgements and Disclosure of Funding
This work is supported in part by the NSF awards IIS-2008461, IIS IIS-2040,,,, IIS-2238714, FAI- 2040880, and research awards from Google, JP Morgan, Amazon, Adobe, Harvard Data Science Initiative, and the Digital, Data, and Design (D") Institute at Harvard. AO is supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-2140745 and UB is funded by the Kempner Institute Graduate Research Fellowship. The views expressed here are those of the authors and do not reflect the official policy position of the funding agencies.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 18-24 Jul 2021.
[2] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09733 2021.
[3] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/23, 1(2):3, 2022.
[4] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
(5) Yoshua Bengio. Deep learning of representations: Looking forward. In International conference on statistical language and speech processing, pages 1-37. Springer, 2013.
[6] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):
c7, 2017.
[7] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International conference on machine learning, pages 2668 2677. PMLR, 2018.
[8] Kyle Hsu, Will Dorrell, James CR Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantization. arXiv preprint arXiv:2305.
2023.
[9] Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't 't trust your eyes: on the (un) reliability of feature visualizations. arXiv preprint arXiv:2306.0477 04719, 2023.
[10] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746-751, 2013.
[11] Kiho Park, Yo Joong Choe, and Vietor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.0366 2023.
[12] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018.
[13] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent vari- able model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
[14] Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah Smith. Sparse over- complete word vector representations. arXiv preprint arXiv: 50550000000 2015.
[15] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why lasion-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference Learning Representations, 2022.
11

---

[16] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint 11X1:2 2022.
[17] Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6820-6829, 2023.
[18] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 1338-5348. PMLR, 2020.
[19] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference on computer vision and pattern recognition, pages 951-958. IEEE, 2009.
[20] Lorenzo Torresani, Martin Szummer, and Andrew Fitzgibbon. Efficient object category recogni- tion using classemes. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part I 111, pages 776-789. Springer, 2010.
[21] Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision, pages 365-372. IEEE, 2009.
[22] Aditya Chattopadhyay, Ryan Pilgrim, and Rene Vidal. Information maximization perspective of orthogonal matching pursuit with applications to explainable ai. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
[23] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06129, 2023.
[24] Konstantinos Panagiotis Panousis, Dino lenco, and Diego Marcos. Sparse linear concept discovery models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2767-2771, 2023.
[25] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3))e00024-000 001, 2020.
[26] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
[27] Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8730-8738, 2018.
[28] Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowl- edge in alphazero. Proceedings of the National Academy of Sciences, 119(47):e2226625119 19, 2022.
[29] Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed, On interpretability of deep learning based skin lesion classifiers using concept activation vectors. 2020 international joint conference on neural networks (IJCNN,, pages 1-10. IEEE, 2020.
[30] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 119-134, 2018.
[31] Thomas Fel, Victor Boutin, Mazda Moayeri, Rémi Cadène, Louis Bethune, Mathieu Chalvidal, Thomas Serre, et al. A holistic approach to unifying automatic concept extraction and concept importance estimation. arXiv preprint arXiv:2306. 07304, 2023.
[32] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, page 2, 2023.
12

---

[33] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[34] Pierre Comon. Independent component analysis, a new concept? Signal processing, 36(3):
287-314, 1994.
[35] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):41 1-430, 2000.
[36] Aleksandar Makelov, Georg Lange, and Neel Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. arXiv preprint arXiv:2311 17030, 2023.
[37] Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi. Text-to-concept (and back) via cross-model alignment. In International Conference on Machine Learning, pages 25037-25060. PMLR, 2023.
[38] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205. / 15480, 2022.
[39] Tian Yun, Usha Bhalla, Ellie Pavlick, and Chen Sun. Do vision-language pretrained models learn composable primitive concepts? arXiv preprint urXiv:2203. /7271, 20222
[40] Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. Stair: Learning sparse text and image representation in grounded tokens. arXiv preprint arXiv:2301.1303 2023.
[41] Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representa- tion via text- decomposition. arXiv preprint urXiv:3310.09916, 2023.
[42] Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprini urXiv:2203.05794 2022.
[43] Brian Murphy, Partha Talukdar, and Tom Mitchell. Learning effective and interpretable semantic models using non-negative sparse embedding. In Proceedings of COLING 2012, pages 1933- 1950, 2012.
[44] Alona Fyshe, Partha P Talulkar,, Brian Murphy, and Tom M Mitchell. Interpretable semantic vectors from a joint model of brain-and text-based meaning. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2014, page 489. NIH Public Access, 2014.
[45] Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell. A compositional and interpretable semantic space. In Proceedings of the 2015 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 32-41, 2015.
[46] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23)
1997.
[47] Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381 (6588))607-609,, 1996.
[48] Vikram V Raaaasammy, Sunnie SY Kim, Ruth Fong, and Olga Russakovsky. Overlooked factors in concept-based explanations: Dataset choice, concept salience, and human capability, arXiv preprint arXiv: 1596.. 2022.
[49] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1 1135-1144, 2016.
[50] David P Vinson and Gabriella Vigliocco. Semantic feature production norms for large set of objects and events. Behavior Research Methods, 40(1):183-190, 2008
[51] Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37 (4):547-559, 2005.
[52] Peter Garrard, Matthew A Lambon Ralph, John R Hodges, and Karalyn Patterson. Prototypi- cality, distinctiveness, and intercorrelation: Analyses of the semantic attributes of living and nonliving concepts. Cognitive neuropsychology. 18(2):125-174, 2001.
13

---

[53] Christoph Schubmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m Open dataset of clip-filterdd 400 million image-text pairs. arXiv preprint arXiv:21 (tw::2/1..020 1.02/ /2/00/1// 14, 2021.
[54] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612-17625, 2022.
[55]
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. doi: 2228///oooo..13777 URL If you use this software, please cite it as
https://doi org/ 10. 2281/zenodo.514373
5143773
below.
[56]
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[57] Phillip Isola, Joseph J. Lim, and Edward H. Adelson. Discovering states and transformations in image collections. In CVPR, 2015.
[58] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[59] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[60] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei--Fii ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[61] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825-288 28300, 2011.
[62] Amir Beck and Mare Teboulle. A fast iterative
algorithm for linear inverse problems. SIAM journal imaging sciences, 2(1):183-202, 2009.
[63] Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Concept- aware mitigation of spurious correlation. In International Conference on Machine Learning, pages 37765-37786. PMLR, 2023.
[64] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case 2019.
generalization. arXiv preprint arXiv:1911.0877
[65] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed opti- mization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3(12:1-122, 2011
[66] Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611, 2006.
[67] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database:
I Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[68] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[69] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. htt:://www.pascal- wwwpe///////////////ooo/oooooooo
[70] Michael Stark, Jonathan Krause, Bojan Pepik, David Meger, James J Little, Bernt Schiele, and Daphne Koller. Fine-grained categorization for 3d scene understanding, International Journal of Robotics Research, 305133:1533-1552 2.
14

---

Appendix
Summary of Appendix Results
Further Details on the Method
When do Sparse Decompositions Exist?
Relationship between cosine similarity and MSE optimization
ADMM for batched on-device LASSO optimization
Effect of Modality Alignment
A.5
Experimental Details
B
Additional Results
B.1
User Study for Human Interpretability
B.2
Performance of SpLiCE on Probing Tasks
B.3
Performance of SpLiCE Retrieval Tasks
B. 4
Additional Zero-Shot Results
B.5
Additional ImageNet Concept Histograms
B. 6
Additional Case Study: Detecting Spurious Correlations
B. 7
Additional Case Study: Spurious Correlation Intervention
B.8
Additional Case Study: Distribution Shift Monitoring
B.
Additional Case Study: Distribution Shift Monitoring
B. 10
Checking the Interpretability of Negative Concepts
B.11
Understanding the Image Mean for Modality Alignment
B.12
Choice of Concept Vocabulary
B.13
Concept Type Distribution
B.14
Experiments on Alternative CLIP Architecture
A Further Details on the Method
A.1 When do Sparse Decompositions Exist?
Proposition 1. Given Assumptions 1-5, CLIP image embeddings f can be written as a sparse linear combination of text embeddings, i.e,
f(x = O" s.t. || S a
where w E Re, and C Bdxk which is the text concept dictionary defined previously.
Proof. Any vector w can be written as w = E wie, where W, E R4, and e, E IR is a one-hot vector with one at the 7hh co-ordinate. Thus we have
f(x!"E) =f himee E) (im)
(Assumption 2)
k
=f himg Σ
wie:
Σ wi f (em)
(Assumption 3)
Here we define c, = f (c;) as the 'image' concept basis vector; analogous to the text concept basis vector c,tt o (c;) already defined. Thus Assumption implies the existence of sparse decomposition of f in terms of 'image' concept vectors dime Additionally, Assumption ensures that this decomposition is sparse, as w is sparse. So far, we have f(xime) = Cime s.t. ||l|| S
From Assumption 4, the image concept vectors and text concept vectors are equal to each other, i.e, imm = f h (e;) = g (e,) c." Finally, from Assumption 5, we have that the text concept C,
vectors ca given simply by word embeddings g of individual words.
15

---

Stringing these arguments together, we have that image representations f(x" can be written as sparse linear combination vectors obtain from CLIP word embeddings c. We finally set w w. thus proving the assertion.
A.2 Relationship between cosine similarity and MSE optimization.
Recall our & relaxed cosine similarity optimization problem from Equ. ①..
Cw
min
||wllo s.t. (z,
2 1 1
(3)
WER
||Cw]1
First we relax the lo constraint to an l penalty.
Cw
max (z, ERRE
A//w|11-
(4)
|Cwl12
By observing that || - (z v y) = (a, x) + (y, 4) 2(x, 3) and that z, [[Ow] are unit-norm, maximizing the above inner product is equivalent minimizing the euclidean norm,
Cw
Cw |Cwl2
min
2112 + 2\//w//1.
weRq
This is non-convex problem, but we can relax this problem to achieve better reconstruction in terms of euclidan distance as shown in Eqn. 2
min Cw 2112 + 2A//w//l
(6)
WERE
This problem will optimize euclidean distance between Cw and z. Consider two vectors æ, y on the unit sphere such that [|l]|] [M] > O. While any vector ay, 0 will have the same cosine similarity score, the optimal vector in terms of euclidean distance æ the vector ay such that p or in other words the projection of x onto Thus, solving for euclidean distance to approximate will find ay which we must then normalize to find the unit-norm solution y. This explains the normalizing process described in Section 4.1
Additionally, view Eqn. (6) applying shrinkage to Cw. Reconverting from euclidean norm to inner product, Eqn. (6) becomes
max (Cw, z) WER
2 (Cw, Cw) - Al|w||1 (Cw, z)
C/ | A|[w|11-
(7)
In conclusion, our optimization problem maximizes the inner product while imposing shinkage penalty and sparsity penalty. Empirically, our reconstructions Cw are low-norm, so we normalize after solving recover the unit-norm reconstruction.
A.3 ADMM for batched on-device LASSO optimization.
As each decomposition requires solving LASSO optimization problem, we implement the Altemat- ing Direction Method Multipliers (ADMM) algorithm Pytorch over batches with GPU support for efficient decomposition of large scale datasets over large numbers of concepts [65] In practice, ADMM achieves primal and dual tolerances of of 1024. We present empirical comparison beterr LASSO and ADMM in 6 where we find both methods to be approximately equivalent.
fewer than 1000 iterations on a batch size
Next we derive the iterates for our ADMM algorithm. Recall our optimization problem,
min
Cw x 22//w/h
(8)
WERS
ADMM breaks down convex optimization problems into multiple sub-problems while penalizing the difference in solutions. We break Eqn. g into two subproblems, one solving the euclidean distance
16

---

objective and one solving the & and nonnegativity constraint. We let w denote the former solution, the latter, and tracks the difference between the two. Our ADMM iterates (w 2 u* are
w ++1 = arg min (f(n(f(w) + E1lu** 2 + 1²1)
(9)
(St/p(u + +)²))++
(10)
" k+1 = n² + k+1 - - =+1
(11)
where S₂ is a soft- soft-thresholding function used to satisfy the LASSO constraints,
a K; a K 0, aa aa+ K, -
(a
(12)
As our optimization function f(w) is quadratic, we can analytically compute w++1
wh+ (2C* C p) (pe + 2Cw),
(13)
where v = ** - n²t In experiments set p 5, and stop when tolerances Cprim ||a***1 2 ||2, Edual || x*) ll are less than 1e 4. Over batch, we iterate until every solver in the batch has reached the above tolerances.
CIFAR10 Zaro Shot Accuracy
CIFAR130 Cosine Reconstruction
288
26
297
Iion, is60
23
leva aften
80
142
22
140
Figure 6: Comparison of ADMM (maroon) and LASSO (yellow) for solving the SpLiCE objective on zero shot accuracy (left) and cosine reconstruction (right) on CIFAR100. Both methods are approximately equal.
A.4 Effect of Modality Alignment
We take MSCOCO images and captions, embed them with CLIP, and compare the cosine similarity between modalities and inter-modality. Before mean-centering and renormalizing, the similarity within modalities is high, with average of around 0.3. This indicates that the image and text embeddings do not span the entire unit-sphere but rather lie on two cones. However, the similarity across modalities has average concentrating around zero, indicating that these two cones non-overlapping. However, after mean-centering and normalizing, we observe that the average cosine similarity for images, text, and between images and text becomes and the modalities are aligned.
CLIP Modality Similarity
CLIP Modality Similarity with Centering
3.0
img
img
text_text
text_test
img img
img
Cosine Similerity
Casine Similarity
Figure 7: Average cosine similarity across pairs of image-text, image-image, and text-text data from MSCOCO. After aligning modalities, the distribution of similarities centered around zero.
17

---

A.5
Experimental Details
All experiments are able to be performed on a single A100 GPU run fast inference with CLIP. After embedding the concept dictionary, all computation can be performed on CPU. Code is made available at httts://githhb. com AILLIFE GROUP/SpLiCE C
B
Additional Results
B.1
User Study for Human Interpretability
We present results from a user study in 8 to assess the human interpretability of SpLiCE. We base our study off of that performed by [23] evaluate Label-Free Concept Bottleneck Models (LF-CBMs)). We benchmark our method against LF-CBMs and IP-OMP [22]. We provided users with twenty randomly chosen, correctly predicted images from ImageNet and explanations from two different methods comprising the top six most important concepts for every image. We then asked users to evaluate and compare the different concept-basse image inputs, (2) their relevance to model predictions, and (3) their informativeness on Likert scales from 5. We found that users significantly preferred explanations generated by SpLiCE to the two baselines for relevance to the images and informativeness, with significance determined via one-sample two-sided t-test and a threshold of p=0.01. We also highlight that our method is able to produce similar/better concept decompositions, in terms of human interpretabilitty than the baselines without needing to train a classification probe or use class labels for concept mining, both of which computationally expensive. This user study ruled exempt by institution's IRB, risks were posed to the users. Participants were able to opt out at any time, and no questions were asked regarding the participants themselves.
explanations for (1) their relevance to the provided
User Study Comparing SpLiCe and LF-CBM
User Study Comparing SpLiCe and IP-OMP
Goonly Apiin
Wheil pater
Biingy
Bing
Percentage
Percentage
Figure 8: Results of a user study evaluating SpLiCE, LF-CBM, and IP-OMP in the style of the user study from LF-CBM. Overall, find that explanations generated by SpLiCE are deemed more relevant to the image, relevant to the prediction, and more informative than prior methods.
B.2 Performance of SpLiCE on Probing Tasks
We evaluate the performance of the decompositions probes trained on both regular CLIP embed- dings as well as decomposed CLIP embeddings for CIFARION 100 in and MIT States in 4 We consider two scenarios: a probe trained CLIP embeddings and tested on SpLiCE embeddings of various sparsities (shown in row CLIP Probe), and a probe both trained and evaluated on SpLiCE embed- dings (shown in row SpLiCE Probe). We report mean for each experiment being less than 0.005. We find that SpLiCE representations closely match the performance of dense CLIP embeddings, with slight drop in performance when probes trained directly on SpLiCE embeddings rather than trained on CLIP embeddings and evaluated on SpLiCE embeddings for CIFAR100.
three runs, with standard deviations
Table 3: Evaluation of Probing Performance on CIFAR100
lp =
lg = 6
lg b 23
1 = 117
CLIP
SPLICE PROBE
0.95
0.95
0.95
0.95
CLIP PROBE
0.96
0.96
0.97
0.97
0.97
18

---

Table Evaluation of Probing Performance on MIT States
l==4 4
10=7
lo 27
CLIP
SPLICE PROBe
0,883
0,883
0.882
CLIP PROBE
0.883
0.883
0.884
0.883
B.3 SpLiCE Performance on Retrieval Tasks
We test the performance of SpLiCE embeddings on text-to-image and image-to-text retrieval tasks. We evaluate retrieval over various 1024 sample subsets of MSCOCO, and assess recall perfor- mance for the top-k closest embeddings of the opposite modality for k f1, 5, 10). We find that baselines when decomposition sparsity is high,
our semantic concept dictionaries outperform but that dictionaries learned over images perform slightly better for text to image retrieval when decompositions have greater than 30 nonzero concepts.
Image to Text Retrioval (1K Test Saf)
Image to Text Retrieval (1k Test Sat)
Imaga Text Ratrioval (1k Test Set)
3.8
3.6
2.4
03
23
90
83
100
120
140
40
90
83
159
120
143
42
83
83
100
120
143
Decorpostion
Dosomposiion
Text to Image Retrioval (1k Test Set)
Text to Image Retrieval (1k Test Set)
Text to Image Retrivval (1k Test Sel)
2.6
2.5
23
83
100
120
143
49
60
150
100
143
0
45
43
100
120
143
Desorpoziionn
Docompoziion
(magee)
(cmsalinn)
classess
Figure 9: Top-1 5, 10 performance of SpLiCE representations on image-to-text (top) and text-to- image (bottom) retrieval on MSCOCO.
B.4 Additional Zero-Shot Results
We present additional results comparing SpLiCE reconstructed vectors and CLIP embeddings the 1altech101 [66]. SUN397 (67), STL10 68), and VOC2007 [69] datasets in We use SpLiCE decompositions with sparsities of 20-35, and we find that they are comparable to the unaltered CLIP embeddings.
Table 5: Additional zero-shot accuracy on baselines from the CLIP paper, for decompositions of sparsity 20-35. Note that at human-interpretable levels of sparsity, we see minor drop in performance.
CALTECH101
SUN397
STL10
VOC 2007
CLIP REPORTED 0.88
0.63
0.97
0.83
CLIP IMPLEMENTED
0.90
0.67
0.96
0.92
SPLICE
0.86
0.66
0.96
0.83
19

---

We further explore the performance of SpLiCE decompositions in the limit they approach the sparsity of the baseline CLIP embeddings (512). We find that SpLiCE completely recovers CLIP zero-shot accuracy at this limit, as shown in 6
Table 6: Zero shot performance at sparsity 512. Note that SpLiCE completely recovers baseline CLIP zero shot accuracy.
CIFAR100
MITSTATES
IMAGENET
CLIP BASELINE SPLICE (512)
0.750 0.768
0.469 0.474
0.552
0.552
B.5 Additional ImageNet Concept Histograms
We present concept histograms for the top seven concepts of five more ImageNet classes: ('Face Powder', 'Feather Boa', "Jack-O-Lantern" -Lantern', "Kimono", 'Dalmation') J. similar to Figure [O] These decompositions give insights both into the distribution of each class as well some biases of CLIP. For example, for the class 'Face Powder', the concept "benefit" is the fifth most common concept, and it indeed a common cosmetic brand name in the images. For the 'Dalmation' class, we see that the decompositions consists of concepts relating to dogs and black and white spots, which together make up the high-level concept of dalmation. Finally, for the class 'Kimono', the concept "doll" the seventh most common, although all of the images in the 'Kimono' class were of real humans, not of dolls. This highlights an implicit bias in CLIP's representations or the descriptions of people wearing kimonos in CLIP's training set.
'Face Powder' Decomposition
'Feather Boa' Decomposition
Jack-O''Lantem Decomposition
coemelic
mardi gres
halloween pumpkin
shadow
fashion portrait
halloween
makoup
bacholoratie party
pumpkins
halloween
seary
benefit
feother
carve
products
fancy gay pride
jack
hair products
glowing
scio vuto 2003 12m AEO:
'Kimono' Decomposition
'Dalmation' Decompasition
kimono
dog breed
spots
japanese girl
cute dog
caremony
spotted
traditonal wedding
japaneee
dog portrait
blackandwhite
200
2.0
0.00
01.0
Figure 10: Example concept histograms of various ImageNet classes. The top concepts for each class are visualized along with their relative weighting, with the average &y norm of individual sample decompositions also being
B.6 Additional Case Study: Detecting Spurious Correlations
We present an additional case study for detecting spurious correlations in CIFAR100. particular, we look at the prevalence of the spurious concept "desert" in the classes 'camel' and 'kangaroo" in Figure II We observe that camels are more frequently pictured in the desert, creating a spurious signal that may be leveraged by downstream classifiers. This figure provides additional example of how we can understand biases and trends in data with SpLiCE decompositions.
B.7 Additional Case Study: Spurious Correlation Intervention
We further test the ability of SpLiCE to enable intervention on intermediate representations and linear classifiers by attempting to remove information pertaining to spurious signals. In particular, we consider the Waterbirds dataset (64), which spuriously correlates landbirds with land backgrounds, resulting trained classifiers performing poorly on waterbirds on land. We thus remove information about whether or not birds on land backgrounds by ablating concept weights "bamboo",
20

---

CIFAR100 'Desert' Concept
Camel
Kangaroo
10
Count
10'
10
0.00
0.05
0.10
0.15
Concept Weight
Figure 11: Distribution of "Desert" concept in 'Camel' and 'Kangaroo' classes of CIFARI00.
Table 7: Evaluation of intervention spurious correlations for Waterbirds dataset. Removing information about land backgrounds improves worst-case subgroup performance.
LANDBIRDS ON LAND
WATERBIRDS ON LAND
LINEAR PROBE
0.98
0.48
INTERVENTION
0.97
0.60
PROBE
"forest", "hiking", and "rainforest" as well as any bigrams containing the word "forest," as shown in Table 7 This significantly improves worst-case subgroup performance for waterbirds land from 0.48 to 0.60.
For both this experiment and the intervention CelebA described in the main paper, train linear probes using the LogisticRegresiooCllssii module in scikit-learn using an & penalty.
B.8 Additional Case Study: Distribution Shift Monitoring
We present a final case study using SpLiCE to monitor distribution shift. This can help identify differences between training and inference distributions or evaluate how a continually sampled dataset changes over time. this experiment we consider the Stanford Cars dataset [70), which contains photos of cars from 1991 to 2012, including their make and year labels. By decomposing photos of cars from each year, we can view how the distribution changed yearly. We visualize the weights of the concepts "convertible" and "yellow" from our decompositions, as well as the actual percentage of cars from each year that were convertibles or ycllow in Figure [2] Note the right-hand y-axis, corresponding the weight of the given concept e; over the of the weights of all concepts E, C.. does not have a meaningful unit of measure or seale. We find that the trends the groundtruth concept prevalence generally closely match that of the predicted/decomposel concepts, allowing us to visualize which years convertibles or yellow cars were popular or out-of distribbuioon with respect that SpLiCE picks up on the -utto--distribution rise in popularity
to other years. Most notably, of brightly colored sports cars in the early 2000s.
B.9 Additional Case Study: Distribution Shift Monitoring
To further verify that SpLiCE allows for identification and tracking of distribution shift, we study the Waterbirds dataset, which is known to have differently balanced train, valodation, and test splits. To identify distribution shifts, we can simply look at the norm of difference between the class decompositions of the two classes for each splot, shown in 8 We find that the validation and test splits are much more similar than the training and validation splits or the training and test splits, which can be verified by the construction process of the Waterbirds dataset.
21

---

Stanford Cars Concepts Distribution Shift
0.25
Groundrutt Concrip Percem
"Witer Cancal Yergt
020
15
Concept
0.10
00 00
Year
Figure 12: Visualization of the presence of convertibles (pink lines) and yellow cars (yellow lines) in Stanford Cars over time. SpLiCE concept weights (dotted) closely track the groundtruth concept prevalence (solid) for both concepts.
Table 8: Study of the differences in distributions between train, validation, and test splits of Waterbirds. The validation and splits are much more similar each other than they are to the train split.
TRAIN, VAL
TRAIN, TEST
VAL, TEST
CLASS LANDBIRD CLASS WATERBIRD
0.0182
0.0182
0.005
0.0229
.0188
0.009
We also find that the most weighted concept the 'landbird' class of the train split "bamboo" but the corresponding weight for "bamboo" in the 'waterbird" class is much lower. The "bamboo" concept weight for both classes and all splits shown below, where we see that the validation and test splits are very similar and mostly evenly balanced, whereas the train split is highly unbalanced.
B.10 Checking the Interpretability of Negative Concepts
We take a set of 71 concept-antonym pairs from the MIT States dataset and embed the terms in CLIP. With and without concept centering, we observe that these concept-antonym pairs have average cosine similarity well above -1, indicating that CLIP does not place antonyms in opposite directions, shown in 10 Next, we take our concept dictionary and prepend "not" all of the words and compare the average cosine similarity between concept and not-concept pairs. Similarly, we observe that with and without centering, concept and not-concept pairs highly similar. Note that the average similarity for true pairs of images and text in MSCOCO is less than the similarity between concepts and not-concepts with and without centering.
B.11 Understanding the Image Mean for Modality Alignment
In order to empirically check that the mean centering of images does not result in a loss of information, we decompose the img mean, Ping- that we used for all experiments. If we decompose with uncentered concepts, the following concepts are highlighted: ("closeup". "flickr", "posed"). The decomposition with centered concepts results the following concepts: ("flickr", "posed", "pics", "angle view", "last post"). These concepts all seem to be generally related to images, with minimal other semantic information, suggesting that centering does not remove any discriminative semantic content of embeddings, but simply removes information about the modality.
Table 9: Study of the prevalence of the concept "bamboo" in the different classes and splits of Waterbirds.
TRAIN
VAL
TEST
CLASS LANDBIRD
0.0196
0.010
0.010
CLASS WATERRIRD
0.0007
0.008
0.008
22

---

Table 10: Evaluation of the similarity of antonyms and negative concepts in CLIP.
PAIRWISE COSINE SIMILARITY (WITHOUT CONCEPT CENTERING)
PAIRWISE COSINE SIMILARITY (WITH CONCEPT CENTERING)
CONCEPT AND ANTONYM
0.7176 = 0.1109
0.1366 + 0.2197 0.6130 + 0.0498
CONCEPT AND "NOT" CONCEPT 0.8661 + 0.0498
B.12 Choice of Concept Vocabulary
We perform a simple ablation study to assess the sensitivity of our method to choices in concept vocabulary. We collect second vocabulary in the same exact manner as the LAION vocabulary from the MSCOCO caption dataset. We consider both the top 10k and top 5k most common words for both, and repeat the zero-shot accuracy and reconstruction cosine similarity experiments from Section 5.2 on CIFAR 100. We see that the MSCOCO10K and LAION 10k vocabularies perform almost exactly the same for both metrics. The smaller vocabularies perform the same for cosine reconstruction but underperform the 10k vocabularies for zero-shot classification tasks.
Effect of Vocab on Cosine Reconstruction
Effect of Vocab on Zero Shot Accuracy
0.75
0.75
Similarit X)
0,70
0.70
0.88
0.50
Acturaty
0.65
0.55
Didhmmayy
0.50
coco (10k)
0.60
Ditionay
0.45
laion (10k)
5000 (10)
0000 (5K)
0000
0.40
laion (5K)
0.55
Bin (5))
CLP
20
40
60
80
100
120
20
40
100
120
Decomposition 10 Norm
Decomposition Norm
Figure 13: Change in SpLiCE performance when considering another semantic concept dictionary derived from MSCOCO as well as a smaller concept vocabulary.
B.13 Concept Type Distribution
In order to better understand any biases produced by the decomposition process or that CLIP itself has, we visualize the types of concepts most commonly activated across multiple datasets, labelling them by part of speech in Figure T4] We see that nouns are by far the most common concepts across datasets, indicating that both CLIP and the decompositions are highly object centric. Note that the low weight on verbs and adjective is due to far fewer concepts of those types being activated (low lo norm) as well as the weight upon those concepts being significantly smaller (low 4 norm). We hypothesize that the information in many adjective and verbs can actually encoded into the noun itself, resulting in this phenomenon. For example, the concept "lemon" is more succinct form of "yellow" and "fruit".
B.14 Experiments on Alternative CLIP Architecture
We present cosine reconstruction and zero-shot accuracy experiments with alternative CLIP architecture from OpenAl with ResNer50 backbone for the vision encoder. Note that these experiments were done with 10000 size vocabulary of only one- merd concepts. We find that results are similar to those presented in 3 save for OpenAl's ResNet50 CLIP performing much worse than OpenCLIP's ViT B/32 backbone general.
23

---

Distribution of Concept Types
200
COCO
175
MIT States
Concept Waights
150
CIFAR100
ImageNet
125
100
75
60
25
0
Noun
Verb
Adjective
Adverb
Part of Speech
Figure 14: SpLiCE decompositions are mostly comprised of nouns across multiple datasets.
CIFAR100
Reconshuctio
ImageNel
2.70
0.75
Ccese Bintrly
0.05
030
8:50
lalor runcom
blar
random
845
tcrcom
Decompoalion 0 Nor
Decompoation d Norm
Decompoation Nar
CIFAR100 Ascuracy
Acouracy
ImageNet Ascurasy
2.40
055
8.10
2.5
Decorposiioo 0 Nor
Doc0: ppopion 0 Morm
Dosorposiion 10 Morm
Figure 15: Performance of SpLiCE decomposition representations on zero-shot classification tasks (bottom row) and cosine similarity between CLIP embeddings and SpLiCE embeddings (top row) for OpenAl's ResNet50 CLIP model.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we introduce a novel method and provide comprehensive experiments demonstrating its utility on downstream tasks as well as various case studies in Sections S
and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In the appendix we note the limitations of our work ??.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification: We provide a detailed set of assumptions, proposition and proof sketch in the main paper section 3 and full proof in section A.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all information required reproduce the main results by providing details on the exact methods and metrics used and providing code to implement SpLiCE In our experiments (See 5) we disclose our datasets. Then, to replicate results, we describe how to compute zero-shot accuracy, cosine similarities, as well as how to generate concept histograms over classes and concepts as shown (Figure These are also included in our code. We also include additional experimental details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
26

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We use publicly-available datasets, MSCOCO, CIFARI00, MIT States, Ima- genet, CelebA, Waterbirds, and Stanford Cars and include code replicate results (link at the end of abstract).
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify hyperparameters and other implementation details in 5.1 We de- scribe how optimize for Eqn. (2) using scikit-learn, as well include implementation in our codebase (linked in the abstract).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Throughout the paper we report error bars when possible. For instance, when computing reconstruction error we are able report error bars (Fig. (3). However, for zero-shot error, this metric is fixed for any dataset we not report error bars there. When possible run experiments 5 times and include error bars.
27

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include details compute resources in the Appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Fthics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////aaapppssccc/pppttttttt
Answer: [Yes]
Justification: We do not have human subjects nor do collect additional data. When conducting semantic decomposition, there is a risk that we include harmful semantics with large dictionary. However, bias to help reduce it in the future rather than perpetuating bias. Semantic decompositions allow for more critical view of our machine learning systems and datasets. In addition, we have open-sourced SpLiCE with essential elements for reproducibility to ensure public governance of the system.
evidenced in Figure 4 our method actually uncovers
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
28

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include discussion broader impact in the (Appendix ??)
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For cxample, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As our method does not release data, nor does it include a generative model, we do not have a high risk for misuse. Thus, we do not have any special safeguards on our method.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all code and models in the paper.
29

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release cleaned and simple-to version of the model with working examples and an API. The code is linked at the end of the abstract.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
30

---

Answer: [Yes]
Justification: We present small-scale user study in our supplementary material, in which were asked rank explanations generated by our method and prior concept-based explanation methods. The study was ruled exempt by our institution's IRB, as no risks were posed the study participants. Further information is given in Appendix B.1
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "LGFAXWnNwd", "reviewer": "Reviewer_Vxod", "summary": "This paper proposes to decompose the representations of the CLIP model using dictionary learning; where the components of the dictionary are composed of human understandable concept directions. The procedure here is as follows: a concept list is constructed from a filtered set of unigrams and bigrams from the LAION-400m captions, the concept dictionary is set to be centered CLIP representations for each pre-specified concept in the list, and an optimization problem that minimizes the reconstruction loss (subject to an L-1 penalty) is specified. Solving that optimization problem results in a weight vector that indicates how the CLIP embedding can be decomposed along the pre-specified concept direction. Given this formulation, the paper solves the problem for the CLIP model and shows that it does not lead to a performance loss given thousands of concepts. In addition, the paper presents several case studies showing how the approach can be used to edit downsteam classifiers built on top of clip representations, and discover spurious correlations.", "questions": "**Unsupervised or residual concepts**: I am surprised that there wasn't any significant loss in performance between the Splice representations and the original CLIP representations. Is it that 15000 concepts is enough to capture the variation? It seemed like one would want to account for the 'residual' directions in the original embedding that the reconstruction doesn't account for. Is this needed? Or do the authors think we can always get away with this kind of supervision?", "limitations": "Appendix A discusses some of the limitations of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**\\\nThis paper is the first that I know to decompose the representations of CLIP with known concepts using dictionary learning. The work is well motivated. \n\n**Quality and Clarity**\\\nThe paper is well-written, and clear. The related work is also discussed, and each component of the workflow presented here is well-explained and clear. \n\n**Significance**\\\nThis work, in my opinion, indicates that CLIP type models can be easily debugged using post hoc supervised concept learning. Overall, I think this opens up the door to more intriguing work along similar lines for larger and more complicated models.", "weaknesses": "Overall, I discuss some of the challenges that I see with this work. I have not much substantive qualms with this work; I just think the presentation could be tightened. However this is up to the authors to either accept or reject.\n\n**Assumptions for the proposition not justified**:  While the assumptions made to prove proposition 1 seem natural at first glance, it is not clear to me that we can claim that they should be true. One, there is a claim that CLIP captures 'semantic' concepts and not 'non-semantic' concepts (assumption 2). In Section 5.2, the authors even confirm experimentally that the assumption is not true. Of course, the model does capture some semantic concepts, but we cannot claim that CLIP cleanly partitions these concepts as described. This is because we know of failure cases where CLIP errs precisely because it does not just capture only semantic concepts. See this paper for example (https://arxiv.org/abs/2306.12105). The failures of the CLIP model that have been observed make it clear that this assumption is not true.  Assumption 3 is somewhat problematic again because we have observed that CLIP representations indeed can be linear for *some* concepts, but it is not linear for all semantic concepts. In fact, how are we to know which concepts that CLIP considers to be semantic, and which it doesn't? Assumption 4 makes sense to me because CLIP is trained to 'align' both encoders. Assumption 5 also difficult to justify in my opinion, but I am willing to live with it. To summarize my challenge with this portion of the work, the proposition is correct, but we can't claim that CLIP behaves as described. I am bothered by these assumptions because the implication is essentially that CLIPs learns the data generating process defined here, which is not the case. This is actually a good thing because, as shown quite nicely in this work, the point of the concept dictionary 'layer' is to 'fix' the challenges with the original CLIP model.  Overall, I don't think this section of the paper can be fully justified.\n\n**task-agnostic and without concept datasets, training, or qualitative analysis of visualizations.**:  The bolded phrase is one of the motivations for the approach described here. This is not quite true. First, you need to collect the concept dataset, the processing done with the LAION dataset is *exactly* the process of collecting a concept dataset. You do need to 'train', in the sense that you can see the process of optimizing to obtain $w$ as training. Lastly, the layer results in concept scores that can be visualized as depicted in Figure 4, so the approach here also allows us to inspect . Overall, I think this claim is hyperbolic and should be relaxed. \n\n**Related Work and Discussion**: This work claims that it is not quite a concept bottleneck, but I disagree. It is exactly translating a concept bottleneck to CLIP. First, the methods does require annotations. To get the dictionary $C$, you need to collect a set of 10000 concepts, which are associated with particular images. The annotation process is exactly the process of manually constructing $C$. The concept list is supervised, meaning it is the authors that determine which concepts to include; of course, the process described here is not manual, but the authors should recognize that it is *still* a supervised process. The intervention procedure described in Section 6 is exactly how that is done in the CBM literature. In fact, the NMF formulation here is not new (see: CRAFT: Concept Recursive Activation FacTorization for Explainability for eg.). Overall, the approach described here is exactly what a CBM is, it is just not applied for classification. Having said all of this, I think it is fine to acknowledge the similarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions! We appreciate your feedback on how to improve this work and address your concerns below.\n\n**Assumptions not justified.** Thank you for this comment. Regarding Assumption 2, it is true that our experiments demonstrate the presence of non-semantic concepts in our decomposition. We clarify this point further in our general rebuttal, but our intent with Section 3 and our listed assumptions was not to claim that CLIP always satisfies these assumptions, but instead to reason from first principles regarding which mathematical properties enable vision-language models to have sparse decompositions. We plan to clarify this, move our discussion of these assumptions to the Appendix, and highlight our empirical investigations of CLIP’s semantic linearity (Assumption 3, experiments in Appendix B.5.) to the main paper, as suggested by reviewer djnP. We direct the reviewer to our general comment for further discussion of this point. \n\n**“Task-agnostic and concept datasets, training, or qualitative analysis”.** Thank you for bringing this to our attention. We will be sure to update and relax our language as suggested, but we would also like to clarify our intent with this phrase here. We say that our method does not require concept datasets, in that traditional Concept Bottleneck Models (along with other methods such as tCAV) require training data that has both concept and class labels to train concept probes, whereas our method does not require a task and concept labeled image dataset. Instead, we construct our dictionary by parsing the text captions of LAION. As such, our method can be applied out of the box for a variety of tasks, without having to collect image datasets for each concept or task.\n\nSecond, while we do need to optimize for our concept weights, we do not need to train any concept probes or classifiers and accordingly do not need any training data, as SpLiCE decompositions perform sufficiently in zero-shot applications. This is where our work differs from many other similar works, such as [22, 23], which require training a sparse linear layer to obtain both explanations and predictions. This also permits SpLiCE to be used post-hoc and in low-data regimes. \n\nThird, our point here was to say that we do not require qualitative analysis to label dictionary elements and thus generate explanations. More specifically, in prior work such as Sparse Autoencoders or neuron analysis in mechanistic interpretability research, dictionary elements are learned, uninterpretable vectors. As such, they require additional analysis to understand the semantics contained by each dictionary element, such as the auto-interpretation done in SAE literature, which involves decomposing large quantities of data, measuring correlations in the decompositions, and then qualitatively describing what each atom encodes. One of the benefits of SpLiCE is that we fix our concepts a priori rather than having to learn and then manually label them post-hoc which can result in errors. However, you are correct that we can use our resulting concept scores in downstream qualitative analysis by inspecting them and visualizing them.\n\nFor a detailed discussion of task-agnosticity, please see our general comment.\n\n**Related work.** The connection you draw between our work and CBMs is very insightful. Our intent in claiming that we are not simply creating a concept-bottleneck model is to distinguish ourselves from works that require training classifier probes and that only consider the predictive case, as noted by the reviewer. We believe that this is a significant contribution of our work, as CLIP is frequently used in many non-predictive settings (retrieval, generation, etc), and because these decompositions allow for dataset summarization for data that is not labeled or does not have an associated task (e.g. unstructured web-scraped data). We agree that our work creates a concept bottleneck to represent CLIP embeddings, and we will discuss this further in the final version. Similarly, we agree that the vocabulary construction process is overseen by the authors, but it is not “supervised” in the traditional ML sense of the word, in that there are no labels or tasks it is optimized for. We also attempted to impose as minimal oversight and constraints on the dictionary as possible, simply ensuring that it was human-interpretable by using only 1- and 2- word concepts and safe by filtering NSFW content. Finally, we thank the reviewer for the provided citation and kindly note that we compare our work to the follow-up work by Fel et al, [31], in our submitted paper, noting that methods such as CRAFT and traditional NMF require feature visualization to understand each concept. We will be sure to clarify this comparison further.  \n\n\n**Residual Concepts.** This is an interesting question and future direction! Given that our dictionary has 15000 elements in 512 dimensions, our problem is very overcomplete, so it is reasonable to assume we can reconstruct CLIP embeddings with little loss even under a sparsity constraint. One way we could account for this residual error is to add it back into our decompositions and return the original CLIP representations. However, doing so results in a component of the representations being unexplained. For example, 98% of an embedding may be explained by our sparse decomposition, but there is a risk that the 2% residual which remains unexplained contains important information for downstream tasks. Depending on the application, one may wish to incur this loss to ensure that all information remains interpretable, or they may add the residual back in for applications that require perfect performance. We find this discussion to be similar in nature to the comparisons between post-hoc explanations and inherently interpretable models, and we believe the choice should be left to the user. We will include this discussion in our final version. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "APC4EoFdlZ", "reviewer": "Reviewer_acbD", "summary": "This paper introduces a method to transform CLIP representations into sparse linear concept embeddings that are interpretable to humans. SpLiCE uses task-agnostic concept sets, demonstrating its versatility over prior works. SpLiCE provides interpretability without compromising zero-shot classification performance and shows further applications, including spurious correlation detection and model editing.", "questions": "[Q1] Does the concept decomposition differ between correct and wrong samples (from the perspective of zero-shot classification) ?\n\n[Q2] As the effectiveness of SpLiCE heavily relies on the quality and comprehensiveness of the concept dictionary derived from the LAION dataset, is there a possibility that limitations or biases in this dictionary could directly impact the performance and interpretability of the method?", "limitations": "Please refer to the weaknesses and questions part.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "SpLiCE suggests a novel method to interpret CLIP embeddings into semantic concepts, which could be a good way to understand the latent space of CLIP. As it is a task-agnostic approach, which does not constrain itself to a certain domain, it can be applied to various datasets to explore the CLIP embeddings. This task-agnostic concept shows its scalability across different datasets, including CIFAR100, MIT States, CelebA, MSCOCO, and ImageNetVal. Also, the authors demonstrate the efficacy of their method both quantitatively and qualitatively.", "weaknesses": "[W1] I believe the choice of datasets is insufficient to show the efficacy of \"task-agnostic\" concpet sets. For example, how does SpLiCE work on bird identification dataset or CelebA? What are the top activated concepts in these datasets?\n\n[W2] Adding class labels to concept dictionary seems inappropriate from the perspecive of \"concept decomposition,\" as in the case of ImageNet. If the top concept is class label itself, what is the need for concept decomposition? \n\n[W3] A user study on the interpretability of SpLiCE would be needed to quantify how interpretable this method is, as in [23].\n\n[Miscellaneous] I think line 96's citation [22] should be changed to [23].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We appreciate your helpful feedback and address your concerns below. We will also correct the typo you mentioned.\n\n**Efficacy of task-agnostic concept sets.** As requested by the reviewer, we include in the additional results concept decompositions of randomly chosen samples from Waterbirds and CelebA (Fig. 3b). We find that the decompositions include fine-grained and detailed concepts, such as the celebrity “Rihanna”, the brand “adidas” (in reference to a branded Adidas sweatband), and the species “pelican”. This shows qualitatively that our concept set is indeed task agnostic or at least broadly applicable to a variety of tasks. While we only include four in the additional rebuttal documents due to space constraints, we will include a more thorough set of examples in the final version. We also include quantitative results for a broader set of tasks from the original CLIP paper [1] in our Additional Materials (Table 1), as suggested by reviewer djnP. These results further indicate the wide applicability of our selected concept set. If the reviewer believes that the term “task-agnostic” is still inappropriate for this method, we are happy to change our language accordingly. \n\n**Adding class labels to the concept set.** Thank you for this comment. For ImageNet, we find that the class names are often fine-grained animal species that are more than two words long (such as “European fire salamander”, “sulphur-crested cockatoo”, “red-backed sandpiper”, etc), making it difficult for one- and two-word concepts to capture. As such, these concepts can be added to the LAION dictionary to allow for full reconstruction, while maintaining the interpretability of all other concepts in the image. We include this augmented vocabulary in our experiments to demonstrate the efficacy of our sparse decomposition method, even if it highlights a limitation of the LAION vocabulary.\n\nWe also note that many images, including those in ImageNet, can be complex and contain semantic information outside of the class object. Even if the top concept is the class label, the  rest of the decomposition may contain information relevant for prediction, especially if there are spurious correlations. Decompositions should include all of this information to allow for full understanding of the semantic content of the image, and more importantly to allow for use cases outside of prediction, such as editing. If you take the example of the Waterbirds dataset, we expect our decompositions to include both the class label and the land/water background concepts. Being able to edit the background concepts while maintaining the class label is useful for improved classification. Furthermore, this semantic information can be useful for other applications, such as dataset summarization, image tagging for retrieval, and exploration of unlabelled data. In summary, concept decompositions that include a combination of class concepts and other semantic information provide utility in a variety of settings beyond prediction; however, we will further clarify this in our final paper. \n\n**User study.** We agree that user studies are important for assessing the human interpretability of concept based explanations. We have included results from a small-scale user study similar to that suggested by the reviewer in [23] (Task 2) in Figure 1 of the additional materials, where we present users with an image and two concept decompositions/explanations and ask the following questions:\n * Which explanation is more relevant to the provided image?\n * Which explanation is more relevant to the model’s prediction?\n * Which explanation is more informative?  \n\nWe benchmark against two other CLIP-interpretability methods: LF-CBMs [23] and IP-OMP [22]. We find that users significantly preferred SpLiCE to LF-CBMs for both (1) and (3), and significantly preferred SpLiCE to IP-OMP for (1, 2, 3). We also highlight that our method is able to produce similar/better concept decompositions, in terms of human interpretability, than the baselines without the need for class labels for concept mining and without training a classification probe, both of which are computationally expensive. For more discussion, please see the general rebuttal.\n\n**Decomposition for correct/incorrect samples.** Thank you for the interesting question. In the additional material, Figure 3a, we include example SpLiCE decompositions for correctly and incorrectly classified samples using zero-shot with SpLiCE. We see that incorrect sample decompositions often contain correlated but slightly incorrect concepts that apply to other classes (“white dog” for an image of a white wolf) or concepts present in the image but not relevant to prediction (“baby toys” and “toddlers” for “abacus”). The former presents an instance where a practitioner might consider intervening on their predictor to improve performance, while the latter is simply an example of a difficult sample due to the noisiness of the image. In both cases, the explanation is useful for understanding the model’s reasoning and mistakes and can be used to intervene and improve performance.\n\n**Limitations of LAION-based concept set.** This is an important limitation of our work. It is correct that limitations and biases in the dictionary will be reflected in downstream performance. We aimed to construct our dictionary to be as comprehensive as possible by using the training set of the model itself (as LAION was used to train Open-CLIP, the model used in this paper), so we are likely to include all concepts learned by the model. We also filtered out any unsafe or NSFW samples from LAION before constructing the dataset to limit harmful content. However, we acknowledge that this is a limitation and will be sure to make this point more clear in the final version. For more discussion please see the general comment.\n\nWe again thank you for your feedback and hope these comments will encourage you to reconsider your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "owa1RDd2cW", "reviewer": "Reviewer_jbkR", "summary": "This paper presents a method to explore semantic concepts in multimodal models of text and images. Specifically, the paper formulate semantic concept discovery  problem as one of sparse recovery and build a novel method, Sparse Linear Concept Embeddings (SpLiCE), for transforming CLIP representations into sparse linear combinations of human-interpretable concepts.", "questions": "1 The connection and difference between your method with a multi model topic model? In my opinion, topic model is good at extracting interpretable concepts. \n\n2 Although a lot of experiments were conducted in the paper, I think how to judge the concept discovery results is still not convincing enough. Whether to add manual evaluation experiments will be more convincing.\n\n3. Why does negative concept weight perform better in Table 5? What does this mean?", "limitations": "see weakness and question", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and reads well.\n\n2. The technical method is generally make sense .\n\n3. The contribution is enough for covering text-image multi-model method to an interpretable model, and the research topic is worth pursuing.\n\n4. Sufficient experiments demonstrate the effectiveness of the proposed simple method.\n\n5. The experiment designed in the paper is make sense.", "weaknesses": "see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We appreciate your feedback and address your concerns below.\n\n**Connection to Multi Modal Topic Models.** This is an interesting connection! MMTMs such as [A] are trained on a corpus of data and use tf-idf statistics to generate multimodal clusters of topics, such as clustering images with text topics. However, these models are designed to explain datasets and not individual samples like SpLiCE. Additionally, this means every image in a cluster gets the exact same image tag and score, which prevents any comparison or differentiation between samples in a dataset despite these images being unique. Furthermore, we could use SpLiCE as a MMTM by generating concept decompositions for each image and applying simple k-means clustering on our interpretable decompositions. Finally, we note that the goals of this work are not to explain a dataset but rather to study CLIP and leverage it to explain individual image embeddings. In doing so we unlock new use cases, one of which is the capability to explain a dataset. We thank you for pointing out this connection, and we will be sure to discuss this work in the final version of our paper.\n\n**Manual evaluation experiments.** We agree that human evaluation studies are important for assessing the human interpretability of concept based explanations. We have included results from a small-scale human evaluation based on the user study presented in [22] in our additional results (Figure 1). We evaluate our method along with two baselines [22, 23] to assess how relevant the concept decompositions are to the input images, how relevant the concept decompositions are to the model’s prediction, and how informative the decompositions are. We find that users significantly prefer our method to the baselines for both relevance to the input images as well as informativeness, further validating the interpretability of our method. Please see our general comment for more information, where we elaborate on the setup and results of this evaluation. \n\n\n**Negative concept weight performance.** Thank you for the observation. As part of our desiderata for interpretability, we impose a nonnegativity constraint on the concept weights, as user studies (including our own) highlight that humans find negative concepts and weights to be confusing and unintuitive. In our optimization problem (2), we try to find a set of weights that minimize our reconstruction error. Constraining this set of weights to be nonnegative will reduce the search space and thus result in a worse reconstruction. So, we see in Table 5 that removing the nonnegativity constraint results in a more accurate reconstruction, but we lose the interpretability of our decompositions. \n\nWe again thank you for your constructive feedback!\n\n[A] Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNqxNYcuHH", "reviewer": "Reviewer_nCHL", "summary": "This paper introduces a method for building sparse embeddings from CLIP in order to improve the interpretability of CLIP’s latent space.  They formulate their objective as a sparse reconstruction problem, and under certain assumptions, demonstrate when finding a sparse representation is possible.  Empirically, the authors show over three datasets that zero-shot accuracy and reconstruction errors are correlated, and performance is similar with sparse embeddings vs. without.  They further illustrate the benefits of sparse embeddings for interpretability noting applications to reducing spurious correlation reliance on a Celeb-A benchmark, and identifying biases through representations.", "questions": "Have the authors demonstrated an evaluation setting where having the more interpretable concept features would improve performance? I believe some OOD setting such as spurious correlation could improve, but there may be many setting this is worse, as you are reliant on the concept features and may be worse for settings such as fine-grained classification where the concepts were not covered.", "limitations": "- An additional limitation of the work not discussed by the authors is the reliance on writing a set list of word-level concepts.  The approach is reliant on this set list having the desired concept when detecting spurious correlations, etc.  As the authors have pointed out this also limits the type of concepts.  \n\n- The authors have also not discussed how practical the assumptions proposed in Section 3 are, and whether they are correct in practice.  This is important for demonstrating applicability of the proposed work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper presents an important study toward understanding the latent space of vision-language models like CLIP which has become a very popular paradigm and backbone for many downstream tasks. This can help to improve the next generation of such models in training and downstream use. The use of text to interpret to solely understand the vision space and perform interventions is also important and makes the method relatively lightweight.  It also makes the method extendable to researchers and practitioners outside of the space as only concept vocabulary is needed. \n\n- The paper is overall well-written stating clearly the assumptions of the work in Section 3, and the proposed method is also clear and intuitive to implement.", "weaknesses": "- The primary weakness of this paper is that it is unclear to me what improvements come from generating sparse CLIP embeddings.  Results indicate that there is no improvement in zero-shot accuracy due to some outstanding reconstruction error.  In contrast, when removing spurious correlations it is typically seen as an improvement to model performance as the model would typically in this case then be evaluated in the OOD setting where the correlation is not present.  From this perspective, I think the authors might benefit from changing to a setting more similar to this, where the results are more aligned with benefits from having an interpretable tunable mechanism, rather than the more ID zero-shot performance settings.  This is started in the model editing, however it doesn’t lead to much improvement from correlation reliance, only mitigation of the capability to classify another attribute.  In contrast performance goes down indicating some capability but perhaps not the right evaluation setting.  \n\n- Similarly, I am uncertain about the case study evaluations for discovering spurious correlations.  Results for the man-woman evaluation appear to only indicate the trend is present in 70/600.  This seems like a relatively low number as the majority then do not have this bias.  It is unclear what the impact would be in terms of downstream performance for classification on CIFAR-100, or a gender classifier trained on CLIP embeddings.  The study seems inconclusive due to the results not being well contextualized 70 out of 600, and the lack of a downstream performance evaluation and improvement.\n\n- Finally, in multiple places, the authors mention improved human interpretability such as 244, 277, etc. however the authors never conduct any human evaluations.  If the authors claim improved human interpretability, the authors should conduct such a human evaluation, otherwise it is only hypothesized improved human interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We appreciate your feedback and address your concerns below.\n\n**Benefits of sparse CLIP embeddings.** Thank you for highlighting this confusion, we hope to clarify this in our response. The primary benefit of SpLiCE is the insight it provides into interpreting the semantic content of the underlying images encoded by ordinary CLIP embeddings. We show in the paper (in Section 6) that this additional insight opens up at least two applications: detecting biases in datasets and intervention on the concepts encoded, both of which are not possible with ordinary CLIP representations. SpLiCE provides a computationally efficient method for understanding the semantics of individual images (See Fig. 2 and additional results Fig 3) and for understanding the semantics of unstructured datasets, allowing for the exploration and summarization of data even without labels. We answer the specific questions that the reviewer has about the two applications below.\n\n**OOD evaluation setting.** Thank you for bringing up this important evaluation setting. We kindly note that we already consider the OOD setting in the Appendix C.5 with the Waterbirds dataset, which has a spurious correlation in its train set and a distribution shift in its test set. SpLiCE allows us to identify distribution shifts in two ways: first, we can evaluate the similarity in distributions between decompositions of test images and validation images (as we did in Table 6 of the Appendix for Waterbirds), or we can assess the weights learned by a linear probe to better understand how correlations in the train and test data might differ. For example, we see that we can detect the spurious correlation of landbirds and land backgrounds in our probe trained on SpLiCE weights, and when intervening on the highly-weighted land-related concepts [“bamboo”, “forest”, “hiking”, “rainforest”, and any bigrams containing “forest”] by setting them to zero in the weights of our probe, we are able to increase accuracy for the subgroup “waterbirds on land” in the test distribution (Appendix Table 5). In other words, SpLiCE’s interpretable sparse decomposition admits the identification of OOD spurious correlations (e.g., landbirds and land backgrounds), and also provides an interpretable strategy for intervention by allowing for fine-grained control via semantic concepts. \n\n**Significance of Woman-Swimwear bias.** Thank you for the question. This study shows that within the “swimwear” attribute, women are significantly overrepresented when compared to men. Another way to view this is that 10% of the images of women in CIFAR100 depict them in swimwear (generally referring to bikinis) or underwear. We believe this is a serious problem of bias in the dataset, and constitutes a “representational harm,” in that these stereotypes present in the dataset will be propagated with the use of this dataset to downstream tasks. This can result in downstream risks of bias such as 1) a generative model learns that when prompted for photos of women, it should generate this woman in revealing clothing or underwear, or 2) an image retrieval or generation algorithm failing to be representative when queried for “A photo of a CEO” by returning primarily men due to women being less likely to be depicted in corporate attire. While the reviewer points out the lack of downstream performance evaluation and improvement, we highlight that the goal of our work and this case study is not to develop a bias intervention but rather to propose an interpretability method and demonstrate its usefulness, and that this bias is a property of the dataset itself, which can result in different impacts for different models and tasks. \n\n**Human evaluations.** We agree that user studies are an essential part of evaluating human interpretability, and we have thus included results for a small-scale human evaluation based on the user study presented in [23] in our additional results (Figure 1). We evaluate our method along with two baselines [22, 23] to assess how relevant the concept decompositions are to the input images, how relevant the concept decompositions are to the model’s prediction, and how informative the decompositions are. We find that users significantly prefer our method to the baselines for both relevance to the input images as well as informativeness, further validating the interpretability of our method. We direct the reviewer to our general comment for more information about the study setup and results. \n\n**Reliance on set list of concepts.** Thank you for this comment. We agree that spurious correlation and bias detection are dependent on the spurious features being present in the concept vocabulary, thus posing a limitation. We note that we attempt to make our vocabulary as general as possible by constructing it from LAION, the training dataset of Open-CLIP, however it is true that this set may still not include the desired concepts for some downstream applications. In addition, we note that we find this vocabulary to be empirically sufficient for a variety of tasks (For more tasks see additional results). However, we will be sure to acknowledge this limitation in the paper and make this point more clear in the final version. For a more detailed discussion on our constructed vocabulary, please see our general comment. \n\n**Practicality of Assumptions.** This is an insightful question. Our empirical evaluations all strive to validate the assumptions proposed in section 3. In addition, we include in Appendix B.5 an additional sanity check of CLIP’s linearity (Assumption 3). We note that we intend to de-emphasize these assumptions in the final version of the paper in favor of our empirical validation from Appendix B.5, as suggested by reviewer djnP. For further discussion of these assumptions, please refer to our general comment. \n\nWe again thank you for your feedback and hope these comments will encourage you to reconsider your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WaYOCcFltH", "reviewer": "Reviewer_djnP", "summary": "Whilst useful for many downstream tasks, CLIP’s vision-language representations are notoriously hard to interpret. The paper proposes to represent CLIP representations in a sparse, non-negative overcomplete basis of learnt interpretable directions using a standard dictionary learning technique. Not only is interpretability increased, but the core claim is that this does not harm the performance of the downstream applications significantly. Such a model would provide interpretability “for free”, in the sense that practitioners could opt for the interpretable version without any performance cost (which would constitute a significant achievement).\n\n---\n\nPost-rebuttal:\nFollowing a productive discussion with the authors (who both clarified many experimental details and proposed revisions to the claims of the paper to better reflect its results), I think the paper will be of great value to the community and recommend its acceptance.", "questions": "# [Q1] Vocabulary choice and dataset dependence\n\nI’m not sure I would agree with the claim that the `efficacy of the decomposition is (in principle) independent of individual datasets`. Isn’t the vocabulary choice made in this work specific to LAION-400m? Perhaps using the WordNET nouns/adjectives as the vocabulary (and bigrams formed by common combinations) would be even more dataset/task agnostic? \n\n# [Q2] Mean centering\n\nAs stated by the authors, the CLIP representations live on a hypersphere. Instead of taking the geometric mean in Euclidean space (as part of addressing the modality gap), did the authors explore taking the Frechet mean? This seems like it would better account for the geodesic distances between the points.\n\n# [Q3] Zero-shot performance in the limit\nWhat happens to the zero-shot performance in Fig. 5 in the limit of the l0 norm being equal to the dimensionality of the space? Shouldn’t we expect matching performance to the CLIP baseline? It would be insightful to extend the x-axis to observe how it performs in the limit.", "limitations": "No, limitations are not adequated addressed, and the current limited discussion should be placed prominently in the main paper rather than the supplementary material.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The work strives to address a vitally important task: to design models that offer interpretability at no cost to performance. Such an endeavor is particularly important practically in allowing practitioners to adopt interpretable models in practice (by not trading off performance).\n- The paper is very well-written, the applied methodology is appropriate, and the focus on CLIP representations means the paper has potentially great relevance and reach to practitioners.\n- Offering more than just transparency, I appreciate the method's multiple use cases--not only for discovering spurious correlations but also for model editing.", "weaknesses": "# [W1] Sparsity/interpretability does trade-off accuracy\n\nI was very excited about the paper’s bold claim in the introduction to `provide interpretability, at no cost to downstream performance` ([L6], but also at [L349]) upon a first read. Unfortunately, from what I came to understand later from the results of the proposed dictionary in yellow plots in Figure 3, this claim appears false.\n\nThere *is* a cost to zero-shot classification performance (which is arguably one of the most important downstream applications of CLIP). For example, small L0 norms lead to as much as ~10% accuracy drop on CIFAR100 (left-most plot), and what appears to be a ~20% performance drop on ImageNET1k using the raw dictionaries. A similar pattern of performance degradation for high sparsity is observed in Fig. 7 for retrieval tasks. Thus, it is simply not true that the method is at “no cost” to performance, and arguably it is not even at “minimal” cost (used later in [L52]). These results unfortunately do not support the paper’s core claims.\n\nCrucially, if I correctly understand the x-axis as quantifying the number of non-zero coefficients, this is even more problematic. High sparsity is desirable for interpretability [L164]—but we see from the experiments that too small an L0 term trades-off accuracy, and sometimes rather significantly so. This fundamental trade-off is a clear critical limitation of the work (in conflict with the paper's core claims) and puts interpretability and accuracy at odds—but it is not ever stated or discussed as a key limitation of the work.\n\nThe authors should revise this core claim in light of the fact there is indeed a cost to performance increasing as a function of sparsity, and include a dedicated discussion of such limitation. Furthermore, I would expect to see many more experiments on the standard zero-shot datasets used in the original CLIP paper [1] to evaluate this core claim, not just on 3 (given this is zero-shot, these experiments are very fast to run with pre-trained dictionaries).\n\nThis weakness in particular swayed my rating of this paper negatively, and I am happy to consider revising my score accordingly, should an adequate answer be provided in response.\n\n# [W2] Limited new technical contributions, but also lacking technical insights/comparisons to alternative existing solutions \n\nThe paragraph in [L229] is rather brief and states that sklean’s Lasso solver is used. In the concrete application of sparse dictionary learning for CLIP representations in the paper, a representative reader would be very interested to see a much richer exploration of alternative approaches to solving the model objective.\n\nFor example, in CLIP, how does this proposed solution compare to non-negative KSVD? How about NMF (with a sparsity penalty)? Or sparse autoencoders? Or learning this end-to-end with projected gradient descent (to project the coefficients onto the non-negative orthant), etc.\n\nGiven that the paper does not offer any novel technical contributions (which is okay!), I think the paper would have been much stronger if at least some technical insights (about existing techniques) as they pertain to representation learning with CLIP were provided. At a minimum, a *discussion* of some of the other techniques listed above (and why they might not be appropriate) would be desirable, and experiments comparing the results from each to validate the proposed solution would make the paper’s contributions even stronger through technical insight.\n\n# [W3] Unnecessary mathematizing (minor)\n\nAs a reader very excited about this paper’s application, I feel the second half of Section 3. “When sparse decompositions exist” is an unnecessary formalization ([L141] onwards). I’m unconvinced of the need to introduce “Proposition 1”—at minimum, it feels like a distraction from the other interesting results in the paper. As far as I can see, this proposition is not used at all in the paper, and no experiments are conducted to show the 5 assumptions are reasonable (aside from Assumption 3).\n\nThe “assumptions” 1-5 listed do appear useful heuristics to reason about when Sparse decompositions are appropriate. As the authors also state [L133], concept linearity seems most crucial here. As a reader, what I care about is how these assumptions hold in CLIP in practice. As such, it would be much more useful to see instead the key experiment in Appendix B.5 discussed here in the main paper to support this critical assumption.\n\nTo summarize this final weakness: my impression is that the paper’s clarity could be improved by (a) considering deferring the formalizations to the supplementary material, or reducing their significance/length in the main paper and (b) considering making the critical experiments validating the key assumption in Appendix B.5 more prominent. I believe the paper would be much more impactful and have greater reach if the key results were better highlighted, and math that is not strictly necessary dropped.\n\n---\n\n* [1]: Radford, Alec et al. “Learning Transferable Visual Models From Natural Language Supervision.” International Conference on Machine Learning (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We appreciate your feedback and address your concerns below.\n\n**Sparsity/interpretability-accuracy tradeoff.** We thank the reviewer for their detailed thoughts on the interpretability-accuracy tradeoff claims and for their willingness to increase their score if this issue is resolved. The reviewer is correct that Figure 3 highlights the existence of tradeoffs between interpretability and accuracy, and we will correct the language around these claims and address this limitation in the final version. However, we note that this tradeoff is small at desirable sparsity levels. For a sparsity level of 20-30 (chosen with respect to a prior human evaluation study [49]), we find a drop in zero-shot accuracy of less than 4% for the CIFAR100 dataset (Fig. 3). Furthermore, this drop is entirely recovered when classification is done via probing instead of zero-shot (please see Appendix C.1). Secondly, this tradeoff is not necessarily fundamental, as it is possible to retain downstream performance by using SPLiCE as a post-hoc concept explanation method. More specifically, we can add the residual error back into the reconstructed embedding to recreate the CLIP embedding before use in downstream applications (as suggested by reviewer Vxod). Thus, we can achieve interpretability without affecting performance. We do note that in this case, the residual will still remain uninterpretable, which is why we do not focus on this setting in the main paper. We will be sure to discuss this in more depth in the final version, including the key limitations of both of these settings. \n\nAs requested by the reviewer, we included experiments on four additional datasets from the CLIP paper [1] (Caltech-101, SUN397, STL10, VOC 2007) in the additional results, Table 1, for sparsities of 20-35. While there is a decrease in performance, we believe it to be relatively small and recoverable by probing or the technique discussed above.\n\n**Technical contributions and exploration of alternative existing solutions.** Thank you for this interesting question. We note that we explored two alternative approaches in the main paper, including an alternative solver, ADMM, and a method for learning the dictionary, FISTA, but we will elaborate on this discussion further in the final version. In Appendix B.2, we discuss the use of Alternating Direction Method of Multipliers (ADMM) in place of scikit-learn's Lasso method to solve the SPLiCE objective with GPU/batched speedup. We present results in Figure 2 of the additional material, illustrating the equivalence of these methods in both cosine reconstruction and zero-shot accuracy for CIFAR100. \n\nWe also explore an alternative dictionary learning method, similar to those suggested by the reviewer (Sparse NMF, KSVD, Sparse Autoencoders, PGD), of sparse dictionary learning with nonnegative weight projection via Fast Iterative Shrinkage-Thresholding (FISTA) in Figures 3, 7. We note that FISTA and the other suggested methods all learn uninterpretable dictionaries that require post-hoc human labeling of the learned concept dictionary atoms. While existing literature such as [31] explores methods for visualizing concepts learned by NMF, PCA, k-Means, and Sparse Autoencoders, this process still requires manual labeling of concepts, which can be both data- and time-intensive. One of the key benefits of SpLiCE is that we fix the concept atoms a priori rather than having to analyze and label components post-hoc. Our results also demonstrate that a learned dictionary results in better reconstruction in terms of cosine similarity, as expected, but surprisingly underperforms our LAION dictionary on zero-shot classification, due to the lack of semantic structure within these learned dictionaries.\n\n**Mathematizing.** Thank you for the comment. We plan to adopt your feedback, move the empirical validation found in Appendix B.5 to the main paper, and reduce the length of our formal analysis, deferring our theoretical results to the appendix. For further discussion, please see our general comment.\n\n**Vocabulary choice.** This is an important point we intend to discuss in the final paper. While we considered WordNet, we preferred LAION as our source of concepts because it is the training dataset of Open-CLIP (the model used in this work), preventing us from including concepts in the dictionary that CLIP may not have learnt. Please see our general comment for a more thorough discussion of this point.\n\n**Mean centering.** We did consider Frechet means, but we found it non-trivial to define a clear notion of mean-centering on the hypersphere. Because the surface of a unit sphere does not contain the “zero” element necessary to define an algebraic field, it is hard to define addition or subtraction on the sphere and, subsequently, mean-centering. We found that Euclidean mean-centering sufficiently closed the modality gap for our purposes; however, if the reviewer has specific suggestions regarding this, we would be happy to update our centering method. \n\n**Zero shot in the limit.** We do expect to recover CLIP’s baseline performance in the limit when our sparsity constraint allows for a 512-dimensional solution, aside from shrinkage incurred by the L1 penalty. In our experiments, we find that sklearn’s Lasso algorithm does not converge with minimal L1 regularization, so we solve with Ridge regression and truncate to the top 512 coefficients. Table 2 in the additional material shows that at 512 coefficients we are indeed able to recover CLIP’s baseline performance. Note that this still provides interpretable decompositions  and so is directly preferable over CLIP. \n\n**Limitations.** We will be sure to move our limitations section up to the main body of the paper for our final submission and will elaborate on the accuracy-interpretability tradeoff you have mentioned above.\n\nWe again thank you for your feedback and hope these comments will encourage you to reconsider your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their thorough assessment of our paper and the AC for facilitating the discussion of our work. We appreciate the reviewers’ recognition that our paper is an “important study toward understanding the latent space of vision-language models like CLIP” and that it will “help to improve the next generation of such models in training and downstream use” [nCHL]. Furthermore, the reviewers note the paper “strives to address a vitally important task” [djnP], that “the research topic is worth pursuing” [jbkR], and that it “opens up the door to more intriguing work” [Vxod]. Finally, we are pleased to note that reviewers appreciated the novelty of the work [acbD, Vxod], the quality of the writing [all reviewers], and the clarity of the work [all reviewers]. In the following section we summarize the main points made by reviewers and respond to their comments. We also present additional experimental results, including results from a user study and evaluation on four additional benchmark datasets in the additional results section and in the body of our responses below.

**Human Evaluation.** Reviewers nCHL, acbD, and jbkR all asked for a human evaluation to validate our claims of improved interpretability. We present results from a small-scale user study in Fig. 1 of the additional results. We base our study off of that provided by reviewer acbD [23], benchmarking our method against two similar CLIP interpretability methods: LF-CBMs [23] and IP-OMP [22]. We provided users with twenty randomly chosen, correctly predicted images from ImageNet and two explanations comprising six concepts each for every image. We then asked users to evaluate the concept-based explanations for their relevance to the provided image inputs, their relevance to model predictions, and their informativeness. We find that users significantly preferred SpLiCE to the two baselines for relevance to the images and informativeness, with significance determined via a one-sample two-sided t-test and a threshold of p=0.01 (Additional material Figure 1). We also highlight that our method is able to produce similar/better concept decompositions, in terms of human interpretability, than the baselines without needing to train a classification probe or use class labels for concept mining, both of which are computationally expensive.

**Limitations of Concept Set.** Reviewers djnP, nCHL, abcD, and Vxod all mention the limitations of our LAION-based vocabulary, mainly related to its “task-agnosticity,” given that we construct this vocabulary using a specific dataset. While there certainly exist tasks that this vocabulary may fail at, it is intended to be a general purpose vocabulary that reflects the concepts learned by CLIP. We pick the LAION dataset because it is the training set of Open-CLIP (the model we use in this work) itself, but we acknowledge that it may still be a suboptimal dataset. Despite this, we find that this dataset performs well on a variety of tasks (CIFAR100, MIT States, WaterBirds, SUN397, Caltech 101, STL10, PASCAL VOC 2007). We agree with the reviewers that "task-agnostic" may be too strong, but it is accurate to state that our method is out-of-the-box applicable to a wide variety of tasks (owing to the generality and scale of LAION), and easily modified for others (i.e., simply listing additional concepts as opposed to labeling images and training a new model). Finally, we benchmark our vocabulary against the previous SOTA concept set, generated by GPT, in Figure 5, and explore other concept sets in Appendix C.10. We also note that our method can accommodate any user-defined vocabulary if the authors’ proposed one is insufficient for the application.  We will be sure to acknowledge these limitations in the final version of our paper and highlight that selecting a proper dictionary is an open question. 

**Necessity of Theoretical Analysis.** Reviewers djnP, nCHL, and Vxod were concerned about the practicality of the assumptions in our theoretical analysis in Section 3 and their necessity to the core results and message of our paper. The intent of Section 3 is to reason from first principles regarding which mathematical properties enable vision-language models to have a sparse decomposition. To this end, we identified five assumptions sufficient to derive Proposition 1, i.e., the sparse decomposition property, similar to past work theoretically characterizing the linear representation hypothesis [11] and word2vec behavior [10]. We did not mean to claim that CLIP always satisfies all these assumptions, but rather that given our empirical results indicating our ability to apply a sparse decomposition to CLIP, we can reasonably conclude that CLIP may approximately satisfy assumptions 1-5. 

We empirically explore the extent to which these assumptions hold in Appendix B.5. Reviewer djnP noted that these “critical” experiments would be much more useful in the main body of the paper as evidence of the practicality of our claims, further improving the paper’s clarity. Based on this feedback, we will move our empirical investigation to Section 3 and transfer the bulk of our theoretical analysis to the appendix, instead writing a shorter paragraph summarizing the basic assumptions and behaviors of CLIP required to admit a sparse decomposition.

Meta-review: Reviewers positively assessed the novelty and significance of providing interpretable embeddings by using dictionary learning to decompose CLIP representations.  Multiple reviewers also expressed the need to include human evaluations in the paper, which were provided by the authors during the rebuttal period.  Two reviewers were concerned about the paper's claims of providing increased interpretability at "no cost" to accuracy.  The authors agreed during the discussion to revise the abstract and several sentences in the paper to convey that the method can provide increased interpretability at a low cost to accuracy.  Given the strengths of the paper and the authors' willingness to make the corrections requested by reviewers, I am recommending acceptance of this paper.
Decision: Accept (poster)