Title: Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)
Venue: NeurIPS 2024
Accepted: yes
Keywords: Interpretable Machine Learning

Dictionary Learning

Representation Learning

Multimodal Models

Interpretability

CLIP
Abstract: CLIP embeddings have demonstrated remarkable performance across a wide range of multimodal applications. However, these high-dimensional, dense vector representations are not easily interpretable,  limiting our understanding of the rich structure of CLIP and its use in downstream applications that require transparency. 
In this work, we show that the semantic structure of CLIP's latent space can be leveraged to provide interpretability, allowing for the decomposition of representations into semantic concepts. 
We formulate this problem as one of sparse recovery and propose a novel method, Sparse Linear Concept Embeddings (SpLiCE), for transforming CLIP representations into sparse linear combinations of human-interpretable concepts. Distinct from previous work, \method is task-agnostic and can be used, without training, 
to explain and even replace traditional dense CLIP representations, maintaining high downstream performance while significantly improving their interpretability. We also demonstrate significant use cases of \method representations including detecting spurious correlations and model editing. Code is provided at https://github.com/AI4LIFE-GROUP/SpLiCE.

Interpreting CLIP with Sparse Linear Concept Embeddings ( SpLiCE )

Abstract

CLIP embeddings have demonstrated remarkable performance across a wide range of multimodal applications. However, these high-dimensional, dense vector representations are not easily interpretable, limiting our understanding of the rich structure of CLIP and its use in downstream applications that require transparency. In this work, we show that the semantic structure of CLIP's latent space can be leveraged to provide interpretability, allowing for the decomposition of representations into semantic concepts. We formulate this problem as one of sparse recovery and propose a novel method, Sparse Linear Concept Embeddings (SpLiCE ), for transforming CLIP representations into sparse linear combinations of humaninterpretable concepts. Distinct from previous work, SpLiCE is task-agnostic and can be used, without training, to explain and even replace traditional dense CLIP representations, maintaining high downstream performance while significantly improving their interpretability. We also demonstrate significant use cases of SpLiCE representations including detecting spurious correlations and model editing. Code is provided at https://github.com/AI4LIFE-GROUP/SpLiCE .

1 Introduction

Natural images include complex semantic information, such as the objects they contain, the scenes they depict, the actions being performed, and the relationships between them. Machine learning models trained on visual data aim to encode this semantic information in their representations to perform a wide variety of downstream tasks, such as object classification, scene recognition, segmentation, or action prediction. However, it is often difficult to enforce explicit encoding of these semantics within model representations, and it is even harder to interpret these semantics post hoc to better understand what models may have learnt and how they leverage this information. Further, model representations can be brittle, encoding idiosyncratic patterns specific to individual datasets and modalities instead of general human-interpretable semantic information. Multimodal models have been proposed as a potential solution to this issue, and methods such as CLIP [1] have empirically been found to provide highly performant, semantically rich representations of image data. The richness of these representations is evident from their high performance on a variety of tasks, such as zero-shot classification and image retrieval [1], image captioning [2], and image generation [3]. However, despite their performance, it remains unclear how to quantify the semantic content contained in their dense representations. In this work, we answer the question: can we decompose

∗ Equal contribution, order by coin flip.

a Kempner Institute for the Study of Natural & Artificial Intelligence

b School of Engineering and Applied Sciences

c Harvard Business School

CLIP

dense

representations,

z

images,

x

img

0.2435

0.5642

0.0489

0.5183

0.7164

0.1235

0.8734

0.9768

0.1871

0.8106

0.4035

0.2819

…

…

0.7400

0.1780

0.4633

0.8931

overcomplete

concept set,

C

{

dog

,

bird

,

grass

,

ocean

, …,

sandy

}

SpLiCE

with a sparse, nonnegative

linear solver

zebra

0.09

africa

0.05

camels

0.04

ostrich

0.03

grazing

0.02

short

0.02

honda cbr

0.08

police

0.07

emergency

services

0.07

police officer

0.04

motorists

0.04

lime

0.02

interpretable

concept

decompositions,

w

CLIP

0.3549

0.4692

0.1286

0.6123

…

…

0.0560

0.3320

CLIP embeddings into human-interpretable representations of the semantic concepts they encode? This can provide insight into the types of tasks CLIP can solve, the biases it may contain, and the manner through which downstream predictions are made.

Existing literature in areas such as concept bottleneck models [4], disentangled representation learning [5], and mechanistic intepretability [6] have proposed various approaches to understanding the semantics encoded by representations. However, these methods generally require predefined sets of concepts [7], data with concept labels [8], or rely on qualitative visualizations, which can be unreliable [9]. Similar to these lines of work, we aim to recover representations that reflect the underlying semantics of the inputs. However, distinct from these works, we propose to do this in a task-agnostic manner and without concept datasets, training, or qualitative analysis of visualizations.

Our method, SpLiCE, leverages the highly structured and multimodal nature of CLIP embeddings for interpretability, and decomposes CLIP representations via a semantic basis to yield a sparse, humaninterpretable representation. Remarkably, these interpretable SpLiCE embeddings have favorable accuracy-interpretability tradeoffs when compared to black-box CLIP representations on metrics such as zero-shot accuracy. Our overall contributions are:

In Sections 3 and 4, we formalize the sufficient conditions under sparse decomposition of CLIP is feasible, and introduce SpLiCE, a novel method that decomposes dense CLIP embeddings into sparse, human-interpretable concept embeddings.

Our extensive experiments in Section 5 reveal that SpLiCE recovers highly sparse 1 , interpretable representations with high performance on downstream tasks, while accurately capturing the semantics of the underlying inputs.

In Section 6, we present two case studies for applying SpLiCE: spurious correlation detection, and model editing. Using SpLiCE, we uncover a spurious correlation in the CIFAR100 dataset, where we find the "woman" concept and the "swimwear" concept to be correlated owing to the prevalence of women in swimwear in CIFAR100.

2 Related Work

Linear Representation Hypothesis. In language modeling, the linear representation hypothesis suggests that many semantic concepts are approximately linear functions of model representations [10, 11, 12, 13, 14]. Recent work has also shown that multimodal models encode concepts additively, behaving like bags-of-words representations [15]. Relatedly, [16, 17] show that there exists a linear mapping between image and text embeddings in arbitrary models. Our work makes use of these distinct but related observations to convert dense CLIP representations to sparse semantic ones.

Concept Bottlenecks and Attribute Learning. Concept Bottleneck Models (CBMs) [18], and attribute-based models [19, 20, 21] learn intermediate representations of scores over concepts or image

'A statue of a cow sitting under

a sign on grass.'

wedding sign

0.06

cow

0.04

shop logo

0.04

sweets

0.04

sweetened

0.03

swing

0.02

'A field that has some snow

and a man skiing in it'

powder

0.08

snowboard

0.07

skiing

0.07

slopes

0.06

winter sports

0.04

ski resort

0.03

'A cat is looking at a bathtub in a

bathroom with a toilet.'

cat

0.05

shower

0.05

tabby cat

0.03

scratching

0.03

shampoo

0.03

sink

0.02

'A knife stabbing an orange with

a sad face.'

vitamin c

0.10

oranges

0.08

fresh orange

0.06

depression

0.04

unhappy

0.04

food background

0.03

macro shot

0.03

'Adult with orange umbrella

walks on bricked open area.'

red umbrella

0.07

umbrella

0.06

woman walks

0.03

businessman

0.03

autumn winter

0.01

orange

background

0.01

paved

0.01

'A foal on the beach laying next to a

horse.'

wild horses

0.10

blackandwhite

bw

0.08

mare

0.07

beach sand

0.03

horse

0.02

resting

0.01

kissing

0.01

SugaongGmeets

attributes for use with a final linear classification head, creating interpretable concept representations. However, these require expert-labeled concept or attribute datasets to train, which is expensive. Recent work on concept-bottlenecks for multimodal models avoids needing such labeled datasets, but still requires concept labels for specific tasks, which is obtained by querying large language models (LLMs) [22, 23, 24], making these methods task-specific and heavily reliant on the domain knowledge and subject to the biases of LLMs. On the other hand, SpLiCE uses a large-scale and overcomplete concept dictionary, avoiding dependence on training, specific domain knowledge, or a downstream task. Consequently, it can even be applied to understand unstructured, unsupervised image datasets in a label-free manner.

Mechanistic Interpretability and Disentanglement. Mechanistic interpretability explains representations through model activations, by labeling circuits and neurons in networks with feature visualization [6, 25] or by measuring concept activations and directions in latent space [26, 27, 7, 28, 29, 30]. Recent work [31] combines these methods, using dictionary learning to extract visual concept activations, whose semantics can be identified via feature visualization. Work in disentangled representation learning has developed architectures that capture independent factors of variation in data [8, 32, 33, 5, 34, 35], allowing for manual probing of disentangled representations for humaninterpretable concepts. In both mechanistic interpretability and disentangled representation learning, methods typically rely on labeled concept sets, manual labeling of visualizations, or computationally intensive searches over data and latent representations or neurons to identify concepts. However, associating human-interpretable semantics with arbitrary neurons or latent directions is challenging, leading to the unreliability [9, 36] exhibited by such methods. Our approach side-steps this issue by decomposing CLIP representations into a predetermined set of concepts.

CLIP Interpretability. Many recent works leverage the semantic structure of CLIP and its text encoder to interpret representations. For example, [37], [38], and [39] construct concept similarity scores of image embeddings for use by downstream CBMs or probes, but these representations are not interpretable due to their lack of sparsity and the presence of negative concepts. Chen et al. [40] create a custom vision-language architecture with a sparse latent dictionary, but it requires training from scratch and cannot be used post-hoc to explain existing models. Gandelsman et al. [41] also leverage the text encoder of CLIP to explain components of the image embedding, but are limited to ViT architectures and take a mechanistic interpretability-style approach requiring a labeled text dataset. Chattopadhyay et al. [22] build concept bottlenecks for specific classification tasks by expressing CLIP image representations as a sparse linear combination of task-specific concept vectors. However, their decomposition includes negative concepts, reducing interpretability, and uses task-specific concept dictionaries. Grootendorst [42] generate textual topics of datasets through multimodal topic modeling, which cannot provide explanations of individual representations. Distinct from these works, SpLiCE is more interpretable due to its sparsity, overcompleteness, and non-negativity, and is task-agnostic, aiming to serve as a drop-in replacement for black-box CLIP representations without requiring training.

3 When do Sparse Decompositions Exist?

In this section, we aim to answer the question: under what conditions can CLIP representations be decomposed into sparse semantic representations ? To do so, we must reason about both the properties of CLIP as well as the properties of the underlying data.

Notation. Let x img ∈ R d i , x txt ∈ R d t be image and text data, respectively. Given the CLIP image encoder f : R d i → R d and text encoder g : R d t → R d , we define CLIP representations in R d as z img = f ( x img ) and z txt = g ( x txt ) . Our method uses dictionary learning to approximate z img with a concept decomposition w ∗ ∈ R c + over a fixed concept vocabulary C ∈ R d × c . We define the resulting reconstruction of z img from C and w ∗ as ˆ z img .

The goal of our method is to approximate f ( x img ) ≈ Cw ∗ , such that w ∗ is non-negative and sparse, and in this section we formalize when this is possible. We begin by considering a datagenerating process for coupled image and text samples. Specifically, we model the generative process parameterized by a k -dimensional latent concept vector ω ∈ R k + and a random noise vector ϵ ∈ R l as

where ρ is a prior distribution over semantic concepts, ϕ is a prior distribution over nonsemantic concepts (such as camera orientation and lighting for images or arbitrary choices between synonyms for text), and h img : R k + l → R d i , and h txt : R k + l → R d t represent the real-world data-generating process from latent variables ( ω, ϵ ) to images and text respectively. Here, each coordinate ω i ∈ R + encodes the degree of prevalence of the i th concept in the underlying data. We now list a set of sufficient conditions for our data-generating process and CLIP that admit a sparse decomposition of images into concepts.

Sufficient Conditions for Sparse Decomposition.

Images and text are sparse in concept space: for some α ≪ k , we have ∥ ω ∥ 0 ≤ α, ∀ ω ∼ ρ .

CLIP captures semantic concepts ω and not ϵ : ∀ ϵ, ϵ ′ , f ◦ h img ( ω, ϵ ) = f ◦ h img ( ω, ϵ ′ ) and similarly for h txt .

CLIP is linear in concept space: g ◦ h txt and f ◦ h img are linear in ω .

CLIP image and text encoders are aligned: for a given ω , f ◦ h img ( ω, ϵ ) = g ◦ h txt ( ω, ϵ ) .

We emphasize that the goal of enumerating a set of sufficient conditions for sparse decomposition is not to claim that these exactly hold in practice, but rather to reason about when sparse decompositionsas done in this work-are appropriate. In the Appendix (Section A.1, Prop. 1) we formalize and prove this claim, but in the interest of simplicity we keep the discussion here informal. We note that many of these are natural; Assumption 1 reflects how real-world images and text are simple and rarely contain complex semantic content, and the CLIP training process optimizes for Assumption 2 and 4 2 . Of these, the most critical one is Assumption 3, which closely relates to the linear representation hypothesis [11], which we investigate below.

Sanity Checking CLIP's Linearity. We provide evidence for the third assumption, the linearity of CLIP, in a toy setting. We begin by asking the following question to confirm the general linearity of CLIP embeddings: 'if two inputs are concatenated, does their joint embedding equal the average

of their two individual embeddings?". For the image domain, we combine two images, x a , x b , to form their composition x ab by placing x a in the top left quarter and x b in the bottom right quarter of a blank image. For the text domain, we simply append text x b to text x a to form x ab . We then embed x a , x b , x ab with CLIP to get z a , z b , z ab . Solving the equation w a ∗ z a + w b ∗ z b = z ab for scalar weights w a , w b then allows us to assess the linearity of z a , z b , z ab . We report w a , w b and the cosine similarity between ˆ z ab = [ z a , z b ] · [ w a , w b ] and z ab in Table 1.

In general, we find that the composition of two inputs results in an embedding that is approximately equal to the average of the two input components, with w a , w b being very close to 0.5 across all datasets and for both modalities, providing preliminary evidence for the linearity of CLIP embeddings for both image and language.

4 Method

In this section, we introduce SpLiCE, a method for expressing CLIP's image representations as sparse, nonnegative, linear combinations of concept dictionary elements. We begin by framing this problem as one of sparse recovery. We then discuss our design choices, including how we choose the concept dictionary and how to address the modality gap between CLIP's images and text representations. Finally, we formalize the optimization problem used in this work.

4.1 Sparse Nonnegative Concept Decomposition

Our goal is to construct decompositions of dense CLIP representations that are human-interpretable, useful, and faithful. To do so, we formulate decomposition as a sparse recovery problem with three main desiderata. First, for the decompositions to be interpretable to humans they must be comprised of human interpretable atoms. We argue that language is a naturally interpretable interface for humans, and construct our concept vocabulary C out of 1- and 2-word atoms, such as 'coffee', 'silver', and 'birthday party'. Second, our decompositions must be simple and concise, which can be formulated as a sparsity constraint on the recovery. A large body of work in computational linguistics [43, 44, 45, 14], neuroscience [46, 47], and interpretability [48, 49, 30] have demonstrated that a human-aligned semantic model should be sparse in representation. Furthermore, [48] found that users can best understand explanations with fewer than 32 concepts while in linguistics, [50, 51, 52] find participants describe concepts and objects with up to 20 semantic properties, motivating our desiderata of sparsity. Third, our decompositions must be constructive, i.e., we must decompose representations in terms of their constituent concepts. For this reason, we require the weights of decompositions to be strictly nonnegative, to avoid having 'negative' concept weights which do not always carry semantic meaning. Furthermore, prior work by Zhou et al. [30] has argued that 'negations of concepts are not as interpretable as positive concepts. ' More specifically, while a small set of concepts have well-defined antonyms which may be viewed as their negative counterparts ( 'day' ↔ 'night' ), negative concepts do not carry semantic meaning in general ( 'tiger' ↔ ?? ). Furthermore, we find that even when antonyms exist, they are not negatives of each other in CLIP latent space (see Appendix B.10). To avoid dependence on negative weights and ensure that all concepts are captured, we construct an overcomplete dictionary containing a wide range of concepts, including antonyms. We build on top of this literature and provide a semantic decomposition satisfying these properties suitable for multimodal models like CLIP.

Concept Vocabulary. Natural language is an intuitive, interpretable, and compact medium for communicating semantic information. Thus, we choose to represent the semantic content contained in CLIP embeddings as combinations of natural language semantic concepts, where we define concepts as semantic units that can be expressed concisely, by one- or two-word phrases. Given that CLIP is used in a wide variety of downstream applications and is trained without a specific task in mind, we want our concept dictionary to be task-agnostic and to span all possible concepts CLIP could have learnt . To construct this vocabulary, we consider the most frequent one- and two-word bigrams in the text captions of the LAION-400m dataset [53], the dataset that most CLIP variants are trained on. We filter the captions to remove any NSFW samples and prune our concept set such that no two concept embeddings have a cosine similarity greater than 0.9. We also remove bigrams highly similar ( > 0 . 9 cosine similarity) to the average of their individual words. We finally choose the top 10 , 000 most common single-word concepts and the top 5000 most common two-word concepts as our concept vocabulary. We note that this vocabulary offers distinct advantages over those used in prior works.

120

120

0

20

40

60

80

100

Decomposition l0 Norm

0.4

0.5

0.6

0.7

0.8

Cosine Similarity (x_hat, x)

CIFAR100 Cosine Reconstruction

0

20

40

60

80

100

120

Decomposition l0 Norm

0.4

0.5

0.6

0.7

0.8

0.9

Cosine Similarity (x_hat, x)

MIT States Cosine Reconstruction

0

20

40

60

80

100

Decomposition l0 Norm

0.4

0.5

0.6

0.7

0.8

0.9

Cosine Similarity (x_hat, x)

Imagenet Cosine Reconstruction

0

20

40

60

80

100

Decomposition l0 Norm

0.2

0.3

0.4

0.5

0.6

0.7

0

20

40

60

80

100

120

Decomposition l0 Norm

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0

20

40

60

80

100

Decomposition l0 Norm

0.1

0.2

0.3

0.4

0.5

Imagenet Zero Shot Accuracy

random

learned

(images)

learned

(text)

laion

CLIP

(baseline)

laion +

imagenet

classes

In particular, it is task-agnostic , meaning that the efficacy of the decomposition is (in principle) independent of individual datasets. Furthermore, this dataset imposes minimal priors from outside curators, such as human experts or LLMs [22, 23, 24]. This allows us to interpret data through the lens of CLIP, to understand the information encoded, including potential biases and mistakes.

Modality Alignment. In order to decompose images into nonnegative combinations of text concepts, we must ensure that our concept set spans the space of possible image embeddings. However, [54] show the existence of a modality gap in CLIP, where image and text embeddings can lie in non-identical spaces on the unit sphere. We empirically find that CLIP image and text embeddings exist on two cones, as the distribution of pairwise cosine similarities between pairs of MSCOCO images and pairs of MSCOCO text captions concentrate at positive values, whereas the distribution of pairwise cosine similarities across modalities concentrates closer to zero. (See Appendix Fig. 7). Not only does this prevent nonnegative decomposition, it also violates Assumption 4 from Section 3. To rectify this, we mean-center CLIP images with the image cone mean, estimated over MSCOCO ( µ img ), and compute decompositions over the mean-centered concept vocabulary ( µ con ). Note that the embeddings need to be re-normalized after centering to ensure they lie on the unit sphere. To convert our decompositions back into dense representations ( ˆ z img ), we uncenter the normalized dense embeddings ˆ z img by adding the image mean back in and normalizing once again, to ensure they lie on the same cone as the original CLIP embeddings ( z img ).

Optimization Problem. Our optimization problem is formulated as follows. Let σ ( x ) = x / ∥ x ∥ 2 be the normalization operation. Given a set of semantic concepts x con = [' dog ', ' tabby cat ', ' cloudy ', · · · ], we construct a centered vocabulary C = [ σ ( g ( x con 1 ) -µ con ) , · · · , σ ( g ( x con c ) -µ con )] , where we recall that g ( · ) is the CLIP text encoder. Now, given the dictionary C and a centered CLIP embedding z = σ ( z img -µ img ) , we seek to find the sparsest solution that gives us a cosine similarity score of at least 1 -ϵ for some small ϵ :

As is standard practice, we relax the ℓ 0 constraint and reformulate this as a minimization of MSE with an ℓ 1 penalty, to construct the following convex relaxation 3 of Eq. (1):

Given the solution to the above problem w ∗ , our reconstructed embedding is: ˆ z img = σ ( Cw ∗ + µ img ) .

5 Experiments

In this section, we evaluate our method to ensure that SpLiCE decompositions are interpretable, performant, and accurately reflect the semantic content of representations.

5.1 Setup

Models. All experiments shown in the main paper are done with the OpenCLIP ViT-B/32 model [55] with results for an additional model in Appendix B.14. For all zero-shot classification tasks, we use the prompt template 'A photo of a {}'. Datasets. We use CIFAR100 [56], MIT States [57], CelebA [58], MSCOCO [59], and ImageNetVal [60] for our experiments with results for additional datasets in the Appendix (Section B.4)

Decomposition. For all experiments involving concept decomposition, we use sklearn's [61] Lasso solver with a non-negativity flag and an l 1 penalty that results in solutions with l 0 norms of 5-20 (around 0.2-0.3 for most datasets).We use a concept vocabulary chosen from a subset of LAION tokens as described in Section 4.1. Both image embeddings and dictionary concepts are centered and normalized as mentioned in Section 4.1, with the image mean used for centering computed over the MSCOCO train set and the concept mean computed over our chosen vocabulary.

5.2 Sparsity-Performance Tradeoffs

We assess the performance of SpLiCE decompositions by evaluating the reconstruction error in terms of cosine similarity between SpLiCE representations and CLIP embeddings, the zero-shot performance of SpLiCE decompositions, and the retrieval performance of SpLiCE embeddings. We compare the performance of decompositions generated from our semantic concept vocabulary to decompositions over random vocabulary and learned dictionary vocabulary baselines. All vocabularies are of size 15,000 concepts. The random vocabulary is sampled from a 512-dimensional normalized Gaussian distribution. The learned vocabularies are generated by using the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) [62] to learn optimal dictionaries given our sparse recovery problem (optimizing Equation (1) for both C and w ). Note that we learn separate dictionaries C img and C text to reconstruct MSCOCO image and text embeddings respectively. In Figure 3, we plot the cosine reconstruction and zero-shot accuracy of image decompositions with the various dictionaries. We evaluate probing performance (Tables 3, 4) and text-to-image and image-to-text retrieval in the Appendix (Figure B.3).

These results overall show SpLiCE efficiently navigates the interpretability-accuracy Pareto frontier and retains much of the performance of black-box CLIP representations with the semantic, humaninterpretable LAION dictionary, significantly outperforming other dictionaries on semantic tasks such as zero-shot classification, probing, and retrieval. At the same time, we find that our semantic LAION dictionary does not result in accurate cosine similarity reconstructions of the original CLIP, often being on par with using random dictionaries. We believe this is because CLIP encodes both semantics of the underlying image and non-semantic "noise", which violates Assumption #2 in Section 3. Given that our SpLiCE decompositions only aim to encode semantics, they are unable to encode non-semantic aspects in the underlying representation, thus causing poor alignment in the cosine similarity sense, while simultaneously exhibiting excellent alignment on semantic tasks such as zero-shot accuracy. For ImageNet, we find that many classes are animal species that cannot easily be described by 1-2 words (e.g. ' red-breasted merganser ', ' American Staffordshire terrier '). Adding these class labels to our concept dictionary increases performance significantly, as shown by the dotted yellow line in Figure 3.

elephant

baby elephant

adult male

south africa

grazing

mating

closeup portrait

females

0.00

0.05

0.10

0.15

'African Elephant' Class

poodle

retriever dog

black dog

dog breed

dog collar

spaniel

black wool

retriever

0.00

0.01

0.02

0.03

0.04

'Curly-coated Retriever' Class

monarch butterfly

butterfly

adult male

lilac flowers

beautiful nature

macro shot

nectar

glass pendant

0.00

0.05

0.10

0.15

0.20

'Monarch Butterfly' Class

timer

countdown

time

number

scoreboard

fm radio

0.00

0.02

0.04

0.06

0.08

'Digital Clock' Class

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Concept Weight

10

0

10

1

10

2

Count

CIFAR100 'Swimwear' Concept

Class

Woman

Man

10

20

30

40

50

Decomposition l0 Norm

0.50

0.55

0.60

0.65

Semantic Relevance

MSCOCO Semantic Relevance

5.3 Ablation Studies

We perform ablation studies to evaluate the effectiveness of the design decisions of SpLiCE, including the choice of vocabulary, the nonnegativity of the decompositions, and the modality alignment by ablating each choice and observing the effect on three metrics: zero-shot accuracy on CIFAR100, cosine similarity between the reconstructed and original embeddings of CIFAR100, and semantic relevance on MSCOCO. The first two metrics are the same as those presented in Figure 3. We compute semantic relevance by tokenizing and filtering stop-words from the MSCOCO human-generated captions and embedding each token with CLIP. Then, we take all non-zero concepts output by SpLiCE and compute the Hausdorff distance between the sets of SpLiCE concepts and caption token embeddings. This essentially measures how aligned decompositions are with human captions. We observe that replacing our dictionary with the LLM-generated concept dictionary used by [22, 24, 23] significantly worsens the decomposition in terms of zero shot accuracy and cosine reconstruction. While allowing for negative concept weights improves cosine reconstruction marginally, it decreases the semantic relevance of the decompositions, as negative concepts frequently correspond to concepts not present in images, and as such, are unlikely to be represented by human captions. Finally, we see that modality alignment is necessary across all three metrics. Overall, these ablation studies show that each aspect of SpLiCE is necessary for creating human-interpretable, semantically relevant and highly performant decompositions.

5.4 Qualitative Assessment of Decompositions

Concept Decompositions for Images. We visualize SpLiCE decompositions to qualitatively assess the semantic content of the images they represent. In Figure 2 we provide six sample decompositions from MSCOCO with their corresponding captions. We display the top seven concepts for each

image and find that they generally well describe the semantics of the images. We also find that these qualitative examples yield interesting and unexpected insights into both CLIP and the data. In the top left image, we see that the decomposition includes the text present on the sign in the image, revealing that CLIP prioritizes text in images over objects. For the bottom left image, the decomposition correctly includes the concept ' macro shot ', revealing that CLIP encodes information regarding geometric perspective. The bottom right decomposition similarly features the concept ' blackandwhite bw ', indicating that CLIP encodes not only the objects present in images but also information about the lighting and color. Overall, these results suggest that SpLiCE may also be used as a zero-shot image tagging method to understand images.

Concept Histograms for Datasets. Beyond concept-based explanations of individual images, we propose that SpLiCE can be used to better understand and summarize collections of images, such as entire datasets. To compute concept decompositions of sets of images, we decompose each individual image and aggregate the results, which we use to generate concept histograms of the dataset. We visualize four concept histograms for the ImageNet classes ' African Elephant ', ' Curly-coated Retriever ', ' Monarch Butterfly ', and ' Digital Clock ', in Figure 4. These decompositions provide information about the distribution of the data and how CLIP represents it. For example, digital clocks are differentiated from analog clocks through the concepts ' lcd display ' and ' countdown '. Monarch butterflies are highly correlated with the concept ' lilac flowers " in ImageNet, which we validated through manual inspection (nearly half of the monarch butterfly images in the validation set feature purple flowers). Interestingly, ' Curly-coated retrievers ' are represented as combinations of ' poodle ', ' retriever dog ', and ' black dog ', which perfectly describe the main characteristics of them: black retrievers with poodle-textured fur.

6 Case Studies and Applications of SpLiCE

In this section, we present two example case studies using SpLiCE: (1) spurious correlation and bias detection in datasets and (2) debiasing classification models. We present additional case studies for (1) and (2), as well as (3) monitoring distribution shift in Appendix B.6, B.7, B.8 B.9. We also present results from a user study to evaluate the human interpretability of SpLiCE in Appendix B.1, where we find that users prefer explanations generated by SpLiCE over existing Concept Bottleneck Model-based methods.

Discovering Spurious Correlations in CIFAR100. Existing methods to detect spurious correlations in datasets generally require subgroup and attribute labels or rely on manual human inspection of images (see [63] for an overview), making it hard to scale to large datasets. SpLiCE, on the other hand, allows for fast automatic detection of such biases, without any labels, training, or even a task. To illustrate this, we study two classes of CIFAR100: 'man' and 'woman', in Figure 4. Upon decomposing these classes, we found that {' bra ',' swimwear ' } were two of the top ten most common concepts in the 'woman' class. On the other hand, the only clothing-related concepts that appear in the top 50 most activated concepts for 'man' are {' uniform ', ' tuxedo ', ' apparel '}. We visualize a histogram of the concept weights on swimwear- and undergarment-related concepts {' swimwear ', ' bra ', ' trunks ', ' underwear '} across both the train and test sets, and find that these concepts are much more likely to be activated for women than men. Manual inspection of CIFAR100 verifies the trend highlighted by SpLiCE, where at least 70 of the 600 images in the 'woman' class feature women in bikinis, underclothes, or even partially undressed , revealing stereotype bias in this popular dataset. We provide a similar study of the concept ' desert ' with respect to the 'camel' and 'kangaroo' classes in CIFAR100 in Appendix B.6.

Model Editing on CelebA Attribute Classifiers. Concept-based representations unlock a key application: being able to intervene on and edit models. This edit can be performed in two equivalent ways: either on the concept representations themselves, where we can zero out a concept or on linear probes built upon the decompositions, where we can edit the weight matrix between concepts and class labels (similar to concept bottleneck models [18]). Here, we evaluate the efficacy of SpLiCE for these forms of model editing. Specifically, we consider two tasks on CelebA, classifying gender and whether the subject is wearing glasses. To test representation editing, we remove the concept of 'eyewear' or 'glasses' from CelebA image representations by zeroing out any weight placed on these concepts in our SpLiCE decompositions and evaluate classifier performance. We report the performance of zero-shot classification and linear probes over our SpLiCE representation in Table

2. In both cases, we find that we can surgically target and remove information pertaining to glasses and reduce classifier performance while preserving information relevant to gender classification. We perform a similar experiment on the Waterbirds dataset [64] to remove spurious background signals in B.7.

7 Discussion

In this work, we show that the information contained in CLIP embeddings can be approximated by a sparse, linear combination of simple semantic concepts, allowing us to interpret representations via sparse recovery. We propose SpLiCE, a method to transform the dense, uninterpretable embeddings of CLIP into human-interpretable sparse concept decompositions.

We empirically demonstrate that SpLiCE allows for an adjustable tradeoff on the interpretabilityaccuracy Pareto frontier, enabling users to decide the loss in performance they are willing to incur for interpretability. Furthermore, we find that the improved interpretability of SpLiCE allows for users to diagnose and fix model mistakes, ideally increasing the effectiveness and performance of the overall system using a VLM. We then provide concrete use cases for SpLiCE: spurious correlation detection and model intervention and editing, showcasing the benefits of using interpretable embeddings with known semantic content. We highlight that SpLiCE embeddings can serve as post-hoc interpretations of CLIP embeddings and can even replace them to ensure full transparency.

Broader Impact. Similar to many works in the field of interpretability, our work provides greater understanding of the behavior of models, including but not limited to the broader implicit biases they perpetuate as well as mistakes made on individual samples. We believe this is particularly salient for CLIP, which is used in a variety of applications that are widely used in practice at this moment. We hope that insights gained from such interpretability allow users to make more informed decisions regarding how they interact with and use CLIP, regardless of their familiarity with machine learning or domain expertise in the task they are using CLIP for. We also highlight that SpLiCE can be used as a visualization-like tool for exploring and summarizing datasets at scale, allowing for easier auditing of spurious correlations and biases in both datasets and models.

Limitations. In this work, we use a large, overcomplete dictionary of one- and two-word concepts, however future work may wish to expand this dictionary or learn a dictionary over tokens (in discrete language space), to capture concepts with more than two words. This may also reduce the size of the dictionary and improve computation time. We note that this dictionary was constructed by looking at token frequency in the LAION-5B dataset, which has its own biases and may not correctly capture all the salient concepts that CLIP encodes. Despite this, we find that SpLiCE performs well on a variety of tasks while outperforming state-of-the-art concept dictionaries (Fig. 5, Appendix Fig. 13) and thus we believe LAION is a good dataset to generate a concept vocabulary from. We also note that this vocabulary can be easily modified by practitioners to consider additional concepts as needed for specific use cases. Finally, SpLiCE also uses an ℓ 1 penalty as the relaxation for ℓ 0 regularization, but future work may consider alternative relaxations or even binary concept weights.

References

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 18-24 Jul 2021.

Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 , 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning , pages 1885-1894. PMLR, 2017.

Yoshua Bengio. Deep learning of representations: Looking forward. In International conference on statistical language and speech processing , pages 1-37. Springer, 2013.

Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill , 2(11): e7, 2017.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pages 2668-2677. PMLR, 2018.

Kyle Hsu, Will Dorrell, James CR Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantization. arXiv preprint arXiv:2305.18378 , 2023.

Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un) reliability of feature visualizations. arXiv preprint arXiv:2306.04719 , 2023.

Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies , pages 746-751, 2013.

Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658 , 2023.

Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics , 6:483-495, 2018.

Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics , 4:385-399, 2016.

Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah Smith. Sparse overcomplete word vector representations. arXiv preprint arXiv:1506.02004 , 2015.

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations , 2022.

Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162 , 2022.

Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6820-6829, 2023.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning , pages 5338-5348. PMLR, 2020.

Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference on computer vision and pattern recognition , pages 951-958. IEEE, 2009.

Lorenzo Torresani, Martin Szummer, and Andrew Fitzgibbon. Efficient object category recognition using classemes. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part I 11 , pages 776-789. Springer, 2010.

Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision , pages 365-372. IEEE, 2009.

Aditya Chattopadhyay, Ryan Pilgrim, and Rene Vidal. Information maximization perspective of orthogonal matching pursuit with applications to explainable ai. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06129 , 2023.

Konstantinos Panagiotis Panousis, Dino Ienco, and Diego Marcos. Sparse linear concept discovery models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2767-2771, 2023.

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill , 5(3):e00024-001, 2020.

David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6541-6549, 2017.

Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8730-8738, 2018.

Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences , 119(47):e2206625119, 2022.

Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. On interpretability of deep learning based skin lesion classifiers using concept activation vectors. In 2020 international joint conference on neural networks (IJCNN) , pages 1-10. IEEE, 2020.

Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 119-134, 2018.

Thomas Fel, Victor Boutin, Mazda Moayeri, Rémi Cadène, Louis Bethune, Mathieu Chalvidal, Thomas Serre, et al. A holistic approach to unifying automatic concept extraction and concept importance estimation. arXiv preprint arXiv:2306.07304 , 2023.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread , page 2, 2023.

Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems , 31, 2018.

Pierre Comon. Independent component analysis, a new concept? Signal processing , 36(3): 287-314, 1994.

Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks , 13(4-5):411-430, 2000.

Aleksandar Makelov, Georg Lange, and Neel Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. arXiv preprint arXiv:2311.17030 , 2023.

Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi. Text-to-concept (and back) via cross-model alignment. In International Conference on Machine Learning , pages 25037-25060. PMLR, 2023.

Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205.15480 , 2022.

Tian Yun, Usha Bhalla, Ellie Pavlick, and Chen Sun. Do vision-language pretrained models learn composable primitive concepts? arXiv preprint arXiv:2203.17271 , 2022.

Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. Stair: Learning sparse text and image representation in grounded tokens. arXiv preprint arXiv:2301.13081 , 2023.

Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. arXiv preprint arXiv:2310.05916 , 2023.

Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794 , 2022.

Brian Murphy, Partha Talukdar, and Tom Mitchell. Learning effective and interpretable semantic models using non-negative sparse embedding. In Proceedings of COLING 2012 , pages 19331950, 2012.

Alona Fyshe, Partha P Talukdar, Brian Murphy, and Tom M Mitchell. Interpretable semantic vectors from a joint model of brain-and text-based meaning. In Proceedings of the conference. Association for Computational Linguistics. Meeting , volume 2014, page 489. NIH Public Access, 2014.

Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell. A compositional and interpretable semantic space. In Proceedings of the 2015 conference of the north american chapter of the association for computational linguistics: Human language technologies , pages 32-41, 2015.

Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research , 37(23):3311-3325, 1997.

Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature , 381(6583):607-609, 1996.

Vikram V Ramaswamy, Sunnie SY Kim, Ruth Fong, and Olga Russakovsky. Overlooked factors in concept-based explanations: Dataset choice, concept salience, and human capability. arXiv preprint arXiv:2207.09615 , 2022.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144, 2016.

David P Vinson and Gabriella Vigliocco. Semantic feature production norms for a large set of objects and events. Behavior Research Methods , 40(1):183-190, 2008.

Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods , 37 (4):547-559, 2005.

Peter Garrard, Matthew A Lambon Ralph, John R Hodges, and Karalyn Patterson. Prototypicality, distinctiveness, and intercorrelation: Analyses of the semantic attributes of living and nonliving concepts. Cognitive neuropsychology , 18(2):125-174, 2001.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems , 35:17612-17625, 2022.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. doi: 10.5281/zenodo.5143773. URL https://doi.org/10.5281/zenodo.5143773 . If you use this software, please cite it as below.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Phillip Isola, Joseph J. Lim, and Edward H. Adelson. Discovering states and transformations in image collections. In CVPR , 2015.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09 , 2009.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences , 2(1):183-202, 2009.

Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Conceptaware mitigation of spurious correlation. In International Conference on Machine Learning , pages 37765-37786. PMLR, 2023.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning , 3(1):1-122, 2011.

Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence , 28(4):594-611, 2006.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 3485-3492. IEEE, 2010.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascalnetwork.org/challenges/VOC/voc2007/workshop/index.html.

Michael Stark, Jonathan Krause, Bojan Pepik, David Meger, James J Little, Bernt Schiele, and Daphne Koller. Fine-grained categorization for 3d scene understanding. International Journal of Robotics Research , 30(13):1543-1552, 2011.

Appendix

Summary of Appendix Results

A . Further Details on the Method

-A.1 . When do Sparse Decompositions Exist?

-A.2 . Relationship between cosine similarity and MSE optimization

-A.3 . ADMMfor batched on-device LASSO optimization

-A.4 . Effect of Modality Alignment

-A.5 . Experimental Details

B . Additional Results

-B.1 . User Study for Human Interpretability

-B.2 . Performance of SpLiCE on Probing Tasks

-B.3 . Performance of SpLiCE on Retrieval Tasks

-B.4 . Additional Zero-Shot Results

-B.5 . Additional ImageNet Concept Histograms

-B.6 . Additional Case Study: Detecting Spurious Correlations

-B.7 . Additional Case Study: Spurious Correlation Intervention

-B.8 . Additional Case Study: Distribution Shift Monitoring

-B.9 . Additional Case Study: Distribution Shift Monitoring

-B.10 . Checking the Interpretability of Negative Concepts

-B.11 . Understanding the Image Mean for Modality Alignment

-B.12 . Choice of Concept Vocabulary

-B.13 . Concept Type Distribution

-B.14 . Experiments on Alternative CLIP Architecture

A Further Details on the Method

A.1 When do Sparse Decompositions Exist?

Proposition 1. Given Assumptions 1-5, CLIP image embeddings f can be written as a sparse linear combination of text embeddings, i.e,

where w ∈ R k + , and C txt ∈ R d × k , which is the text concept dictionary defined previously.

Proof. Any vector ω can be written as ω = ∑ k i =1 ω i e i , where ω i ∈ R + , and e i ∈ R k is a one-hot vector with one at the i th co-ordinate. Thus we have

Here we define c img i = f ◦ h img ( e i ) as the 'image' concept basis vector; analogous to the text concept basis vector c txt i = g ◦ h txt ( e i ) already defined. Thus Assumption 2 implies the existence of a sparse decomposition of f in terms of 'image' concept vectors c img i . Additionally, Assumption 1 ensures that this decomposition is sparse, as ω is sparse. So far, we have f ( x img ) = C img ω s.t. ∥ ω ∥ 0 ≤ α .

From Assumption 4, the image concept vectors and text concept vectors are equal to each other, i.e, c img i = f ◦ h img ( e i ) = g ◦ h txt ( e i ) = c txt i . Finally, from Assumption 5, we have that the text concept vectors c txt i are given simply by word embeddings g of individual words.

Stringing these arguments together, we have that image representations f ( x img ) can be written as a sparse linear combination of vectors obtain from CLIP word embeddings c txt i . We finally set w = ω , thus proving the assertion.

A.2 Relationship between cosine similarity and MSE optimization.

Recall our ℓ 1 relaxed cosine similarity optimization problem from Eqn. (1),

First we relax the ℓ 0 constraint to an ℓ 1 penalty.

By observing that || x -y || 2 2 = ⟨ x -y, x -y ⟩ = ⟨ x, x ⟩ + ⟨ y, y ⟩ -2 ⟨ x, y ⟩ and that z , Cw ∥ Cw ∥ 2 are unit-norm, maximizing the above inner product is equivalent to minimizing the euclidean norm,

This is a non-convex problem, but we can relax this problem to achieve better reconstruction in terms of euclidan distance as shown in Eqn. (2),

This problem will optimize euclidean distance between Cw and z . Consider two vectors x, y on the unit sphere such that ⟨ x || x || , y || y || ⟩ > 0 . While any vector αy , α > 0 will have the same cosine similarity score, the optimal vector in terms of euclidean distance to x is the vector αy such that α = proj y ( x ) , or in other words the projection of x onto y . Thus, solving for euclidean distance to approximate x will find αy which we must then normalize to find the unit-norm solution y . This explains the normalizing process described in Section 4.1.

Additionally, we can view Eqn. (6) as applying shrinkage to Cw . Reconverting from euclidean norm to inner product, Eqn. (6) becomes

In conclusion, our optimization problem maximizes the inner product while imposing a shinkage penalty and sparsity penalty. Empirically, our reconstructions Cw are low-norm, so we normalize after solving to recover the unit-norm reconstruction.

A.3 ADMMfor batched on-device LASSO optimization.

As each decomposition requires solving a LASSO optimization problem, we implement the Alternating Direction Method of Multipliers (ADMM) algorithm in Pytorch over batches with GPU support for efficient decomposition of large scale datasets over large numbers of concepts [65]. In practice, ADMMachieves primal and dual tolerances of 1e -4 in fewer than 1000 iterations on a batch size of 1024 . We present an empirical comparison beterrn LASSO and ADMM in 6, where we find both methods to be approximately equivalent.

Next we derive the iterates for our ADMM algorithm. Recall our optimization problem,

ADMMbreaks down convex optimization problems into multiple sub-problems while penalizing the difference in solutions. We break Eqn. (8) into two subproblems, one solving the euclidean distance

objective and one solving the ℓ 1 and nonnegativity constraint. We let w denote the former solution, z the latter, and u tracks the difference between the two. Our ADMM iterates ( w k , z k , u k ) are

where S κ is a soft-thresholding function used to satisfy the LASSO constraints,

As our optimization function f ( w ) is quadratic, we can analytically compute w k +1 as

where v = z k -u k . In our experiments we set ρ = 5 , and stop when tolerances ϵ prim = || x k +1 -z k +1 || 2 , ϵ dual = || ρ ( z k +1 -z k ) || 2 are less than 1e -4 . Over a batch, we iterate until every solver in the batch has reached the above tolerances.

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.2

0.3

0.4

0.5

0.6

0.7

Dictionary

laion_lasso

learned (images)

learned (text)

random

laion_admm

CLIP

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.4

0.5

0.6

0.7

0.8

Cosine Similarity (x_hat, x)

CIFAR100 Cosine Reconstruction

Dictionary

laion_lasso

learned (images)

learned (text)

random

laion_admm

A.4 Effect of Modality Alignment

We take MSCOCO images and captions, embed them with CLIP, and compare the cosine similarity between modalities and inter-modality. Before mean-centering and renormalizing, the similarity within modalities is high, with an average of around 0.3. This indicates that the image and text embeddings do not span the entire unit-sphere but rather lie on two cones. However, the similarity across modalities has an average concentrating around zero, indicating that these two cones are non-overlapping. However, after mean-centering and normalizing, we observe that the average cosine similarity for images, text, and between images and text becomes zero and the modalities are aligned.

0.0

0.2

0.4

0.6

0.8

Cosine Similarity

0.0

0.5

1.0

1.5

2.0

2.5

Density

CLIP Modality Similarity

Modalities

img_text

text_text

img_img

0.4

0.2

0.0

0.2

0.4

0.6

0.8

Cosine Similarity

0.0

0.5

1.0

1.5

2.0

2.5

Density

CLIP Modality Similarity with Centering

Modalities

img_text

text_text

img_img

A.5 Experimental Details

All experiments are able to be performed on a single A100 GPU to run fast inference with CLIP. After embedding the concept dictionary, all computation can be performed on a CPU. Code is made available at https://github.com/AI4LIFE-GROUP/SpLiCE .

B Additional Results

B.1 User Study for Human Interpretability

We present results from a user study in 8 to assess the human interpretability of SpLiCE. We base our study off of that performed by [23] to evaluate Label-Free Concept Bottleneck Models (LF-CBMs). We benchmark our method against LF-CBMs and IP-OMP [22]. We provided users with twenty randomly chosen, correctly predicted images from ImageNet and explanations from two different methods comprising the top six most important concepts for every image. We then asked users to evaluate and compare the different concept-based explanations for (1) their relevance to the provided image inputs, (2) their relevance to model predictions, and (3) their informativeness on Likert scales from 1 to 5. We found that users significantly preferred explanations generated by SpLiCE to the two baselines for relevance to the images and informativeness, with significance determined via a one-sample two-sided t-test and a threshold of p=0.01. We also highlight that our method is able to produce similar/better concept decompositions, in terms of human interpretability, than the baselines without needing to train a classification probe or use class labels for concept mining, both of which are computationally expensive. This user study was ruled exempt by our institution's IRB, as no risks were posed to the users. Participants were able to opt out at any time, and no questions were asked regarding the participants themselves.

Which explanation

is more relevant to

the image?

Which explanation

is more relevant to

the prediction?

Which explanation

is more informative?

0%

25%

50%

75%

100%

Strongly prefer SpLice

Weakly prefer SpLiCe

No preference

Weakly prefer LF-CBM

Strongly prefer LF-CBM

User Study Comparing SpLiCe and LF-CBM

p < 0.01

p < 0.01

*

*

Percentage of User Evaluations

Which explanation

is more relevant to

the image?

Which explanation

is more relevant to

the prediction?

Which explanation

is more

informative?

0%

25%

50%

75%

100%

Strongly prefer SpLice

Weakly prefer SpLiCe

No preference

Weakly prefer IP-OMP

Strongly prefer IP-OMP

User Study Comparing SpLiCe and IP-OMP

p < 0.01

p < 0.01

p < 0.01

*

*

*

Percentage of User Evaluations

B.2 Performance of SpLiCE on Probing Tasks

We evaluate the performance of the decompositions on probes trained on both regular CLIP embeddings as well as decomposed CLIP embeddings for CIFAR100 in 3 and MIT States in 4. We consider two scenarios: a probe trained on CLIP embeddings and tested on SpLiCE embeddings of various sparsities (shown in row CLIP Probe ), and a probe both trained and evaluated on SpLiCE embeddings (shown in row SpLiCE Probe ). We report mean over three runs, with standard deviations for each experiment being less than 0.005. We find that SpLiCE representations closely match the performance of dense CLIP embeddings, with a slight drop in performance when probes are trained directly on SpLiCE embeddings rather than trained on CLIP embeddings and evaluated on SpLiCE embeddings for CIFAR100.

B.3 SpLiCE Performance on Retrieval Tasks

We test the performance of SpLiCE embeddings on text-to-image and image-to-text retrieval tasks. We evaluate retrieval over various 1024 sample subsets of MSCOCO, and assess recall performance for the top-k closest embeddings of the opposite modality for k = { 1 , 5 , 10 } . We find that our semantic concept dictionaries outperform all baselines when decomposition sparsity is high, but that dictionaries learned over images perform slightly better for text to image retrieval when decompositions have greater than 30 nonzero concepts.

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.1

0.2

0.3

0.4

0.5

0.6

Recall @ 1

Image to Text Retrieval (1k Test Set)

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Recall @ 5

Image to Text Retrieval (1k Test Set)

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Recall @ 10

Image to Text Retrieval (1k Test Set)

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.1

0.2

0.3

0.4

0.5

0.6

Recall @ 1

Text to Image Retrieval (1k Test Set)

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Recall @ 5

Text to Image Retrieval (1k Test Set)

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Recall @ 10

Text to Image Retrieval (1k Test Set)

random

learned

(images)

learned

(text)

laion

CLIP

(baseline)

laion +

imagenet

classes

B.4 Additional Zero-Shot Results

We present additional results comparing SpLiCE reconstructed vectors and CLIP embeddings on the Caltech101 [66], SUN397 [67], STL10 [68], and VOC2007 [69] datasets in 5. We use SpLiCE decompositions with sparsities of 20-35, and we find that they are comparable to the unaltered CLIP embeddings.

We further explore the performance of SpLiCE decompositions in the limit as they approach the sparsity of the baseline CLIP embeddings (512). We find that SpLiCE completely recovers CLIP zero-shot accuracy at this limit, as shown in 6.

B.5 Additional ImageNet Concept Histograms

We present concept histograms for the top seven concepts of five more ImageNet classes: {'Face Powder', 'Feather Boa', 'Jack-O'-Lantern', 'Kimono', 'Dalmation'}, similar to Figure 10. These decompositions give insights both into the distribution of each class as well as some biases of CLIP. For example, for the class 'Face Powder', the concept 'benefit" is the fifth most common concept, and it is indeed a common cosmetic brand name in the images. For the 'Dalmation' class, we see that the decompositions consists of concepts relating to dogs and black and white spots, which together make up the high-level concept of a dalmation. Finally, for the class 'Kimono', the concept 'doll" is the seventh most common, although all of the images in the 'Kimono' class were of real humans, not of dolls. This highlights an implicit bias in CLIP's representations or in the descriptions of people wearing kimonos in CLIP's training set.

cosmetic

eye shadow

makeup

makeup brush

benefit

products

hair products

0.00

0.01

0.02

0.03

'Face Powder' Decomposition

mardi gras

fashion portrait

bachelorette party

halloween costume

feather

fancy dress

gay pride

0.0000

0.0025

0.0050

0.0075

0.0100

0.0125

'Feather Boa' Decomposition

halloween pumpkin

halloween

pumpkins

scary

carve

glowing

0.00

0.05

0.10

0.15

0.20

'Jack-O'-Lantern' Decomposition

traditional dress

japanese girl

tea ceremony

traditional wedding

japanese

doll

0.000

0.025

0.050

0.075

0.100

'Kimono' Decomposition

dog breed

spots

cute dog

spotted

puppy

dog portrait

blackandwhite bw

0.00

0.01

0.02

0.03

'Dalmation' Decomposition

B.6 Additional Case Study: Detecting Spurious Correlations

We present an additional case study for detecting spurious correlations in CIFAR100. In particular, we look at the prevalence of the spurious concept 'desert' in the classes 'camel' and 'kangaroo' in Figure 11. We observe that camels are more frequently pictured in the desert, creating a spurious signal that may be leveraged by downstream classifiers. This figure provides an additional example of how we can understand biases and trends in data with SpLiCE decompositions.

B.7 Additional Case Study: Spurious Correlation Intervention

We further test the ability of SpLiCE to enable intervention on intermediate representations and linear classifiers by attempting to remove information pertaining to spurious signals. In particular, we consider the Waterbirds dataset [64], which spuriously correlates landbirds with land backgrounds, resulting in trained classifiers performing poorly on waterbirds on land. We thus remove information about whether or not birds are on land backgrounds by ablating concept weights on 'bamboo',

0.00

0.05

0.10

0.15

Concept Weight

10

0

10

1

10

2

Count

CIFAR100 'Desert' Concept

Class

Camel

Kangaroo

'forest', 'hiking', and 'rainforest' as well as any bigrams containing the word 'forest,' as shown in Table 7. This significantly improves worst-case subgroup performance for waterbirds on land from 0.48 to 0.60.

For both this experiment and the intervention on CelebA described in the main paper, we train linear probes using the LogisticRegressionClassifier module in scikit-learn using an ℓ 1 penalty.

B.8 Additional Case Study: Distribution Shift Monitoring

We present a final case study using SpLiCE to monitor distribution shift. This can help identify differences between training and inference distributions or evaluate how a continually sampled dataset changes over time. In this experiment we consider the Stanford Cars dataset [70], which contains photos of cars from 1991 to 2012, including their make and year labels. By decomposing photos of cars from each year, we can view how the distribution changed yearly. We visualize the weights of the concepts 'convertible" and 'yellow" from our decompositions, as well as the actual percentage of cars from each year that were convertibles or yellow in Figure 12. Note the right-hand y-axis, corresponding to the weight of the given concept c i over the sum of the weights of all concepts ∑ i c i , does not have a meaningful unit of measure or scale. We find that the trends in the groundtruth concept prevalence generally closely match that of the predicted/decomposed concepts, allowing us to visualize which years convertibles or yellow cars were popular or out-of-distribution with respect to other years. Most notably, we see that SpLiCE picks up on the out-of-distribution rise in popularity of brightly colored sports cars in the early 2000s.

B.9 Additional Case Study: Distribution Shift Monitoring

To further verify that SpLiCE allows for identification and tracking of distribution shift, we study the Waterbirds dataset, which is known to have differently balanced train, valodation, and test splits. To identify distribution shifts, we can simply look at the norm of the difference between the class decompositions of the two classes for each splot, as shown in 8. We find that the validation and test splits are much more similar than the training and validation splits or the training and test splits, which can be verified by the construction process of the Waterbirds dataset.

'93

'08

'10

'09

'12

'07

'98

'99

'00

'01

'02

'06

'11

Year

0.0

0.2

0.4

0.6

0.8

1.0

Groundtruth Concept Percent

0.00

0.05

0.10

0.15

0.20

0.25

Decomposed Concept Weight

Stanford Cars Concepts Distribution Shift

GT 'Convertible' Car Percent

GT 'Yellow' Car Percent

'Convertible' Concept Weight

'Yellow' Concept Weight

We also find that the most weighted concept in the 'landbird' class of the train split is 'bamboo' but the corresponding weight for 'bamboo' in the 'waterbird' class is much lower. The 'bamboo' concept weight for both classes and all splits is shown below, where we see that the validation and test splits are very similar and mostly evenly balanced, whereas the train split is highly unbalanced.

B.10 Checking the Interpretability of Negative Concepts

We take a set of 71 concept-antonym pairs from the MIT States dataset and embed the terms in CLIP. With and without concept centering, we observe that these concept-antonym pairs have an average cosine similarity well above -1, indicating that CLIP does not place antonyms in opposite directions, as shown in 10. Next, we take our concept dictionary and prepend 'not" to all of the words and compare the average cosine similarity between concept and not-concept pairs. Similarly, we observe that with and without centering, concept and not-concept pairs are highly similar. Note that the average similarity for true pairs of images and text in MSCOCO is less than the similarity between concepts and not-concepts with and without centering.

B.11 Understanding the Image Mean for Modality Alignment

In order to empirically check that the mean centering of images does not result in a loss of information, we decompose the img mean, µ img , that we used for all experiments. If we decompose it with uncentered concepts, the following concepts are highlighted: {'closeup", 'flickr", 'posed"}. The decomposition with centered concepts results in the following concepts: {'flickr", 'posed", 'pics", 'angle view", 'last post"}. These concepts all seem to be generally related to images, with minimal other semantic information, suggesting that centering does not remove any discriminative semantic content of embeddings, but simply removes information about the modality.

B.12 Choice of Concept Vocabulary

We perform a simple ablation study to assess the sensitivity of our method to choices in concept vocabulary. We collect a second vocabulary in the same exact manner as the LAION vocabulary from the MSCOCO caption dataset. We consider both the top 10k and top 5k most common words for both, and repeat the zero-shot accuracy and reconstruction cosine similarity experiments from Section 5.2 on CIFAR100. We see that the MSCOCO10k and LAION10k vocabularies perform almost exactly the same for both metrics. The smaller vocabularies perform the same for cosine reconstruction but underperform the 10k vocabularies for zero-shot classification tasks.

0

20

40

60

80

100

120

Decomposition l0 Norm

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

Cosine Similarity (x_hat, x)

Effect of Vocab on Cosine Reconstruction

Dictionary

coco (10k)

laion (10k)

coco (5k)

laion (5k)

0

20

40

60

80

100

120

Decomposition l0 Norm

0.55

0.60

0.65

0.70

0.75

Effect of Vocab on Zero Shot Accuracy

Dictionary

coco (10k)

laion (10k)

coco (5k)

laion (5k)

CLIP

B.13 Concept Type Distribution

In order to better understand any biases produced by the decomposition process or that CLIP itself has, we visualize the types of concepts most commonly activated across multiple datasets, labelling them by part of speech in Figure 14. We see that nouns are by far the most common concepts across datasets, indicating that both CLIP and the decompositions are highly object centric. Note that the low weight on verbs and adjective is due to far fewer concepts of those types being activated (low l 0 norm) as well as the weight upon those concepts being significantly smaller (low l 1 norm). We hypothesize that the information in many adjective and verbs can actually be encoded into the noun itself, resulting in this phenomenon. For example, the concept 'lemon" is a more succinct form of 'yellow" and 'fruit".

B.14 Experiments on Alternative CLIP Architecture

We present cosine reconstruction and zero-shot accuracy experiments with an alternative CLIP architecture from OpenAI with a ResNet50 backbone for the vision encoder. Note that these experiments were done with a 10000 size vocabulary of only one-word concepts. We find that results are similar to those presented in 3, save for OpenAI's ResNet50 CLIP performing much worse than OpenCLIP's ViT B/32 backbone in general.

Noun

Verb

Adjective

Adverb

Part of Speech

0

25

50

75

100

125

150

175

200

Concept Weights

Distribution of Concept Types

Dataset

COCO

MIT States

CIFAR100

ImageNet

0

20

40

60

80

Decomposition l0 Norm

0.45

0.50

0.55

0.60

0.65

0.70

Cosine Similarity (x_hat, x)

CIFAR100 Cosine Reconstruction

Dictionary

laion

random

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.45

0.50

0.55

0.60

0.65

0.70

0.75

Cosine Similarity (x_hat, x)

MIT States Cosine Reconstruction

Dictionary

laion

random

0

50

100

150

200

Decomposition l0 Norm

0.5

0.6

0.7

0.8

Cosine Similarity (x_hat, x)

ImageNet Cosine Reconstruction

Dictionary

laion

random

0

20

40

60

80

Decomposition l0 Norm

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Dictionary

laion

random

CLIP

0

20

40

60

80

100

120

140

Decomposition l0 Norm

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Dictionary

laion

random

CLIP

0

50

100

150

200

Decomposition l0 Norm

0.1

0.2

0.3

0.4

0.5

ImageNet Zero Shot Accuracy

Dictionary

laion

random

CLIP

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, we introduce a novel method and provide comprehensive experiments demonstrating its utility on downstream tasks as well as various case studies in Sections 5 and 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the appendix we note the limitations of our work ?? .

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a detailed set of assumptions, proposition and proof sketch in the main paper section 3 and a full proof in section A.1

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose all information required to reproduce the main results by providing details on the exact methods and metrics used and providing code to implement SpLiCE . In our experiments (Sec 5) we disclose our datasets. Then, to replicate results, we describe how to compute zero-shot accuracy, cosine similarities, as well as how to generate concept histograms over classes and concepts as shown in (Figure 4.) These are also included in our code. We also include additional experimental details in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use publicly-available datasets, MSCOCO, CIFAR100, MIT States, Imagenet, CelebA, Waterbirds, and Stanford Cars and include code to replicate our results (link at the end of abstract).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify hyperparameters and other implementation details in 5.1. We describe how we optimize for Eqn. (2) using scikit-learn, as well as include an implementation in our codebase (linked in the abstract).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Throughout the paper we report error bars when possible. For instance, when computing reconstruction error we are able to report error bars (Fig. 3). However, for zero-shot error, this metric is fixed for any dataset so we do not report error bars there. When possible we run experiments 5 times and include error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include details on compute resources in the Appendix A.5

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We do not have human subjects nor do we collect additional data. When conducting semantic decomposition, there is a risk that we include harmful semantics with our large dictionary. However, as evidenced in Figure 4, our method actually uncovers bias to help reduce it in the future rather than perpetuating bias. Semantic decompositions allow for a more critical view of our machine learning systems and datasets. In addition, we have open-sourced SpLiCE with essential elements for reproducibility to ensure public governance of the system.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We include a discussion of broader impact in the (Appendix ?? )

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: As our method does not release data, nor does it include a generative model, we do not have a high risk for misuse. Thus, we do not have any special safeguards on our method.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite all code and models in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release a cleaned and simple-to use version of the model with working examples and an API. The code is linked at the end of the abstract.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not use human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: We present a small-scale user study in our supplementary material, in which users were asked to rank explanations generated by our method and prior concept-based explanation methods. The study was ruled exempt by our institution's IRB, as no risks were posed to the study participants. Further information is given in Appendix B.1.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhilst useful for many downstream tasks, CLIP’s vision-language representations are notoriously hard to interpret. The paper proposes to represent CLIP representations in a sparse, non-negative overcomplete basis of learnt interpretable directions using a standard dictionary learning technique. Not only is interpretability increased, but the core claim is that this does not harm the performance of the downstream applications significantly. Such a model would provide interpretability “for free”, in the sense that practitioners could opt for the interpretable version without any performance cost (which would constitute a significant achievement).\n\n---\n\nPost-rebuttal:\nFollowing a productive discussion with the authors (who both clarified many experimental details and proposed revisions to the claims of the paper to better reflect its results), I think the paper will be of great value to the community and recommend its acceptance.\n\nSTRENGTHS:\n- The work strives to address a vitally important task: to design models that offer interpretability at no cost to performance. Such an endeavor is particularly important practically in allowing practitioners to adopt interpretable models in practice (by not trading off performance).\n- The paper is very well-written, the applied methodology is appropriate, and the focus on CLIP representations means the paper has potentially great relevance and reach to practitioners.\n- Offering more than just transparency, I appreciate the method's multiple use cases--not only for discovering spurious correlations but also for model editing.\n\nWEAKNESSES:\n# [W1] Sparsity/interpretability does trade-off accuracy\n\nI was very excited about the paper’s bold claim in the introduction to `provide interpretability, at no cost to downstream performance` ([L6], but also at [L349]) upon a first read. Unfortunately, from what I came to understand later from the results of the proposed dictionary in yellow plots in Figure 3, this claim appears false.\n\nThere *is* a cost to zero-shot classification performance (which is arguably one of the most important downstream applications of CLIP). For example, small L0 norms lead to as much as ~10% accuracy drop on CIFAR100 (left-most plot), and what appears to be a ~20% performance drop on ImageNET1k using the raw dictionaries. A similar pattern of performance degradation for high sparsity is observed in Fig. 7 for retrieval tasks. Thus, it is simply not true that the method is at “no cost” to performance, and arguably it is not even at “minimal” cost (used later in [L52]). These results unfortunately do not support the paper’s core claims.\n\nCrucially, if I correctly understand the x-axis as quantifying the number of non-zero coefficients, this is even more problematic. High sparsity is desirable for interpretability [L164]—but we see from the experiments that too small an L0 term trades-off accuracy, and sometimes rather significantly so. This fundamental trade-off is a clear critical limitation of the work (in conflict with the paper's core claims) and puts interpretability and accuracy at odds—but it is not ever stated or discussed as a key limitation of the work.\n\nThe authors should revise this core claim in light of the fact there is indeed a cost to performance increasing as a function of sparsity, and include a dedicated discussion of such limitation. Furthermore, I would expect to see many more experiments on the standard zero-shot datasets used in the original CLIP paper [1] to evaluate this core claim, not just on 3 (given this is zero-shot, these experiments are very fast to run with pre-trained dictionaries).\n\nThis weakness in particular swayed my rating of this paper negatively, and I am happy to consider revising my score accordingly, should an adequate answer be provided in response.\n\n# [W2] Limited new technical contributions, but also lacking technical insights/comparisons to alternative existing solutions \n\nThe paragraph in [L229] is rather brief and states that sklean’s Lasso solver is used. In the concrete application of sparse dictionary learning for CLIP representations in the paper, a representative reader would be very interested to see a much richer exploration of alternative approaches to solving the model objective.\n\nFor example, in CLIP, how does this proposed solution compare to non-negative KSVD? How about NMF (with a sparsity penalty)? Or sparse autoencoders? Or learning this end-to-end with projected gradient descent (to project the coefficients onto the non-negative orthant), etc.\n\nGiven that the paper does not offer any novel technical contributions (which is okay!), I think the paper would have been much stronger if at least some technical insights (about existing techniques) as they pertain to representation learning with CLIP were provided. At a minimum, a *discussion* of some of the other techniques listed above (and why they might not be appropriate) would be desirable, and experiments comparing the results from each to validate the proposed solution would make the paper’s contributions even stronger through technical insight.\n\n# [W3] Unnecessary mathematizing (minor)\n\nAs a reader very excited about this paper’s application, I feel the second half of Section 3. “When sparse decompositions exist” is an unnecessary formalization ([L141] onwards). I’m unconvinced of the need to introduce “Proposition 1”—at minimum, it feels like a distraction from the other interesting results in the paper. As far as I can see, this proposition is not used at all in the paper, and no experiments are conducted to show the 5 assumptions are reasonable (aside from Assumption 3).\n\nThe “assumptions” 1-5 listed do appear useful heuristics to reason about when Sparse decompositions are appropriate. As the authors also state [L133], concept linearity seems most crucial here. As a reader, what I care about is how these assumptions hold in CLIP in practice. As such, it would be much more useful to see instead the key experiment in Appendix B.5 discussed here in the main paper to support this critical assumption.\n\nTo summarize this final weakness: my impression is that the paper’s clarity could be improved by (a) considering deferring the formalizations to the supplementary material, or reducing their significance/length in the main paper and (b) considering making the critical experiments validating the key assumption in Appendix B.5 more prominent. I believe the paper would be much more impactful and have greater reach if the key results were better highlighted, and math that is not strictly necessary dropped.\n\n---\n\n* [1]: Radford, Alec et al. “Learning Transferable Visual Models From Natural Language Supervision.” International Conference on Machine Learning (2021).\n\nQUESTIONS:\n# [Q1] Vocabulary choice and dataset dependence\n\nI’m not sure I would agree with the claim that the `efficacy of the decomposition is (in principle) independent of individual datasets`. Isn’t the vocabulary choice made in this work specific to LAION-400m? Perhaps using the WordNET nouns/adjectives as the vocabulary (and bigrams formed by common combinations) would be even more dataset/task agnostic? \n\n# [Q2] Mean centering\n\nAs stated by the authors, the CLIP representations live on a hypersphere. Instead of taking the geometric mean in Euclidean space (as part of addressing the modality gap), did the authors explore taking the Frechet mean? This seems like it would better account for the geodesic distances between the points.\n\n# [Q3] Zero-shot performance in the limit\nWhat happens to the zero-shot performance in Fig. 5 in the limit of the l0 norm being equal to the dimensionality of the space? Shouldn’t we expect matching performance to the CLIP baseline? It would be insightful to extend the x-axis to observe how it performs in the limit.\n\nLIMITATIONS:\nNo, limitations are not adequated addressed, and the current limited discussion should be placed prominently in the main paper rather than the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for building sparse embeddings from CLIP in order to improve the interpretability of CLIP’s latent space.  They formulate their objective as a sparse reconstruction problem, and under certain assumptions, demonstrate when finding a sparse representation is possible.  Empirically, the authors show over three datasets that zero-shot accuracy and reconstruction errors are correlated, and performance is similar with sparse embeddings vs. without.  They further illustrate the benefits of sparse embeddings for interpretability noting applications to reducing spurious correlation reliance on a Celeb-A benchmark, and identifying biases through representations.\n\nSTRENGTHS:\n- This paper presents an important study toward understanding the latent space of vision-language models like CLIP which has become a very popular paradigm and backbone for many downstream tasks. This can help to improve the next generation of such models in training and downstream use. The use of text to interpret to solely understand the vision space and perform interventions is also important and makes the method relatively lightweight.  It also makes the method extendable to researchers and practitioners outside of the space as only concept vocabulary is needed. \n\n- The paper is overall well-written stating clearly the assumptions of the work in Section 3, and the proposed method is also clear and intuitive to implement.\n\nWEAKNESSES:\n- The primary weakness of this paper is that it is unclear to me what improvements come from generating sparse CLIP embeddings.  Results indicate that there is no improvement in zero-shot accuracy due to some outstanding reconstruction error.  In contrast, when removing spurious correlations it is typically seen as an improvement to model performance as the model would typically in this case then be evaluated in the OOD setting where the correlation is not present.  From this perspective, I think the authors might benefit from changing to a setting more similar to this, where the results are more aligned with benefits from having an interpretable tunable mechanism, rather than the more ID zero-shot performance settings.  This is started in the model editing, however it doesn’t lead to much improvement from correlation reliance, only mitigation of the capability to classify another attribute.  In contrast performance goes down indicating some capability but perhaps not the right evaluation setting.  \n\n- Similarly, I am uncertain about the case study evaluations for discovering spurious correlations.  Results for the man-woman evaluation appear to only indicate the trend is present in 70/600.  This seems like a relatively low number as the majority then do not have this bias.  It is unclear what the impact would be in terms of downstream performance for classification on CIFAR-100, or a gender classifier trained on CLIP embeddings.  The study seems inconclusive due to the results not being well contextualized 70 out of 600, and the lack of a downstream performance evaluation and improvement.\n\n- Finally, in multiple places, the authors mention improved human interpretability such as 244, 277, etc. however the authors never conduct any human evaluations.  If the authors claim improved human interpretability, the authors should conduct such a human evaluation, otherwise it is only hypothesized improved human interpretability.\n\nQUESTIONS:\nHave the authors demonstrated an evaluation setting where having the more interpretable concept features would improve performance? I believe some OOD setting such as spurious correlation could improve, but there may be many setting this is worse, as you are reliant on the concept features and may be worse for settings such as fine-grained classification where the concepts were not covered.\n\nLIMITATIONS:\n- An additional limitation of the work not discussed by the authors is the reliance on writing a set list of word-level concepts.  The approach is reliant on this set list having the desired concept when detecting spurious correlations, etc.  As the authors have pointed out this also limits the type of concepts.  \n\n- The authors have also not discussed how practical the assumptions proposed in Section 3 are, and whether they are correct in practice.  This is important for demonstrating applicability of the proposed work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

comments: SUMMARY:
This paper presents a method to explore semantic concepts in multimodal models of text and images. Specifically, the paper formulate semantic concept discovery  problem as one of sparse recovery and build a novel method, Sparse Linear Concept Embeddings (SpLiCE), for transforming CLIP representations into sparse linear combinations of human-interpretable concepts.

STRENGTHS:
1. The paper is well written and reads well.

2. The technical method is generally make sense .

3. The contribution is enough for covering text-image multi-model method to an interpretable model, and the research topic is worth pursuing.

4. Sufficient experiments demonstrate the effectiveness of the proposed simple method.

5. The experiment designed in the paper is make sense.

WEAKNESSES:
see questions

QUESTIONS:
1 The connection and difference between your method with a multi model topic model? In my opinion, topic model is good at extracting interpretable concepts. 

2 Although a lot of experiments were conducted in the paper, I think how to judge the concept discovery results is still not convincing enough. Whether to add manual evaluation experiments will be more convincing.

3. Why does negative concept weight perform better in Table 5? What does this mean?

LIMITATIONS:
see weakness and question
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to transform CLIP representations into sparse linear concept embeddings that are interpretable to humans. SpLiCE uses task-agnostic concept sets, demonstrating its versatility over prior works. SpLiCE provides interpretability without compromising zero-shot classification performance and shows further applications, including spurious correlation detection and model editing.\n\nSTRENGTHS:\nSpLiCE suggests a novel method to interpret CLIP embeddings into semantic concepts, which could be a good way to understand the latent space of CLIP. As it is a task-agnostic approach, which does not constrain itself to a certain domain, it can be applied to various datasets to explore the CLIP embeddings. This task-agnostic concept shows its scalability across different datasets, including CIFAR100, MIT States, CelebA, MSCOCO, and ImageNetVal. Also, the authors demonstrate the efficacy of their method both quantitatively and qualitatively.\n\nWEAKNESSES:\n[W1] I believe the choice of datasets is insufficient to show the efficacy of \"task-agnostic\" concpet sets. For example, how does SpLiCE work on bird identification dataset or CelebA? What are the top activated concepts in these datasets?\n\n[W2] Adding class labels to concept dictionary seems inappropriate from the perspecive of \"concept decomposition,\" as in the case of ImageNet. If the top concept is class label itself, what is the need for concept decomposition? \n\n[W3] A user study on the interpretability of SpLiCE would be needed to quantify how interpretable this method is, as in [23].\n\n[Miscellaneous] I think line 96's citation [22] should be changed to [23].\n\nQUESTIONS:\n[Q1] Does the concept decomposition differ between correct and wrong samples (from the perspective of zero-shot classification) ?\n\n[Q2] As the effectiveness of SpLiCE heavily relies on the quality and comprehensiveness of the concept dictionary derived from the LAION dataset, is there a possibility that limitations or biases in this dictionary could directly impact the performance and interpretability of the method?\n\nLIMITATIONS:\nPlease refer to the weaknesses and questions part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to decompose the representations of the CLIP model using dictionary learning; where the components of the dictionary are composed of human understandable concept directions. The procedure here is as follows: a concept list is constructed from a filtered set of unigrams and bigrams from the LAION-400m captions, the concept dictionary is set to be centered CLIP representations for each pre-specified concept in the list, and an optimization problem that minimizes the reconstruction loss (subject to an L-1 penalty) is specified. Solving that optimization problem results in a weight vector that indicates how the CLIP embedding can be decomposed along the pre-specified concept direction. Given this formulation, the paper solves the problem for the CLIP model and shows that it does not lead to a performance loss given thousands of concepts. In addition, the paper presents several case studies showing how the approach can be used to edit downsteam classifiers built on top of clip representations, and discover spurious correlations.\n\nSTRENGTHS:\n**Originality**\\\nThis paper is the first that I know to decompose the representations of CLIP with known concepts using dictionary learning. The work is well motivated. \n\n**Quality and Clarity**\\\nThe paper is well-written, and clear. The related work is also discussed, and each component of the workflow presented here is well-explained and clear. \n\n**Significance**\\\nThis work, in my opinion, indicates that CLIP type models can be easily debugged using post hoc supervised concept learning. Overall, I think this opens up the door to more intriguing work along similar lines for larger and more complicated models.\n\nWEAKNESSES:\nOverall, I discuss some of the challenges that I see with this work. I have not much substantive qualms with this work; I just think the presentation could be tightened. However this is up to the authors to either accept or reject.\n\n**Assumptions for the proposition not justified**:  While the assumptions made to prove proposition 1 seem natural at first glance, it is not clear to me that we can claim that they should be true. One, there is a claim that CLIP captures 'semantic' concepts and not 'non-semantic' concepts (assumption 2). In Section 5.2, the authors even confirm experimentally that the assumption is not true. Of course, the model does capture some semantic concepts, but we cannot claim that CLIP cleanly partitions these concepts as described. This is because we know of failure cases where CLIP errs precisely because it does not just capture only semantic concepts. See this paper for example (https://arxiv.org/abs/2306.12105). The failures of the CLIP model that have been observed make it clear that this assumption is not true.  Assumption 3 is somewhat problematic again because we have observed that CLIP representations indeed can be linear for *some* concepts, but it is not linear for all semantic concepts. In fact, how are we to know which concepts that CLIP considers to be semantic, and which it doesn't? Assumption 4 makes sense to me because CLIP is trained to 'align' both encoders. Assumption 5 also difficult to justify in my opinion, but I am willing to live with it. To summarize my challenge with this portion of the work, the proposition is correct, but we can't claim that CLIP behaves as described. I am bothered by these assumptions because the implication is essentially that CLIPs learns the data generating process defined here, which is not the case. This is actually a good thing because, as shown quite nicely in this work, the point of the concept dictionary 'layer' is to 'fix' the challenges with the original CLIP model.  Overall, I don't think this section of the paper can be fully justified.\n\n**task-agnostic and without concept datasets, training, or qualitative analysis of visualizations.**:  The bolded phrase is one of the motivations for the approach described here. This is not quite true. First, you need to collect the concept dataset, the processing done with the LAION dataset is *exactly* the process of collecting a concept dataset. You do need to 'train', in the sense that you can see the process of optimizing to obtain $w$ as training. Lastly, the layer results in concept scores that can be visualized as depicted in Figure 4, so the approach here also allows us to inspect . Overall, I think this claim is hyperbolic and should be relaxed. \n\n**Related Work and Discussion**: This work claims that it is not quite a concept bottleneck, but I disagree. It is exactly translating a concept bottleneck to CLIP. First, the methods does require annotations. To get the dictionary $C$, you need to collect a set of 10000 concepts, which are associated with particular images. The annotation process is exactly the process of manually constructing $C$. The concept list is supervised, meaning it is the authors that determine which concepts to include; of course, the process described here is not manual, but the authors should recognize that it is *still* a supervised process. The intervention procedure described in Section 6 is exactly how that is done in the CBM literature. In fact, the NMF formulation here is not new (see: CRAFT: Concept Recursive Activation FacTorization for Explainability for eg.). Overall, the approach described here is exactly what a CBM is, it is just not applied for classification. Having said all of this, I think it is fine to acknowledge the similarity.\n\nQUESTIONS:\n**Unsupervised or residual concepts**: I am surprised that there wasn't any significant loss in performance between the Splice representations and the original CLIP representations. Is it that 15000 concepts is enough to capture the variation? It seemed like one would want to account for the 'residual' directions in the original embedding that the reconstruction doesn't account for. Is this needed? Or do the authors think we can always get away with this kind of supervision?\n\nLIMITATIONS:\nAppendix A discusses some of the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-31"}