Title: A Concept-Based Explainability Framework for Large Multimodal Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large multimodal models

interpretability

concept extraction

multimodal concepts

multimodal representations

vision-language models

vision-language interpretations
Abstract: Large multimodal models (LMMs) combine unimodal encoders and large language models (LLMs) to perform multimodal tasks. Despite recent advancements towards the interpretability of these models, understanding internal representations of LMMs remains largely a mystery. In this paper, we present a novel framework for the interpretation of LMMs. We propose a dictionary learning based approach, applied to the representation of tokens. The elements of the learned dictionary correspond to our proposed concepts. We show that these concepts are well semantically grounded in both vision and text. Thus we refer to these as ``multi-modal concepts''. 
We qualitatively and quantitatively evaluate the results of the learnt concepts. We show that the extracted multimodal concepts are useful to interpret representations of test samples. Finally, we evaluate the disentanglement between different concepts and the quality of grounding concepts visually and textually. Our implementation is publicly available: https://github.com/mshukor/xl-vlms.

A Concept-Based Explainability Framework for Large Multimodal Models

Abstract

Large multimodal models (LMMs) combine unimodal encoders and large language models (LLMs) to perform multimodal tasks. Despite recent advancements towards the interpretability of these models, understanding internal representations of LMMs remains largely a mystery. In this paper, we present a novel framework for the interpretation of LMMs. We propose a dictionary learning based approach, applied to the representation of tokens. The elements of the learned dictionary correspond to our proposed concepts. We show that these concepts are well semantically grounded in both vision and text. Thus we refer to these as 'multimodal concepts'. We qualitatively and quantitatively evaluate the results of the learnt concepts. We show that the extracted multimodal concepts are useful to interpret representations of test samples. Finally, we evaluate the disentanglement between different concepts and the quality of grounding concepts visually and textually. Our implementation is publicly available. 1

1 Introduction

Despite the exceptional capacity of deep neural networks (DNNs) to address complex learning problems, one aspect that hinders their deployment is the lack of human-comprehensible understanding of their internal computations. This directly calls into question their reliability and trustworthiness [5, 30]. Consequently, this has boosted research efforts in interpretability/explainability of these models i.e. devising methods to gain human-understandable insights about their decision processes. The growth in ability of DNNs has been accompanied by a similar increase in their design complexity and computational intensiveness. This is epitomized by the rise of vision transformers [11] and large-language models (LLMs) [8, 44] which can deploy up to tens of billions of parameters. The effectiveness of these models for unimodal processing tasks has spurred their use in addressing multimodal tasks. In particular, visual encoders and LLMs are frequently combined to address tasks such as image captioning and VQA [2, 25, 29, 43, 45]. This recent class of models are referred to as large multimodal models (LMMs).

For interpretability research, LMMs have largely remained unexplored. Most prior works on interpreting models that process visual data, focus on convolutional neural network (CNN) based systems and classification as the underlying task. Multimodal tasks and transformer-based architectures have both been relatively less studied. LMMs operate at the intersection of both domains. Thus, despite their rapidly growing popularity, there have been very few prior attempts at understanding representations inside an LMM [34, 35, 41, 42].

This paper aims to bridge some of these differences and study in greater detail the intermediate representations of LMMs. To this end, motivated by the concept activation vector (CAV) based approaches for CNNs [14, 15, 17, 22], we propose a novel dictionary-learning based Concept eXplainability method designed for application to LMMs, titled CoX-LMM . Our method is used to learn a concept dictionary to understand the representations of a pretrained LMM for a given word/token of interest (Eg. 'Dog'). For this token, we build a matrix containing the LMM's internal representation of the token. We then linearly decompose this matrix using dictionary learning. The dictionary elements of our decomposition represent our concepts. The most interesting consequence of our method is that the learnt concepts exhibit a semantic structure that can be meaningfully grounded in both visual and textual domains. They are visually grounded by extracting the images which maximally activate these concepts. They can simultaneously be grounded in the textual domain by decoding the concept through the language model of the LMM and extracting the words/tokens they are most associated to. We refer to such concept representations as multimodal concepts . Our key contributions can be summarized as follows:

We propose a novel concept-based explainability framework CoX-LMM , that can be used to understand internal representations of large multimodal models. To the best of our knowledge, this is the first effort targeting multimodal models at this scale.

Our dictionary learning based concept extraction approach is used to extract a multimodal concept dictionary wherein each concept can be semantically grounded simultaneously in both text and vision. We also extend the previous concept dictionary-learning strategies using a Semi-NMF based optimization.

We experimentally validate the notion of multimodal concepts through both, qualitative visualizations and quantitative evaluation. Our learnt concept dictionary is shown to possess a meaningful multimodal grounding covering diverse concepts, and is useful to locally interpret representations of test samples LMMs.

2 Related work

Large Multimodal Models (LMMs) Large language models (LLMs) [8, 21, 33, 44] have emerged as the cornerstone of contemporary multimodal models. Typical large multimodal models (LMMs) [1, 4, 25, 26] comprise three components: LLMs, visual encoders, and light-weight connector modules to glue the two models. Remarkably, recent works have demonstrated that by keeping all pretrained models frozen and training only a few million parameters in the connector (e.g., a linear layer), LLMs can be adapted to understand images, videos, and audios [12, 23, 31, 43, 45], thus paving the way for solving multi-modal tasks. However, there is still a lack of effort aimed at understanding why such frozen LLMs can generalize to multimodal inputs. In this study, we try to decode the internal representation of LLMs when exposed to multimodal inputs.

Concept activation vector based approaches Concept based interpretability aim to extract the semantic content relevant for a model [9]. For post-hoc interpretation of pretrained models, concept activation vector (CAV) based approaches [15, 17, 22, 46-48] have been most widely used. The idea of CAV was first proposed by Kim et al. [22]. They define a concept as a set of user-specified examples. The concept is represented in the activation space of deep layer of a CNN by a hyperplane that separates these examples from a set of random examples. This direction in the activation space is referred to as the concept activation vector. Built upon CA V , ACE [17] automate the concept extraction process. CRAFT [15] proposed to learn a set of concepts for a class by decomposing activations of image crops via non-negative matrix factorization (NMF). Recently, Fel et al. [14] proposed a unified view of CAV-based approaches as variants of a dictionary learning problem. However, these methods have only been applied for interpretation of CNNs on classification tasks. LMMs on the contrary exhibit a different architecture. We propose a dictionary learning based concept extraction method, designed for LMMs. We also propose a Semi-NMF variant of the dictionary learning problem, which has not been previously considered for concept extraction.

Understanding VLM/LMM representations There has been an increasing interest in understanding internal representations of visual-language models (VLM) through the lens of multimodality. Shukor and Cord [42] analyse multimodal tokens and shows that despite being different, visual and perceptual tokens are implicitly aligned inside LLMs. Goh et al. [18] discover neurons termed

LLM

Dictionary learning for concept

extraction in LMMs

FFN

SA

FFN

SA

FFN

SA

Unembed

Z

U

V

dog

Unembed

= {    'black', 'large', 'dark'   }

= {

}

Representation

matrix

Concept

dictionary

Concept

activations

The

is

Image

LMM

Most

activating

Multimodal concept grounding

multimodal , that activate for certain conceptual information given images as input. Recently proposed TEXTSPAN [16] and SpLiCE [7], aim to understand representations in CLIP [38] by decomposing its visual representations on textual representations. For LMMs, Palit et al. [34] extend the causal tracing used for LLMs to analyze information across different layers in an LMM. Schwettmann et al. [41] first proposed the notion of multimodal neurons existing within the LLM part of an LMM. They term the neurons 'multimodal' as they translate high-level visual information to corresponding information in text modality. The neurons are discovered by ranking them by a gradient based attribution score. Pan et al. [35] proposed a more refined algorithm to identify such neurons based on a different neuron importance measure that leverages architectural information of transformer MLP blocks. Instead, we propose to discover a concept structure in the token representations by learning a small dictionary of multimodally grounded concepts. Limiting the analysis to a specific token of interest allows our method to discover fine details about the token in the learnt concepts.

3 Approach

3.1 Background for Large Multimodal Models (LMMs)

Model architecture. We consider a general model architecture for a large multimodal model f , that consists of: a visual encoder f V , a trainable connector C , and an LLM f LM consisting of N L layers. We assume f is pretrained for captioning task with an underlying dataset S = { ( X i , y i ) } N i =1 consisting of images X i ∈ X and their associated caption y i ⊂ Y . X and Y denote the space of images and set of text tokens respectively. Note that caption y i can be viewed as a subset of all tokens. The input to the language model f LM is denoted by the sequence of tokens h 1 , h 2 , ..., h p and the output as ˆ y . The internal representation of any token at some layer l and position p inside f LM is denoted as h p ( l ) , with h p (0) = h p . Note that h p ( l ) is same as the residual stream representation in LLM transformers [13] at position p and layer l . For the multimodal model, the input sequence of tokens for f LM consists of the concatenation of: (1) N V visual tokens provided by the visual encoder f V operating on an image X , followed by the connector C , and (2) linearly embedded textual tokens previously predicted by f LM . For p > N V , this can be expressed as:

where h 1 , . . . , h N V = C ( f V ( X )) , and h p = Emb (ˆ y p -1 ) for p > N V , where Emb denotes the token embedding function. To start the prediction, h N V +1 is defined as the beginning of sentence token. The output token ˆ y p is obtained by normalizing h p ( N L ) , followed by an unembedding layer that applies a matrix W U followed by a softmax. The predicted caption ˆ y consists of the predicted tokens ˆ y = { ˆ y p } p>N V until the end of sentence token.

Training The model is trained with next token prediction objective, to generate text conditioned on images in an auto-regressive fashion. In this work we focus on models trained to "translate" images into text, or image captioning models. These models keep the visual encoder f V frozen and only train the connector C . Recent models also finetune the LLM to improve performance. Our approach can be applied in either case, and in our experiments we consider both type of LMMs. However, we find the generalization of LLMs to multimodal inputs is an interesting phenomenon to understand, thus we focus more on the setup where the LLM is kept frozen.

Transformer representations view Central to many previous approaches interpreting decoderonly LLM/transformer architectures, is the 'residual stream view' of internal representations, first proposed in [13]. Herein, the network is seen as a composition of various computational blocks that 'read' information from the residual stream of token representations h p ( l ) , perform their computation, and add or 'write' their output back in the residual stream. This view can be summarized as:.

a p ( l ) denotes the information computed by attention function at layer l and position p . It has a causal structure and computes its output using h 1 ( l ) , ..., h p ( l ) . m p ( l ) denotes the information computed by the MLP block. It is a feedforward network (FFN) with two fully-connected layers and an intermediate activation function σ , that operates on h p ( l ) + a p ( l ) . The output of σ ( . ) is referred to as FFN activations.

3.2 Method overview

Fig. 1 provides a visual summary of the whole CoX-LMM pipeline. Given a pretrained LMM f and a token of interest t ∈ Y , our method consists of three key parts:

Selecting a subset of images X from dataset S , relevant for target token t . We extract representations by processing samples in X through f . The extracted representations of dimension B are collected in a matrix Z ∈ R B × M , where M is number of samples in X .

Linearly decomposing Z ≈ UV into its constituents, that includes a dictionary of learnt concepts U ∈ R B × K of size K and coefficient/activation matrix V ∈ R K × M .

Semantically grounding the learnt 'multimodal concepts', contained in dictionary U in both visual and textual modalities.

We emphasize at this point that our main objective in employing dictionary learning based concept extraction is to understand internal representations of an LMM. Thus, our focus is on validating the use of the learnt dictionary for this goal, and not to interpret the output of the model, which can be readily accomplished by combining this pipeline with some concept importance estimation method [14]. The rest of the section is devoted to elaborate on each of the above three steps.

3.3 Representation extraction

To extract relevant representations from the LMM about t that encode meaningful semantic information, we first determine a set of samples X from dataset S = { ( X i , y i ) } N i =1 for extraction. We consider the set of samples where t is predicted as part of the predicted caption ˆ y . This allows us to further investigate the model's internal representations of t . To enhance visual interpretability for the extracted concept dictionary, we additionally limit this set of samples to those that contain t in the ground-truth caption. Thus, X is computed as:

Given any X ∈ X , we propose to extract the residual stream representation h p ( L ) from a deep layer L , at the first position in the predicted caption p > N V , such that ˆ y p = t . The representation z j ∈ R B

of each sample X j ∈ X is then stacked as columns of the matrix Z = [ z 1 , ..., z M ] ∈ R B × M . Note that the representations of text tokens in f LM can possess a meaningful multimodal structure as they combine information from visual token representations h p ( l ) , p ≤ N V . In contrast to a p ( l ) and m p ( l ) , that represent residual information at layer l , h p ( L ) contains the aggregated information computed by the LMM till layer L , providing a holistic view of its computation across all previous layers.

3.4 Decomposing the representations

The representation matrix Z ≈ UV , is decomposed as product of two low-rank matrices U ∈ R B × K , V ∈ R K × M of rank K << min( B,M ) , where K denotes the number of dictionary elements. The columns of U = [ u 1 , ..., u K ] are the basis vectors which we refer to as conceptvectors/concepts. The rows of V or columns of V T = [ v 1 , ..., v K ] , v i ∈ R M denote the activations of u i for each sample. This decomposition, as previously studied in [14], can be optimized with various constraints on U , V , each leading to a different dictionary. The most common ones include PCA (constraint: U T U = I ), K-Means (constraint: columns of V correspond to columns of identity matrix) and NMF (constraint: U , V ≥ 0 ). NMF is considered to yield most interpretable results [14]. However, for our use case, NMF is not useful as representation matrix Z is not non-negative. Instead, we propose to employ a relaxed version of NMF, Semi-NMF [10], which allows the decomposition matrix Z and basis vectors U to contain mixed values, and only forces the activations V to be non-negative. Note that given its relations to clustering algorithms [10], enforcing non-negative combinations of decompositions is still valued from an interpretability perspective. Since we expect only a small number of concepts to be present in any given sample, we also encourage sparsity in activations V . The optimization problem to decompose Z via Semi-NMF can be summarized as:

Given any image X where token t is predicted by f , we can now define the process of computing activations of concept dictionary U ∗ for given X , denoted as v ( X ) ∈ R K . To do so, we first extract the token representation for X,z X ∈ R B with the process described in Sec. 3.3. Then, z X is projected on U ∗ to compute v ( X ) . In the case of Semi-NMF, this corresponds to v ( X ) = arg min v ≥ 0 || z X -U ∗ v || 2 2 + λ || v || 1 . The activation of u k ∈ U ∗ is denoted as v k ( X ) ∈ R .

3.5 Using the concept dictionary for interpretation

Multimodal grounding of concepts. Given the learnt dictionary U ∗ and corresponding activations V ∗ , the key objective remaining is to ground the understanding of any given concept vector u k , k ∈ { 1 , ..., K } in the visual and textual domains. Specifically, for visual grounding, we use prototyping [3, 22] to select input images (among the decomposed samples), that maximally activate u k . Given the number of samples extracted for visualization N MAS , the set of maximum activating samples (MAS) for component u k , denoted as X k,MAS can be specified as follows ( | . | is absolute value):

For grounding in textual domain, we note that the concept vectors are defined in the token representation space of f LM . Thus we leverage the insights from 'Lens' based methods [6, 24, 32, 40] that attempt to understand LLM representations. In particular, following [32], we use the unembedding layer to map u k to the token vocabulary space Y , and extract the most probable tokens. That is, we extract the tokens with highest probability in W U u k . The decoded tokens with highest probabilities are then filtered for being an english, non-stop-word with at least 3 characters, to eliminate unnecessary tokens. The final set of words is referred to as grounded words for concept u k and denoted as T k . Fig. 2 illustrates an example of grounding of a concept extracted for token 'Dog' in vision (5 most activating samples) and text (top 5 decoded words).

Most activating concepts for images. To understand the LMM's representation of a given image X , we now define the most activating concepts . Firstly, we extract the representaions z X of the image with the same process as described previously. We then project z X on U ∗ to obtain v ( X ) ∈ R K . We define the most activating concepts, which we denote ˜ u ( X ) , as the set of r concept vectors (in

U ∗ ) whose activations v k ( X ) have the largest magnitude. One can then visualize the multimodal grounding of ˜ u ( X ) . This step could be further combined with concept importance estimation techniques [14] to interpret the model's prediction for token t , however, the focus of this paper is to simply understand the internal representation of the model, for which the current pipeline suffices.

4 Experiments

Models and dictionary learning. In the main paper, we focus on experiments with the DePALM model [45] that is trained for captioning task on COCO dataset [27]. The model consists of a frozen ViT-L/14 CLIP [39] encoder as the visual encoder f V . It is followed by a transformer connector to compress the encoding into N V = 10 visual tokens. The language model f LM is a frozen OPT-6.7B [49] and consists of 32 layers. Additional experiments with LLaVA [28] are in Appendix A. For uniformity and fairness all the results in the main paper are reported with number of concepts K = 20 and for token representations from L = 31 , the final layer before unembedding layer. For Semi-NMF, we set λ = 1 throughout. We consider the 5 most activating samples in X k,MAS for visual grounding for any u k . For text grounding, we consider top-15 tokens for T k before applying the filtering described in Sec 3.5.

The complete dataset consists of around 120,000 images for training, and 5000 each for validation and testing with 5 captions per image following the Karpathy split. We conduct our analysis separately for various common objects in the dataset: 'Dog', 'Bus', 'Train', 'Cat', 'Bear', 'Baby', 'Car', 'Cake'. The extension to other classes/tokens remains straightforward and is discussed in Appendix D. The precise details about number of samples for learning the dictionary, or testing, is available in Appendix C. The implementation of our method is publicly available on GitHub 2

4.1 Evaluation setup

We evaluate the quality of learnt concept dictionary U ∗ on three axes: (i) Its use during inference to interpret representations of LMMs for test samples, (ii) The overlap/entanglement between grounded words of concepts in the dictionary and (iii) the quality of visual and text grounding of concepts (used to understand a concept itself). We discuss concrete details about each axis below:

Concept extraction during inference: To evaluate the use of U ∗ in understanding any test sample X , we first estimate the top r most activating concepts activations, ˜ u ( X ) (Sec. 3.5). We then estimate the correspondence between the test image X and the grounded words T k of u k ∈ ˜ u ( X ) . This correspondence is estimated via two different metrics. The primary metric is the average CLIPScore [20] between X and T k . This directly estimates correspondence between the test image embedding with the grounded words of the top concepts. The secondary metric is the average BERTScore (F1) [50] between the ground-truth captions y associated with X and the words T k . These metrics help validate the multimodal nature of the concept dictionaries. Their use is inspired from [41]. Details for their implementation is in Appendix C.

̸

Overlap/entanglement of learnt concepts: Ideally, we would like each concept in U ∗ to encode distinct information about the token of interest t . Thus two different concepts u k , u l , k = l should be associated to different sets of words. To quantify the entanglement of learnt concepts, we compute the overlap between the grounded words T k , T l . The overlap for a concept u k is defined as an average of its fraction of common words with other concepts. The overlap/entanglement metric for a dictionary

Table 1: Test data mean CLIPScore and BERTScore for top-1 activating concept for all baselines on five tokens. CLIPScore denoted as CS, and BERTScore as BS. Statistical significance is in Appendix D. Our CoX-LMM framework is evaluated with Semi-NMF as underlying dictionary learning method. Higher scores are better. Best score in bold , second best is underlined.

U ∗ is defined as the average of overlap of each concept.

̸

Multimodal grounding of concepts: To evaluate the quality of visual/text grounding of concepts ( X k,MAS , T k ), we measure the correspondence between visual and text grounding of a given concept u k , i.e. the set of maximum activating samples X k,MAS and words T k , using CLIPScore and BERTScore as described above.

Baselines: One set of methods for evaluation are the variants of CoX-LMM where we employ different dictionary learning strategies: PCA, KMeans and Semi-NMF. For evaluating concept extraction on test data with CLIPScore/BERTScore we compare against the following baselines:

-Rnd-Words : This baseline considers Semi-NMF as the underlying learning method. However, for each component u k , we replace its grounded words T k by a set of random words R k such that | R k | = | T k | and the random words also satisfy the same filtering conditions as grounded words i.e. they are non-stopwords from english corpus with more than two characters. We do this by decoding a randomly sampled token representation and adding the top decoded words if they satisfy the conditions.

-Noise-Imgs : This baseline uses random noise as images and then proceeds with exactly same learning procedure as Semi-NMF including extracting activations from the same positions, and same parameters for dictionary learning. Combined with the Rnd-Words baseline, they ablate two parts of the concept extraction pipeline.

-Simple : This baseline considers a simple technique to build the dictionary U ∗ and projecting test samples. It builds U ∗ by selecting token representations in Z with the largest norm. The projections are performed by mapping the test sample representation to the closest element in U ∗ . For deeper layers, this provides a strong baseline in terms of extracted grounded words T k which are related to token of interest t , as they are obtained by directly decoding token representations of t .

We also report score using ground-truth captions (GT captions) instead of grounded words T k , to get the best possible correspondence score. The overlap/entanglement in concept dictionary is compared between the non-random baselines: Simple, PCA, K-Means and Semi-NMF. For evaluating the visual/text grounding we compare against the random words keeping the underlying set of MAS, X k,MAS , same for both.

4.2 Results and discussion

Quantitative results Tab. 1 reports the test top-1 CLIPScore/BERTScore for all baselines and Semi-NMF on different target tokens. Appendix D contains detailed results for other tokens as well as for the PCA and K-Means variants. We report the results for only the top1 activating concept, as the KMeans and Simple baselines map a given representation to a single cluster/element.

Notably, Semi-NMF generally outperforms the other baselines although the Simple baseline performs competitively. More generally, Semi-NMF, K-Means, and Simple tend to clearly outperform RndWords, Noise-Imgs and PCA on these metrics, indicating that these systems project representations of test images to concepts whose associated grounded words correspond well with the visual content.

Tab. 2 reports the overlap between concepts for Simple baseline and PCA, K-Means and Semi-NMF variants of CoXLMM . Interestingly, K-Means and Simple baseline perform significantly worse than Semi-NMF/PCA with a high overlap between grounded words, often exceeding 40%. PCA outperforms other methods with almost no overlap while Semi-NMF shows some overlap. Overall, Semi-NMF strikes the best balance among all the methods, in terms of learning a concept dictionary useful for understanding test image representations, but which also learns diverse and disentangled concepts. Thus, for further CoX-LMM experiments, we consider Semi-NMF as the underlying dictionary learning method.

Fig. 3 shows an evaluation of visual/text grounding of concepts learnt by Semi-NMF. Each point on the figure denotes the CLIPScore (left) or BERTScore (right) for correspondence between samples X k,MAS and words T k for concept u k against random words baseline. We see that for both metrics, vast majority of concepts lie above the y = x line, indicating that grounded words correspond much better to content of maximum activating samples than random words.

CLIPScore

BERT-F1 Score

0.6

0.7

words

words

0.5

0.6

Grounded

6op

Grounded

0.4

6op

cat

cat

0.5

train

0.3

train

car

e

0.4

bus

baby

cake

0.2

baby

cake

snq

bear

0.1

bear

0.4

0.5

0.6

0.7

0.1

0.2

0.3

0.4

0.5

0.6

Random words

Random words

Qualitative results Fig. 4 shows visual and textual grounding of concepts extracted for token 'dog'. For brevity, we select 8 out of 20 concepts for illustration. 2. Grounding for all concepts extracted for 'dog' and other tokens are in Appendix E. The concept visualizations/grounding for

Example concepts for 'Dog'

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

,6op,

'small'

'sausage

'tiny'

'hot'

'plate'

'cute'

'black'

'"furry'

"large'

'hairy'

'dark'

'big'

,buol,

'close'

'fuzzy'

'brown'

'cat'

"large'

'kitten'

,6op,

'tiger'

'tan'

'rabbit'

'golden

,6op,

,6op,

'herd'

'running

'sheep'

'black'

'flock'

'play'

'farm'

'grass'

RIGHT

'shepherd"

'Dog' reveal interesting insights about the global structure of the LMM's representation. Extracted concepts capture information about different aspects of a 'dog'. The LMM separates representation

of animal 'Dog' with a 'hot dog' (Concept 1). Specifically for 'Dog', Concepts (2), (3) capture information about color: 'black', 'brown'. Concept (6) encodes information about 'long hair' of a dog, while concept (5) activates for 'small/puppy-like' dogs. Beyond concepts activating for specific characteristics of a 'dog', we also discover concepts describing their state of actions (Concept (4) 'playing/running'), common scenes they can occur in (Concept (8), 'herd'), and correlated objects they can occur with (Concept (7), 'cat and dog'). We observe such diverse nature of extracted concepts even for other tokens (Appendix E). The information about concepts can be inferred via both the visual images and the associated grounded words, highlighting their coherent multimodal grounding. Notably, compared to solely visual grounding as for CAVs for CNNs, the multimodal grounding eases the process to understand a concept.

Test sample

Top-3concept activations

1.0

0.5

0.0

Cpt 1

Cpt 2

Cpt 3

"furry"

"hairy'

"long'

'fuzzy"

'small'

"tiny”

,Addnd,

'miniature

'cute'

,6op,

'black'

'white'

‘small

"border'

Test sample

Top-3 concept activations

1.0

0.5

0.0

Cpt 1

Cpt 2

Cpt 3

'computer

xsap,

“keyboard

'cat'

'white'

'black'

'cat'

“fat’

'cal'

fluffy'

"long'

furry'

"hairy'

"large'

Test sample

Top-3 concept activations

1.0

0.5

0.0

Cpt 1

Cpt 2

Cpt 3

,snq,

'blue'

'stopped

'large'

,lqnd,

'colorful

‘Thai'

'Asian'

'brightly

'Mexican

'white'

'city'

,snq,

'commutel

'transit'

Fig. 5 illustrates the use of concept dictionaries (learnt via Semi-NMF) to understand test sample representations for tokens 'Dog', 'Cat' and 'Bus'. For each sample we show the normalized activations of the three most activating concepts, and their respective multimodal grounding. Most activating concepts often capture meaningful and diverse features of a given sample. For instance, for first sample containing a 'Dog', the concepts for 'long hair', 'small/tiny/puppy', and 'black/white color' all simultaneously activate. The grounding for first two concepts was also illustrated in Fig. 4. Additional visualizations for test samples are shown in Appendix E, wherein we qualitatively compare interpretations of Semi-NMF to K-Means, PCA variants and Simple baseline.

Layer ablation We analyze the quality of multimodal grounding of concepts across different layers L . The CLIPScore between ( X k,MAS , T k ) , averaged over all concepts u k is shown in Fig. 6, for 'Dog' and 'Cat' for all layers in f LM . For early layers the multimodal grounding is no better than Rnd-Words. Interestingly, there is a noticeable increase around ( L = 20 to L = 25 ), indicating that the multimodal structure of internal token representations starts to appear at this point. This also validates our choice that deeper layers are better suited for multimodal concepts.

AverageCLiPScoreformultimodalgrounding

0.650

cat

0.625

6op

random

0.600

CLIPScore

0.575

0.550

0.525

0.500

0.475

0

5

10

15

20

25

30

Layer L

Additional experiments and discussion. We conduct a preliminary study to analyze the polysemanticity/superposition in concept dictionaries in Appendix B. A qualitative analysis for grounding of extracted concepts for different layers is available in Appendix F. CoX-LMM can be also be applied to understand the processing of visual/perceptual tokens inside the LMM which also exhibit this multimodal structure. The experiment for the same can be found in Appendix G. Limitations of our method are discussed in Appendix H, and the broader societal impacts are discussed in Appendix I.

5 Conclusion

In summary, we have presented a novel dictionary learning based concept extraction framework, useful to understand internal representations of a large multimodal model. The approach relies on decomposing representations of a token inside a pretrained LMM. To this end, we also propose a Semi-NMF variant of the concept dictionary learning problem. The elements of the learnt concept dictionary are grounded in the both text and visual domains, leading to a novel notion of multimodal concepts in the context of interpretability. We quantitatively and qualitatively show that (i) the multimodal grounding of concepts is meaningful, and (ii) the learnt concepts are useful to understand representations of test samples. We hope that our method inspires future work from research community towards designing concept based explainability methods to understand LMMs.

Acknowledgments and Disclosure of Funding

We would like to thank Thomas Fel for his valuable comments on the paper. This work has been partially supported by ANR grant VISA DEEP (ANR-20-CHIA-0022), and HPC resources of IDRIS under the file number AD011014947, allocated by GENCI.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS) , 35:23716-23736, 2022.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35: 23716-23736, 2022.

David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , pages 7775-7784, 2018.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlamingo: An opensource framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 , 2023.

Valérie Beaudouin, Isabelle Bloch, David Bounie, Stéphan Clémençon, Florence d'AlchéBuc, James Eagan, Winston Maxwell, Pavlo Mozharovskyi, and Jayneel Parekh. Flexible and context-specific AI explainability: A multidisciplinary approach. CoRR , abs/2003.07703, 2020.

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112 , 2023.

Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). arXiv preprint arXiv:2402.10376 , 2024.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS) , 33: 1877-1901, 2020.

Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. Advances in neural information processing systems , 35:2832-2845, 2022.

Chris HQ Ding, Tao Li, and Michael I Jordan. Convex and semi-nonnegative matrix factorizations. IEEE transactions on pattern analysis and machine intelligence , 32(1):45-55, 2008.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. Magma-multimodal augmentation of generative models through adapter-based finetuning. arXiv preprint arXiv:2112.05253 , 2021.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread , 1, 2021.

Thomas Fel, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems , 36, 2023.

Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2711-2721, 2023.

Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. In International Conference on Learning Representations , 2024.

Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic conceptbased explanations. In Advances in Neural Information Processing Systems (NeurIPS) , pages 9277-9286, 2019.

Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill , 6(3): e30, 2021.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pages 2668-2677. PMLR, 2018.

Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.13823 , 2023.

Anna Langedijk, Hosein Mohebbi, Gabriele Sarti, Willem H. Zuidema, and Jaap Jumelet. Decoderlens: Layerwise interpretation of encoder-decoder transformers. CoRR , abs/2310.03686, 2023.

Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. URL https://openreview.net/forum?id=SKN2hflBIZ .

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV) , pages 740-755. Springer, 2014.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2023.

Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. Mapl: Parameter-efficient adaptation of unimodal pre-trained models for visionlanguage few-shot prompting. arXiv preprint arXiv:2210.07179 , 2022.

Aniek F Markus, Jan A Kors, and Peter R Rijnbeek. The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies. Journal of biomedical informatics , 113:103655, 2021.

Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162 , 2022.

Nostalgebraist. Interpreting gpt: The logit lens. https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens , 2020. Accessed: [date of access].

OpenAI. Gpt-4 technical report. arXiv , 2023.

Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision-language mechanistic interpretability: A causal tracing tool for blip. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2856-2861, 2023.

Haowen Pan, Yixin Cao, Xiaozhi Wang, and Xun Yang. Finding and editing multi-modal neurons in pre-trained transformer. arXiv preprint arXiv:2311.07470 , 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32:8026-8037, 2019.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikitlearn: Machine learning in python. the Journal of machine Learning research , 12:2825-2830, 2011.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , pages 8748-8763. PMLR, 2021.

Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian T. Foster. Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism. CoRR , abs/2310.16270, 2023.

Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2862-2867, 2023.

Mustafa Shukor and Matthieu Cord. Implicit multimodal alignment: On the generalization of frozen llms to multimodal inputs. arXiv preprint arXiv:2405.16700 , 2024.

Mustafa Shukor, Corentin Dancette, and Matthieu Cord. ep-alm: Efficient perceptual augmentation of language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 22056-22069, October 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Théophane Vallaeys, Mustafa Shukor, Matthieu Cord, and Jakob Verbeek. Improved baselines for data-efficient perceptual augmentation of llms. arXiv preprint arXiv:2403.13499 , 2024.

Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimensional concept discovery (mcd): A unifying framework with completeness guarantees. Transactions on Machine Learning Research , 2023.

Chih-Kuan Yeh, Been Kim, Sercan O Arik, Chun-Liang Li, Pradeep Ravikumar, and Tomas Pfister. On concept-based explanations in deep neural networks. arXiv preprint arXiv:1910.07969 , 2019.

Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept-based explanations for cnn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11682-11690, 2021.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations , 2019.

A Experiments on other LMMs

This section covers our experiments on other types of multimodal models. First, we test our approach on LLaVA to demonstrate that our approach generalizes to more recent networks that also fine-tune f LM on multimodal data. We also test our method on other variants of DePALM with non-CLIP visual encoders to observe their effect on CLIPScore.

A.1 Experiments with LLaVA

We conduct further experiments on LLaVA [28], a popular open-source LMM to demonstrate the generality of our method. The model uses a CLIP-ViT-L-336px visual encoder ( f V ), a 2-layer linear connector ( C ) that outputs N V = 576 visual tokens, and a Vicuna-7B language model ( f LM , 32 layers). We use identical hyperparameters as for DePALM ( K = 20 , λ = 1 , L = 31 ). We report the test CLIPScore for top-1 activating concept, for Rnd-Words, Noise-Imgs, Simple and Semi-NMF, GT-captions in Tab. 3, and Overlap score for non-random baselines in Tab. 4. Quantitatively, we obtain consistent results to those observed for DePALM. Semi-NMF extracts most balanced concept dictionaries with high multimodal correspondence and low overlap. Qualitatively too, the method functions consistently and is able to extract concepts with meaningful multimodal grounding.

Qualitative results and Saliency maps We also show qualitative examples of concepts extracted for token 'Dog' in Fig. 7. More examples for other 'Cat' and 'Train' tokens are given in Fig. 9 and 10. Interestingly, since LLaV A uses a connector C that contains two linear layers, the visual tokens as processed inside f LM preserve the notion of specific image patch representations, i.e. N V = 576 tokens denoting representations for 576 (24 × 24) input patches. This allows us to further explore a simple and computationally cheap strategy of generating saliency maps to highlight which regions a concept vector activates on. To do this one can simply compute the inner product of any given concept vector u k with all visual token representations from corresponding layer L , i.e. u T k [ h 1 ( L ) , ..., h N V ( L ) ] . This can be upscaled to the input image size to visualize the saliency map. We illustrate some qualitative outputs on concepts from 'Dog' in Fig. 8. .

'small'

'brown'

'hair'

'golden'

,un,

'red'

,6op,

'tan'

'close'

'orange'

'black'

,6op,

'German

'border'

"'large'

'animal'

'pit'

'col'

'big'

,sbop,

'golden'

'white'

'yellow'

'small'

'white'

,ueds,

'wet'

'dal'

"light'

,6op,

'black'

,6op.

'flying'

'hot'

,6op,

'sand'

'running

'play'

A.2 DePALM with ViT visual encoders

We further test LMMs which do not contain a CLIP visual encoder to confirm that the high CLIPScore is not due to use of CLIP visual encoders. To test this, we experiment on two different DePALM models with frozen visual encoders different from CLIP, a frozen ViT-L encoder trained on ImageNet [11] and another frozen ViT-L trained as a masked autoencoder (MAE) [19]. Both LMMs use the same pretrained OPT-6.7B language model. Collectively, the three encoders (including CLIP) are pretrained for three different types of objectives. We use Semi-NMF to extract concept dictionaries, with all hyperparameters identical. The results are reported in tables below. 'Rnd-Words' and 'GT-captions' references are reported for each LMM separately, although they are very close to the ones in main paper. The "ViT-L (CLIP)" baseline denotes our system from the main paper that uses CLIP encoder. Importantly, we still obtain similar test CLIPScores as with CLIP visual encoder. The concept dictionaries still possess meaningful multimodal grounding. Many concepts also tend to be similar as for CLIP visual encoder, further indicating that processing inside language model plays a major role in the discovery of multimodally grounded concepts.

nu,

'cal'

'computer

'brown'

'keyboard"

,ds,

'monitor'

'cat'

'black'

'white'

'television'

'cat'

'flat'

'animal'

'remote'

'hair'

'cat'

'fur'

'monitor'

'cat'

'cat'

'pet'

'strip'

,6op,

,6op.

'blank'

'colored"

'drink'

'tab'

'cat'

'black'

GamesforCats

'animal'

,ans,

TolletTricks

,6op,

'travel'

'colored"

,6nl,

,ued,

,6eq.

'reflection'

'park'

'picture'

'street'

'mirror'

'cat'

'window

'tree'

'photo'

'building'

'small'

'gray'

,人qeq.

'grey'

'young'

'cat'

'tiny"

'black'

'white'

'cart'

'sleep'

'fake'

"lay'

'stuff"

'rest'

'picture'

'cat'

'drawing'

'fat'

'orange'

'bath'

'yellow'

'sink'

'golden'

'kitchen'

'cat'

'cat'

'brown'

'white'

20413

B Analyzing polysemanticity in the learnt concepts

We conducted a preliminary qualitative study on some concept vectors in the dictionary learnt for token "Dog" (DePALM model), to analyze if these concept vectors tend to activate strongly for a specific semantic concept (monosemantic) or multiple semantic concepts (polysemantic). In particular, we first manually annotated the 160 test samples for "Dog" for four semantic concepts, for which we knew we had concept vectors in our dictionary, namely "Hot dog" (Concept 2, row 1, column 2 in Fig. 7), "Black dog" (Concept 20, row 10, column 2 in Fig. 7), "Brown/orange dog" (Concept 6, row 3, column 2 in Fig. 7), and "Bull dog" (Concept 15, row 8, column 1 in Fig. 7). For a given semantic concept, we call this set C true . Then, for its corresponding concept vector u k we

'bridge'

'train'

'city'

'colored'

'street'

'electric

,snq.

'passengel

'tram'

'cab'

'steam'

'cargo'

'black'

'container

'smoke'

,buol,

'historic'

'tank'

o,

,peol,

'station'

'red'

'terminal

'Red'

'platform

'German

'European

'hall'

passenge

'train'

passenger'

'birth'

'electric'

'child'

'colored

'fake'

'engine'

,ania,

'city'

'silver'

,!land,

'white'

,snq,

'train'

'passenge

"light'

'train'

'seat'

'train'

,snq,

'model'

'window

'engine'

"layout'

'chair'

'version'

'rust'

'man'

'gra'

'woman'

.

'person'

'broken'

'crowd'

'worn'

,dnoub,

'yellow'

'tour'

'train'

'small'

'British'

'park'

'bright'

pa,

'passenger

'ride'

find the set of test samples for which u k activates greater than a threshold τ . This threshold was set to half of its maximum activation over test samples. We call this set of samples C top . To estimate specificity of the concept vector we compute how many samples in C top lie in the ground-truth set, i.e. | C top | ∩ | C true | / | C top | .

We found Concept 2 ("Hot dog") to be most monosemantic with 100% specificity. For Concept 20 ("Black dog") too, we found high specificity of 93.3%. For concept 15 ("Bull dog") we observed the lowest specificity of 50%. This concept also activated for test images with toy/stuffed dogs. Interestingly, the multimodal grounding of concept 15 already indicates this superposition with maximum activating samples also containing images of 'toy dogs'. Concept 6 ("Brown/orange dog") is somewhere in between, with 76% specificity. This concept vector also activated sometimes for dark colored dogs, which wasn't apparent from its multimodal grounding.

Prominent or distinct semantic concepts seem to be captured by more specific/monosemantic concept vectors, while more sparsely present concepts seem at risk to be superposed resulting in a more polysemantic concept vector capturing them. It is also worth noting that the multimodal grounding can be a useful tool in some cases to identify polysemanticity in advance.

C Further implementation details

C.1 Dictionary learning details

The details about the number of samples used for training the concept dictionary of each token, and the number of samples for testing is given in Tab. 7. The token representations are of dimension B = 4096 .

The hyperparameters for the dictionary learning methods are already discussed in the main paper. All the dictionary learning methods (PCA, KMeans, Semi-NMF) are implemented using scikitlearn [37]. For PCA and KMeans we rely on the default optimization strategies. Semi-NMF is implemented through the DictionaryLearning () class, by forcing a positive code. It utilizes the coordinate descent algorithm for optimization during both the learning of U ∗ , V ∗ and the projection of test representations v ( X ) .

C.2 CLIPScore/BERTScore evaluation

For a given image X and set of words T k associated to concept u k , CLIPScore is calculated between CLIP-image embedding of X and CLIP-text embedding of comma-separated words in T k . We consider a maximum of 10 most probable words in each T k , filtering out non-English and stop words. The computation of the metric from embeddings adheres to the standard procedure described in [20]. Our adapted implementation is based on the CLIPScore official repository, which utilizes the ViT-B/32 CLIP model to generate embeddings.

We found that computing BERTScores from comma-separated words and captions is unreliable. Instead, we adopted a method using the LLaMA-3-8B instruct model to construct coherent phrases from a set of grounded words, T k . Specifically, we provide the LLaMA model with instructions to describe a scene using a designated set of words, for which we also supply potential answers. This instruction is similarly applied to another set of words, but without providing answers. The responses generated by LLaMA are then compared to the captions y using BERTScore. The instruction phrase and an example of the output are detailed in 8. The highest matching score between the generated phrases and the captions of a test sample determines the score assigned to the concept u k . This approach ensures that the evaluation accurately reflects coherent and contextually integrated language use. The metric calculation from embeddings follows the established guidelines outlined in [50]. Our adapted implementation is based on BERTScore official repository, and we use the default Roberta-large model to generate embeddings.

C.3 Resources

DePALM experiments compute Each experiment to analyze a token with a selected dictionary learning method is conduced on a single RTX5000 (24GB)/ RTX6000 (48GB)/ TITAN-RTX (24GB) GPU. Within dictionary learning, generating visualizations and projecting test data, the majority of time is spent in loading the data/models and extracting the representations. For analysis of a single token with ≈ 3000 training samples, it takes around 10-15 mins for this complete process. Evaluation for CLIPScore/BERTScore are also conducted using the same resources. Evaluating CLIPScore for 500 (image, grounded-words) pairs takes around 5 mins. The BERTScore evaluation is in contrast more expensive, consuming around 150 mins for 500 pairs.

LLaVA experiments compute Each experiment to extract a concept dictionary for LLaVA was conducted on a single A100 (80GB) GPU. Representation extraction process for LLaVA is more

expensive compared to DePALM consuming around 90 mins for ≈ 3000 samples. The remaining aspects of dictionary learning, multimodal grounding, representation projection etc. remains relatively cheap. The CLIPScore/BERTScore evaluations are completed with same resources as before.

Licenses of assets The part of the code for representation extraction from LMM is implemented using PyTorch [36]. For our analyses, we also employ the OPT-6.7B model [49] from Meta AI, released under the MIT license, and the CLIP model [39] from OpenAI, available under a custom usage license. Additionally, the COCO dataset [27] used for validation is accessible under the Creative Commons Attribution 4.0 License. We also use CLIPScore [20] and BERTScore [50] for evaluating our method, both publicly released under MIT license. All utilized resources comply with their respective licenses, ensuring ethical usage and reproducibility of our findings.

C.4 Choice for number of concepts K

Our choice of using K = 20 concepts for all tokens was driven by the behaviour of reconstruction error of Semi-NMF on the training samples with different values of K , i.e. || Z -UV || 2 2 . We validate this behaviour on the four target tokens from main text in Fig. 11. We generally found K = 20 as the minimal number of concepts where the reconstruction error drops by at least 50% from K = 0 .

We also conducted an ablation study to test how our method behaves with different values of number of concepts K . Fig. 12 presents the variation of test CLIPScore and Overlap score for K ∈ { 10 , 20 , 30 , 50 } for two target tokens, 'Dog' and 'Cat'. Our method can learn meaningful concepts for different values of K , evident by the consistently high CLIPScore. The Overlap score on the other hand tends to drop more consistently as the number of concepts increase but behaves stably for different choices. Nonetheless they indicate that our method can accommodate dictionaries of larger sizes without compromising the quality of learnt concepts, provided K << M (number of decomposed samples) and at the expense of greater user overhead.

D Evaluation and extension to more tokens

We provide test data mean CLIPScore and BERTScore for top-1 activating concept for all baselines and more tokens: Baby, Car, Cake, and Bear in Tab. 9 (results in the main paper are reported for tokens Dog, Bus, Train, Cat in Tab. 1). Additionally, we also report the macro average over a set of 30 additional COCO-nouns apart from the 8 tokens with individually reported results, denoted

(Test) Reconstruction error by size of dictionary and Category

NumberofConceptsK

6op

cat

60

bus

train

50-

40

30

20

0

20

40

60

80

100

Average Clip Score by Components and Category

Average Clip Score

0.62

0.60

*

dogAvgClipScore

0.58

10

20

30

50

NumberofConceptsK

Average Overlap Score

Average Overlap Score by Components and Category

0.3

dogAvgOverlapScore

catAvgOverlapScore

0.2

0.1

10

20

30

50

Number of ConceptsK

as 'Extra-30'. These nouns are single-token words with at least 40 predicted test samples. We put the filter of single-token words to keep consistency with the presented framework. Extension to multi-token words is straightforward but discussed separately in D.1. The lower bound criterion on test samples is to ensure average test CLIPScore is reliable for each target token. We only report CLIPScore for 'Extra-30' tokens as BERTScore evaluation was more expensive to conduct on large number of dictionaries.

We observe that we consistently obtain higher scores across for Semi-NMF and K-Means. We also report the overlap score between grounded words in Tab. 10 to illustrate the superiority of our method over the simple baseline. As previously noted, we observe a high overlap between grounded

words with KMeans and Simple baselines compared to Semi-NMF/PCA. A low overlap should be encouraged, as it indicates the discovery of diverse and disentangled concepts in the dictionary. Among all the methods, Semi-NMF provides the most balanced concept dictionaries which are both meaningful (high CLIPScore/BERTScore) and diverse (low overlap).

Statistical significance The statistical significance of Semi-NMF w.r.t all other baselines and variants, for CLIPScore/BERTScore evaluation to understand representations of test samples is given in Tab/ 11 (for all tokens separately). We report the p -values for an independent two sided T-test with null hypothesis that mean performance is the same between Semi-NMF and the respective system. The results for Semi-NMF are almost always significant compared to Rnd-Words, Noise-Imgs, PCA. However for these metrics, Simple baseline, K-Means and Semi-NMF all perform competitively and

better than other systems. Within these three systems the significance depends on the target token, but are often not significant in many cases.

D.1 Extending to multi-token words

The presentation of our approach assumes that our token of interest t is a single token. This poses no issues for words which are represented as single tokens but can raise some questions when we wish to extract concept representations for multi-token words. Our approach however, can be easily adapted to this setting. In particular, we extract representation of last token from first prediction of the multi-token sequence. Note that when filtering the training data for samples where ground-truth caption contains the token of interest, we now search for the complete multi-token sequence. The other aspects of the method remain unchanged. While there can also be other viable strategies, the rationale behind this adaptation is that the last token of our sequence of interest can also combines representations from previous tokens in the sequence. We add below results for such examples in Tab. 12 and 13. We observe behaviour consistent with the previous results with Semi-NMF extracting concept dictionaries with high CLIPScore and low overlap.

E Additional visualizations

E.1 Concept grounding

The visual/textual grounding for all tokens in Tab. 1 are given in Figs. 13 ('Dog'), 14 ('Cat'), 15 ('Bus'), 16 ('Train'). All the results extract K = 20 concepts from layer L = 31 . Similar to our analysis for token 'Dog' in main paper, for a variety of target tokens our method extracts diverse and multimodally coherent concepts encoding various aspects related to the token.

E.2 Local interpretations

Here, we qualitatively analyze the local interpretations of various decomposition methods, including PCA, k-means, semi-NMF, and the simple baseline strategy. We select these four as they produce coherent grounding compared to Rnd-Words and Noise-Img baselines. We decompose test sample representations on our learnt dictionary and visualize the top three activating components. Note that in the case of KMeans and Simple baseline, the projection maps a given test representation to a single element of the concept dictionary, the one closest to it. However, for uniformity we show the three most closest concept vectors for both. Figs. 17, 18, 19, 20, 21 are dedicated to interpretations of a single sample each, for all four concept extraction methods.

The inferences drawn about the behaviour of the four baselines from quantitative metrics can also be observed qualitatively. Semi-NMF, K-Means and 'Simple' baseline, are all effective at extracting grounded words can be associated to a given image. However, both K-Means and 'Simple' display similar behaviour in terms of highly overlapping grounded words across concepts. This behaviour likely arises due to both the baselines mapping a given representation to a single concept/cluster. This limits their capacity to capture the full complexity of data distributions. In contrast, Semi-NMF and PCA utilize the full dictionary to decompose a given representation and thus recover significantly more diverse concepts. PCA in particular demonstrates almost no overlap, likely due to concept vectors being orthogonal. However, the grounded words for it tend to be less coherent with the images. As noted previously, Semi-NMF excels as the most effective method, balancing both aspects by extracting meaningful and diverse concepts.

F Qualitative analysis for different layers

We provide a qualitative comparison of multimodal grounding for the token 'dog' across different layers in Fig. 22. As observed in Fig. 6 (main paper), the multimodal nature of token representations for two tokens 'Dog' and 'Cat' starts to appear around layers L = 20 to L = 25 . It is interesting to note that the representations of images still tend to be separated well, as evident by the most activating samples of different concepts. However, until the deeper layers the grounded words often do not correspond well to the visual grounding. This behaviour only appears strongly in deeper layers.

G Analysis for visual tokens

Our analysis in main paper was limited to decomposing representations of text tokens in various layers of an LLM, h p ( l ) , p > N V . This was particularly because these were the predicted tokens of the multimodal model. Nevertheless, the same method can also be used to understand the information

,bop.

,bop,

'black'

'sausage'

'white'

'hot'

'small'

'sandwich'

'border'

'plate'

,bop,

'small'

'black'

'tiny"

'dogs'

'puppy'

'pet'

'miniature'

'cute'

,6op.

'brown'

'running'

'large'

'black'

'dog"

'play'

'tan'

'golden'

OUDI0

RIGHT

'white'

'animal'

"light'

'lab'

,6op.

'eyed'

,Addnd,

'bag'

'herd'

'cup'

'sheep'

'plastic'

'flock'

'stuffed'

'farm'

'hat'

'shepherd'

'room'

'reflection'

'bedroom'

'car'

'living'

'view'

,6op,

'vehicle'

.6op,

'furry'

'cat'

'hairy'"

'kitten'

'fluffy"

'tiger'

,buol,

'rabbit'

,6op,

"lnq.

'bike'

'bicycle'

'giant'

'street'

'dog'

'motorcycle

'large'

'sidewaik'

'woman'

,6op,

'person'

'sandwich'

'man'

'hot'"

'lady"

'food'

'pocket'

.eoq.

'black'

'surf"

'large'

'swim'

'beach'

'big'

'sail'

stored in the visual/perceptual tokens representations as processed in f LM , h p ( l ) , p ≤ N V . An interesting aspect worth highlighting is that while the text token representations in f LM can combine information from the visual token representations (via attention function), the reverse is not true. The causal processing structure of f LM prevents the visual token representations to attend to any information in the text token representations. Given a token of interest t , for any sample X ∈ X t we now only search for first position p ∈ { 1 , ..., N V } , s.t. t = arg max Unembed ( h p ( N L ) ) . Only the samples for which such a p exists are considered for decomposition. The rest of the method to learn U ∗ , V ∗ proceeds exactly as before.

We conduct a small experiment to qualitatively analyze concepts extracted for visual token representations for 'Dog'. We extract K = 20 concepts from L = 31 . The dictionary is learnt with representations from M = 1752 samples, less than M = 3693 samples for textual tokens. As a brief illustration, 12 out of 20 extracted concepts are shown in Fig. 23. Interestingly, even the visual token representations in deep layers of f LM , without ever attending to any text tokens, demonstrate a multimodal semantic structure. It is also worth noting that there are multiple similar concepts that appear for both visual and textual tokens. Concepts 3, 7, 10, 12, 17, 19 are all similar visually and textually to certain concepts discovered for text tokens. This indicates to a possibility that

'gray'

'cat'

'kitten'

'striped'

'bearish'

'cat'

'colored'

.ued,

'street'

'orange'

'outdoor'

'yellow'

'cat'

'cat'

'car'

'golden'

'sidewalk'

'ginger'

'cat'

'kitten'

'kitten'

'small'

'spotted'

'tiny'

'striped'

'baby'

'cal'

'young'

'computer'

'cat'

'keyboard'

'desk'

black'

'pet'

'cat'

'dog'"

,Mol!d,

'room'

'white'

'bed'

'cream'

'wooden'

`light'

'cal'

'bathroom'

'mult'

'toilet'

'cat'

'bath'

'brown'

'cat'

'tort'

'kitchen'

'suitcase'

'black'

'bag"

,abeb6ni.

'cat'

'dark'

'cat'

,a6966eq.

'gray'

,bop,

'person'

'cat'

'woman'

'puppy'

'man'

'horse'

'close'

'pig'

ed,

,buol,

'television'

'cat'

'furry"

'house'

'hairy"

'black'

'fat'

'white'

'fake'

'black'

'stuffed"

'cat'

'cartoon'

'cal"

these concepts are discovered by f LM in processing of the visual tokens and this information gets propagated to predicted text token representations.

H Limitations

We list below some limitations of our proposed method:

The concept dictionaries extracted currently are token-specific. It can be interesting to explore learning concept dictionaries that can encode shared concepts for different tokens.

The current study is conducted mainly on visual captioning models. While we expect the key ideas to generalize to many other types of large multimodal models, this application of our approach to other types of LMMs remains to be explored/confirmed.

We select the most simple and straightforward concept grounding techniques. Both visual and textual grounding could potentially be enhanced. The visual grounding can be improved by enhancing localization of concept activation for any MAS or test sample. Text grounding could be enhanced by employing more sophisticated approaches such as tuned lens [6].

'city'

'bus'

'blue'

'tour'

,snq.

'city"

'large'

sno,

'street'

'transit'

'city'

'bus'

'transit'

'district'

'passenger

'vehicle'

ind,

no,

'traffic'

'truck'

'van'

'vehicle'

'street'

'car'

'white'

'car'

'train'

'subway'

,snq.

'Thai"

'Asian'

'public'

'brightly"

.eas,

'Mexican'

'red'

'vintage'

'Red'

'antique'

,snq.

'older"

'double'

,sn,

lnd,

'classic'

'bus'

'group'

'couple'

'stopped'

'bunch'

'large'

lgnd,

,uew,

'tour'

'bus'

'toy"

'city'

'circus'

'transit'

'mobile'

'shuttle'

'decorated"

'passenger'

'yellow'

"bus'

,snq,

'tour'

'Gold'

'double'

.pjob,

'red'

'city'

'sign'

"building'

'green'

'pole"

BUS

'city"

'parking'

'street"

STOP

'public'

nor,

'white'

'tour'

'city'

'large'

'bus'

'coach'

'commuter'

'grey"

'transit'

While the proposed CLIPScore/BERTScore metrics are useful to validate this aspect, they are not perfect metrics and affected by imperfections and limitations of the underlying models extracting the image/text embeddings. The current research for metrics useful for interpretability remains an interesting open question, even more so in the context of LLMs/LMMs.

I Broader societal impact

The popularity of large multimodal models and the applications they are being employed is growing at an extremely rapid pace. The current understanding of these models and their representations is limited, given the limited number of prior methods developed to understand LMMs. Since interpretability is generally regarded as an important trait for machine learning/AI models deployed in real world, we expect our method to have a positive overall impact. This includes its usage for understanding LMMs, as well as encouraging further research in this domain.

'steam'

'station'

'black'

'platform'

'vintage'

'train'

'train'

'railway'

'large'

'terminal'

'passenger'

'tank'

'subway"

'truck'

'engine'

'rusty'

'box'

,dunp,

'freight'

'bridge'

'train'

'river"

"long'w

'cargo'

'view'

'mountain'

'yellow'

`train'

'train'

'moving'

'train'

'toy'

HERRIN

BABI

'stopped"

'cake'

'passing'

'birthday"

'subway'

oop,

,snq,

'window'

'couch'

'train'

'view'

'subway'

'reflection'

eas,

'yellow'

'commuter'

'man'

'white'

'person'

'subway'

'subway'

abuassed,

'tour'

'red'

'tourist'

'Red"

'Santa'

'redd'

'vintage'

,ania,

'train'

'toy'

'car'

'miniature'

'train'

'amusement'

'platform'

'train'

'station'

sno,

'group'

'train'

'toy'

.oung,

'railroad'

'number'

'model"

'set'

'village'

'vintage'

,ania.

'black'

'passenger'

'historic'

'colorful"

'historical'

anbue,

'yellow'

`train'

网

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose a novel dictionary learning based concept extraction framework for understanding Large Multimodal Model (LMM) representations (Sec. 3). We extend previous dictionary learning-based concept extraction methods by proposing a Semi-NMF based optimization (Sec. 3). The learnt concept dictionary has multimodal semantic structure which we qualitatively and quantitatively validate with our experiments (Sec. 4).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

Semi-NMF

PCA

Concept 12, Concept 3, Concept 0 (dog. L31)

Concept activations

hairy

fuffy

fuzzy

furry

1ong

Concept activations

Asn,

muddy

'dark

'gray'

feral

'small*

'miniature'

‘puppy

tiny'

‘cute”

woman'

knot'

vest'

Cpt 12

Cpt 3

0 d

border

black

"white’

'small

'dog'

Cpt 18

'intelligent'

‘obedient'

“English'

'Afghan'

'Anat'

cept 5, Concept 9, Concept 7 (dog. L31)

Concept activations

small

‘cute’

white

‘dog”

Test sample

Concept activations

fuffy

'puppy”

'small*

'dog'

,Ao1

large'

'small

‘dog'

'puppy”

fluffy

'pet'

dog”

black’

small

"large'

black

'dog'

'puppy”

white'

dog'

brown’

brown’

'small

black’

K-Means

Simple

Semi-NMF

PCA

Concept 13, Concept 19, Concept 11 (dog, L31)

cept 10, Concept 4, Concept 5 (dog. L31)

Test sample

Concept activations

kitten’

tiger

'rabbit'

‘cat”

'dog’

Test sample

Concept activations

reflection’

'reflected”

kitten’

'miror

tooth'

black

'arge

'dark

big”

'close’

‘bedroom”

'gigantie'

.4pno3,

'sofa'

big’

"reflection'

vehicle'

view

‘car”

automotive'

underground’

exploitation

industrial

dog'

'electrical'

Concept 6, Concept 7, Concept 4 (dog, L31)

ept 17, Cot

(TET

‘dog'

'dog'

'small

Targe’

'brown'

'smal

Test sample

Concept activations

Targe

'man’

Test sample

1.0

Concept activations

'puppy

‘cat‘

black

Targe'

fop.

'small*

dogs

dog

‘brown

'cat”

small

Addnd,

'brown'

small

dog

'small

'dog”

'cat”

Targe

brown’

'car*

dogs'

K-Means

Simple

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed separately in Appendix H

Guidelines:

PCA

Simple

Semi-NMF

'stuffed

fake

lamp'

Test sample

1.0

Concept activations

toy

'cat"

Test sample

Concept activations

ladder

child'

‘black’

dark

'cat”

‘close'

'gray"

Cpt 13

television’

house’

black

fat’

'cat'

Cpt 8

'striped’

Marvin*

"peg'

'grass*

tiger

'small

‘cat'”

kitten'

‘'cat'”

Concept activations

white’

black

'dog'

Test sample

Concept activations

'animal

black'

'pant'

‘black’

small*

"large’

'cat’

black'

colored'

kitten'

‘cat’

dog'

house’

black

'gray*

'cat'

kitten’

black

‘cat’

'grey”

‘car

brown’

animal

K-Means

Semi-NMF

PCA

bus'

tour*

'city”

Test sample

transit'

tourist'

Test sample

'red*

Red*

'double'

“bus'

ppa

‘public'

tour

‘circus'

toy

p

red'

'mobile’

'decorated”

Concept activations

tourist'

'double'

'city”

tour

bus'

Fest sample

Concept activations

vehicle'

tour*

cty

red

bus'

‘double'

tour*

'red’

‘bus'

'passenger

public'

city

bus'

'city”

'transit'

'double'

tourist'

tour*

'city”

bus'

,Jafuassed,

'public'

tour

'city

bus'

K-Means

Simple

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution

Semi-NMF

PCA

'city

blue'

'street’

Targe'

bus'

Concept activations

redd

vintage'

'antque’

'older'

abandoned

apa(osqo,

‘defunct'

'rusty'

'railroad’

'station

group'

vintage

'couple'

‘bunch'

'crowd'

'caravan'

'antique’

'salvage'

Concept activations

'street'

large'

'city

,snq,

Concept activations

'passenger*

"public”

city

tour*

bus'

jafuassed,

large’

'smal

white’

bus'

'passenger

transit'*

'city

tour

bus'

public

large'

bus'

'city'

pt 2

Cpt 18

"publie"

double

city

bus'

white’

‘passenger

K-Means

Simple

L 1

L 12

L 20

L 30

,buol.

'set'

'black'

'public'

'head'

'Tamily

'high'

adoad,

.poob,

'couple'

`high'

'set'

,uew,

,uew,

lews,

'time'

,buol.

.peu.

,buol.

`high'

`life'

las,

'test'

.poob,

'time'

'great'

'house'

'black'

'community'

'dark'

'market'

,6op.

aldoad,

'large'

'border'

,uew,

,bop.

,sbop.

'group'

,bujuun,,

'couple'

'play'

'number'

'grass'

.xjw.

'bull

,

,Addnd,

'big'

'type'

'huge'

'photo'

'statue'

is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover

Example concepts for 'Dog' from visual token representations (layer 31)

Concept0（dog,L31)

人ddnd,

'brown'

,6op.

'roll'

'racing'

Concept2(dog,L31)

'white'

,6op.

'yawn'

'toy'

Concept3(dog,L31)

,b6op,

'golden

'waiting]

'guarding

'brown'

Concept5(dog,L31)

,Addnd,

,6op,

'mix'

'Boston

'tail"

Concept6（dog,L31)

'puppy'

,6op,

'black'

'paw'

Concept7(dog,L31)

,6op,

'walk'

'companion

Concept10（dog,L31)

'running]

'chasing

`eaping

'catching

'racing'

Concept11(dog,L31)

,6op,

.lnq,

'bel'

'puppy'

puenb,

Concept12（dog,L31)

'hair'

,6op,

'coated

'bred'

'breed'

Concept14(dog,L31)

'guarding'

,6op,

'barking”

'German

"herd'

Concept17(dog,L31)

'chasing

'sheep'

,6op,

'comingl

Concept19(dog,L31)

'tiny'

'small'

'cat'

'cute'

'puppy'

limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not posit any theoretical results. The paper primarily presents empirical evidence demonstrating the effectiveness of the method rather than focusing on theoretical results. our approach highlights the practical applicability and robust performance of the method through a series of experiments.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have disclosed complete information about our pretrained model, extraction of representations, parameters, and implementation to train the dictionary, evaluation setup, and resource usage for the experiments. Details are in Sec. 4 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code is available at https://github.com/mshukor/xl-vlms Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Major details are in Sec. 4 and all additional details in Appendix C. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We show standard deviation of the key metrics in Tab. 1 and Tab. 9 and discuss the statistical significance comparing baselines and our method in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details are in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed them and adhere to them.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The societal impacts are discussed separately in Appendix I.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We work with models and data already publicly available.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Details available in Appendix C.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We publicly release our code and documentation is available alongside it.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an explainability approach for interpreting the internal representations of large multimodal models (LMMs). The authors train an image captioning model consisting of a pretrained image encoder and language model and a connector model. To extract interpretable representations, the authors use dictionary learning, decomposing representations into lower dimensional $U$ and $V$ matrices using semi non-negative matrix factorization as the optimisation objective. To interpret concepts in $U$ in the textual domain, the authors use the language model unembedding layer to extract the highest probability tokens associated with a given concept vector $u$. Likewise, to visually interpret a concept vector $u$, the authors find the set of images that maximally activate $u$. The authors provide quantitative evidence demonstrating that their method generates concepts which are well aligned with both the input image and the ground-truth captions. Additionally, they demonstrate their approach qualitatively produces well defined concepts with limited overlap between the tokens represented by different concepts.\n\nSTRENGTHS:\n1. This paper introduces a promising novel approach to performing mechanistic interpretability on multimodal models. Given the increasing ubiquity of multimodal models there exists a clear need for interpretability approaches for this style of model. This work presents a viable dictionary-learning based approach I look forward to seeing other researchers building upon in the future.\n2. The evaluation of the framework is comprehensive including both qualitative and quantitative results, both of which are essential for any interpretability method. Additionally, I was impressed to see the authors consider multiple approaches (e.g. PCA/KMeans) and sensible baseline models when evaluating their results, providing more confidence that the final approach taken was an appropriate method.\n3. The paper is very clearly articulated, with the rationale well-defined, the approach clearly outlined and the results succinctly and clearly summarised.\n\nWEAKNESSES:\n1. The authors provide good evidence that the concepts extracted by their method show minimal overlap with other concepts, however they do not address the alternative possibility, that their extracted concepts might represent more than one distinct concept. This phenomenon of feature “superposition” has been well documented in other model interpretability work (see [1][2][3]) and so it seems plausible that it may arise in the approach taken here. This seems especially likely given that the dimensionality of their concept dictionary is lower than the dimensionality of the internal representations. Though I do not think this should detract from the otherwise excellent contributions presented in this paper, I do think this at least warrants a brief discussion, and perhaps more qualitative analysis of the extracted concepts, to assess whether any evidence of feature superposition is observed.\n2. Occasionally the authors make claims that go beyond the scope of this work. For example, in the abstract they state, “we present a novel framework for the interpretation of LMMs”. However, this claim seems too strong given that this approach is only really valid for image captioning models rather than large multimodal models more generally. Additionally, there are a few comments such as “we find the generalization of LLMs to multimodal inputs is an interesting phenomenon to understand” in 3.1 (under “Training”) and “the multimodal structure of internal token representations starts to appear at [later layers]” in 4.2 (under “Layer Ablation”). However these claims do not seem valid as the language model layers are frozen during training. It seems more appropriate to say that these extracted concepts represent language model concepts, and the connector model learns to transform the image representations such that they align with these language model concepts.\n\n[1] Elhage, N, et al. \"Toy Models of Superposition.\" arXiv:2209.10652 (2022)\n\n[2] Arora, S et al. \"Linear Algebraic Structure of Word Senses, with Applications to Polysemy.\" arXiv:1601.03764 (2016)\n\n[3] Elhage, N, et al. \"Softmax Linear Units\" Transformer Circuits Thread (2022)\n\nQUESTIONS:\n1. Why did the authors choose to use Opt-6.7B as the language model rather than more powerful similar sized models such as Llama-7B?\n2. Why do the authors take the absolute activations of $V$ in (5)? Are these activations not guaranteed to be non-negative by the optimisation objective?\n3. Did the authors consider trialling gradient-based feature visualisation approaches [1] in addition to taking images with the highest activations? This could be an alternative approach to build additional confidence that the extracted concepts do represent what qualitative analysis of the highest activating samples appears to suggest. I don’t expect the authors to perform analysis of this kind in this manuscript however it could be worth commenting on this as a future avenue of research, or alternatively raising any valid criticisms of this approach?\n4. There is a typo in figure 1 (“Captionin”)\n\n[1] Olah, C, et al. \"Feature Visualization\", Distill, 2017.\n\nLIMITATIONS:\nThe work presented here is only for a single, relatively small model, trained with a specific objective (image captioning). As such it is not clear that this approach will necessarily generalise to other multimodal models. The authors should touch on this limitation in the discussion or limitations section in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach to understand multimodal concepts learned in LLMs with visual prefixes. To do so, the authors propose a dictionary learning-based approach that decomposes the representation of a word token in the product of two low-rank matrices via Semi-NMF, one representing the concepts and the other representing the activations for a given token. The authors evaluate the representation of the DePALM model on 8 common objects (e.g., dog, bus, cat) showing qualitatively good results, as well as better auto-eval metrics than with related baselines constructed by the authors.\n\nSTRENGTHS:\nOriginality: The paper proposes a novel method to interpret concepts in a multimodal LLM by grounding them in both text and visual spaces. For a given concept, it creates a representation matrix from image–caption pairs, which is then decomposed into two low-rank matrices using Semi-NMF. \n\nQuality: The authors show that extracted concepts for a given token can be interpreted both visually and textually through qualitative examples. Quantitatively, the method outperforms other related baselines introduced by the authors. \n\nClarity: The paper is relatively well written, although some parts (eg, Section 3.4, 4.0 and 4.1) feel very dense and require more time to be processed.\n\nSignificance: The paper provides interesting insights in the representation of a given word token processed by a frozen LLM that is augmented with image understanding via visual prefix learning.\n\nWEAKNESSES:\n1. My main concern is in studying the representation of only eight tokens. It would be interesting to study more words and see if there are any interesting takeaways. These could include the 80 COCO classes, and additional rarer tokens.\n2. The use of Semi-NMF in L165-167 is unclear. Later on, we find out that it works better than other decompositions, but I was left wondering how the authors came up with Semi-NMF in Section 3.4.\n3. It would have been useful to compare the proposed approach with some current interpretability methods that could apply here (or at least discuss why they might not be used here).\n\nQUESTIONS:\n1. Are there any blockers to perform a larger-scale (wrt tokens) analysis?\n2. Interpretability methods like ROME use the last token of a given word – why do you use the first token?\n3. Does the method actually work for multi-token words (i.e., words that are split into multiple tokens by the tokenizer)? If so, are any of your 8 words in that category?\n4. Please double-check your references, some of them appear more than once.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
In this paper, the authors propose a framework for interpreting LMMs. Specifically, they introduce a dictionary learning-based approach applied to the representation of tokens. The elements of the learned dictionary correspond to the proposed concepts. These concepts are semantically well-grounded in both vision and text.

STRENGTHS:
Using concepts to interpret large multimodal models is a promising idea.

WEAKNESSES:
1. I'm sorry, I don't fully understand this field, so I will lower my confidence score.
2. The notation system is quite confusing. I suggest the authors reorganize it for better clarity.
3. The paper directly applies the concept activation vector (CAV) to Large Multimodal Models, but it does not explain the benefits of doing so or why this approach is valid.

QUESTIONS:
The experiments in the paper are relatively few. I suggest conducting further research on the interpretability of multimodal large models such as LLaVA and MiniGPT-4.

LIMITATIONS:
yes, the author explains the limitations of their study and potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

comments: SUMMARY:
The authors propose using dictionary learning to extract concepts from multimodal models and simultaneously ground them in the text and image latent space. They draw on prior work on multimodal neurons and concept activation vectors. The authors provide quantitative results using CLIPScore and BERTScore to measure concept extraction, multimodal grounding, and concept overlap.

STRENGTHS:
Appears to be mathematically correct and well-grounded in prior work.
Well-written and well-illustrated.

WEAKNESSES:
Evaluates only a few tokens (dog, bus, train, cat) thoroughly, which seems like not enough when prior work (Goh et al.) studies thousands of concepts.
Evaluates only one model, DePALM.
Uses two automated metrics, CLIPScore and BERTScore, but no human evaluation. A little worrisome that DePALM uses a CLIP-ViT-L14 and CLIPScore is the primary evaluation for the method.
Overlaps significantly with prior research (Goh et al, Kim et al); multimodal neurons have been described in work dating back several years using non-negative matrix factorization.

QUESTIONS:
In table 2, why is PCA by far the best method for minimizing overlap between learned concepts? Presumably because it’s PCA on only these five concepts, meaning that variance is maximized between only the five concepts and not between the dictionary more broadly?

Do you observe any evidence of polysemanticity, as reported in prior work including Goh et al.?

LIMITATIONS:
I don’t see many negative societal implications of this research.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose to look at the representation of chosen concepts, in multimodal models. They test different automated methods to learn decompositions of a token representations (which they then linearise in a dictionary of concepts). They also provide quantitative and qualitative analysis of a few examples, showing that this method clusters different human-interpretable meaning or usages of a same concept, for both visual and textual modalities.\n\nSTRENGTHS:\nGreat qualitative analysis\nMultiple representation decomposition algorithms are tested, including more well known PCA or very specific but better suited Semi NMF, which is an adaptation to practical realities.\nEvaluation of the method is quite extensive, using a wide range of relevant state of the art interpretability tools to verify their work theoretically and practically.\nEffort is made to put quantitative metrics on concepts and to evaluate very abstract semantic analysis.\nProvided examples are clear and strike curiosity.\n\nWEAKNESSES:\nI worry there is a circularity when evaluating with ClipScore a frozen model with a CLIP component. \n(and even with BERT scoring, where you evaluate an interpretability metric with a non interpretable system of somewhat similar complexity).\n\n\n(Minor)\nVery recent works are relevant to the interpretability of LLM / Multimodal models (Templeton, et al., \"Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet\", Transformer Circuits Thread, 2024 AND Gao, Leo, et al. \"Scaling and evaluating sparse autoencoders.\" arXiv preprint arXiv:2406.04093, 2024) and could be cited.\n\nLine 223 \"the correspondance between a the image\" is probably a typo and should be fixed\n\nQUESTIONS:\nMostly a personal curiosity question: Are there any intuitions from this work allowing to disentangle the data effect of the apparition of the studied concepts from the model/training effect?\n\nLIMITATIONS:\nSample experiments are done on very simple words and concepts, available at pre-training for the tested model. How well this method would adapt to more complex concepts and images is not studied, but is where a lot of interpretability becomes necessary. This is nonetheless a necessary first step in an exciting direction.\nDo you have an intuition on how well this method would scale to this?", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}