A Concept-Based Explainability Framework for Large Multimodal Models
Authors: Jayneel Parekh
Pegah KHAYATAN
Mustafa Shukor
Alasdair Newson
Matthieu Cord
Venue: NeurIPS 2024 poster (2024)
Keywords: Large multimodal models
interpretability
concept extraction
multimodal concepts
multimodal representations
vision-language models
vision-language interpretations
Primary area: interpretability_and_explainability
TL;DR: A dictionary-learning based approach for concept extraction to interpret representations inside a large multimodal model.

Abstract: Large multimodal models (LMMs) combine unimodal encoders and large language models (LLMs) to perform multimodal tasks. Despite recent advancements towards the interpretability of these models, understanding internal representations of LMMs remains largely a mystery. In this paper, we present a novel framework for the interpretation of LMMs. We propose a dictionary learning based approach, applied to the representation of tokens. The elements of the learned dictionary correspond to our proposed concepts. We show that these concepts are well semantically grounded in both vision and text. Thus we refer to these as ``multi-modal concepts''. 
We qualitatively and quantitatively evaluate the results of the learnt concepts. We show that the extracted multimodal concepts are useful to interpret representations of test samples. Finally, we evaluate the disentanglement between different concepts and the quality of grounding concepts visually and textually. Our implementation is publicly available: https://github.com/mshukor/xl-vlms.

A Concept-Based Explainability Framework for Large Multimodal Models
Jayneel Parekh'
Pegah Khayatan'
Mustafa Shukor'
Alasdair Newson'
Matthieu Cord 1,2
IIRR, Sorbonne Université, Paris, France Valeo. Paris, France
[ayyeell payyeel.parekh, pegah. thayyatan/asorbome-uuueeer
Abstract
Large multimodal models (LMMs) combine unimodal encoders and large language models (LLMs) to perform multimodal tasks. Despite recent advancements towards the interpretability of these models, understanding internal representations of LMMs remains largely a mystery. In this paper, we present a novel framework for the interpretation of LMMs. We propose a dictionary learning based approach, applied to the representation of tokens. The elements of the learned dictionary correspond to our proposed concepts. We show that these concepts are well semantically grounded in both vision and text. Thus we refer to these as "multi- modal concepts". We qualitatively and quantitatively evaluate the results of the learnt concepts. We show that the extracted multimodal concepts are useful to interpret representations of test samples. Finally, we evaluate the disentanglement between different concepts and the quality of grounding concepts visually and textually. Our implementation is publicly available
Introduction
Despite the exceptional capacity of deep neural networks (DNNs) address complex learning prob- lems, one aspect that hinders their deployment is the lack of numan-comprehensible understanding of their internal computations. This directly calls into question their reliability and trustworthiness Consequently, this has boosted research efforts in interpetabiliyyexplannbbb models i.e. devising methods to gain human- uumaa--nderstandablle insights about their decision processes. The growth in ability of DNNs has been accompanied by similar increase in their design complexity and computational intensiveness. This is epitomized by the rise of vision transformers [1I] and large-language models (LLMs) [8 which can deploy up to tens of billions of parameters. The effectiveness of these models for unimodal processing tasks has spurred their use in addressing multimodal tasks. In particular, visual encoders and LLMs are frequently combined address tasks such as image captioning and VQA 2 25 large multimodal models (LMMs).
5. 30].
of these
45). This recent class of models are referred to as
For interpretability research, LMMs have largely remained unexplored. Most prior works on interpret- ing models that process visual data, focus on convolutional neural network (CNN) based systems and classification as the underlying task. Multimodal tasks and transformer- architectures have both been relatively less studied. LMMs operate at the intersection of both domains. Thus, despite their rapidly growing popularity, there have been very few prior attempts at understanding representations 42).
inside LMM 34
Project page and code: htttps://jayneellaaeek .ithhggittuu.....
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

This paper aims to bridge some of these differences and study in greater detail the intermediate representations of LMMs. To this end, motivated by the concept activation vector (CAV) based approaches for CNNs [14 15
17 22] we propose a novel dictionary-learning based Concept eXplainability method designed for application to LMMs, titled COX-LMM. Our method is used to learn concept dictionary to understand the representations of a pretrained LMM for given word/token of interest (Eg. 'Dog'.. For this token, we build a matrix containing the LMM's internal representation of the token. We then linearly decompose this matrix using dictionary learning. The dictionary elements of our decomposition represent our concepts. The most interesting consequence of method is that the learnt concepts exhibit semantic structure that can be meaningfully grounded in both visual and textual domains. They are visually grounded by extracting the images which maximally activate these concepts. They can simultaneously be grounded in the textual domain by decoding the concept through the language model of the LMM and extracting the words/tokens they are most associated to. We refer such concept representations multimodal concepts. Our key contributions can be summarized as follows:
We propose a novel concept- explainability framework CoX-LMM, that can be used to understand internal representations of large multimodal models. To the best of our knowledge, this is the first effort targeting multimodal models at this scale.
Our dictionary learning based concept extraction approach is used extract a multimodal concept dictionary wherein each concept can be semantically grounded simultaneously in both text and vision. We also extend the previous concept dictionary-leaming strategies using a Semi-NMF based optimization.
We experimentally validate the notion of multimodal concepts through both, qualitative visualizations and quantitative evaluation. Our learnt concept dictionary is shown to possess a meaningful multimodal grounding covering diverse concepts, and is useful to locally interpret representations of test samples LMMs.
2
Related work
Large Multimodal Models (LMMs) Large language models (LLMs) 8 21 33 have emerged as the cornerstone of contemporary multimodal models. Typical large multimodal models (LMMs) 4 25, [66] comprise three components: LLMs, visual encoders, and light-weight connector modules to glue the two models. Remarkably, recent works have demonstrated that by keeping all pretrained models frozen and training only a few million parameters in the connector (e.g., a linear layer), LLMs can be adapted to understand images, videos, and audios 12 23 43 45),
thus paving the way for solving multi-modal tasks. However, there is still a lack of effort aimed at understanding why such frozen LLMs can generalize multimodal inputs. In this study, we try to decode the internal representation of LLMS when exposed to multimodal inputs.
Concept activation vector based approaches Concept based interpretability aim extract the semantic content relevant for a model (9. For post-hoc interpretation of pretrained models, concept activation vector (CAV) based approaches [15] [7 22 46 have been most widely used. The idea of CAV was first proposed by Kim et al. (22). They define a concept as a set of user-specified examples. The concept represented in the activation space of deep layer of CNN by a hyperplane that separates these examples from a set of random examples. This direction in the activation space is referred as the concept activation vector. Built upon CAV, ACE [7] automate the concept extraction process. CRAFT [15] proposed to learn set of concepts for a class by decomposing activations of image crops via nen-negative matrix factorization (NMF). Recently, Fel et al. [14] proposed a unified view of CAV-based approaches as variants of a dictionary learning problem. However, these methods have only been applied for interpretation of CNNs on classification tasks. LMMs on the contrary exhibit a different architecture. We propose dictionary learning based concept extraction method, designed for LMMs. We also propose a Semi-NMF variant of the dictionary learning problem, which has not been previously considered for concept extraction.
Understanding VLM/LMM representations There has been an increasing interest in understand- ing internal representations of visual-language models (VLM) through the lens of multimodality. Shukor and Cord 42) analyse multimodal tokens and shows that despite being different, visual and perceptual tokens are implicitly aligned inside LLMs. Goh et al. [18] discover neurons termed

---

dog
Dictionary learning for concept extraction in LMMs
Z
U
Unsmbee
U1
oOo
X
UKT
FPM
LMM
UK
f
LM
Representation
matrix
Concept dictionary
activations
Concept
Multimodal concept grounding
26
LLM
'black', 'large', 'dark'
A
Tour Patice
Uk
X KEAAAS
Image
X
Figure I: Overview of multimodal concept extraction and grounding in COX-LMM. Given pretrained LMM for captioning and a target token (for eg. 'Dog'), our method extracts internal representations of f about t. across many images. These representations are collated into matrix Z. We linearly decompose Z to learn a concept dictionary U and its coefficients/actiiatioss V. Each concept ue U, multimodally grounded in both visual and textual domains. For text grounding, we compute the set of most probable words Tr by decoding ug through the unembedding matrix Wy. Visual grounding AS is obtained via Uk as the set of most activating samples.
multimodal, that activate for certain conceptual information given images as input. Recently proposed TEXTSPAN [16] and SpLiCE (7). aim understand representations in CLIP 38 by decomposing
its visual representations on textual representations. For LMMs, Palit et al. 34 extend the causal tracing used for LLMs to analyze information across different layers in LMM. Schwettmann et al. [41] first proposed the notion of multimodal neurons existing within the LLM part of an LMM. They term the neurons "multimodal" they translate high-level visual information to corresponding information in text modality. The neurons are discovered by ranking them by a gradient based attribution score. Pan al. (35] proposed
refined algorithm identify such neurons based a different neuron importance measure that leverages architectural information of transformer MLP blocks. Instead, we propose discover a concept structure the token representations by learning a small dictionary of multimodally grounded concepts. Limiting the analysis to specific token of interest allows our method discover fine details about the token in the learnt concepts.
3 Approach
3.1 Background for Large Multimodal Models (LMMss)
Model architecture. We consider general model architecture for large multimodal model f. that consists of: a visual encoder fv, a trainable connector C, and an LLM fLM consisting of N; layers. We assume f is pretrained for captioning task with underlying dataset S {(X,, v:) v:)) > consisting of images X; E X' and their associated caption images and set of tokens respectively. Note that caption yi can viewed The input to the language model fLM is denoted by the sequence of tokens h22 1,2 :..., hP and the output as The internal representation of any token at some layer and position p inside fLM is denoted as hu with ho = h², Note that hi is same as the residual stream representation in LLM transformers [33 at position p and layer l. For the multimodal model, the input sequence of tokens for FLM consists of the concatenation of: (1) Ny visual tokens provided by the visual encoder fv operating image X, followed by the connector C, and (2) linearly embedded textual tokens previously predicted by fLM. For > Nv, this can be expressed as:
C ). X' and y denote the space of
subset of all tokens.
gd fLM(h ,h2 ..... Nv ..... h2),
(I)
3

---

where .......,,, (X)v and ht = Emb ( for p Nv. where Emb denotes the token embedding function. To start the prediction, h &v+1 defined the beginning of sentence token. The output token jj" is obtained by normalizing ( followed by an unembedding layer that applies a matrix Wy followed by a softmax. The predicted caption ÿ consists of the predicted tokens f = [0")p>Ny Ip>Nv until the end of sentence token.
Training The model is trained with next token prediction objective, to generate text conditioned on images an auto-regressive fashion. this work we focus on models trained to "translate" images into text, or image captioning models. These models keep the visual encoder fv frozen and only train the connector C. Recent models also finetune the LLM to improve performance. Our approach can be applied in either case, and in our experiments find the generalization of LLMs to multimodal inputs is an interesting phenomenon to understand, thus focus more on the setup where the LLM is kept frozen.
consider both type of LMMs. However,
Transformer representations view Central to many previous approaches interpreting decoder- only LLM/transformer architectures, is the "residual stream view" of internal representations, first proposed in [3]. Herein, the network is seen as composition of various computational blocks that "read" information from the residual stream of token representations M[y perform their computation, and add or "write" their output back in the residual stream. This view can be summarized as:.
M(++1) h) + do m()
(n) denotes the information computed by attention function at layer I and position p. It has a causal structure and computes its output using h/y:.... BBy mLy denotes the information computed by the MLP block. It is a feedforward network (FFN) with two fully-connected layers and an intermediate activation function σ, that operates hi +o)-- The output of o.)) is referred to as FFN activations.
3.2 Method overview
Fig. D
provides a visual summary of the whole CoX-LMM pipeline. Given pretrained LMM f and token of interest / E y, our method consists of three key parts:
Selecting a subset of images X from dataset S, relevant for target token t. We extract representations by processing samples in X through f. The extracted representations of dimension B are collected in a matrix Z E R RxM, where M number of samples in X.
2. Linearly decomposing Z R UV into its constituents, that includes dictionary of learnt concepts U E BBBKKK of size and coefficient/actiiation matrix E K
3. Semantically grounding the learnt "multimodal concepts", contained dictionary in both visual and textual modalities.
We emphasize at this point that our main objective in employing dictionary learning based concept extraction is understand internal representations of LMM. Thus, our focus is validating the use of the learnt dictionary for this goal, and not to interpret the output of the model, which can be readily accomplished by combining this pipeline with some concept importance estimation method [1]. The rest of the section is devoted to elaborate on each of the above three steps.
3.3 Representation extraction
To extract relevant representations from the LMM about t that encode meaningful semantic infor- mation, we first determine a set of samples X from dataset S = {(X;, y/ y, for extraction. We consider the set of samples where is predicted part of the predicted caption ý. This allows to further investigate the model's internal representations of t. To enhance visual interpretability for the extracted concept dictionary, we additionally limit this set of samples to those that contain & in the ground-truth caption. Thus, X is computed as:
X (X; E ((X)) and i y, and (X,, E S).
Given any E X. we propose to extract the residual stream representation hi/,) from deep layer L. at the first position in the predicted caption p Nv. such that j t. The representation 23 E R

---

of each sample X, E X is then stacked as columns of the matrix Z = [21, _M] E R Bx M RBM, Note that the representations of text tokens in fI.M can possess meaningful multimodal structure as they combine information from visual token representations "1) I Nv. In contrast to d(j) and (i that represent residual information at layer I, Mi contains the aggregated information computed by the LMM till layer L, providing a holistic view of its computation across all previous layers.
3.4 Decomposing the representations
The representation matrix Z = UV, is decomposed as product of two low-rank matrices U E BBx V E RRKM of rank K << min( B, M), where K denotes the number of dictionary [M1,...,ux; ux] are the basis vectors which we refer to as concept-
elements. The columns of U vectors/coneepts. The rows of V or columns of V = ["1, ...; "x], 04 E TR denote the activations of u, for each sample. This decomposition, various constraints on U, V. each leading to a different dictionary. The most common ones include PCA (constraint: U = D, K-Means (constraint: columns of V correspond to columns of identity matrix) and NMF (constraint: U. V 2 0). NMF is considered to yield most interpretable results [14). However, for our use case, NMF is not useful as representation matrix Z is not non-negative. Instead, we propose to employ a relaxed version of NMF, Semi-NMF [TO]. which allows the decomposition matrix Z and basis vectors U to contain mixed values, and only forces the activations V to be non-negative. Note that given its relations to clustering algorithms [I0], enforcing non-negative combinations of decompositions is still valued from interpretability perspective. Since expect only a small number of concepts to be present in any given sample, we also encourage sparsity in activations V. The optimization problem decompose Z via Semi-NMF can be summarized as:
previously studied in [14,, can be optimized with
U" V = arg min U.V
Z - UV + A||VII 8.t. V 2 0, and ||u/l22 S 1 Ak E {l,...,K} (4)
Given any image X where token t is predicted by f, we can now define the process of computing activations of concept dictionary U for given X, denoted as v(X) E R To do so. we first extract the token representation for X, E R with the process described in Sec. 3.3 Then, 2x is projected on U to compute v(X). In the case of Semi-NMF, this corresponds to (X)= = arg min ||=x U' + A|[u||1. The activation of uj U is denoted as (X) R.
3.5 Using the concept dictionary for interpretation
Multimodal grounding of concepts. Given the learnt dictionary U* and corresponding activations V*, the key objective remaining is to ground the understanding of any given concept vector uk- k E -... K} in the visual and textual domains. Specifically, for visual grounding, we use prototyping 3 12 to select input images (among the decomposed samples), that maximally activate ug. Given the number of samples extracted for visualization NM the set of maximum activating samples (MAS) for component ug, denoted as XR,MA can be specified as follows (l.) is absolute value):
XRMAS
argmax
Σ (X)(
& XC XEX
For grounding in textual domain, we note that the concept vectors are defined in the token representa- tion space of JT.M- Thus we leverage the insights from "Lens" based methods (6] 24 32 40 that attempt to understand I.LM representations. In particular, following 132,, we use the unembedding layer to map uy to the token vocabulary space V, and extract the most probable tokens. That is, we extract the tokens with highest probability in Wo The decoded tokens with highest probabilities are then filtered for being an english, non-stop-word with at least characters, to climinate unneces- sary tokens. The final set of words is referred to as grounded words for concept 4y and denoted as Te. Fig. 2 Iillustrates an example of grounding of a concept extracted for token "Dog" in vision (5 most activating samples) and text (top 5 decoded words).
Most activating concepts for images. To understand the LMM's representation of given image X, we now define the most activating concepts. Firstly, we extract the representaions 2x of the image with the same process as described previously. We then project ex on U* to obtain o(X) E K We define the most activating concepts, which we denote ü(X), as the set of r concept vectors (in
5

---

'hite''
'light' 'luffy'
'oodde
'dog'
Figure 2: Example of multimodal concept grounding in vision and text. Five most activating samples (among decomposed in Zi and five most probable decoded words shown.
U*) whose activations vg (X) have the largest magnitude. One can then visualize the multimodal grounding of ū(X). This step could be further combined with concept importance estimation techniques [14 interpret the model's prediction for token f, however, the focus of this paper is to simply understand the internal representation of the model, for which the current pipeline suffices.
Experiments
Models and dictionary learning. In the main paper, focus experiments with the DePALM model [45 that is trained for captioning task on COCO dataset
(27). 'he model consists of a frozen
ViT-L/14 CLIP 33] encoder the visual encoder fy. It is followed by transformer connector to
compress the encoding into Ny = 10 visual tokens. The language model fLM a frozen OPT-6.7B
49 and consists of 32 layers. Additional experiments with LLaVA
[28] in Appendix A For
uniformity and fairness all the results in the main paper are reported with number of concepts K 20 and for token representations from L 31, the final layer before unembedding layer. For Semi-NMF, we set À = throughout. We consider the 5 most activating samples in XRMAS for visual grounding for any Ug. For text grounding, we consider top-15 15 tokens for Ta before applying the filtering described in Sec[3.5]
The complete dataset consists of around 120,000 images for training, and 5000 each for validation and testing with 5 captions per image following the Karpathy split. We conduct our analysis separately for various common objects in the dataset: "Dog". "Bus", "Train", "Cat", "Bear", "Baby", "Car", "Cake". The extension to other classes/tokens remains straightforward and is discussed in Appendix D The precise details about number of samples for learning the dictionary, or testing, is available in Appendix C
The implementation of our method is publicly available on GitHub 2
4.1 Evaluation setup
We evaluate the quality of learnt concept dictionary U on three axes: (i) Its use during inference to interpret representations of LMMs for test samples, (ii) The overlap/entanglement between grounded words of concepts in the dictionary and (ii)) the quality of visual and text grounding of concepts (used understand concept itself). We diseuss conerete details about each axis below:
Concept extraction during inference: To evaluate the use of U* in understanding any test sample X, we first estimate the top most activating concepts activations, ū(X) (Sec. (.5). We then estimate the correspondence between the test image X and the grounded words Ta of uu ū(X). This correspondence is estimated via two different metrics. The primary metric is the average CLIPScore 20 between and T₁. This directly estimates correspondence between the test image embedding with the grounded words of the top concepts. The secondary metric is the average BERT (F1) 50 between the ground-trutt captions y associated with X and the words Tg. These metries help validate the multimodal nature of the concept dictionaries. Their use is inspired from [T. Details for their implementation is in Appendix C
Overlap/entanglement of learnt concepts: Ideally, we would like each concept in U to encode distinct information about the token of interest f. Thus two different concepts Uk ut le & I should be associated to different sets of words. To quantify the entanglement of learnt concepts, we compute the overlap between the grounded words T2, Tr. The overlap for a concept ид defined as an average of its fraction of common words with other concepts. The overlap/entanglement metric for a dictionary
httt::///gtthub...om/ssskk//////

---

Token
Metric
BordWords
Nois
Simple
Semi-NMF (Ours)
GT-captions
CS top-1 (f)
0.519 + 0.05
0,425 + 0.06
0,546 + 0.08
0.610 + 0.09
0.783 + 0.06
Dog
BS top-1 (f)
0.201 + 0.04
0.306 1 0.05
0.346 + 0.08
0.405 1 0.07
0.511 1 0.11
CS top-1 (f)
0.507 + 0.05
0.425 + 0.08
0.667 0.06
0.634 + 0.08
0.736 + 0.05
Bus
BS top-1 (f)
0.200 + 0.05
0.303 + 0.06
0.390 0.05
0.404 + 0.07
0,466 + 0.11
CS top-1 (f)
0.496 + 0.05
0.410
0.07
0.642 + 0.06
0.646 1 0.07
0.727 + 0.05
Train
BS top-1 (f)
0.210 + 0.06
0.253 0.06
0.392 0.07
0.378 + 0.07
0.436 + 0.08
CS top-1 1 (1)
0.539 + 0.04
0.461 + 0.04
0.589 0.07
0.627 + 0.06 0.798 + 0.05
Cat
BS top-1 (1)
0.207 +: 0.07
0.307 1: 0.03
0.425 + 0.10
0.437 1 0.08
0.544 + 0.10
Table 1: Test data mean CLIPScore and BERTScore for top-1 activating concept for all baselines on five tokens. CLIPScore denoted as CS, and BERTScore as BS. Statistical significance is in Appendix
D
Our CoX-LMM framework is evaluated with Semi-NMF underlying dictionary learning method. Higher scores are better. Best score in bold, second best is underlined.
U defined as the average of overlap of each concept.
K Σ (K - 1) Im
T TA
Overlap(U") K Σ Overlap(ux), Overlap(ux)
T
To evaluate the quality of visual/text grounding of concepts (AS Tx), we measure the correspondence between visual and text grounding of given concept Uk. i.e. the set of maximum activating samples M and words Tr, using CLIPScore and BERTScore as described above.
Multimodal grounding of concepts:
One set of methods for evaluation are the variants CoX-LMM where we employ different dictionary learning strategies: PCA, KMeans and Semi-NMF. For evaluating concept extraction test data with CLIPScore/BERTSorr we compare against the following basclines:
Baselines:
the underlying learning method. However, for each component ukg we replace its grounded words Tr by a set of random words R2 such that R = [xx] and the random words also satisfy the same filtering conditions grounded words iee they are non-stopwords from english corpus with more than two characters. We do this by decoding a randomly sampled token representation and adding the top decoded words if they satisfy the conditions.
Word:W This bascline considers Semi-NMF
Nooise-mgs: This baseline uses random noise images and then proceeds with exactly same learning procedure as Semi-NMF including extracting activations from the same positions, and same parameters for dictionary learning. Combined with the End-Words baseline, they ablate two parts of the concept extraction pipeline.
Simple: This baseline considers a simple technique to build the dictionary U and projecting test samples. It builds U by selecting token representations in Z with the largest norm. The projections are performed by mapping the test sample representation to the closest element in U*. For deeper layers, this provides a strong baseline in terms of extracted grounded words Tr which are related to token of interest 4, as they are obtained by directly decoding token representations of t. We also report score using ground-truth captions (GT captions) instead of grounded words T,, get the best possible correspondence score. The overlap/entanglement between the non-random baselines: Simple, PCA, K-Means and Semi-NMF. For evaluating the visual/text grounding we compare against the random words kceping the underlying set of MAS, MA, same for both.
concept dictionary is compared
4.2 Results and discussion
Quantitative results Tab. Treports the test top-1 CLLPPcoor/BERTTSco for all baselines and Semi-NMF on different target tokens. Appendix D contains detailed results for other tokens as well as for the PCA and K-Means variants. We report the results for only the top-1 activating concept, as the KMeans and Simple baselines map given representation to single cluster/element.
7

---

Notably, Semi-NMF generally outperforms the other baselines although the Simple baseline performs competitively, More generally, Semi-NMF, K-Means, and Simple tend to clearly outperform Rnd- Words, Noise-Imge and PCA on these metrics, indicating that these systems project representations of test images to concepts whose associated grounded words correspond well with the visual content.
Tab. reppots the overlap between concepts for Simple base-
line and PCA. K-Means and Semi-NMF variants of CoX- LMM. Interestingly, K-Means and Simple baseline perform significantly worse than Semi-NMF/PCA with high over- Dog
Token
Simple
PCA
KMMnnn
Semi-NMF
0.371
0.004
0.501
0.086
lap between grounded words, often exceeding 40%. PCA outperforms other methods with almost no overlap while Semi-NMF shows some overlap. Overall, Semi- strikes the best balance among all the methods, in terms of learning concept dictionary useful for understanding test image representations, but which also learns diverse and disentan- gled concepts. Thus, for further CoX-LMM experiments, we consider Semi-NMF as the underlying dictionary learning method.
Bus
0.622
0.002
0.487
0.177
Train
0.619
0.015
0.367
0.107
Cat
0.500
Table 2: Overlap between learnt con- cepts. Lower is better. Best score in bold, second best underlined.
Fig. B] shows an evaluation of visual/text grounding of concepts learnt by Semi-NMF. Each point on the figure denotes the CLIPP Score (left) or BERTScore (right) for correspondence between sam- ples AA and words Ta for concept Un against random words baseline. We see that for both mer- rics, vast majority of concepts lie above the y = x line, indicating that grounded words correspond much better to content of maxi- mum activating samples than ran- dom words.
CUP Score
BERTIF1 Score
0.6
Grounded words
0.5
66- Ce
Grounder
0.
0.3
Gila
0.2
basy
0.1
0.1
0.2
words
Evaluating visual/text grounding (CLIP- Score/BERTScore) Each point denotes score for grounded words of a concept (Semi-NMF) vs Rnd-Words w.r.t the same visual grounding.
Figure 3:
Qualitative results Fig. 4 shows visual and textual grounding of concepts extracted for token 'dog". For brevity, select 8 out of 20 concepts for illustration. Grounding for all concepts extracted for 'dog' and other tokens are in Appendix The concept visualizationn/gruunnnng for
Example concepts for 'Dog'
'dog'
small
Sausaqu
Unn
hat"
(5) toupp
"sendenne
mimmetur
black
Tuny
"ange"
"hairy
"aakk"
(6)
"Nuffy
'tlg'
'long"
'close'
"uzzy"
brown
'cat'
larget
Sitten
'tiger
loolden
tong'
'dog'
'herd!
Yunning
Sheep
(4)
(8)
Tlock
'pay'
*orm!
'grese'
Sshepher
Figure 4: Visual/textual grounding for 8 out of 20 concepts for 'Dog' token (layer 31). For each concept we illustrate the set of 5 most activating samples and 5 most probable decoded words.
'Dog' reveal interesting insights about the global structure of the L.M''s representation. Extracted concepts capture information about different aspects of a 'dog'. The LMM separates representation
8

---

of animal 'Dog' with 'hot dog' (Concept 1). Specifically for 'Dog', Concepts (2). (3) capture information about color: 'black', 'brown'. Concept (6) encodes information about 'long hair' of a dog, while concept (5) activates for "small/puppy-like dogs. Beyond concepts activating for specific characteristics of 'dog', we also discover concepts describing their state of actions (Concept (4) "playing/runing") ), common scenes they can occur in (Concept (8), 'herd"), ) and correlated objects they can occur with (Concept (7), "cat and dog'). ) We observe such diverse nature of extracted concepts even for other tokens (Appendix E The information about concepts can be inferred via both the visual images and the associated grounded words, highlighting their coherent multimodal grounding. Notably, compared to solely visual grounding for CAVs for CNNs, the multimodal grounding eases the process to understand a concept.
Test cample
Tp corcept actiation.
Tap3 concept ocinatior
surple
Top-3 concest ectivtioss
BAL
Tooks
Tha
Slock
Figure 5: Local interpretations for test samples for different tokens ('Dog', 'Cat', 'Bus') with Semi- NMF (layer 31). Visual/text grounding for three highest concept activations (normalized) is shown.
Fig. 5 iillustrates the use of concept dictionaries (learnt via Semi-NMF to understand test sample representations for tokens 'Dog'. 'Cat" and 'Bus'. For each sample we show the normalized activations of the three most activating concepts, and their respective multimodal grounding. Most activating concepts often capture meaningful and diverse features of a given sample. For instance, for first sample containing a 'Dog', the concepts for "long hair", "small/tin/puppyy" and "black/white color" all simultaneously activate. The grounding for first two concepts was also illustrated in Fig. 4 Additional visualizations for test samples are shown in Appendix E. wherein we qualitatively compare interpretations of Semi-NMF to K-Means, PCA variants and Simple baseline.
Layer ablation We analyze the qual- ity of multimodal grounding of con- cepts across different layers L. The CLIPScore between (XR.MAS; Ts), av- eraged over all concepts un is shown in Fig. 6 for 'Dog' and 'Cat' for all layers in fLW. For early layers the multimodal grounding no better than nords. Interestingly, there is noticeable in- crease around (L- = 20 to 25), indi- cating that the multimodal structure of internal token representations starts to appear at this point. This also validates our choice that deeper layers are better suited for multimodal concepts.
Average CLIPScore for mutmmdda grounding
0.25
8800
0375
0520
0.25
0500
0.45
30
Lager
Figure Mean CLIPScore between visual/text grounding X MAS, Tç for all concepts (Semi-NMF), across differ- ent layers L. Results are for tokens 'Dog' and 'Cat'.
Additional experiments and discussion. We conduct a preliminary study to analyze the polyseman- ticity/superposition in concept dictionaries in Appendix B A qualitative analysis for grounding of extracted concepts for different layers is available in Appendix [ CoX-LMM can be also be applied to understand the processing of visual/perceptual tokens inside the LMM which also exhibit this multimodal structure. The experiment for the same can be found Appendix G Limitations of our method are discussed in Appendix H and the broader societal impacts are discussed in Appendix I

---

Conclusion
In summary, we have presented a novel dictionary learning based concept extraction framework, useful understand internal representations of large multimodal model. The approach relies on decomposing representations of a token inside a pretrained LMM. To this end, we also propose a Semi-NMF variant of the concept dictionary learning problem. The elements of the learnt concept dictionary are grounded in the both text and visual domains, leading to a novel notion of multimodal concepts in the context of interpretabiliity. We quantitatively and qualitatively show that (i) the multimodal grounding of concepts is meaningful, and (ii) the learnt concepts are useful to understand representations of test samples. We hope that method inspires future work from research community towards designing concept based explainability methods to understand LMMs.
Acknowledgments and Disclosure of Funding
We would like to thank Thomas Fel for his valuable comments on the paper. This work has been partially supported by ANR grant VISA DEEP (ANR-20-CH)) 20--CHHA---022,, and HPC resources of IDRIS under the file number ADOI 1014947, allocated by GENCI.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 16-2336,,,,,,, 2022.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736, 2022.
[3] David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 7775-7784, 2018.
[4] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 2023.
[5] Valérie Beaudouin, Isabelle Bloch, David Bounic, Stéphan Clémençon, Florence d' d'Alché- Buc, James Eagan, Winston Maxwell, Pavlo Mozharovskyi, and Jayneel Parekh. Flexible and context-specific Al explainability: A multidisciplinary approach. CORR, abs/2003.07703, 2020.
[6] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt, Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08// 2023.
[7] Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P Calmon, and Himabindu Lakkaraju. In- terpreting clip with sparse linear concept embeddings (splice). arXiv preprint arXiv:2402.1033 2024.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33: 1877-1901, 2020.
[9] Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What cannot predict, do not understand: human-centered evaluation framework for explainability methods. Advances in neural information processing systems, 35:2832-2845. 2022.
10

---

[10] Chris HQ Ding, Tao Li, and Michael Jordan. Convex and semi cemi-nonnegative matrix fac- torizations. IEEE transactions on pattern analysis and machine intelligence, 32(1):45-55, 2008.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010. 1929,, 2020.
[12] Constantin Eichenberg. Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. Magma-multimon augmentation of generative models through adapter-base based finetuning. arXiv 2021.
preprint
[13] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1, 2021.
[14] Thomas Fel, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems, 36, 2023.
[15] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for ex- plainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2711-2721, 2023.
[16] Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representa- tion via text-based decomposition. International Conference on Learning Representations, 2024.
[17] Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explanations. In Advances in Neural Information Processing Systems (NeurIPS), pages 9277-9286, 2019.
[18] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3): e30, 2021.
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[20] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, 2021.
[21] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-opting large language models. arXiv preprint 2rx.x:2203.155555 2022.
[22] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pages 2668 2677. PMLR, 2018.
[23] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried, Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.138223 2023.
[24] Anna Langedijk, Hosein Mohebbi, Gabriele Sarti, Willem H. Zuidema, and Jaap Jumelet. Decoderlens: Layerwise interpretation of encoder-decoder transformers. CoRR, 2023.

---

[25] Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL h/tpp://///perreieww net/forum? Sdd
[26] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprini arXiv:2301. /2597, 2023.
[27] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision (ECCV), pages 740-755, Springer, 2014.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2023.
[29] Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. Mapl: Purameter-efficient adaptation of unimodal pre-trained models for vision- language few-shot prompting, arXiv preprint arXiv:2210.07179, 2022.
[30] Anick F Markus, Jan A Kors, and Peter R Rijnbeek. The role of explainability in creating trustworthy artificial intelligence for health care: comprehensive survey of the terminology, design choices, and evaluation strategies. Journal of biomedical informatics, 1 113:103655, 2021.
[31] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint wrxi:2209.11162, 2022.
[32] Nostalgebraist. Interpreting gpt: The logit lens. https://www www. www.sweogg.co/oosseeeteeeoeoeeee com/posts/ baIRBBBunpaaarevrrrrr/ttttpeeepn thapptttteee--ttt-ttoe logit lens 2020. Accessed: [date of access].
the
[33] OpenAI. Gpt-4 technical report. arXiv, 2023.
[34] Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision-languae mechanistic interpretability: A causal tracing tool for blip. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2856-2861, 2023.
[35] Haowen Pan, Yixin Cao, Xiaozhi Wang, and Xun Yang. Finding and editing multi-modal neurons in pre-trained transformer. arXiv preprint arXiv:2311.074777 2023.
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037, 2019.
[37] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning python. the Journal of machine Learning research, 12:2825-2830, 2011.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 1166. PMLR, 2021.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748-8763. PMLR, 2021.
[40] Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian T. Foster. Attention lens: tool for mechanistically interpreting the attention head information retrieval mechanism. CoRR, abs/23 16270 2023.
12

---

[41] Sarah Schwettmann, Neil Chowdhury. Samuel Klein, David Bau, and Antonio Torralba. Multi- modal neurons pretrained text-only transformers. Proceedings of the IEEE/CVE Interna- lional Conference on Computer Vision, pages 2862-2867, 2023.
[42] Mustafa Shukor and Matthieu Cord. Implicit multimodal alignment: On the generalization of frozen Iims to multimodal inputs. arXiv preprint arXiv:2405.16700 2024.
[43] Mustafa Shukor, Corentin Dancette, and Matthieu Cord. ep-alm: Efficient perceptual aug- mentation of language models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 22056-22069, October 2023.
[44] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wwXi::2307.09288, 2023.
[45] Théophane Vallaeys, Mustafa Shukor, Matthieu Cord, and Jakob Verbeek. Improved baselines for data-efficient perceptual augmentation of llms. arXiv preprint arXiv:2403. 13499, 2024.
[46] Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimensional concept discovery (med): A unifying framework with completeness guarantees. Transactions on Machine Learning Research, 2023.
[47] Chih-Kuan Yeh, Been Kim, Sercan O Arik, Chun-Liang Li, Pradeep Ravikumar, and Tomas Pfis- On concept-based explanations in deep neural networks. arXiv preprint arXiv: 1910.07969, 2019.
[48] Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept-base based explanations for enn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11682-11690, 2021.
[49] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01065 2022.
[50] Tianyi Zhang, Varsha Kishore, Felix WU, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert, In International Conference on Learning Representations, 2019.
13

---

A Experiments on other LMMs
This section covers our experiments on other types of multimodal models. First, we test our approach on LLaVA to demonstrate that our approach generalizes to more recent networks that also fine-tune fLM on multimodal data. We also test our method on other variants of DePALM with non-CLIP visual encoders to observe their effect on CLIPScore.
A.1 Experiments with LLaVA
We conduct further experiments on LLaVA [28], a popular open-source LMM to demonstrate the generality of our method. The model uses LLIP-VIT--3336p visual encoder (fv), 2-layer linear connector (C) that outputs Ny = 576 visual tokens, and a Vicuna-7B language model (fLMs 32 layers). We use identical hyperparameters for DePALM (K 20, À L, = 31). We report the test CLIPScore for top-1 1 activating concept, for words, Noise Imes, Simple and Semi-NMF, GT-captions in Tab. 3 and Overlap score for non-random baselines in Tab. Quantitatively, obtain consistent results to those observed for DePALM. Semi-NMF extracts most balanced concept dictionaries with high multimodal correspondence and low overlap. Qualitatively too, the method functions consistently and is able to extract concepts with meaningful multimodal grounding.
Token
Metric
FordWords
Noise-Imgs
Simple
Semi-NMF (Ours)
GT-cappions
Dog
CS top-1 1 (f)
0.537 + 0.03
0,530 0.05
0,567 0.08
0.595 + 0.07
0.777 + 0.06
BS top-1 (f)
0.205 + 0.07
0.227 0.06
0.331 0.07
0.305 + 0.07
0.519 + 0.11
CS top-1 (f)
0.509 + 0.04
0.487 + 0.05
0.619 + 0.06
0.591 + 0.08
0.742 + 0.05
Bus
BS top-1 (f)
0.198 +: 0.07
0.253 +: 0.06
0.319 + 0.04
0.306 + 0.06
0.460 +: 0.10
CS top-1 (f)
0.518 0.03
0.505
0.04
0.633
0.05
0.640 1 0.07
0.725 1 0.05
Train
BS top-1 (f)
0.177 + 0.07
0.221 + 0.04
0.310 0.05
0.293 + 0.05
0.432 + 0.08
CS top-1 1 (f)
0.536 + 0.03
0.545 + 0.04
0.625 + 0.06
0.621 + 0.07
0.795 + 0.05
Cat
BS top-1 (f)
0.142 1 0.06
0.235 1 0.05
0.306 1 0.06
0.329 1 0.07
0.540 1 0.11
Table 3: Concept extraction on LLaVA-v1.5: Test data mean CLIPScore reported for top-1 activating concept for same baselines and tokens in main paper table 1. Higher scores are better. Best score in bold, second best is underlined.
Token
Simple
PCA
KMeans
Semi-NMF
Dog
0.435
0.008
0.429
0.149
Bus
0.464
0.010
0.518
0.124
Train
0.315
0.024
0,382
0.087
Cat
0.479
0.013
0.554
0.166
Table 4: Overlap evaluation (LLaVA). Lower is better. Best score in bold, second best underlined.
Qualitative results and Saliency maps We also show qualitative examples of concepts extracted for token 'Dog' in Fig. 7 More examples for other 'Cat' and "Train' tokens are given in Fig. and Interestingly, since LLa VA uses a connector C' that contains two linear layers, the visual tokens as processed inside JLM preserve the notion of specific image patch representations, i.e. Nv 576 tokens denoting representations for 576 (24 24) input patches. This allows to further explore simple and computationally cheap strategy of generating saliency maps to highlight which regions a concept vector activates on. To do this one can simply compute the inner product of any given concept vector ug with all visual token representations from corresponding layer L, i.e. uf [1] [1] kMyy) This can upscaled to the input image size to visualize the saliency map. We illustrate some qualitative outputs concepts from 'Dog' in Fig, g
10
14

---

small"
'brown'
'hair'
'golden'
'red'
"dog"
'tan'
'close'
orange
'black'
'dog'
German
'border
'large'
animal
'pit'
'col'
'big'
'dogs'
golden'
White'
yellow
white'
'span'
'dal'
'lght'
'dog"
Figure 7: Multimodal grounding for example concepts for 'Dog' token (layer 31) LLaVA.
'black' 'flying' "dog"
'dog'
'hot'
'running
'play'
Figure 8: Examples of generating visual concept saliency maps for two 'Dog' concepts for LLa Red denotes high activations, blue denotes low activation (bottom row)
A.2 DePALM with VIT visual encoders
We further test LMMs which do not contain CLIP visual encoder to confirm that the high CLIPScore not due to use of CLIP visual encoders. To test this, we experiment on two different DePALM models with frozen visual encoders different from CLIP, frozen ViT-L encoder trained ImageNet [ and another frozen ViT-L trained as a masked autoencoder (MAE) [19]. Both LMMs use the same pretrained OPT-6.7B language model. Collectively, the three encoders (including CLIP) are pretrained for three different types of objectives. We use Semi-NMF to extract concept dictionaries, with all hyperparameters identical. The results reported in tables below. 'Rnd- Ford- and "GT-captions' references are reported for each LMM separately, although they are very close to the ones in main paper. The "ViT-L (CLIP)" baseline denotes our system from the main paper that uses CLIP encoder. Importantly, we still obtain similar test CLIPScores as with CLIP visual encoder. The concept dictionaries still possess meaningful multimodal grounding. Many concepts also tend be similar as for CLIP visual encoder, further indicating that processing inside language model plays major role in the discovery of multimodally grounded concepts.
Token
Rnd-Words
VIT-L (ImageNet)
(CLIP)
GT-captions
Dog
0.514 0.05
0.611 0.09
0.610 + 0.09
0.783
Bus
0.498 + 0.05
0.644 + 0.07
0.634 + 0.08
0.739 + 0.05
Train
0.494 0.05
0.617 0.07
0.646 + 0.07
0.728 +
Cat
0.539 + 0.05
0.628 + 0.07
0.627 + 0.06
0.794 + 0.06
Table 5: Test CLIPScore evaluation for DePALM with VIT-L frozen image encoder trained on ImageNet: Scores reported for top-1 activating concept of Semi-NMF for Rnd-Words, GT-captions and ViT-L. (CLIP) which denotes the system in main text.
15

---

mmult
'computer
brown'
keyboerd
'strip''
monntorr
"cat"
'black'
white'
"selevision"
"cat
'laa''
animal
'remote"
hhirr
"cat"
furr
"monitor
"cat'
'cat'
pee'
'strip'
dog'
'dog'
blank
collree
drink
'tab'
'cat'
Gameos sers
animall
'suit
'dog'
travel
colored
'lug'
'pan"
"bag'
reflection
'park'
sicture"
Street"
mirrer"
window
'tree'
photo'
'buildiing
small
'gray'
baby
"grey"
young'
ttiy"
bblakk'
white'
sstrip'
"fakee
lay"
stuff
rest
picture'
'cat'
drawing
foat"
orange
"bath'
'ellow
'galden'
'kitchen
'cat'
'cat'
'brown'
'white'
Figure Multimodal grounding for example concepts for 'Cat' token (layer 31) on LLaVA.
Token
Rnd- Words
ViT-L (MAE)
VIT-L (CLIP)
GT-captions
Dog
0.515 + 0.05
0.602 0.07
0.610 + 0.09
0.784 = 0.06
Bus
0.501 + 0.05
0.627 + 0.07
0.634 + 0.08
0.737 = 0.05
Train
0.483 + 0.06
0.618 0.08
0.646 + 0.07
0.726 + 0.05
Cat
0.541 + 0.04 0.629 + 0.09
0.627 + 0.06
0.795 - 0.06
Table 6: Test CLIPScore evaluation for DePALM with ViT-L frozen image encoder trained as masked autoencoder (MAE): Scores reported for top-1 activating concept of Semi-NMF for und-Words, GT-captions and ViT-L (CLIP) which denotes the system in main text.
B Analyzing polysemanticity in the learnt concepts
We conducted preliminary qualitative study on some concept vectors in the dictionary learnt for token "Dog" (DePALM model), to analyze if these concept vectors tend to activate strongly for a specific semantic concept (monosemantic) or multiple semantic concepts (polysemantic). In particular, we first manually annotated the 160 test samples for "Dog" for four semantic concepts, for which we knew we had concept vectors our dictionary, namely "Hot dog" (Concept 2, row 1, column 2 in Fig. 7), "Black dog" (Concept 20, row 10, column 2 in Fig. 7), "Brown/orange dog" (Concept 6, row 3. column 2 in Fig. 7), and "Bull dog" (Concept 15, row column Fig. 7). For given semantic concept, call this set Crrue- Then, for its corresponding concept vector uş
16

---

bbrigge'
"city"
'colored
'street'
'electric
"bus'
ppasserige
tram'
'cab"
'stemm'
"cargo'
black'
'containe
'mokee"
'long'
historic
"tank'
'coal'
'load'
'tationn
termine
'Red'
platform
German
'usy''
'Errppaa
passenge
"tratn'
Thomas
bassenger
'brth''
elecctiic
'child'
colored
'fake'
enginee
'blue'
"city"
'sivver'
pubbicc
'white'
'bus'
'bullet'
'train'
pessennee
'lghh"
'train"
scet'
'train'
'bus
'model"
window
'engine'
'ben'
'layout"
chair
'version
'rust'
"gra'
woman
"dlitty'
'person"
broken"
'crowd'
woom
'group'
yelloo
"tour'
'train'
'small"
Britishi
'parke'
'bright'
'red'
ppsssenge
'ride'
Figure 10: Multimodal grounding for example concepts for 'Train' token (layer 31) on LLaVA.
find the set of test samples for which uı activates greater than threshold This threshold set to half of its maximum activation over test samples. We call this set of samples Ctop- To estimate specificity of the concept vector compute how many samples in Ctop lie in the ground-truie -truh set, i... |Ctopl n
We found Concept 2 ("Hot dog") to be most monosemantic with 100% specificity. For Concept 20 ("Black dog") too, we found high specificity of 93.3%. For concept 15 ("Bull dog") we observed the lowest specificity of 50%. This concept also activated for test images with toy/stuffed dogs. Interestingly, the multimodal grounding of concept 15 already indicates this superposition with maximum activating samples also containing images of 'toy dogs'. Concept 6 (" ("Brown/orange dog") is somewhere between, with 76% specificity. This concept vector also activated sometimes for dark colored dogs, which wasn't apparent from its multimodal grounding.
Prominent or distinct semantic concepts seem to be captured by more speciffi/monoosemntii concept vectors, while more sparsely present concepts seem at risk to be superposed resulting in a more polysemantic concept vector capturing them. also worth noting that the multimodal grounding can be useful tool in some cases to identify polysemanticity in advance.
17

---

Split
Dog
Bus
Train
Cat
Baby
Car
Cake
Bear
3693
2382
1529
Test
161
91
147
167
44
79
86
55
Table 7: Number of samples training/testing samples for each token for DePALM
C Further implementation details
C.1 Dictionary learning details
The details about the number of samples used for training the concept dictionary of each token, and the number of samples for testing is given in Tab. 7 The token representations are of dimension B 4096.
The hyperparameters for the dictionary learning methods are already discussed in the main paper. All the dictionary learning methods (PCA, KMeans, Semi-NMF) are implemented using scikit- learn (37]. For PCA and KMeans we rely on the default optimization strategies. Semi-NMP is implemented through the DictionaryLearing() class, by forcing positive code. utilizes the coordinate descent algorithm for optimization during both the learning of U*, V and the projection of test representations u( (X).
C.2 CLIPScore/BERTScore evaluation
For given image X and set of words Ty associated to concept uy, CLIPScore is calculated between CLIP-image embedding of and CLIPPtext embedding of comma-separated words in Tr. We consider maximum of 10 most probable words in each Tr, filtering out non-English and stop words. The computation of the metric from embeddings adheres to the standard procedure described in 220) Our adapted implementation is based on the CLIPScore official repository which utilizes the ViT-B/32 CLIP model to generate embeddings.
We found that computing BERTScores from comma-separatee words and captions is unreliable. Instead, adopted method using the LLMMA-3-8B instruct model to construct coherent phrases from a set of grounded words, Tg- Specifically, we provide the LLaMA model with instructions to describe scene using a designated set of words, for which also supply potential answers. This instruction is similarly applied to another set of words, but without providing answers. The responses generated by LLaMA and an example of the output are detailed in The highest matching score between the generated phrases and the captions of test sample determines the score assigned to the concept ug- This approach ensures that the evaluation accurately reflects coherent and contextually integrated language use. The metric calculation from embeddings follows the established guidelines outlined in (50]. Our adapted implementation is based on BERTScore official repository and we use the default Roberta-large model to generate embeddings.
then compared the captions using BERTScore. The instruction phrase
C.3 Resources
DePALM experiments compute Each experiment to analyze token with selected dictionary learning method is conduced on single RTX5000 (24GB) RTX6000 (48GB) TITAN-RTX (24GB) GPU. Within dictionary learning, generating visualizations and projecting test data, the majority of time is spent in loading the data/models and extracting the representations. For analysis of a single token with x 3000 training samples, it takes around 10-15 mins for this complete process. Evaluation for TT.PPScor/BERRTTSS 500 (image, moounded-words pairs takes around mins. The BERTScore evaluation is in contrast. more expensive, consuming around 150 mins for 500 pairs.
are also conducted using the same resources. Evaluating CLIPScore for
L.LaVA VA experiments compute Each experiment to extract concept dictionary for LLaVA was conducted single A100 (80GB) GPU. Representation extraction process for LLaVA is more
18

---

Table Generating contextually and grammatically coherent phrases Using the LLaMA Model for BERTScore Evaluation
Instruction to LLaMA
Nature of Response
Generate three distinct phrases, each incorporating the words dog, white, people. Ensure each phrase is clear and contextually meaningful. Number each phrase as follows: 1. 2. 3.
1. white dog is standing next to people. 2. People are standing next to a white dog. 3. The dog is standing next to people wearing white clothes.
Generate three distinct phrases, cach incorporating the words List of words Ensure each phrase is clear and contextually meaningful. Number each phrase as follows: 1. 2. 3.
LLaMA autonomously creates relevant description, demonstrating comprehension and creative integration of the new words.
Generate three distinct phrases, each incorporating the words brown, black, large, fluffy, cat Ensure each phrase is clear and contextually meaningful. Number each phrase as follows: 1. 2. 3.
1. A large, fluffy black cat is sleeping on the brown couch. 2. The brown cat is curled up next to large, fluffy black cat. 3. The large, fluffy cat's brown fur stands out against the black background.
expensive compared to DePALM consuming around 90 mins for 3000 samples. The remaining aspects of dictionary learning, multimodal grounding, representation projection etc. remains relatively cheap. The CLIPScorr/BERRTSScn evaluations are completed with same resources as before.
Licenses assets The part of the code for representation extraction from LMM implemented using PyTorch [36]. For our analyses, we also employ the OPT-6.7B 7.B model 49 from Meta AI, released under the MIT license, and the CLIP model (39 from OpenAI, available under custom
usage license. Additionally, the COCO dataset (27 used for validation is accessible under the Creative Commons Autribution 4.0 License. We also use CLIPScore (20] and BERTScore (50) for evaluating our method, both publicly released under MIT license. All utilized resources comply with their respective licenses, ensuring ethical usage and reproducibility of our findings.
C.4
Choice for number of concepts K
Our choice of using 20 concepts for all tokens was driven by the behaviour of reconstruction
error of Semi-NMF the training samples with different values of K, i.c ||Z UV 1|.. We validate this behaviour on the four target tokens from main text in Fig. 11 We generally found K = 20 as the minimal number of concepts where the reconstruction error drops by at least 50% from K 0.
We also conducted an ablation study to test how our method behaves with different values of number of concepts K. Fig. [22 presents the variation of test CLIPScore and Overlap score for K E (10, 20, 30, 50) for two target tokens, 'Dog' and 'Cat'. Our method can learn meaningful concepts for different values of K, evident by the consistently high CL.PPScore. The Overlap score on the other hand tends to drop more consistently as the number of concepts increase but behaves stably for different choices. Nonetheless they indicate that our method can accommodate dictionaries of larger sizes without compromising the quality of learnt concepts, provided << A (number of decomposed samples) and at the expense of greater user overhead.
D Evaluation and extension to more tokens
provide test data mean CLIPScore and BERTScore for top-1 activating concept for all baselines and more tokens: Baby, Car, Cake, and Bear in Tab. p (results in the main paper are reported for tokens Dog, Bus, Train, Cat Tab. [T]. Additionally, we also report the maero average over a set of 30 additional COCO-nouns apart from the tokens with individually reported results, denoted
19

---

(Test) Reconstruction error by size of dictionary and Category
doy
cat
80
train
40
30
20
20
Number of Concepts K
Figure 11: Variation of reconstruction error with number of concepts K for decompositions different target tokens.
Average Clip Score by Components and Category
Average Clip Score
0.62
0.60
dog Avg Clip Score cat Avg Clip Score
0.58
10
30 Number of Concepts K
20
50
Average Overiap Score
Average Overlap Score by Components and Category
dog Avg Overlap Score cat Avg Overlap Score
0.2
0.1
10
20
50
Number Concepts
Figure 12: Test CLIPScore and Overlap score ablation with number of concepts K. CLIPScore remains consistently high and drops slightly only for very small K. Overlap score generally improves with higher K.
'Extra-30'. These nouns single-token words with at least 40 predicted test samples. We put the filter of single-token words to keep consistency with the presented framework. Extension to multi-token words is straightforward but discussed separately in D.I The lower bound criterion on test samples to ensure average test CLIPScore reliable for each target token. We only report CLIPScore for 'Extra-30' tokens BERTScore evaluation more expensive to conduct large number dictionaries.
We observe that we consistently obtain higher scores across for Semi-NMF and K-Means. We also report the overlap score between grounded words in Tab. 10 to illustrate the superiority of our method over the simple baseline. As previously noted, observe high overlap between grounded
20

---

words with KMeans and Simple baselines compared to Semi-NME/PCC. A low overlap should be encouraged, as indicates the discovery of diverse and disentangled concepts in the dictionary. Among all the methods, Semi-NMF provides the most balanced concept dictionaries which are both meaningful (high CL.PScore/BERTT SSoree) and diverse (low overlap).
Token
Metric
Knd Words
Noise Lins
Simple
PCA (Curs)
KMeuss (Ours)
Semi NMP (Cuus)
GT-captions
CS top-1 cf)
0.199 0.05
0.425 + 0.06
0.546 + 0.08
0.559 = 0.06
0,599 + 0,07
0.610 + 0:09
0,783 + 0.06
Dog
BS top-1 (1)
0.201 0.04
0,306 +: 0.05
0.346 0.08
0.353 0.10
0.398 :: 0.06
0,405 = 0.07
0.511 = 0.11
CS lop-1
0.307 0.05
0,425 + 0.08
0,667 0.06
0.309 0.05
0.645 + 0.08
0.634 + 0.08
0.736 0.05
Bus
BS top-1 (t)
0.200 0.05
0.303 +: 0.06
0,500 +: 0.05
0.380 = 0.13
0:401 + 0.06
0.404 = 0.07
0.466 = 0.11
CS lop-1 (f)
0.496 0.05
0,410 +: 0.07
0.642 0.06
0.554 = 0.08
+ 0.06
0.646 = 0.07
0.727 = 0.05
Train
BS (1)
0.210 = 0.06
0,253 +: 0.06
10,922 + 0.07
0.334 0.09
0,375 - 0.07
0.37% = 0.07
0.436 =: 0.08
CS top-1 (1)
0.539 0.04
0.461 + 0.04
0.589 + 0.07
0:541 0.08
0.608 = 0.08
0.627 = 0.06
0.798 = 0.05
Ci
top-1 (f)
0.207
+ 0.03
0.425
0.13
0.08
0.08
0.544
CS top-1 (f)
0,532 0.04
0.471 + 0.05
0.631
0.575 0:07
+ 0.05
0.621 = 0.06
0.811 =
Baby
BS top (t)
0.192 0.03
0,379 d: 0.05
0,471 + 0.06
0.338 0.06
0,405 +: 0.07
0.426 +: 0.07
0.530 :: 0.14
CS top (t)
0.518 0.05
0.461 4: 0.08
0.605 + 0.04
0.547 0.08
0,602 4: 0.05
0.614 =: 0.05
0.766 +: 0.06
BS top-1
0.192
0,336 + 0.05
0.448 0.07
0:370 0.10
0.435 + 0.08
0:08
0.485
CS top
0.188
0.08
0.611
0.06
0.772
Cake
BS top-1 (t)
0.186 = 0.04
0.366 + 0.07
0,375 + 0.10
0.243 0.08
0.334 + 0.07
0.334 = 0:08
0.414 = 0.13
CS top-1 (t)
0.526 =: 0.04
0,526 +: 0.06
0.611 + 0.04
0.564 = 0.06
0.680 + 0.05
0.660 = 0:07
0.798 =:: 0.06
Rear
BS (f)
0.255 0.10
0.396 1: 0,08
0.434 0.05
0.420 0.10
0.474 4: 0.08
0.494
0.10
0.541 c: 0.10
Extro-30
CS top-1 (f)
0.566
0,521 + 0.03
0.626
0.547 0:06
+: 0.06
0.631
0.96
0.763
Table 9: Test data mean CLIPScore and BERT for top-1 activating concept CLIPScore denoted as CS, BERTScore denoted as BS for all concept extraction baselines considered. Analysis for layer L = 31. Best score indicated in bold and second best is underlined. "Extra-30" denotes a set of 30 additional single-token COCO nouns (apart from previous 8) with least 40 predicted test samples by JLM- For 'Extra-30' tokens we report the macro average and standard deviation of mean test data CLIPScore, taken over the set of 30 tokens.
Token
Simple
PCA
KMeans
Semi-NMF
Baby
0.645
0.006
0.502
0.187
Car
0.246
0.001
0.322
0.097
Cake 0.415
0.005
0.398
0.147
Bear
0.556
0.002
0.360
0,203
Extra-30
0.443
0.050
0.452
0.156
Table 10: Overlap/entanglemmet between grounded words of learnt concepts for different dictionary learning methods. Results are for additional tokens. Lower is better. Best score indicated in bold and second best is underlined. For 'Extra-30' tokens (additional 30 single-token COCO nouns) we report the macro average of Overlap score, taken over the set.
Statistical significance The statistical significance of Semi-NMF w.r.t all other baselines and evaluation to understand representations of test samples is given
variants, for CLIPScore/BERT Tab/ TT (for all tokens separately). We report the p-values for an independent two sided T-test with null hypothesis that mean performance is the same between Semi-NMF and the respective system. The results for Semi-NMF are almost always significant compared to Rnd-Words Noise-Imgs, PCA. However for these metrics, Simple baseline, K-Means and Semi-NMF all perform competitively and
21

---

better than other systems. Within these three systems the significance depends on the target token, but are often significant in many cases.
Token
Metric
Rnd Winds
Nobe lonse-Imgs
Simple
PCA
KM
GT-exptions
CS top-4
0.901
<0.001
00001
0.001
0.1
<0.001
BS up-1
0,001
<0.001
0.001
0.001
<0.001
CS
0,001
<0.001
0.002
0.001
<0.001
Dus
top-1
5.1
lop-1
0.001
0.000
00.
0.001
0.001
Tran
BS top-1
0.001
<0.001
0.08
0.001
0.1
0.001
CS top-1
0.0001
<0.001
00001
00000
0,018
0.001
wp-1
2.0.1
top-1
00,001
0.001
Haby
BS up-1
0.001
0.001
0.001
0.001
0.1
0.001
CS top-1
0.001
<0.001
0.1
0.000
0.1
0.001
wp-1
0.001
lpp-1
0.001
Cake
BS top-1
0.001
0,005
0.003
0.001
>0.1
0.001
CS lop-1
0.001
0.001
2.0
00001
0.072
0.001
0,000
0.0001
0.0001
0.0001
0.015
Table 11: Statistical significance of Semi-NMF w.r. other baselines for test data CLIP- Score/BERTScore. p-values for two sided T-test are reported. Significant values (p-value < 0.05) are indicated in bold. The values do not indicate which system has better mean score but just that if the difference is significant.
D.1 Extending to multi-token words
The presentation of our approach assumes that our token of interest t is single token. This poses no issues for words which are represented as single tokens but can raise some questions when we wish to extract concept representations for multi-token words. Our approach however, can be easily adapted to this setting. In particular, we extract representation of last token from first prediction of the multi-token sequence. Note that when filtering the training data for samples where ground-truth caption contains the token of interest, we now search for the complete multi-token sequence. The other aspects of the method remain unchanged. While there can also be other viable strategies, the rationale behind this adaptation that the last token our sequence of interest can also combines representations from previous tokens in the sequence. We add below results for such examples in Tab. 12 and 13) We observe behaviour consistent with the previous results with Semi-NMF extracting concept dietionaries with high CLIPScore and low overlap.
Multi-token word
Rud- Words
Noise-Imgs
Simple
Semi-NMF
GT-captions
Traffic light
0.516 0.03
0.525 0.03
0.664 1 0.06
0.634 1: 0.05
0.744 1 0.04
Cell phone
0.542 0.04
0.547 + 0.03
0.598 +: 0.04
0.598 + 0.05
0.765 4: 0.06
Stop sign
0.533
0.03 0.549
0.03
0.617 + 0.08
0.616 +: 0.05
0.775 + 0.04
Table 12: Test mean CLIPScore (t) reported for top-1 activating concept for multi-token words. Higher scores are better. Best score in bold, second best is underlined.
22

---

Multi-token word
Simple
PCA
K-Mcans
Semi-NMF
Traffic light
0.704
0.050
0.579
0.174
Cell phone
0.623
0.051
0.746
0.164
Stop sign
0,461
0.058
0.704
0.109
Table 13:
Overlap score ()) reported for top-1 activating concept for multi-token words. Higher scores are better. Best score in bold, second best is underlined.
E
Additional visualizations
E.1
Concept grounding
The visual/textual grounding for all tokens in Tab. are given in Figs. T3 ('Dog',, 14 ('Cat'),
("Bus"). [ ("Train"). All the results extract K = 20 concepts from layer L = 31. Similar to our analysis for token 'Dog' in main paper, for variety of target tokens our method extracts diverse and multimodally coherent concepts encoding various aspects related to the token.
E.2 Local interpretations
Here, we qualitatively analyze the local interpretations of various decomposition methods, including PCA, k-means, Semi-NMF, and the simple baseline strategy. We select these four as they produce coherent grounding compared
and Noise-Ing baselines. We decompose test sample representations on our learnt dictionary and visualize the top three activating components, Note that in the case of KMeans and Simple baseline, the projection maps given test representation element of the concept dictionary, the one closest to it. However, for uniformity we show the three are dedicated interpretations of
single
most closest concept vectors for both. Figs. 17 single sample each, for all four concept extraction methods.
18 19
20
The inferences drawn about the behaviour of the four baselines from quantitative metries also be observed qualitatively. Semi-NMF, K-Means and 'Simple' baseline, are all effective at extracting grounded words be associated to given image. However, both K-Means and 'Simple' display similar behaviour in terms of highly overlapping grounded words across concepts. This behaviour single concept/cluster. This
likely arises due to both the baselines mapping given representation limits their capacity to capture the full complexity of data distributions. In contrast, Semi-NMPF and PCA utilize the full dictionary decompose given representation and thus recover significantly more diverse concepts. PCA in particular demonstrates almost no overlap. likely due to concept vectors being orthogonal. However, the grounded words for it tend be less coherent with the images. As noted previously, Semi-NMF excels as the most effective method, balancing both aspects by extracting meaningful and diverse concepts.
Qualitative analysis for different layers
We provide qualitative comparison of multimodal grounding for the token 'dog' across different layers in Fig. 22 As observed in Fig. (main paper), the multimodal nature of token representations for two tokens "Dog" and 'Cat" starts appear around layers L 20 to L 25. It is interesting to note that the representations of images still tend to be separated well, as evident by the most activating samples of different concepts. However, until the deeper layers the grounded words often do not correspond well to the visual grounding. This behaviour only appears strongly in deeper layers.
G Analysis for visual tokens
Our analysis in main paper was limited decomposing representations of text tokens in various layers of an LLM, bu P Ny. This was particularly because these were the predicted tokens of the multimodal model. Nevertheless, the same method can also be used to understand the information
23

---

Sausage ToE "laree
wuiite "border"
"maal" Vnimotorr
"dlg"
"hrowN "dog
tplay"
pupppy
'hag'
"hendd "tarm"
satuffed
'reea" "iving rdog
'ratecttion
"Vew""
"halry "Tuzay"
'itt 'dog'
"eult"
"Tikee 'motorcycion
"eomann tean igir
hat" poocket
"ourt"
targe"
"aall"
"chse'
Figure 13: multimodal concept grounding in vision and text for the token 'Dog'. The five most activating samples and the five most probable decoded words for each component ug, & E {1,.... are shown. The token representations are extracted from L=31 of the L.LM section of our LMM.
stored in the visual/perceptuul tokens representations processed in SLM, ( S Ny. An interesting aspect worth highlighting is that while the text token representations in fLM can combine information from the visual token representations (via attention function), the reverse is not true. The causal processing structure of fLM prevents the visual token representations attend any information in the text token representations. Given a token of interest f, for any sample X E X we now only search for first position p E f1...., Nv), s.t. t arg max Unembed(h" (k())) Only the samples for which such a exists are considered for decomposition. The rest the method to learn U*, V proceeds exactly as before.
We conduct small experiment to qualitatively analyze concepts extracted for visual token repre- sentations for 'Dog'. We extract K 20 concepts from L = 31. The dictionary is learnt with representations from M 1752 samples, less than M 3693 samples for textual tokens. As brief illustration, 12 out of 20 extracted concepts are shown in Fig. 23 Interestingly, even the visual token representations in deep layers of fr.M. without ever attending any text tokens, demonstrate multimodal semantic structure. It is also worth noting that there are multiple similar concepts that appear for both visual and textual tokens. Concepts 3, 7, 10. 12, 17, 19 are all similar visually and textually to certain concepts discovered for text tokens. This indicates to possibility that

---

verppedd
beeeraa "pant"
"autwooo"
gellon "iinger
Teldewaa all""
tral" vetrppdd
"small" "baby"
ccmmuter"
"caat "Mlockk"
"koybooadd" "cal"
heddoomm
"huutty
"pea" "oocch"
"caff
Teatheoom "DoleT
"nuut
"EMM 'altchen'
"tore"
'suitcase'
"Mlack" "clese"
baag gaggage
"pali"
tong
leadt
large"
"alack""
white'
"luke"
"he"
"tay"
Figure 14: multimodal concept grounding in vision and text for the token 'Cat'. The five most activating samples and the five most probable decoded words for each component ug, k E {1, 20, are shown. The token representations are extracted from L=31 of the LLM section our LMM.
these concepts are discovered by fLM in processing of the visual tokens and this information gets propagated to predicted text token representations.
H Limitations
We list below some limitations of our proposed method:
The concept dictionaries extracted currently are token-specific. It can be interesting to explore learning concept dictionaries that can encode shared concepts for different tokens.
The current study is conducted mainly on visual captioning models. While we expect the key ideas to generalize many other types of large multimodal models, this application of our approach to other types LMMs remains to be explored/confirmed.
We select the most simple and straightforward concept grounding techniques. Both visual and textual grounding could potentially be enhanced. The visual grounding can be improved by enhancing localization concept activation for any MAS or test sample. Text grounding could be enhanced by employing more sophisticated approaches such tuned lens (6].
25

---

WRY Tag
Spurts:
valich
vas
Teat
"heas"
Preana
dec Mecorated stodd
"pessenger
BUS STOP
wiy
"our"
Teamsil
Figure 15: multimodal concept grounding in vision and text for the token 'Bus'. The five most activating samples and the five most probable decoded words for each component ug. k E {I...., 20} shown. The token representations extracted from L=31 of the LLM section of our LMM.
While the proposed CLIPScore/BERRTScore metrics are useful to validate this aspect, they are not perfect metrics and affected by imperfections and limitations of the underlying models extracting the image/text embeddings. The current research for metrics useful for interpretability remains an interesting open question, even more so in the context of LLMs/LLMM.
I Broader societal impact
The popularity large multimodal models and the applications they are being employed is growing at extremely rapid pace. The current understanding of these models and their representations limited, given the limited number of prior methods developed to understand L.Ms.. Since interpretability is generally regarded in real world, we expect our method to have a positive overall impact. This includes its usage for understanding LMMs, well as encouraging further research in this domain.
important trait for machine learning/Al models deployed
26

---

steam trale
station 'allway'
Taank" "austty 'dummp''
rengine
teaii yollow
Weteer Imountain "train"
ttraker ppacoing"
'tay' wrthday
deoruay
"Couc""
man" aabbeay "erass"
White passenger
"oor Ecnorrddo
iamurement
Ioletforan
Tourlee
couple 'set
'ailrad
villnge
ventage
"Hue" yellow
Iileorrcaa
Figure 16: multimodal concept grounding in vision and text for the token 'Train'. The five most activating samples and the five most probable decoded words for each component ug, & E [I,..., 20, are shown. The token representations are extracted from L=31 of the LLM section of our LMM.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose novel dictionary learning based concept extraction framework for understanding Large Multimodal Model (LMM) representations (Sec. 3 We extend
previous dictionary learning-base -baeed concept extraction methods by proposing Semi-NMF based optimization (Sec. (3). The learnt concept dictionary has multimodal semantic structure which we qualitatively and quantitatively validate with our experiments (Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
27

---

Semi-NMF
PCA
Towper
The
K-Means
Simple
Figure 17: Local interpretations for test sample of token 'Dog' with SemiNMF, KMeans, PCA, and Simple baselines (layer 31). Visual/text grounding for the three highest concept activations (normalized) shown. SemiNMF baseline provides the most visually and textually consistent results, while other baselines provide components that not well disentangled (Simple and KMMaan baseline), or the text grounding not closely related to the test image.
Semi-NMF
PCA
K-Means
Simple
Figure 18: Local interpretations for test sample 37 of token 'Dog' with SemiNMF, KMeans, PCA, and Simple baselines (layer 31). Visual/text grounding for the three highest concept activations (normalized) is shown.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings. is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations discussed separately in Appendix H
Guidelines:
28

---

Semi-NMF
PCA
Yonal
K-Means
Simple
Figure 19: Local interpretations for test sample 43 of token 'Cat' with SemiNMF, KMeans, PCA, and Simple baselines (layer 31). Visual/text grounding for the three highest concept activations (normalized) is shown.
Semi-NMF
PCA
Our
K-Means
Simple
Figure 20: Local interpretations for test sample 6 of token 'Bus' with SemiNMF, KMeans, PCA. and Simple baselines (layer 31). Visual/text grounding for the three highest concept activations (normalized) is shown.
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution
29

---

Semi-NMF
PCA
Teier
My
Voder
K-Means
Simple
Figure 21: Local interpretations for test sample 12 of token 'Bus' with SemiNMF, KMeans, PCA, and Simple baselines (layer 31). Visual/text grounding for the three highest concept activations (normalized) is shown.
Tarmayy
public
'good"
"sight
'commun
"poapie""
"high"
'couple
"hard"
'time!
"small
Tuny"
"nigh"
lll
'ong'
"Imme'
"hard"
'house'
"great"
L1
L12
fariily"
"blackk"
'comunityy
"dark"
market
'dog"
ppoopee'
Tange
"dag"
'person'
'dogs'
'gruup'
"running
'couple'
"play"
"raas
'mle'
"bult
cross
"glont"
pumpy
"alg"
Type'
Thuge
phuto"
L00
L 30
Figure 22: Examples of multimodal grounding across different layers for concepts with similar visual grounding (target token 'Dog'). The grounded words from early layers do not correspond well to the most activating samples a concept.
is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover
30

---

Example concepts for 'Dog' from visual token representations (layer 31)
Corcest U dogg ta11
1dog
Trunning
brown
"chesing
dog'
tesping
"cacahing
white
(dog,
'log'
'dog'
ttll
"ywwn
"oet
"tay'
curled
60ogg (-1)
Concest
dao'
golden
"willing
ccated
guarding
towa
breed
Concapt idog L:11
Suardi
Concapt 24 (dog 1:31
tail
German
/dog La5)
hand
'suppy
chesing
Shoup'
ddog
coomnng
Cansupp (dog, LS3
'dog'
Small
walk
compani
'pupp
Figure 23: Example concepts extracted for 'Dog" from visual token representations in layer 31.
limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper does not posit any theoretical results. The paper primarily presents empirical evidence demonstrating the effectiveness of the method rather than focusing on theoretical results. approach highlights the practical applicability and robust performance of the method through a series of experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
31

---

Justification: We have disclosed complete information about our pretrained model, extraction of representations, parameters. and implementation train the dictionary, evaluation setup, and resource usage for the experiments. Details are in Sec. 4 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g.. with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is available allatpee//gichuu..co////suuoo//s
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
mublic/guides Poodeeubmissioooolieee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https www.b//i//guiieee/oeeeutuuueeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
32

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Major details are in Sec. and all additional details in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We show standard deviation of the key metrics Tab. [and Tab. and discuss
the statistical significance comparing baselines and our method in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are in Appendix
C
33

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed them and adhere to them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The socictal impacts are discussed separately in Appendix !
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
34

---

Answer: [NA]
Justification:
We work with models and data already publicly available.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Details available in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We publicly release code and documentation is available alongside it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
35

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "4nGaA0KUJF", "reviewer": "Reviewer_8A1z", "summary": "Authors propose to look at the representation of chosen concepts, in multimodal models. They test different automated methods to learn decompositions of a token representations (which they then linearise in a dictionary of concepts). They also provide quantitative and qualitative analysis of a few examples, showing that this method clusters different human-interpretable meaning or usages of a same concept, for both visual and textual modalities.", "questions": "Mostly a personal curiosity question: Are there any intuitions from this work allowing to disentangle the data effect of the apparition of the studied concepts from the model/training effect?", "limitations": "Sample experiments are done on very simple words and concepts, available at pre-training for the tested model. How well this method would adapt to more complex concepts and images is not studied, but is where a lot of interpretability becomes necessary. This is nonetheless a necessary first step in an exciting direction.\nDo you have an intuition on how well this method would scale to this?", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Great qualitative analysis\nMultiple representation decomposition algorithms are tested, including more well known PCA or very specific but better suited Semi NMF, which is an adaptation to practical realities.\nEvaluation of the method is quite extensive, using a wide range of relevant state of the art interpretability tools to verify their work theoretically and practically.\nEffort is made to put quantitative metrics on concepts and to evaluate very abstract semantic analysis.\nProvided examples are clear and strike curiosity.", "weaknesses": "I worry there is a circularity when evaluating with ClipScore a frozen model with a CLIP component. \n(and even with BERT scoring, where you evaluate an interpretability metric with a non interpretable system of somewhat similar complexity).\n\n\n(Minor)\nVery recent works are relevant to the interpretability of LLM / Multimodal models (Templeton, et al., \"Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet\", Transformer Circuits Thread, 2024 AND Gao, Leo, et al. \"Scaling and evaluating sparse autoencoders.\" arXiv preprint arXiv:2406.04093, 2024) and could be cited.\n\nLine 223 \"the correspondance between a the image\" is probably a typo and should be fixed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and intriguing questions. We respond to them pointwise below: \n\n**CLIPScore and LMM with frozen CLIP encoder:** We believe the meaningful grounding is influenced more by the language model. We conducted experiments with two non CLIP visual encoders to support this hypothesis. Please find further details in our global response. \n\n**Very recent works on LLM interpretability:** Thanks, we make a note of them.\n\n**Line 223 typo:** Thanks for pointing out the error. We’ll fix it.\n\n**Disentangling data effect/model effect:** Indeed, the question of the impact of the available data vs the model architecture/training, w.r.t the concepts, is interesting. Qualitatively, in our initial experiments we observed cases where the type of input data clearly affected the quality of extracted concepts. For instance, when extracting concepts for token 'light', we observed most concepts only contained images of 'traffic light', even though we expected more diversity. This was because most input images in our dataset with predicted token 'light' were of 'traffic light'. \n\nOn a separate note, it could be worth tracking the evolution of the learnt concepts of our approach during training. This could possibly help to disentangle the training/optimization effect. But we leave that as a future research direction to pursue. \n\n**Expanding to more complex concepts:** Yes, this is indeed an important direction we are considering to expand in future. One possible way to extend to more complex or abstract words could be to associate the complex word to a set of target tokens. We could then consider decomposing representations for the all tokens in the set simultaneously. Our first instinct is to use a linear decomposition approach. Nevertheless, if this is not sufficient, non-linear decomposition methods could also be explored.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oemfG7o2ri", "reviewer": "Reviewer_Go37", "summary": "The authors propose using dictionary learning to extract concepts from multimodal models and simultaneously ground them in the text and image latent space. They draw on prior work on multimodal neurons and concept activation vectors. The authors provide quantitative results using CLIPScore and BERTScore to measure concept extraction, multimodal grounding, and concept overlap.", "questions": "In table 2, why is PCA by far the best method for minimizing overlap between learned concepts? Presumably because it’s PCA on only these five concepts, meaning that variance is maximized between only the five concepts and not between the dictionary more broadly?\n\nDo you observe any evidence of polysemanticity, as reported in prior work including Goh et al.?", "limitations": "I don’t see many negative societal implications of this research.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Appears to be mathematically correct and well-grounded in prior work.\nWell-written and well-illustrated.", "weaknesses": "Evaluates only a few tokens (dog, bus, train, cat) thoroughly, which seems like not enough when prior work (Goh et al.) studies thousands of concepts.\nEvaluates only one model, DePALM.\nUses two automated metrics, CLIPScore and BERTScore, but no human evaluation. A little worrisome that DePALM uses a CLIP-ViT-L14 and CLIPScore is the primary evaluation for the method.\nOverlaps significantly with prior research (Goh et al, Kim et al); multimodal neurons have been described in work dating back several years using non-negative matrix factorization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address their concerns pointwise below:\n\n**Evaluates only on DePALM:** We have now also conducted experiments on LLaVA. We are able to extract meaningful multimodal concepts and achieve quantitatively consistent results as for DePALM. Further details regarding the same are available in the global response and 1-page PDF. \n\n**Uses two automated metrics, but no human evaluation:** Human evaluation would be a good way to further evaluate the performance of our approach. The current paper was rather focused on developing dictionary learning for LMMs, extracting multimodal concepts through dictionary learning, and studying various methods for this goal in a structured way. To the best of our knowledge, the feasibility of these aspects has not been explored before, and was one of the main concerns of the paper. Nevertheless, we would like to consider human evaluation in future developments.\n\n**Using CLIP as visual encoder and CLIPScore as metric:** We believe the meaningful grounding is influenced more by the language model. We conducted experiments with two non CLIP visual encoders to support this hypothesis. Please find further details in our global response.\n\n**Overlap/difference with prior research (Goh et al., Kim et al.) and multimodal neurons literature:**  We discuss both Goh et al., and TCAV (Kim et al.) in Sec. 2 along with our differences with them. Yes, the notion of multimodal neurons has been introduced previously. However, we respectfully disagree with the assessment that there is a significant overlap with it. Our approach performs dictionary learning on internal representations of a LMM, to extract a set of concepts about a target token. This methodology is quite unrelated to work in (Goh et al.) that analyzes activations of individual neurons in CLIP ResNet encoder (final conv layer) for various input images, to determine what conceptual information in image they activate for. In general, dictionary learning based concept extraction approaches and individual neuron analysis approaches are methodologically different but complementary. They are complementary approaches to analyze internal representations of a network. However, dictionary learning methods discover directions in the representation space useful to decompose data representations, while neuron analysis approaches study activation patterns of individual neurons in detail. We list some more significant differences with Goh et al.:\n\n- The ‘multimodality’ discussed in Goh et al. is different from ‘multimodality’ here or notion of multimodal neurons in Schwettmann et al. In particular, Goh et al. analyze neuron activations only for images where certain conceptual information can be present in different forms/modalities (eg. celebrity as a face in the image, as caricatures in image, as text in the image etc.), while the multimodality here or in Schwettman et al. concerns with explicit grounding of a concept vector or neuron in vision and text. \n    \n- The two papers analyze very different types of models (LMMs vs CLIP-ResNet encoder) with different architectures, training objectives and tasks they solve. \n\n**Low overlap for PCA:** PCA is best for minimizing overlap possibly because its concept vectors by design are orthogonal. Thus, when decoded, they typically yield different sets of grounded words. We would also like to state for clarity that PCA is not learnt for just five concepts, nor do the target tokens ('dog', 'bus', 'cat', 'train') represent concepts of our dictionary. We apply PCA (or any dictionary learning method) to learn a different dictionary of $K=20$ concept vectors for each target token separately. Thus the overlap for dictionary learning methods is reported separately for each token and calculated between 20 concepts learnt for it. \n\n**Evidence of polysemanticity:** Our preliminary qualitative analysis suggests that concept vectors for prominent and distinct semantic concepts seem to be more monosemantic (eg. 'hot dog', 'black dog'). We also found concept vectors which are more polysemantic in nature, and capture more sparsely present semantic concepts. Such concepts might be identifiable in some cases while extracting the multimodal grounding. For instance, row 8, left column in Fig. 7 illustrates a concept discovered for 'Dog' token that activates for 'bull-dogs', 'pugs' but also for 'toy/stuffed dogs'. More details about the analysis can be found in our comment to Reviewer 17ab on feature superposition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u9RPXswJ7j", "reviewer": "Reviewer_XzMc", "summary": "In this paper, the authors propose a framework for interpreting LMMs. Specifically, they introduce a dictionary learning-based approach applied to the representation of tokens. The elements of the learned dictionary correspond to the proposed concepts. These concepts are semantically well-grounded in both vision and text.", "questions": "The experiments in the paper are relatively few. I suggest conducting further research on the interpretability of multimodal large models such as LLaVA and MiniGPT-4.", "limitations": "yes, the author explains the limitations of their study and potential negative societal impact.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Using concepts to interpret large multimodal models is a promising idea.", "weaknesses": "1. I'm sorry, I don't fully understand this field, so I will lower my confidence score.\n2. The notation system is quite confusing. I suggest the authors reorganize it for better clarity.\n3. The paper directly applies the concept activation vector (CAV) to Large Multimodal Models, but it does not explain the benefits of doing so or why this approach is valid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We add our response for each point below:\n\n**Notation system:** We will try our best to reorganize some notations for clarity. We remain open to incorporate any particular suggestions. \n\n\n**Benefits of CAV-based approach for LMMs:** Concept based explainability approaches in general are preferred as they highlight what semantic representations a model extracts rather than which regions of input are important for a model [1]. In order to gain understanding about internal representations of an LMM, CAV-type approach were the only viable option for concept extraction as the other types of concept explainability methods (eg. Concept bottleneck models) focus on training interpretable networks by design. Furthermore, the effectiveness of recent works ([2]) in learning concepts dictionaries for CNNs also motivated us to develop a concept extraction approach that can be applied to LMMs.\n\n**New experiments:** We have conducted new experiments on LLaVA. We are able to extract meaningful multimodal concepts and achieve quantitatively consistent results as for DePALM. The global response and 1-page pdf present more details regarding the same.\n\n[1] J. Colin, T. Fel, R. Cadène, and T. Serre. \"What I cannot predict, I do not understand: A human-centered evaluation framework for explainability methods.\" NeurIPS 2022.\n\n[2] T. Fel, V. Boutin, L. Béthune, R. Cadène, M. Moayeri, L. Andéol, M. Chalvidal, and T. Serre. \"A holistic approach to unifying automatic concept extraction and concept importance estimation.\" NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f4s39SvJFH", "reviewer": "Reviewer_afLg", "summary": "This paper proposes a new approach to understand multimodal concepts learned in LLMs with visual prefixes. To do so, the authors propose a dictionary learning-based approach that decomposes the representation of a word token in the product of two low-rank matrices via Semi-NMF, one representing the concepts and the other representing the activations for a given token. The authors evaluate the representation of the DePALM model on 8 common objects (e.g., dog, bus, cat) showing qualitatively good results, as well as better auto-eval metrics than with related baselines constructed by the authors.", "questions": "1. Are there any blockers to perform a larger-scale (wrt tokens) analysis?\n2. Interpretability methods like ROME use the last token of a given word – why do you use the first token?\n3. Does the method actually work for multi-token words (i.e., words that are split into multiple tokens by the tokenizer)? If so, are any of your 8 words in that category?\n4. Please double-check your references, some of them appear more than once.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper proposes a novel method to interpret concepts in a multimodal LLM by grounding them in both text and visual spaces. For a given concept, it creates a representation matrix from image–caption pairs, which is then decomposed into two low-rank matrices using Semi-NMF. \n\nQuality: The authors show that extracted concepts for a given token can be interpreted both visually and textually through qualitative examples. Quantitatively, the method outperforms other related baselines introduced by the authors. \n\nClarity: The paper is relatively well written, although some parts (eg, Section 3.4, 4.0 and 4.1) feel very dense and require more time to be processed.\n\nSignificance: The paper provides interesting insights in the representation of a given word token processed by a frozen LLM that is augmented with image understanding via visual prefix learning.", "weaknesses": "1. My main concern is in studying the representation of only eight tokens. It would be interesting to study more words and see if there are any interesting takeaways. These could include the 80 COCO classes, and additional rarer tokens.\n2. The use of Semi-NMF in L165-167 is unclear. Later on, we find out that it works better than other decompositions, but I was left wondering how the authors came up with Semi-NMF in Section 3.4.\n3. It would have been useful to compare the proposed approach with some current interpretability methods that could apply here (or at least discuss why they might not be used here).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the interesting feedback and positive comments: \n\n**Computational scaling for large number of tokens:** Our experiments use single GPU. The BERTScore evaluation does not scale well. It can take upto 3-4 hours to evaluate all baselines on some target tokens (COCO). The representation extraction process scales fine with DePALM (upto 15 minutes for some tokens), but poorly with models like LLaVA (upto 3-5 hours for some tokens). The core of the method itself, dictionary learning and multimodal grounding, is computationally cheap.  \n\n**Experiments with more tokens:**\nWe didn't experiment earlier with more because of expensive BERTScore evaluation. However, we have now conducted experiments with 30 additional COCO classes (apart from 8 in the paper) for CLIPScore and Overlap , with identical hyperparameters. We considered these COCO classes based on the criteria that these are single token words with at least 40 predicted test samples by $f_{LM}$. Single token criterion is to keep our experimental setup consistent. Lower bound criterion on test samples is to ensure average test CLIPScore is reliable for each target token. We report the macro average of test CLIPScores and overlap over the 30 extra target tokens in tables below. We'll add the detailed results in appendix for all baselines. We obtain results consistent with those in main paper, with Semi-NMF extracting concepts with good balance between high CLIPScore and low overlap.\n\n| Metric (Macro Avg)| Rnd-Words | Noise-Imgs | Simple | Semi-NMF |GT-captions|\n|:---:|:--:|:---:|:---:|:---:|:---:|\n| test CLIPScore $(\\uparrow)$  | 0.515 | 0.521 | 0.625 | 0.630 | 0.762 |\n\n\n| Metric (Macro Avg) | Simple | PCA | KMeans | Semi-NMF |\n|:--:|:--:|:---:|:---:|:---:|\n| Overlap $(\\downarrow)$  | 0.443 | 0.050 | 0.451 | 0.155 |\n\n\n**Application to multi-token words:** It is an interesting question. None of our evaluated tokens were in this category. However, our approach can be easily adapted to this. In particular, we extract representation of last token from first prediction of the multi-token sequence. The other aspects of the method remain unchanged. While there can also be other viable strategies, the rationale behind this adaptation is that the last token of our sequence of interest can also attend/combine representations from previous tokens in the sequence. We add below results for such examples. Detailed results will be added in appendix):\n\n|Metric|Token |Rnd-Words|Noise-Imgs|Simple|Semi-NMF|GT-captions|\n|:---:|:---:|:--:|:--:|:--:|:--:|:--:|\n|test CLIPScore $(\\uparrow)$|traffic light|0.516 ± 0.03|0.525 ± 0.03|0.664 ± 0.06|0.634 ± 0.05|0.744 ± 0.04|\n|test CLIPScore $(\\uparrow)$|cell phone|0.542 ± 0.04|0.547 ± 0.03|0.598 ± 0.04|0.598 ± 0.05|0.765 ± 0.06|\n|test CLIPScore $(\\uparrow)$|stop sign|0.533 ± 0.03|0.549 ± 0.03|0.617 ± 0.08|0.616 ± 0.05|0.775 ± 0.04|\n\n| Metric | Token | Simple | PCA | K-Means | Semi-NMF |\n|:--:|:--:|:---:|:---:|:---:|:---:|\n| Overlap $(\\downarrow)$|traffic light|0.704|0.050|0.579|0.174|\n| Overlap $(\\downarrow)$|cell phone|0.623|0.051|0.746|0.164|\n| Overlap $(\\downarrow)$|stop sign|0.461|0.058|0.704|0.109|\n\n**Motivation to use Semi-NMF:** The idea to consider Semi-NMF was inspired from the effectiveness of NMF for various interpretability applications ([1-4]). Given the positive and negative values in $h^p_{(L)}$, it was not possible to effectively apply NMF. Thus, Semi-NMF arose as a natural option by relaxing non-negativity on $\\mathbf{U}$.  We expected the constraint to only positively combine the dictionary elements should still be useful from an interpretability perspective. Theoretical connections between Semi-NMF being generalized version of K-Means [5] also supported our confidence. We'll expand Sec 3.4 to include some of the motivation for more clarity. We'd also like to add that we did consider all three approaches (PCA, KMeans, Semi-NMF) initially and qualitatively found Semi-NMF dictionaries most balanced (diverse and multimodally meaningful).\n\n\n**Applicability of other current interpretability methods:** Related works (Sec. 2) discuss in detail the applicability of previous CAV-based approaches and previous approaches to understand VLMs/LMMs. Within general interpretability literature, the closest related methods to CAV based concept extraction are input autoencoder-based concept methods or concept bottleneck models. However these are used almost exclusively as by-design interpretable networks and thus out of scope for understanding pretrained models. We can add a brief discussion about them in Sec. 2.\n\n\n**Representations extraction + Relation to ROME:** We selected representation from first position where target token is predicted following Schwettmann et al. [6] who analyzed neurons for a given caption at the first predicted noun. In practice, what is essential is extracting representations where the target token is predicted. In this regard we are not in conflict with ROME but rather aligned. ROME analyzes factual associations of the form (subject, object, relation). The input prompt to the language model consists of the subject and relation. The model predicts the next token which is expected to be the object and thus ROME also analyzes representation at a position where their token of interest is predicted.\n\n\n**Repeated references:** Thanks. We'll correct them. \n\n[1] G. Trigeorgis et al. \"A deep semi-nmf model for learning hidden representations.\" ICML 2014\n\n[2] J. Parekh et al. \"Listen to interpret: Post-hoc interpretability for audio networks with nmf.\" NeurIPS 2022\n\n[3] T. Fel et al. \"Craft: Concept recursive activation factorization for explainability.\" CVPR 2023\n\n[4] YT. Guo et al. \"The rise of nonnegative matrix factorization: algorithms and applications.\" Information Systems 2024.\n\n[5] CHQ. Ding et al. \"Convex and semi-nonnegative matrix factorizations.\" IEEE TPAMI 2008\n\n[6] S. Schwettmann et al. \"Multimodal neurons in pretrained text-only transformers.\" ICCVW 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ipj9VchU1O", "reviewer": "Reviewer_17ab", "summary": "This paper introduces an explainability approach for interpreting the internal representations of large multimodal models (LMMs). The authors train an image captioning model consisting of a pretrained image encoder and language model and a connector model. To extract interpretable representations, the authors use dictionary learning, decomposing representations into lower dimensional $U$ and $V$ matrices using semi non-negative matrix factorization as the optimisation objective. To interpret concepts in $U$ in the textual domain, the authors use the language model unembedding layer to extract the highest probability tokens associated with a given concept vector $u$. Likewise, to visually interpret a concept vector $u$, the authors find the set of images that maximally activate $u$. The authors provide quantitative evidence demonstrating that their method generates concepts which are well aligned with both the input image and the ground-truth captions. Additionally, they demonstrate their approach qualitatively produces well defined concepts with limited overlap between the tokens represented by different concepts.", "questions": "1. Why did the authors choose to use Opt-6.7B as the language model rather than more powerful similar sized models such as Llama-7B?\n2. Why do the authors take the absolute activations of $V$ in (5)? Are these activations not guaranteed to be non-negative by the optimisation objective?\n3. Did the authors consider trialling gradient-based feature visualisation approaches [1] in addition to taking images with the highest activations? This could be an alternative approach to build additional confidence that the extracted concepts do represent what qualitative analysis of the highest activating samples appears to suggest. I don’t expect the authors to perform analysis of this kind in this manuscript however it could be worth commenting on this as a future avenue of research, or alternatively raising any valid criticisms of this approach?\n4. There is a typo in figure 1 (“Captionin”)\n\n[1] Olah, C, et al. \"Feature Visualization\", Distill, 2017.", "limitations": "The work presented here is only for a single, relatively small model, trained with a specific objective (image captioning). As such it is not clear that this approach will necessarily generalise to other multimodal models. The authors should touch on this limitation in the discussion or limitations section in the appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper introduces a promising novel approach to performing mechanistic interpretability on multimodal models. Given the increasing ubiquity of multimodal models there exists a clear need for interpretability approaches for this style of model. This work presents a viable dictionary-learning based approach I look forward to seeing other researchers building upon in the future.\n2. The evaluation of the framework is comprehensive including both qualitative and quantitative results, both of which are essential for any interpretability method. Additionally, I was impressed to see the authors consider multiple approaches (e.g. PCA/KMeans) and sensible baseline models when evaluating their results, providing more confidence that the final approach taken was an appropriate method.\n3. The paper is very clearly articulated, with the rationale well-defined, the approach clearly outlined and the results succinctly and clearly summarised.", "weaknesses": "1. The authors provide good evidence that the concepts extracted by their method show minimal overlap with other concepts, however they do not address the alternative possibility, that their extracted concepts might represent more than one distinct concept. This phenomenon of feature “superposition” has been well documented in other model interpretability work (see [1][2][3]) and so it seems plausible that it may arise in the approach taken here. This seems especially likely given that the dimensionality of their concept dictionary is lower than the dimensionality of the internal representations. Though I do not think this should detract from the otherwise excellent contributions presented in this paper, I do think this at least warrants a brief discussion, and perhaps more qualitative analysis of the extracted concepts, to assess whether any evidence of feature superposition is observed.\n2. Occasionally the authors make claims that go beyond the scope of this work. For example, in the abstract they state, “we present a novel framework for the interpretation of LMMs”. However, this claim seems too strong given that this approach is only really valid for image captioning models rather than large multimodal models more generally. Additionally, there are a few comments such as “we find the generalization of LLMs to multimodal inputs is an interesting phenomenon to understand” in 3.1 (under “Training”) and “the multimodal structure of internal token representations starts to appear at [later layers]” in 4.2 (under “Layer Ablation”). However these claims do not seem valid as the language model layers are frozen during training. It seems more appropriate to say that these extracted concepts represent language model concepts, and the connector model learns to transform the image representations such that they align with these language model concepts.\n\n[1] Elhage, N, et al. \"Toy Models of Superposition.\" arXiv:2209.10652 (2022)\n\n[2] Arora, S et al. \"Linear Algebraic Structure of Word Senses, with Applications to Polysemy.\" arXiv:1601.03764 (2016)\n\n[3] Elhage, N, et al. \"Softmax Linear Units\" Transformer Circuits Thread (2022)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive and insightful comments. Please find our response pointwise below:\n\n**Qualitative analysis for feature superposition/specificity of concept vectors:** Thanks for the interesting suggestion. We conducted a preliminary qualitative study on some concept vectors in the dictionary learnt for token \"Dog\" to analyze if these concept vectors tend to activate strongly for a specific semantic concept (monosemantic) or multiple semantic concepts (polysemantic). In particular, we first manually annotated the 160 test samples for \"Dog\" for four semantic concepts, for which we knew we had concept vectors in our dictionary, namely \"Hot dog\" (Concept 2, row 1, column 2 in Fig. 7), \"Black dog\" (Concept 20, row 10, column 2 in Fig. 7), \"Brown/orange dog\" (Concept 6, row 3, column 2 in Fig. 7), and \"Bull dog\" (Concept 15, row 8, column 1 in Fig. 7). For a given semantic concept, we call this set $C_{true}$. Then, for its corresponding concept vector $u_k$ we find the set of test samples for which it activates greater than a threshold $\\tau$. This threshold was set to half of its maximum activation over test samples. We call this set of samples $C_{top}$. To estimate specificity of the concept vector we compute how many samples in $C_{top}$ lie in the ground-truth set, i.e. $|C_{top} \\cap C_{true}|/|C_{top}|$. We find Concept 2 (\"Hot dog\") to be most monosemantic with 100\\% specificity. For Concept 20 (\"Black dog\") too, we found high specificity of 93.3\\%. For concept 15 (\"Bull dog\") we observed the lowest specificity of 50\\%. This concept also activated for test images with toy/stuffed dogs. Interestingly, the multimodal grounding of concept 15 already indicates this superposition with maximum activating samples also containing images of 'toy dogs'. Concept 6 (\"Brown/orange dog\") is somewhere in between, with 76\\% specificity. This concept vector also activated sometimes for dark colored dogs, which wasn't apparent from its multimodal grounding. \n\nIn summary, your expectation of superposition/polysemanticity is fair. Prominent or distinct semantic concepts seem to be captured by more specific/monosemantic concept vectors, while more sparsely present concepts seem at risk to be superposed resulting in a more polysemantic concept vector capturing them. We can add this discussion in appendix.\n\n\n**Qualifying certain claims:** We accept your concerns about the three claims/statements. We will modify them to qualify their extent and scope. \n\n\n**Powerful language model + Generalization to other LMMs:** We are pleased to share that we conducted new experiments LLaVA-v1.5. It uses Vicuna-7B (closely related to LLaMA) as the language model. We are able to extract meaningful multimodal concepts and achieve quantitatively consistent results as for DePALM. The global response presents more details regarding the same.\n\n\n**Absolute activations in Eq. (5):** We wrote Eq. (5) this way so that it remains applicable even for methods without the constraint for non-negativity of activations (eg. PCA). For Semi-NMF it indeed does not make a difference.\n\n**Trialling gradient-based feature visualisation:** This is a fair suggestion and a useful future direction to build upon. For now, we have explored generating saliency maps with LLaVA (not via gradients) for concept activations by computing the inner product of concept vector $u_k$ with all visual token representations from corresponding layer $L$, i.e. $u_k^T [h^1_{(L)}, ..., h^{N_V}_{(L)}]$. This vector of size $N_V =576$ is reshaped to 24 $\\times$ 24 and resized to original input. This is feasible for LLaVA as all visual token representations fed to the language model preserve their patch based identity. Sample visualizations are available in 1-page PDF (Fig. 2) in global response. \n\n**Typo in Fig. 1:** Thanks for pointing it out. We'll correct it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all the reviewers for their great interest and useful feedback. We address most reviewer comments individually. In this global post we would like to address two key concerns, each raised by at least two reviewers:

1. **New experiments on LLaVA (Reviewers Go37, XzMc, 17ab):**  We have now conducted experiments on LLaVA-v1.5 model. The model uses a CLIP-ViT-L-336px visual encoder, a 2-layer linear mapper that outputs $N_V=576$ visual tokens, and a Vicuna-7B language model (32 layers). We use identical hyperparameters as for DePALM ($K=20, \lambda=1, L=31$). The **attached 1-page PDF** contains the figures/tables regarding this experiment. For now, we report (i) the test CLIPScore for top-1 activating concept, for Rnd-Words, Noise-Imgs, Simple and Semi-NMF, GT-captions (Tab. 1 in 1-page PDF) and (ii) Overlap score for non-random baselines (Tab. 2 in 1-page PDF). We also show qualitative examples of concepts extracted for token `Dog' (Fig. 1 in 1-page PDF). More detailed quantitative and qualitative results will be added in appendix. Quantitatively, we obtain **consistent results** to those observed for DePALM. Semi-NMF extracts most balanced concept dictionaries with high multimodal correspondence and low overlap. Qualitatively too, the method functions consistently and is able to extract concepts with meaningful multimodal grounding.

2. **CLIP visual encoder and CLIPScore as metric (Reviewers 8A1z, Go37):** The meaningful concept representations, we believe, are more due to their alignment with language model representations than the visual encoder. To support our hypothesis, we conducted experiments with 2 different DePALM models with frozen visual encoders different from CLIP, a frozen ViT-L encoder trained on ImageNet [1] and another frozen ViT-L trained as a masked autoencoder (MAE) [2]. Both LMMs use the same pretrained OPT-6.7B language model. Collectively, the three encoders (including CLIP) are pretrained for three different types of objectives. We use Semi-NMF to extract concept dictionaries, with all hyperparameters identical. The results are reported in tables below. 'Rnd-Words' and 'GT-captions' references are reported for each LMM separately, although they are very close to the ones in main paper. The "ViT-L (CLIP)" baseline denotes our system from the main paper that uses CLIP encoder. Importantly, we still obtain similar test CLIPScores as with CLIP visual encoder. The concept dictionaries still possess meaningful multimodal grounding. The various concepts also tend to be similar as for CLIP visual encoder. We'll add the qualitative visualizations and complete quantitative results for the non-CLIP encoders in appendix.  

| Token         | Rnd-Words  | ViT-L (ImageNet)  | ViT-L (CLIP)  | GT-captions |
|:-------------:|:----------:|:---------------:|:--------------:|:-----------:|
| Dog           | 0.514 ± 0.05 | 0.611 ± 0.09 | 0.610 ± 0.09 | 0.783 ± 0.06 |
| Bus           | 0.498 ± 0.05 | 0.644 ± 0.07 | 0.634 ± 0.08 | 0.739 ± 0.05 |
| Train         | 0.494 ± 0.05 | 0.617 ± 0.07 | 0.646 ± 0.07 | 0.728 ± 0.05 |
| Cat           | 0.539 ± 0.05 | 0.628 ± 0.07 | 0.627 ± 0.06 | 0.794 ± 0.06 |

| Token         | Rnd-Words  | ViT-L (MAE)  | ViT-L (CLIP)  | GT-captions |
|:-------------:|:----------:|:--------------:|:------------:|:-----------:|
| Dog           | 0.515 ± 0.05 | 0.602 ± 0.07 | 0.610 ± 0.09 | 0.784 ± 0.06 |
| Bus           | 0.501 ± 0.05 | 0.627 ± 0.07 | 0.634 ± 0.08 | 0.737 ± 0.05 |
| Train         | 0.483 ± 0.06 | 0.618 ± 0.08 | 0.646 ± 0.07 | 0.726 ± 0.05 |
| Cat           | 0.541 ± 0.04 | 0.629 ± 0.09 | 0.627 ± 0.06 | 0.795 ± 0.06 |

[1] A. Dosovitskiy et al. "An image is worth 16x16 words: Transformers for image recognition at scale." ICLR 2021

[2] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick. "Masked autoencoders are scalable vision learners." CVPR 2022

Meta-review: This paper proposes a dictionary-learning approach to analyzing multimodal LMs.  A qualitative/quantitative analysis shows the emergence of clearly interpretable multi-modal concepts.

There was some discrepancy in the reviewers' opinion about the paper, and some important concerns were voiced, pertaining to the limited scope of the evaluation, the fact that there is no subject-based manual validation of the results, and the general issue of the danger of interpreting concepts in a monosemic way when representation superposition is a widespread phenomenon in neural networks. However, the method is clever and leads to promising results, and it could generally help people working on LM interpretability. Moreover, the reviewers that provided the more detailed reviews and more actively engaged with the authors were also those that have a more positive opinion of the paper. I would tend to agree with them, and I would like to see this work at the conference.
Decision: Accept (poster)