Understanding Information Storage and Transfer in Multi-Modal Large Language Models

Authors: samyadeep  basu
martin  grayson
cecily  morrison
besmira  nushi
soheil  feizi
daniela  massiceti
d  levy
c m li
r  lim
m  lin
s  lin
m  litwin
t  lopez
r  lowe
p  lue
a  makanju
k  malfacini
s  manning
t  markov
y  markovski
b  martin
k  mayer
a  mayne
b  mc- grew
s m mckinney
c  mcleavey
p  mcmillan
j  mcneil
d  medina
a  mehta
j  menick
l  metz
a  mishchenko
p  mishkin
v  monaco
e  morikawa
d  mossing
t  mu
m  murati
o  murk
d  mély
a  nair
r  nakano
r  nayak
a  neelakantan
r  ngo
h  noh
l  ouyang
c  o'keefe
j  pachocki
a  paino
j  palermo
a  pantuliano
g  parascandolo
j  parish
e  parparita
a  passos
m  pavlov
a  peng
a  perelman
f  de avila
belbute  peres
m  petrov
h p de
oliveira  pinto
m  pokrass
v h pong
t  powell
a  power
b  power
e  proehl
r  puri
a  radford
j  rae
a  ramesh
c  raymond
f  real
k  rimbach
c  ross
b  rotsted
h  roussez
n  ryder
m  saltarelli
t  sanders
s  santurkar
g  sastry
h  schmidt
d  schnurr
j  schulman
d  selsam
k  sheppard
t  sherbakov
j  shieh
s  shoker
p  shyam
s  sidor
e  sigler
m  simens
j  sitkin
k  slama
i  sohl
b  sokolowsky
y  song
n  staudacher
f p such
n  summers
i  sutskever
j  tang
n  tezak
m b thompson
p  tillet
a  tootoonchian
e  tseng
p  tuggle
n  turley
j  tworek
j f c uribe
a  vallone
a  vijayvergiya
c  voss
c  wainwright
j j wang
a  wang
b  wang
j  ward
j  wei
c  weinmann
a  welihinda
p  welinder
j  weng
l  weng
m  wiethoff
d  willner
c  winter
s  wolrich
h  wong
l  workman
s  wu
j  wu
m  wu
k  xiao
t  xu
s  yoo
k  yu
q  yuan
w  zaremba
r  zellers
c  zhang
m  zhang
s  zhao
t  zheng
j  zhuang
w  zhuk
b  zoph
  gpt-
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 20:02:59

Abstract: Understanding the mechanisms of information storage and transfer in Transformer-based models is important for driving model understanding progress. Recent work has studied these mechanisms for Large Language Models (LLMs), revealing insights on how information is stored in a model's parameters and how information flows to and from these parameters in response to specific prompts. However, these studies have not yet been extended to Multi-modal Large Language Models (MLLMs). Given their expanding capabilities and real-world use, we start by studying one aspect of these models -- how MLLMs process information in a factual visual question answering task. We use a constraint-based formulation which views a visual question as having a set of visual or textual constraints that the model's generated answer must satisfy to be correct (e.g. What movie directed by \emph{the director in this photo} has won a \emph{Golden Globe}?). Under this setting, we contribute i) a method that extends causal information tracing from pure language to the multi-modal setting, and ii) \emph{VQA-Constraints}, a test-bed of 9.7K visual questions annotated with constraints. We use these tools to study two open-source MLLMs, LLaVa and multi-modal Phi-2. Our key findings show that these MLLMs rely on MLP and self-attention blocks in much earlier layers for information storage, compared to LLMs whose mid-layer MLPs are more important. We also show that a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. We validate these mechanisms by introducing MultEdit a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs by targeting these causal blocks. We will publicly release our dataset and code.

Main paper: 
 introduction: 
 Multi-modal Large Language Models (MLLMs) trained on both text and images are rapidly moving from research into deployment and are being used by millions of people. Yet, while there have been some advances in understanding how Large Language Models (LLMs) work, much less has been done to understand MLLMs. This paper begins to close this gap by studying how information is stored and transferred in MLLMs. To do this through the lens of a factual Visual Question Answering (VQA) task -a very common use case of MLLMs today [26, 35, 27] .MLLMs process factual information in two steps: information storage and information transfer. Information storage refers to how facts from a pre-training dataset are stored in a model's parameters -its so-called 'parametric' memory. Information transfer describes how information from input prompts are propagated through these storage locations to the model's final output. Understanding We find that very early MLP layers [1] [2] [3] [4] have high indirect estimation effects to outputs (i.e., they are causal) in LLaVa-7B, whereas the middle MLP layers [4] [5] [6] [7] are causal in LLaMA (Vicuna)-7B. For LLaMA, a larger window size (e.g., 5 ) is also required to find causal sites, compared to a window size of 1 for LLaVA-7B.these mechanisms can have many benefits, including ensuring models are factually grounded and informing better evaluation protocols.Extensive works have explored how LLMs store and transfer factual information [12, 29, 24] , however, this has not been studied for multi-modal inputs. For example, it is suggested that auto-regressive Transformer-based LLMs store factual information in their mid-layer MLP parameters [12, 24] . However, MLLMs and LLMs are different: an MLLM involves an additional (continuous) image input, alongside a (discrete) text prompt, and requires additional modules to process it [26, 18, 19, 16, 7] . Typically, a vision encoder (e.g. CLIP) is used to convert this image into either visual tokens via a projection layer [18, 19, 11] or cross-attention layers [3, 7, 16, 36] which are then integrated into the language encoder. These differences suggest that our existing understanding of LLM information storage and transfer may not map directly to MLLMs.In this work, we use a factual VQA task to study the mechanisms of multi-modal information storage and transfer. We use a constraint-based formulation which views a visual question as having a set of either visual or textual constraints (e.g. What movie directed by this director has won a Golden Globe?). The information retrieved by the model should satisfy these constraints (e.g. this director, Golden Globe) for the answer to be factually correct. This formulation, therefore, offers a systematic way of understanding a model's behavior. Under this framework, we propose MULTI-MODALCAUSALTRACE, which extends LLM causal tracing [12, 28, 33] to the multi-modal setting, to understand information storage, as well as leverage attention contribution methodologies [37, 8] to study information transfer in MLLMs. We also introduce VQA-Constraints, a new dataset of 9.7k factual questions annotated with constraints, spanning natural images (from OK-VQA [22] , WikiMovies [37] , and Known [12] ). With these tools, we study how a widely-used MLLM family processes multi-modal information, specifically LLaVa [18, 19] and multi-modal Phi-2 [11] 1 .Our key findings are that MLLMs, in contrast to LLMs: 1) retrieve information from earlier MLP layers (i.e. layers 1-4 vs layers 4-7 in a LLM) (see Fig. 1 ); and 2) use less parametric memory (require a smaller window size to retrieve this information), when answering a multi-modal question. We also find that information is transferred from a given image to these early MLP blocks through a consistent subset of visual tokens (e.g. last ∼36 tokens from LLaVa's CLIP encoder), and that the self-attention blocks in the middle layers are primarily responsible for shuttling this information to the last token.Finally, we demonstrate that by editing these early causal MLPs, we can correct errors and insert new factual information into an MLLM. Specifically, we propose a new model-editing algorithm, MULTEDIT, which modifies the projection matrix of the early causal MLPs with a closed-form update. We empirically show MULTEDIT's effectiveness on questions from VQA-Constraints and Encyclopedic VQA [25] .In summary, our contributions are:  
related works: 
 Multimodal Large Language Models.. We consider a MLLM to be a model that takes an image and text as input, and generates a text output [2] . Over the last year, such models have made tremendous advances in tasks like VQA and image captioning, including BLIP [15] , BLIP-2 [16] , Instruct-BLIP [7] , LLaVA [18, 19] , Flamingo [3] and multi-modal Phi-2 (from the Bunny repo) [11] . These MLLMs can broadly be categorized into two families based on how their visual information is integrated into the language model: (i) by embedding the vision encoder's output into each layer of the language model with a cross-attention layer (e.g., Flamingo, BLIP) or, (ii) by mapping the vision encoder's output into "visual tokens" in the language model's input space (i.e. alongside the text tokens) via a projection layer (e.g., LLaVA, Bunny). Both families are widely used, however, the projection layer family has recently shown stronger performance on popular benchmark [18, 19, 11] . We, therefore, focus our study of information storage and transfer on this model family.A well-established arm of model interpretability examines the relationship between a model's performance and its internals. A range of recent works have studied the internal mechanisms of information storage [23, 34, 24] and transfer [9, 37] in LLMs. However, to the best of our knowledge, only a few works [30, 31] have studied the interpretability of MLLMs, with none specifically investigating the relationship between a model's outputs and its internal states. [30] , for example, designs an interactive interface to visualize the attention maps in an MLLM, while [31] explores the shortcomings of the CLIP vision encoder in MLLMs. Neither consider the influence of both vision and text inputs on model internals or offer causal insights, as our work does. Our model editing approach which targets the projection layer MLLM family, is complemented by [6] , who propose baselines for inserting information into the cross-attention layer MLLM family. 
a constraint-based framework for studying information storage and transfer in mllms: 
 In this section, we describe the constraint-based formulation we use to study information storage and transfer in MLLMs. Under this framing, we introduce MULTIMODALCAUSALTRACE, a novel causal information tracing technique which we use to study multi-modal information storage. We also describe how we use attention contributions [8] to study multi-modal information transfer. Finally, we describe VQA-Constraints, a new test-bed of visual questions annotated with constraints. Prior works have used a constraint satisfaction framework to study LLMs [37, 20, 14] as they offer a systematic way of studying model behavior. This framing defines a constraint as a set of words in the question. The model must retrieve information that satisfies this constraint from its parametric memory in order to generate the correct answer. For example, for the question "What city is the Space Needle in?", the model must retrieve information relevant to the constraint "Space Needle". In the multi-modal setting, we consider these textual constraints as well as introduce visual constraints.We define a visual constraint to be a set of words in the question which refers to an entity in the image. The model must similarly retrieve information about this entity to generate the correct answer.For example, given an image of Christopher Nolan and the question "Name a movie directed by this director in 2006?", the visual constraint is "this director" (and the text constraint is "2006"). We refer to a question involving both a visual and text constraint as a multi-constraint question, and ones with only a visual constraint as a single-constraint question.We use this framework to study the widely-used "projection layer" MLLMs family. These models are composed of a visual encoder f θ , a large language model g ϕ and a projection head p γ . The projection head p γ is responsible for mapping the output of the visual encoder into the input space of the language model, as so-called "visual tokens". Given an image-text pair denoted as (x, y), the language model processes them as g ϕ (p γ (f θ (x)), h(t(y))), where p γ (f θ (x)) = {v i } N i=1 is the set of visual token embeddings and t(y) = {t i } M i=1 is the tokenized text inputs for the language model. These text tokens are processed by an embedding layer h to obtain text token embeddings as {e i } M i=1 ∈ R d . Because we are interested in studying the outputs of specific layers in response to specific tokens, we use g ϕ (.) k,ℓ to denote the output layer embedding corresponding to the k th token position and the layer ℓ. We denote the output of a MLP layer as g ϕ (.) k,ℓ mlp and the output of a self-attention block as g ϕ (.) k,ℓattn .Causal tracing, derived from the causality literature [28] , can be combined with a constraint-based framework to gain causal insights on how information propagates through a model with respect to specific constraint tokens. This has been used to identify where information is stored in LLMs [23, 24] and text-to-image generative models [4] . The central idea of causal tracing is to corrupt a clean model by perturbing the input prompt. The activations of a small subset of layers are then iteratively copied from the clean model to the corrupted model, until the corrupted model restores its output probability to match the clean model's. In this way, we can identify which layers are used to retrieve information relevant to the constraints in the prompt (i.e. causally relate the input to the output).In LLMs, the model is corrupted by adding a small amount of Gaussian noise to the embeddings of the textual constraint tokens (usually less than 5) [23] . In MLLMs, however, noise must be added to a much larger number of token embeddings -those of the visual tokens (e.g., 576 in LLaVa and 729 in multi-modal Phi-2) from the projection head, and the textual tokens of the visual constraint (e.g. "this director"). Our experiments show that this large noise injection makes it difficult to revert the MLLM to a clean state and recover relevant causal traces (see Fig. 7 ).We, therefore, introduce MULTIMODALCAUSALTRACE (see Fig. 2 ) to address this. Rather than adding Gaussian noise to the embeddings, we instead corrupt the visual constraint token IDs by replacing them with token IDs from a separate word or phrase, such that the visual information is ignored. We illustrate this through an example. (1) Clean Model: Given an image x of the Space Needle, and the question y, "Which city is this building located in?", we compute the probability of the model's output O (e.g., "Seattle") as P clean (O). (2) Corrupted Model: We substitute the visual constraint with an alternative such that the question does not require information from the image to be answered (e.g., "this building" is replaced with "Taj Mahal"). For a multi-constraint question which also has a textual constraint, we can either add Gaussian noise to the textual constraint tokens' embeddings (since there are only a few) or we can similarly replace the token IDs. After all replacements, we ensure that the question still makes semantic sense. We then measure the probability of original output O as P corr (O). With the right corruption, P corr (O) is expected to be low. (3) Restored Model: We then iteratively copy layer activations g ϕ (.) k,ℓ mlp and g ϕ (.) k,ℓattn ∀k ∈ [1, M + N ] from the clean model to the corrupted model, for each layer in turn. For a given layer ℓ and token position k, we denote the restored probability as P restored (O) k,ℓ . After a layer is copied, we observe if P restored (O) k,ℓ is high -indicating that layer ℓ has a strong causal association to the output O. In some cases, no layers have a causal association. We note that this copy operation can be performed over a window of layers {ℓ i } W i=1 at a time, where W is the window size. A window size of 1 copies only one layer at a time. Similar to [23] , we track the indirect estimation effect for a layer ℓ as P restored (O) k,ℓ -P corr (O) and use it as a metric to track causal states. Intuitively, this measures the difference in the probability of O under the corrupted model and when a layer ℓ is restored to its original clean state. A high value indicates that the copied layer/s can restore the model to its original clean state (i.e. the layer is causal).MULTIMODALCAUSALTRACE enables us to identify the specific layers a model retrieves information from in order to answering a visual question. A second component of understanding how MLLMs process factual information is understanding how input prompts are propagated through these storage locations to the model's final output. For this, we use attention contributions [8, 37] which compute how much one set of input tokens influences a set of output tokens during the self-attention operation.Specifically, we use this to track (i) how information is transferred from the visual tokens to the causal layers, and (ii) from these layers to the final token, where the final probabilities are computed. [32] consists of the query, value, key and output weight matrices:W ℓ q , W ℓ v , W ℓ k , W ℓ o . Each of these are divided into H heads as W ℓ,h q , W ℓ,h v , W ℓ,h k ∈ R d×d h , W ℓ,h o ∈ R d h ×d, where d is the dimension of the internal token embeddings and d h is the dimensionality of the token embedding for a particular attention head h. We define the attention contribution from a token j to token i in layer ℓ as follows:EQUATIONwhere the attention matrix for layer ℓ and head h is defined as:EQUATIONwhere g ϕ (. +N ) . For understanding information transfer from the visual tokens to the causal layers, we use j ∈ [1, M ] and i = c constraint , where c constraint corresponds to the last token in the visual constraint. For understanding the information transfer to the last token, we set j = c constraint and i = c last , where c constraint corresponds to visual constraint's last token and c last corresponds to the last token in the question.) 1:(M +N ),ℓ-1 ∈ R (M +N )×d and A ℓ,h ∈ R (M +N )×(MAlongside the above tools, we also introduce a new test-bed called VQA-Constraints to enable our analyses. The test-bed consists of 9.7K natural images paired with factual questions, where each question is annotated with visual and textual constraints (see Sec. 3.1). Specifically, we source image-question pairs from the following datasets i) OK-VQA [21] which covers general knowledge questions, ii) Movies [37] which includes questions about movie directors and awards from Wikipedia, and iii) Known [23] which covers questions about countries, famous people, and places. The questions from the Movies and Known datasets are not originally multi-modal, so we modify them to refer to images which we source from Bing. We provide more details on the dataset construction and the constraint annotation in the Appendix B.We leverage GPT-4 [27] to annotate the textual and visual constraints in the visual questions in VQA-Constraints. We first manually annotate 100 examples from each dataset with constraints. We provide this as context to GPT-4 and prompt it (see Appendix B) to annotate the constraints in new questions from Multimodal Known and OK-VQA. Due to the templated prompts in Multimodal Movies (e.g., "Name a movie directed by this director? "), the constraints are constant ("this director") across all examples and does not require external annotations. 
key findings in how mllms store and transfer information: 
 Using the tools presented in Sec. 3.1, we present our key findings in how MLLMs retrieve information from internal layers and how this information is transferred across the model.We find that information required to answer a visual question is mainly retrieved from the early-layer MLP and self-attention blocks of a MLLM. This is confirmed by the high indirect estimation effects which MULTIMODALCAUSALTRACE assigns to the early layers in both LLaVa (see Fig. 3 , Fig. 8 ) and multi-modal Phi-2 (see Appendix E) across the three datasets in VQA-Constraints.This contrasts earlier results for LLMs which have been shown to retrieve information from mid-layer MLPs to answer factual questions [23, 24] . To obtain a fairer comparison, we apply MULTIMODAL-CAUSALTRACE to LLaMA and LLaVa which use the same language backbone. We run both on the same set of questions -LLaMA on the Known dataset [23] , and LLaVa on our multi-modal version of Known which modifies the questions to refer to an image (see Sec. We also find that causal traces can be extracted from LLaVA with a minimum window size of 1, while LLaMa requires a minimum window size of 5 to obtain any significant causal traces.Although we find a smaller window size of 1 to provide relevant causal traces (see Fig. 8 ), we find that a slightly larger window size of 3 results in consistent causal traces across all the questions. In Fig. 3 , we report the results using a window size of 3, where we find that the early MLPs as well as self-attention layers are used to retrieve relevant knowledge to answer a visual question. We note that this observation is consistent for all the datasets in VQA-Constraints.For multi-constraint questions that consist of a visual and textual constraint, information corresponding to the textual constraint is retrieved from a broader set of layers -both the early and mid-layer MLP and self-attention blocks (see Fig. 4 ). We also find that a larger window size (at least of 6) is required to obtain any causal traces. This suggests that more parametric memory is required to meet both a visual and textual constraint in a given question.4.2 Finding 2: Only a subset of visual tokens are involved in transferring information from the image to the early causal MLP layers.We also find that the late visual tokens transfer information from the image to the early causal MLP layers (where information is mainly stored) via the first self-attention layer. We show this in Fig. 5 by visualizing the attention values in the early self-attention layers between each visual tokens and the final token in the visual constraint using the method described in Sec. 3.3. We see that the values are the highest in LLaVa's first self-attention layer (which occurs just before the first causal MLP layer), specifically for the last subset of visual tokens (indexes 540-576 out of 576 in total). We hypothesize that these tokens may be summarizing image information that is relevant to the given question before it is transferred and then stored in the MLPs, however we leave this to future study. We note that this pattern holds across all three datasets from VQA-Constraints.4.3 Finding 3: Mid-layer self-attention layers are involved in transferring information from the early causal layers to the question's final tokenRather counter-intuitively, we find that even though information is stored in the early layers, the self-attention blocks in the middle layers (rather than layers immediately after) are responsible for propagating this information to the question's final token. The model's answer is sampled at this point, hence the information present here likely influences the ultimate generation. We see this in Fig. 15 and Fig. 16 , which plots the attention contribution values between the last visual constraint token and the last token in the question across all the layers, using the methodology in Sec. 3.3. For LLaVa, the self-attention blocks in layers 16-17 are most active. This behaviour is similar in LLMs which also use mid-layer self-attention blocks to transfer information from (mid-layer) stored locations to the last token position.4.4 Finding 4: Mid-layer self-attention contributions can be used to predict whether a MLLM will generate a correct answer, but model confidence is a more reliable predictor When a MLLM generates a correct answer, we observe that the self-attention contributions in its middle layers are higher to when it generates an incorrect answer (see Fig. 15 ). For LLaVa, this is specifically the attention contributions between the last constraint token and the last prompt token in the 16th and 17th layer. This holds potential to detect when a model will answer correctly without running a full inference pass, therefore enabling "early" failure mode detection. To inves- tigate this, we compute the scalar average of the attention contributions from these two layers as and find that it can classify a correctly generated answer with an AUROC of 0.63 on the Known dataset in VQA-Constraints (see Fig. 16 ). We find, however, that the model's confidence -the probability of the generated output token at the final layer -is a slightly stronger predictor, with an AUROC of 0.76 (see Fig. 17 ). This parallel's previous work in LLMs [37] which used the attention contributions from all the LLM's layers (via a linear model) to predict the generated answer's correctness. In comparison, our results suggest that a subset of MLLM's middle layers alone can be leveraged as a coarse "early" failure mode detector.Takeaway. MLLMs behave differently in terms of information retrieval from their parametric memory however quite similarly to LLMs in terms of information transfer to the final token. 
correcting and inserting long-tailed information in mllms: 
 Previous works have shown that counterfactual information can be inserted into LLMs by editing their causal layers [23, 24] . In this section, we verify if a similar approach can be used to edit a MLLM. Specifically, we introduce MULTEDIT, which applies a closed-form update to the early causal MLP layers we identified in Sec. 4 . We show that our approach can effectively both (i) fix erroneous answers, and (ii) insert new long-tailed information in LLaVa in a VQA task.Given an image-question (x, y), we denote a MLLM's generated answer as O. MULTEDIT updates a few parameters in the model such that it generates a new answer O * . Similar to [23, 13] , we update the W ℓ proj matrix at specific causal MLP layers such that the probability P(O * ) increases. Specifically, we view W ℓ proj as a linear-associated memory (where the matrix's input are treated as keys and its output as values) and uses a closed-form update to map the original keys to new (correct) values. Below, we outline the process for acquiring both the keys and values.Obtaining keys. Let {v i } N i=1 be the visual token embeddings and {e i } M i=1 the text token embeddings. Given a causal layer ℓ and the last token of a constraint c, we refer to the input of the layer's W ℓ proj matrix as its keys. Specifically, we define the key k c,ℓ to be the input embedding to the W ℓ proj matrix corresponding to the last token of the constraint. This can be obtained with a simple forward pass with the visual and text token embeddings.Obtaining values. Given a causal layer ℓ, we refer to the output of the layer's W ℓ proj matrix as its values. Specifically, we define z c,ℓ as the output embedding corresponding to the last token of the constraint. We optimize z c,ℓ such that the probability of the correct answer P(O * ) increases as:EQUATIONwhere L(z c,ℓ ) is the standard next-token prediction loss used to train LLMs:EQUATIONMULTEDIT modifies the W ℓ proj matrix such that the old keys k c,ℓ are mapped to the new optimized values z * c,ℓ which increase P(O * ). We define MULTEDIT's editing objective as:EQUATIONThe second regularization term ensures that W ℓ ′ proj , the weights before the edit, do not deviate too much from W ℓ proj . This helps to preserve performance on unrelated VQA pairs.We experimentally validate MULTEDIT in two real-world model-editing applications:Fixing incorrect answers to common questions. We test MULTEDIT on a set of ∼450 visual questions which LLaVa answers incorrectly (detected using incorrect VQA accuracy) from the multi-modal Known dataset in VQA-Constraints. These are generally questions about well-known places, persons and brands and companies.Inserting long-tailed VQA knowledge. We test MULTEDIT on a set of visual questions from the Encyclopedia-VQA dataset [25] . These query fine-grained knowledge about rare landmarks around the world, which MLLMs have been shown to struggle on [25] .For both settings, we compare MULTEDIT to the fine-tuning baselines:(i) fine-tuning from [23] which fine-tunes all the layers using the language modeling objective, and (ii) fine-tuning with constraints from [38] which fine-tunes the layers in a language model with a constraint on the weights to ensure local loss continuity.We measure the success of each edit operation using the following metrics: (i) Editing Efficacy, which uses P(O * ) to measure the edited model's ability to generate the correct answer for imagequestion (x, y). (ii) Generalization, which uses P(O * ) to measure the edited model's ability to generate the correct answer for the question y paraphrased using a language model (see Appendix for details), and (iii) Specificity, which uses VQA accuracy [1] to measure the edited model's performance on unrelated VQA questions. We consider unrelated questions to be those from the OK-VQA and Movies datasets in VQA-Constraints (see further details in Appendix G).Overall, our results show that updating the projection matrix using MULTEDIT at just a single early (causal) MLP layer can be a very effective approach for both correcting incorrect answers and inserting new knowledge in MLLMs.Fixing incorrect answers. In Fig. 6 (left), we show that MULTEDIT is able to successfully fix the generations for all questions with wrong answers. In editing efficacy, the average probability of the correct answer improves from 0.07 to 0.82 after the edit. We also observe strong generalization, with a probability of 0.76 for the correct answer even when the question is paraphrased. Although we see a small drop of 1.5% accuracy on unrelated image-question pairs, we note that MULTEDIT outperforms fine-tuning with and without constraints on all the metrics.Inserting long-tailed information. In Fig. 6 (right), we show that MULTEDIT is able to reliably insert new long-tailed knowledge in the model with an editing efficacy of 0.83. We see similar strong generalization when paraphrasing questions with an efficacy of 0.79. Similar to above, MULTEDIT incurs a small drop of 1.4% on unrelated questions but is less affected than other methods.In Fig. 14 , we also provide ablations showing that editing the early causal MLP layers leads to better editing efficacies than editing the middle or the later MLP layers.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will provide the final cleaned code with the camera-ready version of the paper. However, for the time being, we have provided all the experimental details in fine-grained details to reproduce our results. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We have added a section on Ethics in the paper.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusion: 
 Our paper takes a closer look at how MLLMs process multi-modal information. We contribute a novel multi-modal causing tracing methodology and test-bed, VQA-Constraints, as well as a range of novel insights on how MLLMs retrieve and transfer information. We also introduce a novel model-editing algorithm, MULTEDIT, which can effectively fix errors or introduce long-tailed knowledge in MLLMs using a simple closed-form update which targets the early causal MLPs. Overall, this work deepens our scientific understanding of recent MLLM architectures, and enables future work in this direction. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have added a section (Limitations and Societal Impact) in the paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The findings of our work are limited to factual questions on natural images with short-form answers. Future work should investigate other types of questions (e.g. subjective), image domains (e.g. charts), and longer-form answers. Our work also does not map visual tokens to specific concepts in the input image -another important research direction. Our editing method, MULTEDIT, while effective for correcting and adding new information, could also be used to add false or harmful information into a MLLM. Work is needed on how to reliably detect misinformation in multi-modal settings.In Sec. 3.1, we introduce the constraint framework for the task of factual VQA. In particular, there are two types of constraints: (i) Visual Constraint: A set of words in a question which refer to an entity in an image; (ii) Text Constraint: A set of words in a question which together with the visual constraint are responsible for answering a question. In our VQA-Constraints dataset there are two types of questions: (i) Single Constraint Questions: These questions comprise of only the visual constraint; (ii) Multi-Constraint Questions: These questions comprise of both the visual as well as the text constraint. In our paper, we primarily focus on the Single Constraint questions, therefore a majority of questions in our test bed of VQA-Constraints consist of only the visual constraints. Our VQA dataset VQA-Constraints which is annotated with constraints comprise of the following three parts:OK-VQA: We annotate the questions in the test-set of OK-VQA with the constraints. In total, this part consists of 5k questions. We use the same images as those used in the original OK-VQA test-set.Multimodal Movies: We use the text-only WikiMovies dataset from [37] and download images using the Bing API using the name of the directors. Our team filters and validates that the downloaded images are of the director itself and there's no noise in the downloaded images. We use a set of 1.5k questions in the final dataset.Multimodal Known: For understanding knowledge storage in language models, [23] use a probe dataset (known.json) consisting of 1.2k questions. We first replace the subject in the question with a constraint and then download images from Bing API. Our team validates that the downloaded images are correct and uses it to create the Multimodal Known dataset.Multi-Constraint Questions: The multi-constraint questions in VQA-Constraints are created from OK-VQA and Multimodal Movies. In particular, there are 150 multi-constraint questions for OK-VQA and 500 multi-constraint questions from Multimodal Movies. Given that the primary focus of our paper is on Single Constraint questions, this set of Multi-Constraint questions is relatively small. In total, VQA-Constraints consist of 9.7K VQA questions with Single Constraints and 650 Multi-Constraint VQA questions which we use for interpretability analysis. Annotating and Validating the Visual Constraints. To automatically annotate the constraints in the visual questions from VQA-Constraints, we use a strong language model such as GPT-4. In particular, we annotate only the Multimodal Known and OK-VQA sub-parts of the VQA-Constraints dataset. First from each dataset, we annotate 100 examples which we use as in-context examples to the language model to annotate new questions. Given these annotations, our team then verified if the annotation is correct and if incorrect, modified the constraint annotation to make it correct. In total, the visual constraints are annotated for 9.7k VQA questions.In Fig. 7 , we find that the standard causal tracing approach, which adds Gaussian noise to the span of visual tokens and the constraint in the text does not recover any causal states, even with a large window size of 10. This is true for recovering both the MLP as well as the self-attention causal layers.Layers Layers Token Index Token Index Causal Importance of Layers Causal Importance of Layers We find that using the standard causal tracing procedure from [23] is not able to recover causal states. Averaged across all the datapoints in Multimodal Known and Multimodal Movies from VQA-Constraints. A window size of 10 is used.In this section, we present various qualitative plots corresponding to MULTIMODALCAUSALTRACE showing that our method is able to recover causal states in both the MLP as well as self-attention layers very early on in the model. We note that even using a small window size of 1 is sufficient to recover causal states. We highlight that earlier works [23, 24] show that for language models, a larger window size of 10 is required and also the middle layers are used to recover the relevant knowledge. However, we find that in the presence of visual prompts, relevant knowledge is retrieved from the early layers. We use MULTIMODALCAUSALTRACE on multimodal phi-2, where the language model is phi-2 [17] , a small language model capable of obtaining strong performances and often close to large language models for certain language tasks. Similar to our earlier observations for LLaVa, we find that even in multimodal phi-2, the early MLP and self-attention layers are causal. This shows the generalizability of our results for latest multimodal language models. We also note that a small window size of 1, is sufficient to recover causal states in multimodal language models. In this section, we present qualitative plots for the multi-constraint questions from OK-VQA. These questions consist of a visual constraint as well as a text-constraint which in conjunction are used to answer a given question. Given that we earlier observe early causal layers corresponding to the visual constraint -in this section, we show results corresponding to the text-constraint. We have two primary observations: (i) A larger window size (at least a size of 6) is required to recover causal states, which highlights that a set of internal layers are used to retrieve relevant information; (ii) The causal layers span the early as well as the middle layers. This is true for both the MLP as well as self-attention layers. is better than the middle or the later layers. However, we find that editing the middle or the later layers also lead to a non-trivial editing accuracy, though it's lesser in terms of effectiveness than editing the early causal layers. Recent work [10] also observe this behaviour of non-trivial editing accuracy at locations which are not causal layers.In this section, we first show qualitative in Fig. 15 that the attention contributions from the constraint tokens to the last token is significantly more for correct answers than for incorrect answers. This is quantitatively highlighted and shown in Fig. 16 where we find that Layer 16 and Layer 17 to be the main orchestrator of this difference. A recent work [37] use attention contributions as a signal to detect hallucinations in language models. They show that the attention contribution values when used together with a linear model can be used to predict the correctness of the answer. In fact, they find this metric to be close to the confidence metric (which is the probability of the generated token). However, [37] necessitates the use of all self-attention layers which reduces it to be used as an "early" failure mode detection metric. For multimodal language models, instead of using all the self-attention layers, we use the average attention contributions from Layer 16 and Layer 17 as an "early" failure mode detection metric. From Fig. 17 , we find that the average attention contributions indeed lead to a non-trivial AUROC of 0.63, but it lags behind the confidence metric which obtains AUROC of 0.76. Our early results show that using the model internals can be used to "early" detect if the model is going towards a failure mode, but it lags behind the confidence although that requires the full forward pass.Layers Layers Correct Answers Incorrect AnswersThe editing dataset consists of two parts: (i) VQA questions from Multimodal Known on which the LLaVa-7B gives incorrect answers; and (ii) VQA questions from Encyclopedia-VQA which tests the long-tailed knowledge of the model. The long-tailed questions are primarily about landmarks. Below we present some qualitative examples on the long-tailed VQA questions:Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Taken from the Bunny repository. 


Decision: The submission received scores of Accept, Accept, Weak Accept and Borderline Accept. 

The paper applies and adapts interpretability techniques known for LLMs for Multimodal LLMs. The paper is thorough in providing insights about how MLLMs encode information but also how to leverage this knowledge to adapt them to new concepts. The reviewers are mostly positive about the work but point out that the contribution is limited as it mostly adapts techniques know for LLMs. However overall still recognize the value of adapting these techniques for LLMs and the insights derived from this exercise.

Reviews: review: paper_summary: paper_summary: This manuscripts studies mechanistic interpretability in autoregressive vision-language models. Towards this, the authors propose `MultiModalCausalTrace`, an extension of the causal tracing technique for analyzing text-only LLMs, which is to perturb "visual constraint" tokens with a set of semantically coherent, but irrelevant tokens, and measure its change effects in model behavior. The authors observe that VLMs store and transfer information at early layers, while LLMs operate in early-to-mid layers. Based on this observation, the paper proposes `MultEdit`, a technique that injects long-tailed information in these causal layers of VLMs.
strength_weakness: strength_weakness: - While `MultEdit` has demonstrated impressive editing efficiency and generalization performance, the technical novelty is somewhat limited  as it is an application of a well-tested technique for editing LLMs. I would also like to see its generalization performance not only on `VQA-Constraints`, but also on other standard VLM benchmarks, such as MMMU.
- While the authors finding clearly indicates VLMs store and transfer information differently than text-only LLMs, I'm hoping that the authors could give a more detailed explanation to the cause of this. Since all tested VLMs are fine-tuned from LLMs, it is conceivable that they would operate in a similar fashion.
- This work currently only examines multimodal fusion at the embedding level, while there are other popular approaches such as the Flamingo architecture. I'm curious to see whether the authors finding would still hold under these different architectural choices.
questions: questions: See weaknesses.
limitations: limitations: The authors have adequately addressed limitations and potential societal impacts of their work
paper_summary: paper_summary: This paper studies the information storage and transfer for the multi-modality large language model (MLLM). The author provides a comprehensive empirical study leveraging the causal tracking method, i.e., corrupting a clean model by perturbing the input prompt, to identify which layers are used to retrieve information relevant to the constraints in the prompt. The author also leverages the attention contributions to compute how much one set of input tokens influences a set of output tokens to track how information is transferred from visual tokens to the causal layers. To provide new insights, a new dataset called VQA-Constraints has been created to support the empirical study. The author provides many new insights that are different from the existing research in LLM that the MLLMs reply on MLP and self-attention block in much earlier layers for information storage and a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. The mechanism revealed in this study also inspires a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs.
strength_weakness: strength_weakness: This is a strong paper in general, and the review does not realize the critical weakness of the present paper. The reviewer may be curious whether the code will be public upon the acceptance.
questions: questions: Please refer to the Weaknesses sections for more details.
limitations: limitations: N/A.
paper_summary: paper_summary: The paper introduces MULTIMODALCAUSALTRACE, an algorithm designed to identify hidden states in large language models (LLMs) that store factual information, specifically extending this capability to vision-language models where images are encoded as visual tokens. Building on previous work in information storage identification, MULTIMODALCAUSALTRACE constructs three models: a clean model, a corrupted model (where visual tokens are randomly replaced, perturbing the hidden states), and a restored model (where some corrupted hidden states are replaced with clean ones). By conducting mediation analysis on the causal relationship between state corruption and next-token prediction outcomes, the algorithm can pinpoint layers associated with different facts. The study reveals that multi-modal LLMs exhibit distinct behaviors in information storage and propagation, such as storing facts in middle layers rather than earlier transformer layers.

Additionally, the paper contributes the VQA-constraints dataset, derived from existing visual QA datasets and annotated with constraints.

The paper also proposes MULTEDIT, an algorithm for inserting or correcting factual knowledge in multi-modal LLMs. MULTEDIT optimizes the projection matrix in the MLP layers of transformers to minimize the mean squared difference between the projection output and the value vector that maximizes the prediction probability of the desired output. Empirical results demonstrate that MULTEDIT effectively corrects factual errors in VQA tasks.
strength_weakness: strength_weakness: + The uniqueness and impact of multi-modal models are not thoroughly demonstrated. As stated in the paper, the proposed methods is largely based on the canonical Causal Trace(CT) and Rank-One Model Editing (ROME), with modifications in implementation allowing them to work for vision-language models. However, the choice (or absence) of most of such modifications are not well-motivated and no ablation study provided to understand their exact contribution. For example, the proposed MULTEDIT largely resembles the existing ROME method, with differences in how the key vector is found and how the optimal value vector is acquired. But it is not clear from the text why such changes are necessary and/or make the method works better for visual language model. 

+ The lack of discussion on multi-modal properties hinders the paper's novelty. The overall structure of the paper is similar to [23], and the main text does not substantially differentiate the methodology from prior work. 

+ Minor formatting issue in the reference: journal/conference names are missing in some items (such as [23] [24])

(The reference indices of the original submission are used throughout this review)
questions: questions: + In vanilla ROME editing, the value vector is found by optimizing both the probability of the desired output and similarity to the essence of the original sentence. But in MULTEDIT the later term is dropped. What is the purpose of such change and how does the updated objective preserve the understanding of the overall prompt?

+ In the creation of VQA-Constraints, does the authors manually correct all annotations generated by GPT-4? If so, what is the main advantage of starting with automated annotations?
limitations: limitations: The authors included a limitation section in the appendix covering drawbacks of the proposed method and potential negative societal impacts.
paper_summary: paper_summary: This paper presents an approach to investigate the layer in which multi-modal large language models retrieve factual information and show several insights into their behavior. To investigate causal tracing, they propose replacing the input text tokens with different ones so that the model can respond differently from the correct answer with the replaced text tokens, then they copy activations from the clean layers until the model can reconstruct the correct answer. Through this investigation, their primary finding is that information to answer a visual question is mainly retrieved from early MLP and self-attention layers in MLLMs, which is different from the insight that LLMs retrieve factual information from middle layers. They also reveal several facts: only a subset of visual tokens are involved in sending information to the LLM's early layers, and mid-layer self-attention layers are involved in transferring information from the early causal layers to the question’s final token. They also demonstrate that an approach similar to editing factual knowledge of LLMs can be used to overwrite the knowledge of MLLMs.
strength_weakness: strength_weakness: 1. They offer an interesting observation that MLLM retrieves factual information from early layers, but they lack insight into why this happens. I think it is very important to give intuitions on why their behavior is different from LLMs. 

2. They present the insight from their approach using specific examples as in Fig. 3. However, if my understanding is correct, they do not provide numerical stats of which layers are responsible for retrieving factual information. Since they construct a new dataset, I guess it is easy to compute quantitative values to compare in which layers LLM and MLLM are retrieving information. 

3. The overall idea of their approach is not very novel. They borrow the idea from prior work and adapt it to MLLM. The novel part of their approach is replacing input text tokens so that the model makes an incorrect answer. 

4. I think section 5 does not improve the value of this submission. The section focuses on how to edit the knowledge of MLLM following prior work. The topic is related, yet different from their core contributions.
questions: questions: At this phase, I am on the borderline. I acknowledge the importance of analyzing the behavior of MLLMs and think the proposed approach is simple and reasonable. But, I also think the findings from this paper are not very novel. 

1. Related to 2 in weaknesses, is the value of Fig. 3 all from a specific example? or is it averaged over many examples? 

2. Please respond to all the weaknesses above since I might misunderstand some parts.
limitations: limitations: Yes.

Comments: comment: Thank you for your detailed response! I have increased my score accordingly.