Understanding Information Storage and Transfer in Multi-Modal Large Language Models
Authors: Samyadeep Basu
Martin Grayson
Cecily Morrison
Besmira Nushi
Soheil Feizi
Daniela Massiceti
Venue: NeurIPS 2024 poster (2024)
Keywords: interpretability
multimodal generative models
VQA
Primary area: interpretability_and_explainability
TL;DR: framework for understanding information storage and flow in multimodal language models

Abstract: Understanding the mechanisms of information storage and transfer in Transformer-based models is important for driving model understanding progress. Recent work has studied these mechanisms for Large Language Models (LLMs), revealing insights on how information is stored in a model's parameters and how information flows to and from these parameters in response to specific prompts. However, these studies have not yet been extended to Multi-modal Large Language Models (MLLMs). Given their expanding capabilities and real-world use, we start by studying one aspect of these models -- how MLLMs process information in a factual visual question answering task. We use a constraint-based formulation which views a visual question as having a set of visual or textual constraints that the model's generated answer must satisfy to be correct (e.g. What movie directed by \emph{the director in this photo} has won a \emph{Golden Globe}?). Under this setting, we contribute i) a method that extends causal information tracing from pure language to the multi-modal setting, and ii) \emph{VQA-Constraints}, a test-bed of 9.7K visual questions annotated with constraints. We use these tools to study two open-source MLLMs, LLaVa and multi-modal Phi-2. Our key findings show that these MLLMs rely on MLP and self-attention blocks in much earlier layers for information storage, compared to LLMs whose mid-layer MLPs are more important. We also show that a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. We validate these mechanisms by introducing MultEdit a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs by targeting these causal blocks. We will publicly release our dataset and code.

Understanding Information Storage and Transfer in Multi-modal Large Language Models
Samyadeep Basu University of Maryland
Martin Grayson
Microsoft Research
Cecily Morrison Microsoft Research
Besmira Nushi Microsoft Research
Soheil Feizi
Daniela Massiceti Microsoft Research
University of Maryland
Abstract
Understanding the mechanisms of information storage and transfer in Transformer- based models is important for driving model understanding progress. Recent work has studied these mechanisms for Large Language Models (LL.Ms), revealing insights on how information stored in model's parameters and how information flows to and from these parameters in response to specific prompts. However, these studies have not yet been extended to Multi-modal Large Language Models (MLLMs). Given their expanding capabilities and real-world use, we start by studying one aspect of these models - how MLLMs process information in a factual visual question answering task. We use constraint-basee formulation which views a visual question as having a set of visual or textual constraints that the model's generated answer must satisfy to be correct (e.g. What movie directed by the director in this photo has won a Golden Globe?). Under this setting, we contribute i) method that extends causal information tracing from pure language to the multi-modal setting, and ii) VQA-Constraints, a test-bed of 9.7K visual questions annotated with constraints. We use these tools to study two open-source MLLMs, LLa Va and multi-modal Phi-2. Our key findings show that these MLLMs rely on MLP and self-attention blocks in much earlier layers for information storage, compared to LLMs whose mid-layer MLPs are more important. We also show that consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. We validate these mechanisms by introducing MULTEDIT, a model-editing algorithm that can correct errors and insert new long-tailed information into MILMMs by targeting these causal blocks.
Introduction
Multi-modal Large Language Models (MLLMs) trained both text and images are rapidly moving from research into deployment and are being used by millions of people. Yet, while there have been some advances in understanding how Large Language Models (LLMs) work, much less has been done to understand MLLMs. This paper begins to close this gap by studying how information is stored and transferred in MLLMs. To do this through the lens of factual Visual Question Answering (VQA) task very common use case of MLLMs today [26, 35, 27].
MLLMs process factual information in steps: information storage and information transfer. Information storage refers to how facts from a pre-training dataset are stored in a model's parameters its so-called 'parametric' memory. Information transfer describes how information from input prompts are propagated through these storage locations to the model's final output. Understanding
38th Conference on Neural Information Processing Systems (NcurIPS 2024),, Vancouver.

---

Constraint
Causal Importance Layers
Tokan
Causallmportance of Layers
1.0
Takan
Maximum Restored Prob:
0.8
Space
0.8
located
Very early
0.6
Need
Maximum Restored
0.6
causal sites
let
Answe
0.4
is
Early causel sites
werds
0.2
located
0.2
Ass
Image
0
5
10
15
20
25
30
0
5
10
15
20
25
Layers
Layers
LLavA-7B: Multimodal LLM
LLaMA (Vicuna)-7B
Figure 1: MILMS retrieve information from earlier internal layers compared to their L.MM counterparts. We find that very early MLP layers [1-4] have high indirect estimation effects to outputs (i.e.. they are causal) LLaVa-7B, whereas the middle MLP layers [4-7] are causal in LLaMA (Vicuna)-7B. For LLaMA, a larger window size (e.g.. 5) is also required to find causal sites, compared to window size of for LLaVA-7B.
these mechanisms can have many benefits, including ensuring models are factually grounded and informing better evaluation protocols.
Extensive works have explored how LLMs store and transfer factual information [12, 29, 24], however, this has been studied for multi-modal inputs. For example, it is suggested that auto-regressive Transformer-based LLMs store factual information in their mid-layer MLP parameters [12, 24). However, MLLMS and LLMs different: MLLM involves additional (continuous) image input, alongside a (discrete) text prompt. and requires additional modules to process it [26, 18, 19, 16, 7]. Typically, vision encoder (e.g. CLIP) is used convert this image into either visual tokens via projection layer [18, 19, 11] cross-attention layers [3, 7, 16, 36] which are then integrated into the language encoder. These differences suggest that our existing understanding of LLM information storage and transfer may not map directly to MLLMs.
In this work, and transfer. We use constraint-based formulation which views a visual question as having of either visual or textual constraints (e.g- What movie directed by this director has won Golden Globe?). The information retrieved by the model should satisfy these constraints (e.g. this director, Golden Globe) for the answer to be factually correct. This formulation, therefore, offers systematic way of understanding model's behavior. Under this framework, we propose MULTI- MODALCAUSALTRANN. which extends LLM causal tracing [12, 28, 33] to the multi-modal setting, to understand information storage, as well as leverage attention contribution methodologies [37, 8] study information transfer in MILMs. We also introduce T0A-Constraints, new dataset of 9.7k factual questions annotated with constraints, spanning natural images (from OK-VQA [22], WikiMovies [37], and Known [12].. With these tools, we study how a widely-used MLLM family processes multi-mode modal information, specifically LLaVa [18, 19] and multi-modal Phi-2 [11]*.
factual VQA task to study the mechanisms of multi-modal modal information storage
Our key findings that MILMs, in contrast LLMs: 1) retrieve information from earlier MI.P layers (i.c. layers 1-4 vs layers 4-7 in LLM) (see Fig. 1); and 2) use less parametric memory (require smaller window size retrieve this information), when answering a multi-moda, modal question. We also find that information is transferred from a given image to these early MLP blocks through a consistent subset visual tokens (e.g. last ~36 tokens from VL' CLIP encoder), and that the self-attention blocks in the middle layers are primarily responsible for shuttling this information to the last token.
Finally, demonstrate that by editing these early causal MLPs, new factual information into an MILM. Specifically, we propose a new model-editing algorithm, MULTEDIT, which modifies the projection matrix of the early causal MLPs with closed-form update. We empirically show MUl.EDIT effectiveness on questions from VQA-Constraints and Encyclopedic VQA [25].
correct errors and insert
In summary, contributions
Taken from the Bunny repository.

---

Prob/Amtarctica
Causal Importance of Layers
Clean Model (a)
0.86
1.0
Restored Model [c] Copy a layer from the clean to corrupted model
Prompt Tokens
0.8
Early Leyers
0.6
contina
Answer
0.4
Visunl Takens
Corrupted (b)
ProbjAntayétion) 0.005
0.2
MIP
Selt Attemtinn
0
10
20
30
Layers
This place: Vinson Massif
Visual Tokens
Replacement the constraint such that visual toleana are ignored
a causal tracing method understand informa- tion storage in MILMs. clean model is corrupted by replacing the question's 's constraint with an incorrect one for the given image (e.g. "This place" -> "Paris city" for an image of "Vinson Massif").. " The activations of windows of layers are then iteratively copied from the clean to the corrupted model until the corrupted model restores its output probability match the clean model's.
Figure 2: We introduce
1. A novel multi-modal causal tracing methodology that can be used study information storage from image and text inputs in MLLMs.
2. A new dataset, VQ4-Constraints, of 9.7K factual visual questions about natural images annotated with constraints to support future research in these directions.
3. A suite of novel insights on the mechanisms underlying and retrieval in MLLMs.
information storage
4. A eddting method, MULTEDIT, which we demonstrate can precisely correct erroneous information and insert new long-tailed information in an MLLM.
2 Related Works
Multimodal Large Language Models.. We consider MILM to be model that takes an image
and text as input, and generates text output [2]. Over the last year, such models have made tremendous advances in tasks like VQA and image captioning. including BLIP [15], BLIP-2 [16], Instruct-BLIP [7], LLaVA [18, 19], Flamingo [3] and multi-modal Phi-2 (from the Bunny repo) [11]. These MLLMs can broadly be categorized into two families based on how their visual information is integrated into the language model: (i) by embedding the vision encoder's 's output into each layer of the language model with cross-attention layer (e.g., Flamingo, BLIP) or, (ii) by mapping the vision encoder's ' output into "visual tokens" in the language model's input space (i.c. alongside the text tokens) via projection layer (e.g., LLaVA, Bunny). Both families widely used, however, the projection layer family has recently shown stronger performance on popular benchmark [18, 19, 11]. We, therefore, focus study of information storage and transfer this model family.
Interpretability MLLMs. A well-established arm of model interpretability examines the rela- tionship between a model's 's performance and its internals. A range of recent works have studied the internal mechanisms information storage [23, 34, 24] and transfer [9, 37] LLMs. However, to the best of our knowledge, only few works [30, 31] have studied the interpretability of MLLMs, with none specifically investigating the relationship between a model's outputs and its internal states. [30], for example, designs an interactive interface to visualize the attention maps an MLLM, while [31] explores the shortcomings of the CLIP vision encoder in MI.LMs. Neither consider the influence of both vision and text inputs model internals offer causal insights,
work does. Our model editing approach which targets the projection layer MILM family, is complemented by [6], who propose baselines for inserting information into the cross-attention layer MLLM family.
A Constraint-Basee Framework for Studying Information Storage and Transfer in MILMs
In this section, we describe the constraint-based formulation we use to study information storage and transfer in MILMs. Under this framing, we introduce MULTIMODAL AAMMMAAAAAUSAA.TRR...... ACE, a novel causal information tracing technique which we use to study multi-modal information storage. We also describe how we use attention contributions [8] to study multi-modal information transfer. Finally, describe VQA-Cone DDA-Conttraints, a new test-bed of visual questions annotated with constraints.
3

---

Restoring MLPs
Restoring MLPs
Restoring MLPs
1.0
1hot
0.8
Condradd
Very early laysal
0.8
Very earty Causal layers
0.6
Vary early leyars
0.6
Tntore
0.6
Dite:
0.4
cther
044
Coostcky
Toter
Last Takon
Layers
20
10
Layers
20
Layers
Restoring Self-Attention
1.0
Restoring Self-Attention
1.0
Restoring Self- Attentinn
Nare
uSoh
0.8
Very early Causal layers
0.8
Comstrand
Very earty layers
Very eerly Causel layers
0.6
0.6
Traass
0.4
One Touors
0.4
Dee Toker
0.4
Cossteir,
Tolars
0.2
0.2
0.2
Lust Tokan
10
20
30
10
Layers
20
30
10
Layers
30
Layers
Dataset: Movies
Muttimodal Known
Dataset: OKVQA
Information to answer a visual question with a single constraint is mainly retrieved from early MLP and self-attention layers in MLLMs. MULTIMODAL.CAUU CAUSALTRACE obtains high indirect estimation effect values LLaVa's early MLP and self-attention blocks corresponding to visual constraint, across all datasets in "QA-Constraints. This suggests these layers are causally important for information storage. The causal traces emerge with window size of 3 (see results with window size of I in Appendix C).
Figure 3:
3.1 A Multi-modal Constraint-base Framework
Prior works have used a constraint satisfaction framework to study LLMs [37, 20, 14] as they offer a systematic way of studying model behavior. This framing defines a constraint the question. The model must retrieve information that satisfies this constraint from its parametric memory in order to generate the correct answer. For example, for the question "What city is the Space Needle in?", the model must retrieve information relevant to the constraint "Space Needle". In the multi-modal setting, consider these textual constraints as well as introduce visual constraints. We define a visual constraint to be a set of words in the question which refers to an entity in the image. The model must similarly retrieve information about this entity to generate the correct answer. For example, given an image Christopher Nolan and the question "Name a movic directed by this director in 2006?", the visual constraint is "this director" (and the text constraint is "2006"). We refer to question involving both a visual and text constraint as a monti-constrrint question, and ones with only visual constraint single-cone conttaccnnttraitt question.
set of words in
We use this framework to study the widely-used "projection layer" MILMs family. These models are composed of visual encoder fo, a large language model Je and projection head py- The projection head py is responsible for mapping the output of the visual encoder into the input space of the language model, so-called "visual tokens". Given image-text pair denoted as (x. y), the language model processes them as g (Px h(((y))), where p.y (fe())) = {v,) is the == is the tokenized text inputs for the language
set of visual token embeddings and ((y) {4,,/1 model. These text tokens are processed by embedding layer / to obtain text token embeddings as { E Idd. Because we are interested in studying the outputs of specific layers in response to specific tokens, we use (-).. denote the output layer embedding corresponding to the kth token position and the layer é. We denote the output of MLP layer g0 g and the output of a self-attention block as (-)
3.2 MULTIMOOAL.CAUSAATT
Studying Information Storage in MLLMS
Causal tracing, derived from the causality literature [28], can be combined with constraint- based framework to gain causal insights on how information propagates through model with respect to specific constraint tokens. This has been used to identify where information is stored in LLMS [23, 24] and text-to-image generative models [4]. The central idea of causal tracing is to corrupt clean model by perturbing the input prompt. The activations of small subset of layers are then iteratively copied

---

Restoring MLPs
Restoring Self-Attention
1.0
1.0
0.8
0.8
Text
Early-layers
Early layy-
Tokens
Conotrinn
Mid-layers
0.6
Constrain Toxons
Mid-layers
0.6
Other
Other
0.4
Taxan
Tokans
0.2
0.2
Last Tokema
Last
Tokens
10
20
30
0
10
20
30
Layers
Layers
Figure 4: Information to answer a visual question with a visual and textual constraint is retrieved from early and middle MLP and self-attention layers in MLLMs. This suggests that meeting multiple constraint requires more parametric memory compared to single constraints. We show that MULTIMODAL- CAUSALTRAN obtains high indirect estimation effect values in the early and middle layers in LLaVa's ' the OK-VQA dataset VQA- VQA-Construints (see multi-constraint results from the Movies dataset in Appendix F).
from the clean model the corrupted model, until the corrupted model restores its output probability to match the clean model's. s. In this way, we can identify which layers are used to retrieve information relevant to the constraints the prompt (i.e. causally relate the input to the output).
In LLMs, the model is corrupted by adding a small amount of Gaussian noise to the embeddings of the textual constraint tokens (usually less than 5) [23]. In MLLMs, however, noise must be added to a much larger number of token embeddings those of the visual tokens (e.g., 576 in LLaVa and 729 in multi-modal Phi-2) from the projection head, and the textual tokens of the visual constraint (e.g. "this director"). Our experiments show that this large noise injection makes it difficult to revert the MLLM to a clean state and recover relevant causal traces (see Fig. 7).
We. therefore, introduce CULTIMODALCCUUSALTRR adding Gaussian noise to the embeddings, we instead corrupt the visual constraint token IDs by replacing them with token IDs from a separate word or phrase, such that the visual information is ignored. We illustrate this through an example. (1) Clean Model: Given image of the Space Needle, and the question y. "Which city is this building located in?", we compute the probability of the model's output O (e.g.. "Seattle") as Potean (O). (2) Corrupted Model: We substitute the visual constraint with an alternative such that the question does not require information from the image be answered (e.g.. "this building" is replaced with "Taj Mahal"). For multi-constraint question which also has a textual constraint, we can either add Gaussian noise to the textual constraint tokens' embeddings (since there are only few) or we can similarly replace the token IDs. After all replacements, we ensure that the question still makes semantic sense. We then measure the probability of original output O (O)) With the right corruption, P (O) is expected be low. (3) Restored Model: We then iteratively copy layer activations Is (.)..tmm and (.)..... 4k E [1, M,,M+ + N] from the clean model to the corrupted model, for each layer in turn. For given layer E and token position k. denote the restored probability Prestorea (O),.e After layer copied, we observe if Prestorea(O)a k. is high indicating that layer has strong causal association to the output O. In some cases, no layers have causal association. We note that this copy operation can be performed over window of layers W at a time, where W is the window size. A window size of I copies only one layer at time. Similar to [23], we track the indirect estimation effect for layer E as Prestored (O)k. Peorr (O) and use it as metric to track causal states. Intuitively, this measures the difference in the probability of under the corrupted model and when a layer is restored its original clean state. A high value indicates that the copied layer/s can restore the model to its original clean state (i.e. the layer is causal).
(see Fig. 2) to address this. Rather than
3.3 Studying Information Transfer in MLLMs with Attention Contributions
enables us to identify the specific layers model retrieves information from in order to answering a visual question. A second component of understanding how MILMs process factual information is understanding how input prompts are propagated through these storage locations to the model's ''s final output. For this, we use attention contributions [8. 37] which compute how much one set of input tokens influences a set of output tokens during the self-attention operation.
COLLTMODALLCAUALTTEE
5

---

Specifically, we use this to track (i) how information is transferred from the visual tokens to the causal layers, and (ii) from these layers the final token, where the final probabilities are computed.
Defining Attention Contributions. The attention operation in a Transformer [32] consists of the query, value, key and output weight matrices: W W W W Each of these are divided into H qa heads as W.A.A W.S., wet E Rdxdi Whthh E Rd xd where d is the dimension of the internal token embeddings and d, is the dimensionality of the token embedding for a particular attention head h. We define the attention contribution from a token j to token i in layer l as follows:
d (o W
(1)
where the attention matrix for layer and head h is defined as:
(oo()IM+NNG W "gga"Moo(MMMi MoO W
A = sof softmarl
Vd/H
where yo(:) Mo()1(M++)))-111 E RMM+N)xd and A E (R(M+M)( For understanding informa- tion transfer from the visual tokens to the causal layers, we use j E [1. M] and i Coonstraint. where Coonstraint corresponds to the last token in the visual constraint. For understanding the information transfer to the last token, we set = Cronstraint and i = Clas where Coonstaint corresponds to visual constraint's last token and Cass corresponds to the last token in the question.
3.4 VQA-Constraints: A Constraint Annotated Test-Bed for VQA
Alongside the above tools, we also introduce new test-bed called VQA-Constraints to enable our analyses. The test-bed consists of 9.7K natural images paired with factual questions, where each question is annotated with visual and textual constraints (see Sec. 3.1). Specifically, we source image-question pairs from the following datasets i) OK-VQA [21] which covers general knowledge questions, ii) Movies [37] which includes questions about movie directors and awards from Wikipedia, and iii) Known [23] which covers questions about countries, famous people, and places. The questions from the Movies and Known datasets are not originally multi-modal, so we modify them to refer to images which we source from Bing. We provide more details on the dataset construction and the constraint annotation in the Appendix B.
We leverage GPT-4 [27] annotate the textual and visual constraints in the visual questions in VQA- Constraints. We first manually annotate 100 examples from each dataset with constraints. We provide this as context GPT-4 and prompt it (see Appendix B) to annotate the constraints in new questions from Multimodal Known and OK- Due to the templated prompts in Multimodal Movies (e.g., "Name movie directed by this director? "), the constraints are constant ("this director") across all examples and does not require external annotations.
4
Key Findings in how MLLMs Store and Transfer Information
Using the tools presented in Sec. 3.1, we present our key findings in how MLLMs retrieve information from internal layers and how this information is transferred across the model.
4.1 Finding 1: Early MLPs and self-attention layers are causal
We find that information required to answer a visual question is mainly retrieved from the early-laye MLP and self-attention blocks of MLLM. This is confirmed by the high indirect estimation effects which MULTIMODALCAUS. ALTRACE assigns to the early layers in both LLa (see Fig. 3, Fig. 8) and multi-modal Phi-2 (see Appendix E) across the three datasets in VQA-Constraiins
This contrasts earlier results for LLMs which have been shown retrieve information from mid-layer MLPs to answer factual questions [23, 24). To obtain a fairer comparison, we apply MULTIMODAL- CAUSALTR ACE to LLaMA and LL which use the same language backbone. We run both on the same set of questions I.LMMA on the Known dataset [23], and LL on our multi-modal version of Known which modifies the questions to refer to an image (see Sec. 3.4). Fig. 1, we show that the MLPs in the first 4 layers are causal for LLa Va while the MLPs in layers 4-7 are causal for LLaMa.

---

Dataset: Movies. Late
Dataset: Multi-Known
Late visual tokens
Dataset: OKVQA
Late
Earty Actention Leyers
0
0.20
Early Actention Layers
P 2 0
0.20
Early Altanttoo Layere
0
0.20
2
0.15
4-
4-
0.10
0.10
0.10
6-
6
0.05
80
6
10
0.00
0.00
10
0.00
200
400
200
400
200
400
Visual Token Index
Visual Token Index
Visual Token Index
Figure 5: The late visual tokens are primarily responsible for transferring information from the image to the early causal layers, via the first self-attention layer. We visualize attention contributions (see Eq.(1) (1) from the visual tokens to the visual constraint token averaged across the three datasets in VQA-Constraints.
We also find that causal traces can be extracted from LLaVA with a minimum window size of 1, while LLaMa requires minimum window size of 5 to obtain any significant causal traces.
Although we find smaller window size of 1 to provide relevant causal traces (see Fig. 8), we find that a slightly larger window size of 3 results in consistent causal traces across all the questions. In Fig. 3. we report the results using a window size of 3, where we find that the early MLPs as well as self-attention layers are used to retrieve relevant knowledge to answer a visual question. We note that this observation is consistent for all the datasets VQA-Constraini
For multi-constraint questions that consist of a visual and textual constraint, information correspond- ing to the textual constraint is retrieved from a broader set of layers both the early and mid-layer MLP and self-attention blocks (see Fig. 4). We also find that a larger window size (at least of 6) is required to obtain any causal traces. This suggests that more parametrie memory is required to meet both a visual and textual constraint in a given question.
4.2 Finding 2: Only subset of visual tokens are involved in transferring information from the image to the early causal MLP layers.
We also find that the late visual tokens transfer information from the image to the early causal MLP layers (where information is mainly stored) via the first self-attention layer. We show this in Fig. 5 by visualizing the attention values in the early self-attention layers between each visual tokens and the final token in the visual constraint using the method described in Sec. 3.3. We see that the values are the highest in LLa first self-attention layer (which occurs just before the first causal MLP layer), specifically for the last subset of visual tokens (indexes 540-576 out of 576 in total). We hypothesize that these tokens may be summarizing image information that is relevant the given question before it is transferred and then stored in the MLPs, however we leave this to future study. We note that this pattern holds across all three datasets from VQA-Consinn UQA-Constraints.
4.3
Finding Mid-layer self-attention layers are involved in transferring information from the early causal layers to the question's 's final token
Rather counter-ituuitieely, we find that even though information stored in the early layers, the self-attention blocks in the middle layers (rather than layers immediately after) are responsible for propagating this information to the question's 's final token. The model's answer is sampled at this point, hence the information present here likely influences the ultimate generation. We see this in Fig. 15 and Fig. 16, which plots the attention contribution values between the last visual constraint token and the last token in the question across all the layers, using the methodology in Sec. 3.3. For LLaVa, the self-attention blocks in layers 16-17 are most active. This behaviour is similar in LLMs which also use mid-layer self-attention blocks to transfer information from (mid-layer) stored locations to the last token position.
4.4 Finding 4: Mid-layer self-attention contributions can be used to predict whether a MLLM will generate a correct answer, but model confidence is a more reliable predictor
When a MLLM generates a correct answer, we observe that the oeff-attention contributions in its middle layers are higher to when it generates an incorrect answer (see Fig. 15). For LLaVa, this is specifically the attention contributions between the last constraint token and the last prompt to- ken in the 16th and 17th layer. This holds potential to detect when a model will answer correctly without running a full inference pass, therefore enabling "carly" failure mode detection. To inves-
7

---

Fixing Error Cases
Introducing LoingTaiiled Knowledge
0.8
I HAu
Leadted Teatura MJAE:1 (00m
0.8
Teatur
Metric
0.6
Metric
0.6
0.4
0.4
0.2
0.2
Editing Efficacy Generalization
Specificity
Editing Efficacy Generalization
Specificity
Figure 6: MULTEDIT can correct error cases (left) and insert long-tailed factual knowledge (right) by
editing the early causal MLP layers in an MILM. We use the average probability of the correct token O as the metric for Editing Efficacy and Generalization, and VQA-Aceuracy for Specificity (higher the better for all).
tigate this, we compute the scalar average of the attention contributions from these two layers as a.8 to
" "ramttin Svontuin. and find that it can classify correctly generated answer with AUROC of 0.63 on the Known dataset in VQA- IDA-Constraints (see Fig. 16). We find, however, that the model's confidence the probability of the generated output token the final layer is slightly stronger predictor, with an AUROC of 0.76 (see Fig. 17). This parallel's 's previous work in LLMs [37] which used the attention contributions from all the L.LM's s layers (via a linear model) to predict the generated answer's correctness. In comparison, our results suggest that a subset of MLLM's middle layers alone can be leveraged as a coarse "early" failure mode detector.
Takeaway. MLLMs behave differently in terms of information retrieval from their parametric memory however quite similarly LLMs terms of information transfer the final token.
5 Correcting and Inserting Long- Tailed Information in MILMs
Previous works have shown that counterfactual information can be inserted into LLMs by editing their causal layers [23, 24]. In this section, we verify a similar approach can be used to edit a introduce MUL.TÉDIT, which applies a closed-form update the early
MILMM. Specifically, causal MLP layers we identified in Sec. 4. We show that our approach can effectively both (i) fix erroneous answers, and (ii) insert new long-tailed information in LLa Va in VQA task.
5.1 MULTEDIT
Given imagc-question (x. y). denote a MLLM's generated answer as O. MULTEDIT updates a few parameters in the model such that it generates new answer O Similar to [23, 13], we update the proo matrix at specific causal MLP layers such that the probability P(O*) increases. Specifically, we view W as a linear- associated memory (where the matrix's 's input are treated as keys and its output values) and uses closed-form update map the original keys to (correct) values. Below, we outline the process for acquiring both the keys and values.
Obtaining keys. Let Jo be the visual token embeddings and {e: J the text token embeddings. Given causal layer é and the last token of a constraint we refer to the input of the layer's W proj matrix as its keys. Specifically, corresponding to the last token of the constraint. This can be obtained with simple forward pass with the visual and text token embeddings.
define the key Ri, be the input embedding the Wrroj matrix
Obtaining values. Given causal layer we refer to the output of the layer's W proj matrix as its values. Specifically, define Ze,t the output embedding corresponding to the last token of the constraint. We optimize že,e such that the probability of the correct answer P(O*) increases as:
f arg min L(z.,))
(3)
where C(zc,e) is the standard next-token prediction loss used to train LLMs:
L(ze.) = log P(O" Po00P(O"luueeeee "U.""||u..NNEE......
(4)

---

MULTEDIT modifies the W proj matrix such that the old keys ke, are mapped to the new optimized values Eee which increase P(O"). We define MULTEDIT'S editing objective as:
Wer proj arg min I| W
proy Ke,e - =,,ell2 1 X|[ proj W proj ll
(5)
The second regularization term ensures that W proj the weights before the edit, do not deviate too much from W proj' This helps to preserve performance on unrelated VQA pairs.
5.2 Experimental details
We experimentally validate MULTEDIT in two real- model-editing applications:
Fixing incorrect answers common questions. We test MULTEDIT on a set of <-50 visual questions which LLaVa answers incorrectly (detected using incorrect VQA accuracy) from the multi-modal Known dataset in VQA- (QA-Constraints. These are generally questions about well-known places, persons and brands and companies.
Inserting long-tailed VQA knowledge. We test MULTEDIT on a of visual questions from the Encyclopedia- VQA dataset [25]. These query fine-grained knowledge about rare landmarks around the world, which MILMS have been shown to struggle [25].
For both settings, we compare MULTEDIT to the fine-tuning baselines:()) fine-tuning from [23] which fine-tunes all the layers using the language modeling objective, and (ii) fine-tuning with constraints from [38] which fine-tunes the layers in a language model with a constraint on the weights to ensure local loss continuity.
We measure the success of each edit operation using the following metrics: (i) Editing Efficacy. which uses P(O") to measure the edited model's ability to generate the correct answer for image- question (x, 3). (ii) Generalization, which uses P(O*) to measure the edited model's ability to generate the correct answer for the question paraphrased using language model (see Appendix for details), and (ii) Specificity, which uses VQA accuracy [1] to measure the edited model's performance unrelated VQA questions. We consider unrelated questions to be those from the OK-VQA and Movies datasets in VOA-Constraints (see further details in Appendix G).
5.3 Results
Overall, results show that updating the projection matrix using MULTEDIT at just single early (causal) MLP layer can be a very effective approach for both correcting incorrect answers and inserting new knowledge in MLLMs.
Fixing incorrect answers. In Fig. 6 (left), we show that MULTEDIT is able to successfully fix the generations for all questions with wrong answers. In editing efficacy, the average probability of the correct answer improves from 0.07 to 0.82 after the edit. We also observe strong generalization, with probability of 0.76 for the correct answer even when the question is paraphrased. Although we see a small drop of 1.5% accuracy on unrelated image-question pairs, we note that MULTEDIT outperforms fine-tuning with and without constraints on all the metrics.
Inserting long-tailes information. In Fig. 6 (right), we show that MULTEDIT is able to reliably insert new long-tailed knowledge in the model with an editing efficacy of 0.83. We see similar strong generalization when paraphrasing questions with an efficacy of 0.79. Similar to above, MULTEDDI incurs a small drop of 1.4% unrelated questions but is less affected than other methods.
In Fig. 14, also provide ablations showing that editing the early causal MLP layers leads to better editing efficacies than editing the middle or the later MLP layers.
6 Conclusion
Our paper takes a closer look at how MLLMs process multi- modal information. We contribute a novel multi-modal causing tracing methodology and test-bed, VOA-Constraints, as well as range of novel insights on how MLLMS retrieve and transfer information. We also introduce a novel model-cditing algorithm, MULTEDIT, which can effectively fix errors introduce long- sai-tailed knowledge in MLLMs using a simple closed-form update which targets the early causal MLPs. Overall, this work deepens our scientific understanding of recent MLLM architectures, and enables future work in this direction.

---

Author Contributions
Samyadeep Basu conceived the idea, implemented the algorithms, curated the dataset, ran experiments and wrote the paper. Daniela Massiceti provided critical feedback at every stage of the project (algorithms, dataset) and worked with Samyadeep write the paper. Besmira Nushi, Soheil Feizi and Cecily Morrison provided critical feedback on the project direction, paper writing and presentation. Martin Grayson helped the team with experiments on curating the probe dataset.
Acknowledgements
This project was supported in part through a part-time internship at Microsoft Research and in part was supported in part by grant from NSF CAREER AWARD 1942230, ONR YIP award N00014- 22-1-2271, 1, ARO's is Early Career Program Award 310902-00006 Army Grant No. W91 W91INF2120076 the NSF award SCF2212245, NSF Award No. 2229885 (NSF Institute for Trustworthy Al Law and Society, TRAILS), an Amazon Research Award and an award from Capital One. We thank Hamid Palangi for proofreading the draft and providing feedback.
References
[1] A. Agrawal, J. Lu, S. Antol, M. Mitchell, C. L. Zitnick, D. Batra, and D. Parikh. VQA: Visual Question Answering, 2016.
[2] A. Agrawal, I. Kajić, E. Bugliarello, E. Davoodi, A. Gergely, P. Blunsom, and A. Nematzadeh. Reassessing Evaluation Practices in Visual Question Answering: A Case Study Out-of- Distribution Generalization, 2023.
[3] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, and M. Reynolds. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems, volume 35, pages 23716-23736, 2022.
[4] S. Basu, N. Zhao, V. Morariu, S. Feizi, and V. Manjunatha. Localizing and Editing Knowledge in Text-to-Image Generative Models, 2023.
[5] S. Basu, K. Rezaei, P. Kattakinda, R. Rossi, C. Zhao, V. Morariu, V. Manjunatha, and S. Feizi. On Mechanistic Knowledge Localization in Text-to-Iman Image Generative Models, 2024.
[6] S. Cheng, B. Tian, Q. Liu, X. Chen, Y. Wang, H. Chen, and N. Zhang. Can We Edit Multimodal Large Language Models?, 2024.
[7] W. Dai, J. Li, D. Li, A. M. H. Tiong. J. Zhao, W. Wang. B. Li, N. Fung, and S. Hoi. InstructBLIP: Towards general-purper vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, volume 36, 2024.
[8] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield- -atdeeld-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish and C. Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
[9] M. Geva, J. Bastings, K. Filippova, and A. Globerson. Dissecting Recall of Factual Associations in Auto Regressiee Language Models, 2023.
[10] Hase, M. Bansal, B. Kim, and A. Ghandeharioun. Does Localization Inform Editing? Surprising Differences in Causality- -Based Localization vs. Knowledge Editing in Language Models, 2023.
[11] M. He, Y. Liu, B. Wu, Yuan, Y. Wang, T. Huang, and B. Zhao, Efficient Multimodal Learning from Data-centric Perspective. arXiv preprint arXiv:2402.1 11530, 2 2024.
[12] Kevin Meng and David Bau and Alex Andonian and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances Neural Information Processing Systems, 33:57 17372, 2022.
10

---

[13] T. Kohhnen. Correlation Matrix Memories. IEEE Transactions on Computers, C-21:353-359,
1972.
[14) C. Lawless, Schoeffer, L. Le, K. Rowan, S. Sen, C. S. Hill, J. Suh, and B. Sarrafzadeh. "I Want It That Way": Enabling Interactive Decision Support Using Large Language Models and Constraint Programming. arXiv preprint arXiv:2312.06908. 2023.
[15] J. Li, D. Li, C. Xiong, and S. Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation, 2022.
[16] Li, D. Li, Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference Machine Learning, pages 19730-19742. PMLR, 2023. ISBN 2640-3498.
[17] Y. Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and T. Lee. Textbooks Are All You Need II: phi- 1.5 technical report, 2023.
[18] H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved Baselines with Visual Instruction Tuning, arXiv preprint arXiv:2310.03744, 10 2023.
[19] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[20] A. Lu, H. Zhang, Y. Zhang, X. Wang, and D. Yang. Bounding the capabilities of large language models in open generation with prompt constraints. arXiv preprint arXiv:2302.09188, 2023.
[21] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. CoRR, abs/1906.00067, 2019. URL http: //arxiv// hrggabs/1906..0m
[22] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3195-3204, 2019.
[23] K. Meng, D. Bau, A. Andonian, and Belinkov. Locating and editing factual associations in gpt, 2023.
[24] K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau. Mass-Editing Memory in Transformer, 2023.
[25] T. Mensink, J. Uijlings, L. Castrejon, A. Goel, F. Cadar, H. Zhou, F. Sha, A. Araujo, and V. Ferrari. Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories. In VQA, pages 31 3113-3124, 2023.
[26] OpenAl. Gpt-4v(ision) system card, 9 2023.
[27] OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, Babuschkin, S. Balaji, V. Balcom, P. Bal- tescu, H. Bao, M. Bavarian, Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung. D. Cummings. J. Currier, Y. Dai, C. Decareaux, Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou,
D. Farhi, L. Fedus, N. Felix, S. Fishman, Forte, Fulford, L. Gao, Georges, Gibson,
V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes. J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, Hallacy, Han, J. Harris, He, M. Heaton, Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, Jain, Jang, A. Jiang, R. Jiang, Jin, Jin, Jomoto, B. Jonn, H. Jun, T. Kaftan, Lukasz Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, I.. Kilpatrick, J. W. Kim, C. Kim, Kim, J. H. Kirchner, Kiros, M. Knight, D. Kokotajlo, Fukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee. J. Leike, J. Leung,
11

---

D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, Mayne, B. Mc- Grew, S. M. McKinney, C. eavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menic,, L.. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo. H. Noh, L.. Ouyang, C. O'Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo,
Parish,
E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Prochl. R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. SchnurT, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shich, S. Shoker, P. Shyam, S. Sidor, F. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang. J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph. GPT-4 Technical Report, 2024.
[28] Pearl. Direct and Indirect Effects. In Proceedings of the Seventeenth Conference on Uncer- tainty in Artificial Intelligence, UAI'01, page 41 11 1-420, 2001. ISBN 1558608001.
[29] A. S. Sharma, D. Atkinson, and D. Bau. Locating and Editing Factual Associations in Mamba. arXiv preprint arXiv:2404.03664,, 2024.
[30] G. B. M. Stan, R. Y. Rohekar, Y. Gurwicz, M. I. Olson, A. Bhiwandiwalla, E. Aflalo, C. Wu, N. Duan, S.-Y. Tseng, and V. Lal. LVL.M-Inrrppre:: An Interpretability Tool for Large Vision- Language Models, 2024.
[31] S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, 2024.
[32] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. CoRR, abs/1706.03762, 2017. URL http://axxiv. 03762.
[33] J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. M. Shieber. Causal Medi- ation Analysis for Interpreting Neural NLP: The Case of Gender Bias. CoRR, abs/2004.12265 2020. URL https: http:///axxvvvoooo www./oog//////.....o.o.oooe /2g/4. 12265.
[34] K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the Wild:
a Cireuit for Indirect Object Identification in GPT-2 small, 2022.
[35] Z. Yang. L. Li, K. Lin, Wang. C.-C. Lin, Z. Liu, and L. Wang. The dawn of LMMs: Preliminary explorations with GPT-4V (ision). arXiv preprint 174.1 2023.
[36] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. CoCa: Contrastive Captioners are Image-Texi Text Foundation Models. arXiv preprint arXiv:2205.019//1 5 2022.
[37] M. Yuksekgonul, V. Chandrasekaran, E. Jones, S. Gunasekar, R. Naik, H. Palangi, E. Kamar, and B. Nushi. Attention Satisfies: A Canstrain-SSatisacttin Lens on Factual Errors of Language Models. In International Conference on Learning Representations, 2023.
[38] C. Zhu, A. S. Rawat, M. Zaheer, S. Bhojanapalli, D. Li, F. X. Yu, and S. Kumar. Modifying memories in transformer models. CoRR, abs/2012.00363, 2020. URL https://arxiv, org/ abb/22012000663. 00363.
12

---

A Limitations and Ethical Considerations
The findings of our work are limited to factual questions on natural images with short-form answers. Future work should investigate other types of questions (e.g. subjective), image domains (e.g. charts), and longer-form answers. Our work also does not map visual tokens to specific concepts in the input image another important research direction. Our editing method, MULTEDIT, while effective for correcting and adding new information, could also be used to add false or harmful information into a MLLM. Work is needed on how to reliably detect misinformation in multi-modal settings.
B VQA-Constraints Details
In Sec. 3.1, we introduce the constraint framework for the task of factual VQA. In particular, there are two types of constraints: (i) Visual Constraint: A set of words in a question which refer to an entity in an image; (ii) Text Constraint: A set of words in a question which together with the visual constraint are responsible for answering question. In our VQA- "QA-Constraintss dataset there are two types of questions: (i) Single Constraint Questions: These questions comprise of only the visual constraint; (ii) Multi-Constraint Questions: These questions comprise of both the visual as well as the text constraint. In our paper, we primarily focus on the Single Constraint questions, therefore a majority of questions in our test bed of VQA-Constraints consist of only the visual constraints. Our VQA dataset VQA-Construintss which is annotated with constraints comprise of the following three parts:
OK-VQA: We annotate the questions in the test-set of OK-VQA with the constraints. In total, this part consists of 5k questions. We use the same images as those used in the original OK-VQA test-set.
Multimodal Movies: We use the text-only WikiMovies dataset from [37] and download images using the Bing API using the name of the directors. Our team filters and validates that the downloaded images are of the director itself and there's noise in the downloaded images. We use set of 1.5k questions in the final dataset.
For understanding knowledge storage in language models, [23] use probe dataset (known. json) consisting of 1.2k questions. We first replace the subject in the question with a constraint and then download images from Bing API. Our team validates that the downloaded images are correct and uses it to create the Multimodal Known dataset.
Multimodal Known:
Multi-Constraint Questions: The multi-constrrint questions in VOA-Constraints are created from OK-V OK-VQA and Multimodal Movies. In particular, there are 150 multi-constraint questions for OK- VQA and 500 multi-constraie questions from Multimodal Movies. Given that the primary focus of questions relatively small.
our paper on Single Constraint questions, this set of
In total, VQA- UQA-Constsaints consist of 9.7K VQA questions with Single Constraints and 650 Multi- Constraint VQA questions which we use for interpretability analysis.
Deus plaa teniiinnii Vigs Conmsaaas
Devrppiin
Inample
Inaeple
Inampee
Wil
Natrendo Kreve
Redi Coneecin
Phrrpmen in entuccse
Tho soperts wit mn
The cip tsl citrcters
Mallas Mert
andie by Acmoo?
are cano dinteal by Sis Aooooo?
Waich du careeu ope- ind/daa
Hon doo penaatee 2/ compert Murriaa
Mraaii ed bancas pezus Killost
Meriar
2007
divata deoor 1991
Table 1: Description of the different VQA questions in VQA- -BA--onstsaints. The constraints are marked in italic in the columns.
Annotating and Validating the Visual Constraints. To automatically annotate the constraints in the visual questions from VOA-Constraints, we use a strong language model such as GPT-4. In particular, we annotate only the Multimodal Кnown and OK-VQA sub-parts of the VOA-Constraints dataset. First from each dataset, we annotate 100 examples which we use as in-context examples to the language model to annotate new questions. Given these annotations, our team then verified if the annotation is correct and if incorrect, modified the constraint annotation to make it correct. In total, the visual constraints are annotated for 9.7k VQA questions.
13

---

C Standard Causal Tracing Does Not Recover Causal States
In Fig. 7, we find that the standard causal tracing approach, which adds Gaussian noise to the span of visual tokens and the constraint in the text does not recover any causal states, even with large window size of 10. This is true for recovering both the MLP as well as the self-attention causal layers.
Iap of Layers
Causal Imporrance of Layers
0.5
0.5
Token Index
Token Index
5
0.4
0.4
10
No causal states
0.3
10
No causal states
0.3
15
0.2
15
0.2
20
0.1
20
0.1
0
10
20
30
10
20
30
Layers
Layers
Figure 7: Standard Causal Tracing: We find that using the standard causal tracing procedure from [23] is not able recover causal states. Averaged across all the datapoints in Multimodal Known and Multimodal Movies from VOA-Constraints. A window size of 10 is used.
D MULTIMODAL CAUSALTRACE. MULTTMODAA.CASSA.TR..CCC Qualitative Plots
In this section, we present various qualitative plots corresponding to MULTTMODALCAUSS.IRR showing that our method is able to recover causal states in both the MLP as well as self-attention layers very early in the model. We note that even using small window size of is sufficient to recover causal states. We highlight that earlier works [23, 24] show that for language models, larger window size of 10 is required and also the middle layers used to recover the relevant knowledge. However, we find that the presence of visual prompts, relevant knowledge is retrieved from the early layers.
1.0
60
1.0
1981
Meyoatycesadluc
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
10
20
30
1.0
0821
Waycarly sazadllggr
0.8
1838
0.8
0.8
0.6
0.6
0.6
0.4
Inw
0.4
0.2
10
20
30
10
20
30
Figure 8: MULTIMODAL.AAUAALEE with Window Size Early MLP layers are causal.
Qualitative tracing results for examples with single constraint questions (containing only a visual constraint).
14

---

1.0
1.0
1.0
Early Cousal Layer
0.8
0.8
165
Causal Layer
0.8
10100
0.6
0.6
0.6
0.4
0.4
0.4
2000
10
20
1.0
1.0
Early Causal Layer
0.8
Early Cauasi Layer
0.8
US2
Early Ceusal Layer
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
10
20
30
Layers
Figure 9: CEATUMMMDDLCAASALTTT....C with Window Size 1: Early self-attention layers are weakly causal compared to causal MLPs. Qualitative tracing results for examples with single constraint questions (containing only visual constraint).
E Multimodal Phi-2 Tracing Results
We use BUTLTIMODA.CAATAA.TT small language model capable of obtaining strong performances and often close large language models for certain language tasks. Similar to our earlier observations for LLaVa, we find that even in multimodal phi-2, the early MLP and self-attention layers causal. This shows the generalizability of our results for latest multimodal language models. We also note that a small window size of 1, is sufficient to recover causal states in multimodal language models.
on multimodal phi-2, where the language model phi-2 [17],
Very earty cousel layer layer
1,0
I5C3
Very causal
0.6
0032
1.27
cousal
0.8
kc40
me1-23
0.6
0113
0.6
0.4
0.1
1940
0.2
10
20
30
Very cany tayer
carly cousallayer
0.8
Very cousal
0.8
0.8
0.6
0.6
0.6
0.4
0.4
30
10
20
Layers
Figure 10: MOLTMMODALCAUSALTRR. with Window Size 1: Early MLP layers are causal for Multimodal Phi-2, similar to LLa Va-7B. Qualitative tracing results for examples with single constraint questions (containing only visual constraint).
15

---

1.0
1.0
1.0
Very early
layer
Very carty cousal layer
0.8
(27041)
0.8
0.8
A11a
0.4
0.4
0.4
0.2
0.2
0.2
30
10
20
1.0
0321
earyearlyco cousal
1901
Very earty causal
0.8
114
0.8
0.8
11:00
0.6
0.6
0.6
A1A
0.2
0.2
0.2
10
20
30
10
20
30
10
20
30
Layers
Figure 11: MULTIMODAL with Window Size Early Self-Attention layers are also causal for Multimodal Phi-2, similar to LLaVa-7B. Qualitative tracing results for examples with single constraint questions (containing only visual constraint).
F Multi-constraint Questions Plots
In this section, we present qualitative plots for the multi-constraint questions from OK-VQA These questions consist of a visual constraint as well as a text-constraint which in conjunction are used given question. Given that earlier observe early causal layers corresponding the visual constraint - in this section, we show results corresponding to the text-constraint. We have two primary observations: (i) larger window size (at least a size of 6) is required recover causal states, which highlights that set of internal layers are used to retrieve relevant information; (ii) The causal layers span the early as well as the middle layers. This is true for both the MLP as well as self-attention layers.
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
Fn5
0.4
Armc
0.2
0.2
10
20
30
30
1.0
1.0
606
0.8
0.6
0.6
0.4
0.4
Anowe
-90
0.2
0.2
who
0.2
365
0
10
20
30
10
20
30
30
Layers
Figure 12: MULTIMMODAL.AUSS..T... with Window Size 6: Early and mid MLP layers are causal. Qualitative tracing results for examples with multiple constraint questions (containing visual constraint and text constraint). Here text-constraint position. For multi-constraint questions, larger window size is needed to recover any causal states.
ACE is performed at the
16

---

1.0
1.0
1351
098
0.6
/60
0.6
0.6
0.4
0.2
0.2
-11
10
20
30
10 20
30
Layers
Figure 13: MULTMMOAACCAUSALTTR. with Window Size 6: Early and mid self-attention layers are causal for a majority of the visual questions. Qualitative tracing results for examples with multiple constraint questions (containing visual constraint and text constraint). Here MULTIMODALCAUSA.TRR
is performed at the text-constraint position. For multi-constraint questions, a larger window size is needed to recover any causal states.
G Model Editing Formulation and Further Results
As described in Sec. 5.1, MULTEDIT modifies the W proj matrix such that the old keys ka. are mapped to the new optimized values zn, such that the new values can drive the model towards increasing P(O*). We define the editing objective in MULTEDIT below:
W Vf = arg min ! Pros - 20 A/ proj
We proj
WE
term
term2
where W proj denotes the weights of the W proj matrix before the edit optimization. The second term
acts regularization which ensures that W proj does not deviate too much from W proj
From Eq. (6), observe that it can be solved using simple closed form update as follows:
Whe proj (AW proj k., ke,ck".,)''
Given access to the keys and values, this closed-form optimization can be solved CPU. We note that this closed-form update for model editing has been recently used for text-to-image models [4, 5]. However to the best of our knowledge, ours the first work use such an update for MI.LMs.
pyperparameters. We use a learning rate of 0.1 to optimize for the values using Adam Optimizer. For the regularization factor A, we use 0.01 after grid search. Amongst the set of early causal layers between 0-5, we find editing Layer 2 to result in the best editing efficacy.
In Fig. 14, show that editing the early causal MLP layers
Editing Efficacy
0.4
Layers
Figure 14: Editing early causal layers leads to better editing efficacy, though editing non-causal layers also leads to non-trivial editing efficacy. Similar results for language models have been highlighted in [10]. The editing performed the position of the last token of the visual constraint.
17

---

is better than the middle or the later layers. However, we find that editing the middle or the later layers also lead non-trivial editing accuracy, though it's lesser terms of effectiveness than editing the early causal layers. Recent work [10] also observe this behaviour of non-trivial editing accuracy at locations which are not causal layers.
H Information Transfer and Early Failure Mode Detection Qualitative Plots
In this section, we first show qualitative in Fig. 15 that the attention contributions from the constraint tokens to the last token is significantly more for correct answers than for incorrect answers. This quantitatively highlighted and shown in Fig. 16 where we find that Layer and Layer 17 to be the main orchestrator of this difference. A recent work [37] attention contributions to detect hallucinations language models. They show that the attention contribution values when used together with linear model be used to predict the correctness of the answer. In fact, they find this metric to be close to the confidence metric (which is the probability of the generated token). However, [37] necessitates the use all self-attention layers which reduces failure mode detection metric. For multimodal language models, instead of using all the self-attention layers, we use the average attention contributions from Layer 16 and Layer 17 as "early" failure mode detection metric. From Fig. 17, we find that the average attention contributions indeed lead to non-trivial AUROC of 0.63, but it lags behind the confidence metric which obtains AUROC of 0.76. Our early results show that using the model internals can be used to "early" detect the model is going towards failure mode, but lags behind the confidence although that requires the full forward pass.
signal
be used as "early"
Correct Answers
Layers
10
Layers
20
Layers
2
2
2
0.5
1.5
Text Constraint Toklen Index
Text Constraint Token Index
Text Constraint Token Index
Incorrect Answers
Layers
Layers
30
30
20
20
2
2
1
0.0
0.5
1.0
2.0
1.5
Text Constraint Token Indax
Text Constraint Token Index
Visual Taloon Index
Figure 15: Correct answers have stronger attention contributions from the constraint tokens (X-axis) to the last token, when compared to incorrect answers. These qualitative examples are from the Multimodal Known dataset COA-CCostraiins.
18

---

Correct Answers
Incorrect Answers
Avg. Attention Contribution
2.00
Avg. Attention Contributipn
7767
answers have higher attention contributions mid-layers (16 and 17 layer)
mm--
1.30
1.25
123
1.00
3.5
0.55
3.50
2:0
029
300
30
Layers
Layers
Attention Contributions
Attention Contributions
Levers
Layers
noxt-Constraiit Tokens
Text- Constraint Tokens
Figure 16: Correct answers have stronger attention contributions from the constraint tokens (X-axis) the last token, when compared to incorrect answers on an average. Computed on the Multimodal Known dataset in IQA-Constraints In particular the layer 16 and layer 17 are the distinguishing layers which have higher attention contributions the correct answers.
Attention Contribution Metric
Confidence Metric
Correct
Incornect
AUROC: 0.63
12
AUROC: 0.76
Count
Count
62
1.0
Norm of Attention Contributions
Probability
Figure 17:
Attention Contribution Metric although lags behind Confidence Metric, it can be used as a coarse "early" failure mode detection metric. Dataset used: Multimodal Known from
We use the average attention contributions from Layer 16 and Layer 17 for computing the failure mode metric.
I
Editing Dataset
The editing dataset consists of two parts: (i) VQA questions from Multimodal Known on which the Va-7B gives incorrect answers; and (ii) VQA questions from Encyclopedia- which tests the long-tailed knowledge of the model. The long-tailed questions are primarily about landmarks, Below present qualitative examples on the long-tailed VQA questions:
19

---

In which country is this house located?
In which country is this island located?
Inchcolm
Croome Court
In which country is this hotel located?
In which country is this house located?
Hilton Prague
Cragside
In which country is this island located?
In which country is this lake located?
Giannutri
Kandy Lake
Figure 18:
Qualitative Examples from the long-tailed knowledge editing subset. The questions are sourced from the questions involving landmarks in Encyclopedia-VQA.
J Compute
All our experiments are performed Nvidia-A6000 and A5000 GPUs.
20

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our abstract and introduction clearly state the claims made and they are backed by empirical results and plots in the subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification: Our paper has section on the Limitations of our work which also discusses ethical considerations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not have any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our paper provides all the experimental details in the main paper as well as the Appendix. All the small implementation details mentioned in the paper can be used to reproduce our results.
22

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will provide the final cleaned code with the camera-ready version of the paper. However, for the time being, we have provided all the experimental details in fine-gres ane-grained details to reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ Seuuides//odeeuumii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wttppplcccgguuueee//e/eeeeeeeeel wttcc//gggee//eee wwt////oobiiisiiioooeooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
23

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the experimental details have been clearly stated in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All the variability in the experimental settings have been clearly stated in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or /figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding //igures tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Our paper clearly states the GPU resources used the experimentation.
Guidelines:
The answer NA means that the paper does not include experiments.

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics attpss//nne http://enuripppppee httppplli//tiisssuieeeeeee
Answer: [Yes]
Justification:
We have added section Ethics in the paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have added section (Limitations and Societal Impact) in the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
If the authors
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
25

---

Justification: We have described the safeguards for responsible release of the data or models in the Appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have aptly credited and cited the original works and assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www./oo////////// conddatasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets in the paper are well documented and the documentation is provided the main paper as well as the Appendix.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
26

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "b56txbW0aw", "reviewer": "Reviewer_ieTM", "summary": "This manuscripts studies mechanistic interpretability in autoregressive vision-language models. Towards this, the authors propose `MultiModalCausalTrace`, an extension of the causal tracing technique for analyzing text-only LLMs, which is to perturb \"visual constraint\" tokens with a set of semantically coherent, but irrelevant tokens, and measure its change effects in model behavior. The authors observe that VLMs store and transfer information at early layers, while LLMs operate in early-to-mid layers. Based on this observation, the paper proposes `MultEdit`, a technique that injects long-tailed information in these causal layers of VLMs.", "questions": "See weaknesses.", "limitations": "The authors have adequately addressed limitations and potential societal impacts of their work", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper's presentation is very clear and tells a coherent story.\n- The main techniques (multimodal causal tracing) are reasonable; they are built upon well-tested frameworks for mechanistic interpretability of LLMs.\n- The findings are intriguing as the authors observe VLMs behave differently in terms of information storage and transferring, compared to LLMs.\n- `VQA_Constraint` is a valuable contribution to the interpretability community, and binding visual input to a natural language reference is a reasonable way to evaluate the knowledge for VQA, and is different from the many algorithmic tasks studied in prior mechanistic interpretability works on text-only LLMs.", "weaknesses": "- While `MultEdit` has demonstrated impressive editing efficiency and generalization performance, the technical novelty is somewhat limited  as it is an application of a well-tested technique for editing LLMs. I would also like to see its generalization performance not only on `VQA-Constraints`, but also on other standard VLM benchmarks, such as MMMU.\n- While the authors finding clearly indicates VLMs store and transfer information differently than text-only LLMs, I'm hoping that the authors could give a more detailed explanation to the cause of this. Since all tested VLMs are fine-tuned from LLMs, it is conceivable that they would operate in a similar fashion.\n- This work currently only examines multimodal fusion at the embedding level, while there are other popular approaches such as the Flamingo architecture. I'm curious to see whether the authors finding would still hold under these different architectural choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing constructive feedback and appreciating the paper’s presentation, techniques and the findings. We are glad that the reviewer feels that VQA-Constraint is a valuable contribution to the research community. \n\nBelow we address the weaknesses raised by the reviewer:\n\n**While MultEdit has demonstrated impressive editing efficiency and generalization performance, the technical novelty is somewhat limited as it is an application of a well-tested technique for editing LLMs**: While the high-level idea of employing a closed-form approximation to update (edit) a targeted localized weight matrix in the network is similar to LLMs, our proposed editing method has two key distinctions:  (i) our editing method does not require a sample of Wikipedia text to be cached. In contrast, in LLM editing techniques like ROME [23], this is required for computing the uncentered covariance to ensure that the values corresponding to keys for unrelated texts remain similar (see Eq (2)). In our case, we enforce this condition with an L2 regularizer which ensures that the updated weight matrices do not deviate significantly from the original weight matrices (controlled by a hyperparameter \\lambda - see Eq (5) in our paper). We find that this simple modification leads to strong editing results. One could use ROME to update MLLMs, but that would require caching the embeddings from a multimodal Wikipedia type dataset (which might not be readily available and clean for our use-case) thus adding an additional operation.   (ii) Our method *only* optimizes the multimodal language modeling next-token prediction loss. In contrast, LLM editing techniques optimize the language modeling loss along with a KL divergence loss which preserves the essence of the subject. In fact, during our early experimentations in the project, we used an additional KL divergence term to preserve the essence of the visual constraint. In particular, the objective of the KL divergence loss was to maintain the output probability distribution with the prompt (visual prompt (from visual tokens) + <visual-constraint> is a ) between the original MLLM and the MLLM whose value vector is optimized. However, empirically we did not see an improvement in the editing performance. Therefore we use the value vectors obtained with just the multimodal language modeling next-token prediction loss. \n\nOverall, MultEdit is simpler to implement and does not require caching a multimodal Wikipedia entry while leading to strong editing performances. We will add these distinctions in the final version of our paper. \n\nWe note that MultEdit is one of our paper’s contributions. The others include MultimodalCausalTrace, a multi-modal casual tracing methodology, and the dataset VQA-Constraints. Together these contributions advance our understanding of how large multimodal language models process information. Each required technical novelty in terms of handling multimodal information and architectures, besides adapting current techniques.\n\n**I would also like to see its generalization performance not only on VQA-Constraints, but also on other standard VLM benchmarks, such as MMMU**: Thank you for the suggestion! We have evaluated the edited LLaVa on MMMU and report the following results:  LLaVa-7B (unedited) obtains 34.4% on the MMMU validation set, and LLaVa-7B (averaged across multiple edits) obtains 33.8% on the MMMU validation set. This result highlights that targeted model editing does not impact generalization performance significantly. We use the evaluation scripts from https://github.com/BAAI-DCAI/Bunny for the evaluation. \n\n**While the authors finding clearly indicates VLMs store and transfer information differently than text-only LLMs, I'm hoping that the authors could give a more detailed explanation to the cause of this. Since all tested VLMs are fine-tuned from LLMs, ....**: The reviewer raises a good point. Through conducting our analyses, we developed several hypotheses as to why MLLMs retrieve information differently compared to LLMs. We suspect that the language model elicits distinct internal circuits (a set of MLPs and attention heads) in the presence and absence of visual information. If the visual information is not present, the language model relies on one type of circuit, but in the presence of information from visual tokens, this circuit gets overridden by another.  It can also be crucial to study what happens to the visual tokens, after the projection stage (i.e., what type of information flows into the final architecture and how are the constraints encoded) as that is the potential orchestrator towards eliciting a different circuit in the model. Our work lays down a strong foundation and a practical set of tools (that we plan to open source) to study these hypotheses. We believe that making these results and tools available will also help us initiate a discussion and welcome hypotheses proposals from the community that can be studied in future work. \n\n**This work currently only examines multimodal fusion at the embedding level, while there are other popular approaches such as the Flamingo architecture. I'm curious to see whether the authors finding would still hold under these different architectural choices**: We scoped our paper to focus on the embedding-level fusion MLLM family (e.g., LLaVa, Multimodal-Phi3) because it has generally demonstrated stronger performance across a wide range of multi-modal tasks, compared to other families. Models like Flamingo fall into another MLLM family where visual tokens are fused with the language decoder at different layers. Because of these mechanistic differences, we expect that the causal layers will be different, however we leave this to future work to confirm. We note that since our framework and methodologies (including the probe datasets) can be applied to *any* MLLM architecture, they could be used to conduct these future analyses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ys9pFZxjaE", "reviewer": "Reviewer_3JTJ", "summary": "This paper studies the information storage and transfer for the multi-modality large language model (MLLM). The author provides a comprehensive empirical study leveraging the causal tracking method, i.e., corrupting a clean model by perturbing the input prompt, to identify which layers are used to retrieve information relevant to the constraints in the prompt. The author also leverages the attention contributions to compute how much one set of input tokens influences a set of output tokens to track how information is transferred from visual tokens to the causal layers. To provide new insights, a new dataset called VQA-Constraints has been created to support the empirical study. The author provides many new insights that are different from the existing research in LLM that the MLLMs reply on MLP and self-attention block in much earlier layers for information storage and a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. The mechanism revealed in this study also inspires a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs.", "questions": "Please refer to the Weaknesses sections for more details.", "limitations": "N/A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, this paper is very well written, and the main messages are conveyed smoothly. The findings and takeaway message have been delivered clearly.\n\n2. The paper's novelty stems not only from the research problem but also from the design of the exploration method, the novel insights, and the corresponding proposed model-editing method. To the best of my knowledge, this should be the first work that provides a comprehensive study of knowledge tracing on MLLM, providing a great foundation for future exploration.\n\n3. The research findings are very interesting and insightful. They are validated by different datasets and the newly proposed dataset, making the insights solid and sound.\n\n4. The proposed model-editing method is effective and can partially validate the mechanism identified in the paper.", "weaknesses": "This is a strong paper in general, and the review does not realize the critical weakness of the present paper. The reviewer may be curious whether the code will be public upon the acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback, in particular acknowledging that this is the “first work that provides a comprehensive study of knowledge tracing on MLLM”. We are also glad that the reviewer finds our paper well-written and feels that the work can provide a solid foundation for future exploration in the space of MLLM interpretability. We will certainly be releasing the code publicly, and will link it in the final camera-ready version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3nAoi3vB2f", "reviewer": "Reviewer_BKyn", "summary": "The paper introduces MULTIMODALCAUSALTRACE, an algorithm designed to identify hidden states in large language models (LLMs) that store factual information, specifically extending this capability to vision-language models where images are encoded as visual tokens. Building on previous work in information storage identification, MULTIMODALCAUSALTRACE constructs three models: a clean model, a corrupted model (where visual tokens are randomly replaced, perturbing the hidden states), and a restored model (where some corrupted hidden states are replaced with clean ones). By conducting mediation analysis on the causal relationship between state corruption and next-token prediction outcomes, the algorithm can pinpoint layers associated with different facts. The study reveals that multi-modal LLMs exhibit distinct behaviors in information storage and propagation, such as storing facts in middle layers rather than earlier transformer layers.\n\nAdditionally, the paper contributes the VQA-constraints dataset, derived from existing visual QA datasets and annotated with constraints.\n\nThe paper also proposes MULTEDIT, an algorithm for inserting or correcting factual knowledge in multi-modal LLMs. MULTEDIT optimizes the projection matrix in the MLP layers of transformers to minimize the mean squared difference between the projection output and the value vector that maximizes the prediction probability of the desired output. Empirical results demonstrate that MULTEDIT effectively corrects factual errors in VQA tasks.", "questions": "+ In vanilla ROME editing, the value vector is found by optimizing both the probability of the desired output and similarity to the essence of the original sentence. But in MULTEDIT the later term is dropped. What is the purpose of such change and how does the updated objective preserve the understanding of the overall prompt?\n\n+ In the creation of VQA-Constraints, does the authors manually correct all annotations generated by GPT-4? If so, what is the main advantage of starting with automated annotations?", "limitations": "The authors included a limitation section in the appendix covering drawbacks of the proposed method and potential negative societal impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper tackles an important task of identifying and correcting factual information in visual language model, a topic that received relatively less attention than in language modeling.\n\n+ The paper provided strong motivation to the problem and background it addresses, as well as clear visual and formal presentation of the methodology adapted. \n\n+ The proposed solution is comprehensive and encompassed various aspect of the information storage analysis problem. It delivers valuable insights to some practical challenges of adapting existing LLM-based methods to vision-language models such as the increased number of constraint tokens due to visual encoding.", "weaknesses": "+ The uniqueness and impact of multi-modal models are not thoroughly demonstrated. As stated in the paper, the proposed methods is largely based on the canonical Causal Trace(CT) and Rank-One Model Editing (ROME), with modifications in implementation allowing them to work for vision-language models. However, the choice (or absence) of most of such modifications are not well-motivated and no ablation study provided to understand their exact contribution. For example, the proposed MULTEDIT largely resembles the existing ROME method, with differences in how the key vector is found and how the optimal value vector is acquired. But it is not clear from the text why such changes are necessary and/or make the method works better for visual language model. \n\n+ The lack of discussion on multi-modal properties hinders the paper's novelty. The overall structure of the paper is similar to [23], and the main text does not substantially differentiate the methodology from prior work. \n\n+ Minor formatting issue in the reference: journal/conference names are missing in some items (such as [23] [24])\n\n(The reference indices of the original submission are used throughout this review)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback, and for acknowledging the comprehensiveness of our findings and insights. We are glad that they also see the importance of being able to identify and correct factual information in MLLMs, which is currently an understudied problem. Below we address the points raised by the reviewer:\n\n**“The uniqueness and impact of multi-modal models are not thoroughly demonstrated…..’**: Our two methodologies, i) MultimodalCausalTrace for multi-modal causal tracing, and ii) MultEdit for multimodal editing method, are novel in their ability to work with multimodal (image-text) inputs. Towards i), we found that applying existing causal tracing methodologies out-of-the-box did not elicit relevant causal traces for multimodal inputs (attributed to large noise in the corrupted model due to large number of visual tokens) - see Fig.(7) - Appendix. This required us to extend these methodologies in a non-trivial way for multi-modal inputs. Towards ii) we introduce a novel editing approach that is based on ROME, but improves it with an updated loss objective that does not rely on caching a Wikipedia dataset. This is computationally preferred in a multi-modal setting, as images have a larger memory footprint than text alone. We also introduce a novel probe dataset, VQA-Constraints, to conduct our analyses and can be used to drive future impact in multimodal interpretability studies. \n\n**As stated in the paper, the proposed methods is largely based on the canonical Causal Trace(CT) and Rank-One Model Editing (ROME),.....**: We note that MultEdit has certain distinctions from the ROME method which we detail as follows:  While the high-level idea of employing a closed-form approximation to update (edit) a targeted localized weight matrix in the network is similar to LLMs, there are certain major distinctions in the editing method itself: (i) If looked at closely, the edit-optimization equations in ROME [23] - Eq. (2) is different from Eq. (5) of MultEdit in our paper. In principle, our editing method does not require caching a sample of Wikipedia text for computing the uncentered covariance in their equation. This term in the LLM editing works (e.g., ROME)  ensure that the values corresponding to keys for unrelated texts remain similar. However, in our case, we enforce this condition with the L2 regularizer ensuring that the updated weight matrices do not deviate significantly from the original weight matrices (controlled by a hyperparameter \\lambda). We find that this simple modification leads to strong editing results. One can also use the editing equation from ROME to update MLLMs, but that would require caching the embeddings from a multimodal Wikipedia type dataset (which is not readily available and requires curation/cleaning) which might be inefficient and also incur an additional operation.  (ii) We also find that obtaining the values by *only* optimizing the multimodal language modeling next-token prediction loss is sufficient towards obtaining good embeddings of values – which lead to good editing performance on using it with Eq. (5) in our paper.  During our experiments, we did add a KL divergence loss where the objective function was to maintain the output probability distribution with the prompt (visual prompt + <visual-constraint> is a ) between the original MLLM and the MLLM whose value vector is optimized. However, empirically we did not see an improvement in the editing performance. \n\nOverall, MultEdit is simpler in implementation and  does not require caching a multimodal Wikipedia entry while leading to strong editing performances.  We will add these distinctions from the LLM editing works such as ROME in the final version of our paper. We also note that our paper has a package of contributions (including MultimodalCausalTrace and VQA-Constraints) that as a whole advance the understanding of large multimodal language models.  Each of these steps required technical novelty in terms of handling multimodal information and architectures, besides adapting current techniques.\n\n**Minor formatting issue in the reference: journal/conference names are missing in some items (such as [23] [24]):**: Thank you for pointing this out. We will fix these formatting issues in the final version of the paper.\n\n**The lack of discussion on multi-modal properties hinders the paper's novelty. The overall structure of the paper is similar to [23], ...**: We point the reviewer to Section 3.2 in our paper, where we describe the causal tracing methods used for language models and how our method MultiModalCausalTrace differs and extends this to work for multi-modal inputs. In the final version of the paper, we will make the distinctions between Causal Trace and MultimodalCausalTrace more clear for better readability. We also point the reviewer to the first point and second point in the rebuttal where we discuss the distinctions and the uniqueness of the various components in our paper.\n\n**In the creation of VQA-Constraints, does the authors manually correct all annotations generated by GPT-4? If so, what is the main advantage of starting with automated annotations**: For the Movies dataset in VQA-Constraints, the questions are templated (e.g., questions are of the form ‘Name of movie directed by this director’) so the visual constraint (e.g., this director) is defined by default. For the OK-VQA and Known datasets in VQA-Constraints, we conducted an MTurk study to filter and correct erroneous annotations. Initially, we used automated annotations, finding that GPT-4 achieved approximately 96% accuracy in annotating constraints in VQA questions with as few as 50 in-context examples (which we manually annotate with constraints). Based on this high efficacy, we used GPT-4 for the initial annotations and then used MTurk to correct any remaining errors (<3% of the total examples in VQA-Constraints). We will include these filtering details in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xkedp0x5of", "reviewer": "Reviewer_eXiq", "summary": "This paper presents an approach to investigate the layer in which multi-modal large language models retrieve factual information and show several insights into their behavior. To investigate causal tracing, they propose replacing the input text tokens with different ones so that the model can respond differently from the correct answer with the replaced text tokens, then they copy activations from the clean layers until the model can reconstruct the correct answer. Through this investigation, their primary finding is that information to answer a visual question is mainly retrieved from early MLP and self-attention layers in MLLMs, which is different from the insight that LLMs retrieve factual information from middle layers. They also reveal several facts: only a subset of visual tokens are involved in sending information to the LLM's early layers, and mid-layer self-attention layers are involved in transferring information from the early causal layers to the question’s final token. They also demonstrate that an approach similar to editing factual knowledge of LLMs can be used to overwrite the knowledge of MLLMs.", "questions": "At this phase, I am on the borderline. I acknowledge the importance of analyzing the behavior of MLLMs and think the proposed approach is simple and reasonable. But, I also think the findings from this paper are not very novel. \n\n1. Related to 2 in weaknesses, is the value of Fig. 3 all from a specific example? or is it averaged over many examples? \n\n2. Please respond to all the weaknesses above since I might misunderstand some parts.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Their main strength is providing an approach to locate the layers in which MLLM models are retrieving factual information. Since the research on MLLM is popular these days, the technique may be a good one to analyze the behavior of MLLM. \n\n2. They present a new dataset to study the issue. \n\n3. The insight obtained from their approach is interesting. Unlike LLMs which retrieve factual information from middle layers, MLLMs retrieve factual information from early layers. This fact can be new to the community. \n\n4. Other observations such as how visual tokens are used in MLLMs are insightful too. These facts might be useful to think about the design of MLLMs.", "weaknesses": "1. They offer an interesting observation that MLLM retrieves factual information from early layers, but they lack insight into why this happens. I think it is very important to give intuitions on why their behavior is different from LLMs. \n\n2. They present the insight from their approach using specific examples as in Fig. 3. However, if my understanding is correct, they do not provide numerical stats of which layers are responsible for retrieving factual information. Since they construct a new dataset, I guess it is easy to compute quantitative values to compare in which layers LLM and MLLM are retrieving information. \n\n3. The overall idea of their approach is not very novel. They borrow the idea from prior work and adapt it to MLLM. The novel part of their approach is replacing input text tokens so that the model makes an incorrect answer. \n\n4. I think section 5 does not improve the value of this submission. The section focuses on how to edit the knowledge of MLLM following prior work. The topic is related, yet different from their core contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. We are glad that they acknowledge the strength and importance of our proposed methodologies and findings for future MLLM development and interpretability. Below we address the specific points raised by the reviewer: \n\n**They offer an interesting observation that MLLM retrieves factual information from early layers, but they lack insight into why this happens. I think….**: Through conducting our analyses, we developed several hypotheses as to why MLLMs retrieve information differently compared to LLMs. We suspect that the language model elicits distinct internal circuits (a set of MLPs and attention heads) in the presence and absence of visual information. If the visual information is not present, the language model relies on one type of circuit, but in the presence of information from visual tokens, this circuit gets overridden by another. This could be because the visual tokens contain fine-grained information about the constraint that forces the model to take a different path to obtain the final answer. Given the circuits are different, the nodes (e.g., MLPs/attention heads) in the circuit will therefore also be different, hence we see different causal layers activated for MLLMs versus LLMs. \n\nValidating these hypotheses fell outside the scope of this paper, however, our work lays down a strong foundation towards such future work. We will add a discussion on these hypotheses in the Appendix of the final version of the paper and lay down some possible directions for future work.\n\n**They present the insight from their approach using specific examples as in Fig. 3. However, if my understanding is correct, they do not provide numerical stats of which layers are...is the value of Fig. 3 all from a specific example?**: Fig. 3 is averaged across the examples in VQA-Constraints, thus providing a robust (visual) estimate of the causal layers in MLLMs for a factual VQA task. As suggested by the reviewer, we will augment this plot with numerical statistics to improve the readability of the results in the final version of the paper. \n\n**The overall idea of their approach is not very novel. They borrow the idea from prior work and adapt it to MLLM.**: Our proposed methodology, MultiModalCausalTrace, proposes a non-trivial adaptation to existing causal tracing methodologies in order for it to work for multi-modal inputs. Without it, no causal traces would have emerged (see Fig 7 - Appendix where we show this result) and our subsequent insights on MLLM information storage and retrieval would not have been possible.\n\nWe also note that building on existing methodologies, even in small ways, drives the field meaningfully forwards. For example, the Vision Transformer is different from the Transformer only in how it views image inputs as a grid of visual tokens, yet this architecture has unlocked enormous advances across a wide range of computer vision tasks. We believe that MultiModalCausalTrace along with our probe-dataset can similarly unlock rich mechanistic interpretability studies for MLLMs in the future.   \n\n**I think section 5 does not improve the value of this submission. The section focuses on how to edit the knowledge of MLLM following prior work. The topic is related, yet different from their core contributions**: Thank you for the feedback! We included Section 5, due to the following reasons:\n1) First, it allowed us to validate the multi-modal causal tracing methodology proposed in Section 4. If our identified causal traces were spurious or not meaningful, then any subsequent model editing would have immediately revealed this.  \n2) Second, it introduces (and validates) one practical application where these causal traces could be used in the real-world. Correcting errors and adding new information has been widely explored for LLMs in light of the huge costs (including environmental) that are associated with re-training or fine-tuning these models. We, therefore, felt it was an important direction to direct the research community’s attention towards. Interestingly, our experiments suggest that targeted MLLM model editing is a better alternative to fine-tuning which we hope will inform future research in this direction. The finding is particularly relevant for long tail knowledge, for which it may not even be possible to have sufficient data for training or fine tuning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their constructive feedback and comments. We have individually addressed the comments in their respective sections.  

We want to highlight that our paper proposes a package of novel contributions that work together to advance our understanding of how state-of-the-art multimodal models process visual and textual information. Our proposed methodologies, MultiModalCausalTrace and MultEdit are carefully designed, extending existing approaches for causal tracing and model editing to multimodal inputs and are generalizable across multiple MLLM families. Together with our proposed dataset, VQA-Contraints, we present a wide suite of novel insights on how MLLMs store/retrieve information, and how information can be inserted into them in a computationally efficient way. We will open source all our methods, datasets and code to the community to enable the community to drive further advances in MLLM interpretability.

Meta-review: The submission received scores of Accept, Accept, Weak Accept and Borderline Accept. 

The paper applies and adapts interpretability techniques known for LLMs for Multimodal LLMs. The paper is thorough in providing insights about how MLLMs encode information but also how to leverage this knowledge to adapt them to new concepts. The reviewers are mostly positive about the work but point out that the contribution is limited as it mostly adapts techniques know for LLMs. However overall still recognize the value of adapting these techniques for LLMs and the insights derived from this exercise.
Decision: Accept (poster)