Title: Understanding Information Storage and Transfer in Multi-Modal Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: interpretability

multimodal generative models

VQA
Abstract: Understanding the mechanisms of information storage and transfer in Transformer-based models is important for driving model understanding progress. Recent work has studied these mechanisms for Large Language Models (LLMs), revealing insights on how information is stored in a model's parameters and how information flows to and from these parameters in response to specific prompts. However, these studies have not yet been extended to Multi-modal Large Language Models (MLLMs). Given their expanding capabilities and real-world use, we start by studying one aspect of these models -- how MLLMs process information in a factual visual question answering task. We use a constraint-based formulation which views a visual question as having a set of visual or textual constraints that the model's generated answer must satisfy to be correct (e.g. What movie directed by \emph{the director in this photo} has won a \emph{Golden Globe}?). Under this setting, we contribute i) a method that extends causal information tracing from pure language to the multi-modal setting, and ii) \emph{VQA-Constraints}, a test-bed of 9.7K visual questions annotated with constraints. We use these tools to study two open-source MLLMs, LLaVa and multi-modal Phi-2. Our key findings show that these MLLMs rely on MLP and self-attention blocks in much earlier layers for information storage, compared to LLMs whose mid-layer MLPs are more important. We also show that a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. We validate these mechanisms by introducing MultEdit a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs by targeting these causal blocks. We will publicly release our dataset and code.

Understanding Information Storage and Transfer in Multi-modal Large Language Models

Abstract

Understanding the mechanisms of information storage and transfer in Transformerbased models is important for driving model understanding progress. Recent work has studied these mechanisms for Large Language Models (LLMs), revealing insights on how information is stored in a model's parameters and how information flows to and from these parameters in response to specific prompts. However, these studies have not yet been extended to Multi-modal Large Language Models (MLLMs). Given their expanding capabilities and real-world use, we start by studying one aspect of these models - how MLLMs process information in a factual visual question answering task. We use a constraint-based formulation which views a visual question as having a set of visual or textual constraints that the model's generated answer must satisfy to be correct (e.g. What movie directed by the director in this photo has won a Golden Globe ?). Under this setting, we contribute i) a method that extends causal information tracing from pure language to the multi-modal setting, and ii) VQA-Constraints , a test-bed of 9.7K visual questions annotated with constraints. We use these tools to study two open-source MLLMs, LLaVa and multi-modal Phi-2. Our key findings show that these MLLMs rely on MLP and self-attention blocks in much earlier layers for information storage, compared to LLMs whose mid-layer MLPs are more important. We also show that a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. We validate these mechanisms by introducing MULTEDIT, a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs by targeting these causal blocks.

1 Introduction

Multi-modal Large Language Models (MLLMs) trained on both text and images are rapidly moving from research into deployment and are being used by millions of people. Yet, while there have been some advances in understanding how Large Language Models (LLMs) work, much less has been done to understand MLLMs. This paper begins to close this gap by studying how information is stored and transferred in MLLMs. To do this through the lens of a factual Visual Question Answering (VQA) task - a very common use case of MLLMs today [26, 35, 27].

MLLMs process factual information in two steps: information storage and information transfer. Information storage refers to how facts from a pre-training dataset are stored in a model's parameters - its so-called 'parametric' memory. Information transfer describes how information from input prompts are propagated through these storage locations to the model's final output. Understanding

LLaVA-7B: Multimodal LLM

LLaMA (Vicuna)-7B

Layers

Layers

Constraint

Token

Very early

causal sites

Early causal sites

Constraint

Token

Maximum Restored Prob: 0.86

Maximum Restored Prob: 0.39

Image

1.0

1.0

<S>

USER

This

0.8

iSpace*

0.8

building

located

0.6

Need*

0.6

Answer

in

？

一

le*

few

In

a

0.4

is

0.4

words

located

0.2

0.2

in

ANT

0

5

10

0

5

10

1520

25

these mechanisms can have many benefits, including ensuring models are factually grounded and informing better evaluation protocols.

Extensive works have explored how LLMs store and transfer factual information [12, 29, 24], however, this has not been studied for multi-modal inputs. For example, it is suggested that auto-regressive Transformer-based LLMs store factual information in their mid-layer MLP parameters [12, 24]. However, MLLMs and LLMs are different: an MLLM involves an additional (continuous) image input, alongside a (discrete) text prompt, and requires additional modules to process it [26, 18, 19, 16, 7]. Typically, a vision encoder (e.g. CLIP) is used to convert this image into either visual tokens via a projection layer [18, 19, 11] or cross-attention layers [3, 7, 16, 36] which are then integrated into the language encoder. These differences suggest that our existing understanding of LLM information storage and transfer may not map directly to MLLMs.

In this work, we use a factual VQA task to study the mechanisms of multi-modal information storage and transfer. We use a constraint-based formulation which views a visual question as having a set of either visual or textual constraints (e.g. What movie directed by this director has won a Golden Globe ?). The information retrieved by the model should satisfy these constraints (e.g. this director , Golden Globe ) for the answer to be factually correct. This formulation, therefore, offers a systematic way of understanding a model's behavior. Under this framework, we propose MULTIMODALCAUSALTRACE, which extends LLM causal tracing [12, 28, 33] to the multi-modal setting, to understand information storage, as well as leverage attention contribution methodologies [37, 8] to study information transfer in MLLMs. We also introduce VQA-Constraints , a new dataset of 9.7k factual questions annotated with constraints, spanning natural images (from OK-VQA [22], WikiMovies [37], and Known [12]). With these tools, we study how a widely-used MLLM family processes multi-modal information, specifically LLaVa [18, 19] and multi-modal Phi-2 [11] 1 .

Our key findings are that MLLMs, in contrast to LLMs: 1) retrieve information from earlier MLP layers (i.e. layers 1-4 vs layers 4-7 in a LLM) (see Fig. 1); and 2) use less parametric memory (require a smaller window size to retrieve this information), when answering a multi-modal question. We also find that information is transferred from a given image to these early MLP blocks through a consistent subset of visual tokens (e.g. last ∼ 36 tokens from LLaVa's CLIP encoder), and that the self-attention blocks in the middle layers are primarily responsible for shuttling this information to the last token.

Finally, we demonstrate that by editing these early causal MLPs, we can correct errors and insert new factual information into an MLLM. Specifically, we propose a new model-editing algorithm, MULTEDIT, which modifies the projection matrix of the early causal MLPs with a closed-form update. We empirically show MULTEDIT's effectiveness on questions from VQA-Constraints and Encyclopedic VQA [25].

In summary, our contributions are:

Layers

…...

…

Visual Tokens

…

…...

…

Visual Tokens

…

Replacement of the constraint such that

visual tokens are ignored

Clean Model (a)

Prob(Antarctica)

= 0.86

Prob(Antarctica)

= 0.003

Restored Model

(c)

Copy a layer

from the

clean

to

corrupted

model

Early Layers

MLP

Self-Attention

This place

:

Vinson Massif

Copy Operation

Prompt Tokens

1.0

<S>

USER

This

0.8

place

Tocated

the

0.6

Answer

0.4

0.2

0

10

30

A novel multi-modal causal tracing methodology that can be used to study information storage from image and text inputs in MLLMs.

A new dataset, VQA-Constraints , of 9.7K factual visual questions about natural images annotated with constraints to support future research in these directions.

A suite of novel insights on the mechanisms underlying multi-modal information storage and retrieval in MLLMs.

Amodel-editing method, MULTEDIT, which we demonstrate can precisely correct erroneous information and insert new long-tailed information in an MLLM.

2 Related Works

Multimodal Large Language Models. . We consider a MLLM to be a model that takes an image and text as input, and generates a text output [2]. Over the last year, such models have made tremendous advances in tasks like VQA and image captioning, including BLIP [15], BLIP-2 [16], Instruct-BLIP [7], LLaVA [18, 19], Flamingo [3] and multi-modal Phi-2 (from the Bunny repo) [11]. These MLLMs can broadly be categorized into two families based on how their visual information is integrated into the language model: (i) by embedding the vision encoder's output into each layer of the language model with a cross-attention layer (e.g., Flamingo, BLIP) or, (ii) by mapping the vision encoder's output into 'visual tokens' in the language model's input space (i.e. alongside the text tokens) via a projection layer (e.g., LLaV A, Bunny). Both families are widely used, however, the projection layer family has recently shown stronger performance on popular benchmark [18, 19, 11]. We, therefore, focus our study of information storage and transfer on this model family.

Interpretability of MLLMs. A well-established arm of model interpretability examines the relationship between a model's performance and its internals. A range of recent works have studied the internal mechanisms of information storage [23, 34, 24] and transfer [9, 37] in LLMs. However, to the best of our knowledge, only a few works [30, 31] have studied the interpretability of MLLMs, with none specifically investigating the relationship between a model's outputs and its internal states. [30], for example, designs an interactive interface to visualize the attention maps in an MLLM, while [31] explores the shortcomings of the CLIP vision encoder in MLLMs. Neither consider the influence of both vision and text inputs on model internals or offer causal insights, as our work does. Our model editing approach which targets the projection layer MLLM family, is complemented by [6], who propose baselines for inserting information into the cross-attention layer MLLM family.

3 A Constraint-Based Framework for Studying Information Storage and Transfer in MLLMs

In this section, we describe the constraint-based formulation we use to study information storage and transfer in MLLMs. Under this framing, we introduce MULTIMODALCAUSALTRACE, a novel causal information tracing technique which we use to study multi-modal information storage. We also describe how we use attention contributions [8] to study multi-modal information transfer. Finally, we describe VQA-Constraints , a new test-bed of visual questions annotated with constraints.

Restoring MLPs

Layers

Dataset

: Movies

Dataset

: Multimodal Known

Layers

Restoring MLPs

Constraint

Tokens

Constraint

Tokens

Last Token

Last Token

Other

Tokens

Other

Tokens

Constraint

Tokens

Constraint

Tokens

Very early Causal layers

Very early Causal layers

Very early Causal layers

Very early Causal layers

Layers

Layers

Layers

Layers

Very early Causal layers

Very early Causal layers

Constraint

Tokens

Constraint

Tokens

Other

Tokens

Other

Tokens

Dataset

: OKVQA

Restoring MLPs




1.0

1.0

1.0

Name

USER

0.8

0.6

0.6

0.8

0.8

0.6

0.4

0.4

0.4

0.2

0.2

F0.2

0

10

30

0

10

30

0

10

30

1.0

1.0

1.0

Name

USER

0.6

0.8

0.6

0.8

0.6

Ea

0.4

0.4

0.4

0.2

0.2

0.2

0

10

30

0

10

30

0

10

30

3.1 A Multi-modal Constraint-based Framework

Prior works have used a constraint satisfaction framework to study LLMs [37, 20, 14] as they offer a systematic way of studying model behavior. This framing defines a constraint as a set of words in the question. The model must retrieve information that satisfies this constraint from its parametric memory in order to generate the correct answer. For example, for the question 'What city is the Space Needle in?", the model must retrieve information relevant to the constraint 'Space Needle". In the multi-modal setting, we consider these textual constraints as well as introduce visual constraints. We define a visual constraint to be a set of words in the question which refers to an entity in the image. The model must similarly retrieve information about this entity to generate the correct answer. For example, given an image of Christopher Nolan and the question 'Name a movie directed by this director in 2006 ?', the visual constraint is ' this director ' (and the text constraint is ' 2006 '). We refer to a question involving both a visual and text constraint as a multi-constraint question, and ones with only a visual constraint as a single-constraint question.

We use this framework to study the widely-used 'projection layer' MLLMs family. These models are composed of a visual encoder f θ , a large language model g ϕ and a projection head p γ . The projection head p γ is responsible for mapping the output of the visual encoder into the input space of the language model, as so-called 'visual tokens'. Given an image-text pair denoted as ( x , y), the language model processes them as g ϕ ( p γ ( f θ ( x )) , h ( t ( y ))) , where p γ ( f θ ( x )) = { v i } N i =1 is the set of visual token embeddings and t ( y ) = { t i } i M =1 is the tokenized text inputs for the language model. These text tokens are processed by an embedding layer h to obtain text token embeddings as { e i } i M =1 ∈ R d . Because we are interested in studying the outputs of specific layers in response to specific tokens, we use g ϕ ( . ) k,ℓ to denote the output layer embedding corresponding to the k th token position and the layer ℓ . We denote the output of a MLP layer as g ϕ ( . ) k,ℓ mlp and the output of a self-attention block as g ϕ ( . ) k,ℓ attn .

3.2 MULTIMODALCAUSALTRACE: Studying Information Storage in MLLMs

Causal tracing, derived from the causality literature [28], can be combined with a constraint-based framework to gain causal insights on how information propagates through a model with respect to specific constraint tokens. This has been used to identify where information is stored in LLMs [23, 24] and text-to-image generative models [4]. The central idea of causal tracing is to corrupt a clean model by perturbing the input prompt. The activations of a small subset of layers are then iteratively copied

Text

Constraint

Tokens

Text

Constraint

Tokens

Other

Tokens

Other

Tokens

Last

Tokens

Last

Tokens

Restoring MLPs

Early-layers

Mid-layers

Early-layers

Mid-layers

Layers

Layers

1.0

1.0

0.8

0.8

0.6

0.4

0.4

0.2

0.2

0

10

30

0

10

30

from the clean model to the corrupted model, until the corrupted model restores its output probability to match the clean model's. In this way, we can identify which layers are used to retrieve information relevant to the constraints in the prompt (i.e. causally relate the input to the output).

In LLMs, the model is corrupted by adding a small amount of Gaussian noise to the embeddings of the textual constraint tokens (usually less than 5) [23]. In MLLMs, however, noise must be added to a much larger number of token embeddings - those of the visual tokens (e.g., 576 in LLaVa and 729 in multi-modal Phi-2) from the projection head, and the textual tokens of the visual constraint (e.g. ' this director '). Our experiments show that this large noise injection makes it difficult to revert the MLLM to a clean state and recover relevant causal traces (see Fig. 7).

We, therefore, introduce MULTIMODALCAUSALTRACE (see Fig. 2) to address this. Rather than adding Gaussian noise to the embeddings, we instead corrupt the visual constraint token IDs by replacing them with token IDs from a separate word or phrase, such that the visual information is ignored. We illustrate this through an example. (1) Clean Model : Given an image x of the Space Needle, and the question y , 'Which city is this building located in?', we compute the probability of the model's output O (e.g., 'Seattle') as P clean ( O ) . (2) Corrupted Model: We substitute the visual constraint with an alternative such that the question does not require information from the image to be answered (e.g., 'this building' is replaced with 'Taj Mahal'). For a multi-constraint question which also has a textual constraint, we can either add Gaussian noise to the textual constraint tokens' embeddings (since there are only a few) or we can similarly replace the token IDs. After all replacements, we ensure that the question still makes semantic sense. We then measure the probability of original output O as P corr ( O ) . With the right corruption, P corr ( O ) is expected to be low. (3) Restored Model: Wethen iteratively copy layer activations g ϕ ( . ) k,ℓ mlp and g ϕ ( . ) k,ℓ attn ∀ k ∈ [1 , M + N ] from the clean model to the corrupted model, for each layer in turn. For a given layer ℓ and token position k , we denote the restored probability as P restored ( O ) k,ℓ . After a layer is copied, we observe if P restored ( O ) k,ℓ is high - indicating that layer ℓ has a strong causal association to the output O . In some cases, no layers have a causal association. We note that this copy operation can be performed over a window of layers { ℓ i } i W =1 at a time, where W is the window size. A window size of 1 copies only one layer at a time. Similar to [23], we track the indirect estimation effect for a layer ℓ as P restored ( O ) k,ℓ -P corr ( O ) and use it as a metric to track causal states. Intuitively, this measures the difference in the probability of O under the corrupted model and when a layer ℓ is restored to its original clean state. A high value indicates that the copied layer/s can restore the model to its original clean state (i.e. the layer is causal).

3.3 Studying Information Transfer in MLLMs with Attention Contributions

MULTIMODALCAUSALTRACE enables us to identify the specific layers a model retrieves information from in order to answering a visual question. A second component of understanding how MLLMs process factual information is understanding how input prompts are propagated through these storage locations to the model's final output. For this, we use attention contributions [8, 37] which compute how much one set of input tokens influences a set of output tokens during the self-attention operation.

Specifically, we use this to track (i) how information is transferred from the visual tokens to the causal layers, and (ii) from these layers to the final token, where the final probabilities are computed.

Defining Attention Contributions. The attention operation in a Transformer [32] consists of the query, value, key and output weight matrices: W ℓ q , W ℓ v , W ℓ k , W ℓ o . Each of these are divided into H heads as W ℓ,h q , W ℓ,h v , W ℓ,h k ∈ R d × d h , W ℓ,h o ∈ R d h × d , where d is the dimension of the internal token embeddings and d h is the dimensionality of the token embedding for a particular attention head h . We define the attention contribution from a token j to token i in layer ℓ as follows:

where the attention matrix for layer ℓ and head h is defined as:

where g ϕ ( . ) 1:( M + N ) ,ℓ -1 ∈ R ( M + N ) × d and A ℓ,h ∈ R ( M + N ) × ( M + N ) . For understanding information transfer from the visual tokens to the causal layers, we use j ∈ [1 , M ] and i = c constraint, where c constraint corresponds to the last token in the visual constraint. For understanding the information transfer to the last token, we set j = c constraint and i = c last, where c constraint corresponds to visual constraint's last token and c last corresponds to the last token in the question.

3.4 VQA-Constraints : A Constraint Annotated Test-Bed for VQA

Alongside the above tools, we also introduce a new test-bed called VQA-Constraints to enable our analyses. The test-bed consists of 9 . 7 K natural images paired with factual questions, where each question is annotated with visual and textual constraints (see Sec. 3.1). Specifically, we source image-question pairs from the following datasets i) OK-VQA [21] which covers general knowledge questions, ii) Movies [37] which includes questions about movie directors and awards from Wikipedia, and iii) Known [23] which covers questions about countries, famous people, and places. The questions from the Movies and Known datasets are not originally multi-modal, so we modify them to refer to images which we source from Bing. We provide more details on the dataset construction and the constraint annotation in the Appendix B.

We leverage GPT-4 [27] to annotate the textual and visual constraints in the visual questions in VQAConstraints . We first manually annotate 100 examples from each dataset with constraints. We provide this as context to GPT-4 and prompt it (see Appendix B) to annotate the constraints in new questions from Multimodal Known and OK-VQA. Due to the templated prompts in Multimodal Movies (e.g., 'Name a movie directed by this director ? '), the constraints are constant (' this director' ) across all examples and does not require external annotations.

4 Key Findings in how MLLMs Store and Transfer Information

Using the tools presented in Sec. 3.1, we present our key findings in how MLLMs retrieve information from internal layers and how this information is transferred across the model.

4.1 Finding 1: Early MLPs and self-attention layers are causal

We find that information required to answer a visual question is mainly retrieved from the early-layer MLP and self-attention blocks of a MLLM. This is confirmed by the high indirect estimation effects which MULTIMODALCAUSALTRACE assigns to the early layers in both LLaVa (see Fig. 3, Fig. 8) and multi-modal Phi-2 (see Appendix E) across the three datasets in VQA-Constraints .

This contrasts earlier results for LLMs which have been shown to retrieve information from mid-layer MLPs to answer factual questions [23, 24]. To obtain a fairer comparison, we apply MULTIMODALCAUSALTRACE to LLaMA and LLaVa which use the same language backbone. We run both on the same set of questions - LLaMA on the Known dataset [23], and LLaVa on our multi-modal version of Known which modifies the questions to refer to an image (see Sec. 3.4). In Fig. 1, we show that the MLPs in the first 4 layers are causal for LLaVa while the MLPs in layers 4-7 are causal for LLaMa.

Late visual tokens

Dataset : Multi-Known

Dataset : Movies

Visual Token Index

Early Attention Layers

Early Attention Layers

Late visual tokens

Dataset : OKVQA

Late visual tokens

Visual Token Index

Visual Token Index

Early Attention Layers

0

0.20

0

0.20

0

0.20

2

0.15

2

0.15

2

0.15

4

4

4

0.10

0.10

0.10

6

6

6

0.05

0.05

0.05

8

8

8

0.00

0.00

10

0.00

0

0

0

We also find that causal traces can be extracted from LLaVA with a minimum window size of 1, while LLaMa requires a minimum window size of 5 to obtain any significant causal traces.

Although we find a smaller window size of 1 to provide relevant causal traces (see Fig. 8), we find that a slightly larger window size of 3 results in consistent causal traces across all the questions. In Fig. 3, we report the results using a window size of 3, where we find that the early MLPs as well as self-attention layers are used to retrieve relevant knowledge to answer a visual question. We note that this observation is consistent for all the datasets in VQA-Constraints .

For multi-constraint questions that consist of a visual and textual constraint, information corresponding to the textual constraint is retrieved from a broader set of layers both the early and mid-layer MLP and self-attention blocks (see Fig. 4). We also find that a larger window size (at least of 6) is required to obtain any causal traces. This suggests that more parametric memory is required to meet both a visual and textual constraint in a given question.

4.2 Finding 2: Only a subset of visual tokens are involved in transferring information from the image to the early causal MLP layers.

We also find that the late visual tokens transfer information from the image to the early causal MLP layers (where information is mainly stored) via the first self-attention layer. We show this in Fig. 5 by visualizing the attention values in the early self-attention layers between each visual tokens and the final token in the visual constraint using the method described in Sec. 3.3. We see that the values are the highest in LLaVa's first self-attention layer (which occurs just before the first causal MLP layer), specifically for the last subset of visual tokens (indexes 540-576 out of 576 in total). We hypothesize that these tokens may be summarizing image information that is relevant to the given question before it is transferred and then stored in the MLPs, however we leave this to future study. We note that this pattern holds across all three datasets from VQA-Constraints .

4.3 Finding 3: Mid-layer self-attention layers are involved in transferring information from the early causal layers to the question's final token

Rather counter-intuitively, we find that even though information is stored in the early layers, the self-attention blocks in the middle layers (rather than layers immediately after) are responsible for propagating this information to the question's final token. The model's answer is sampled at this point, hence the information present here likely influences the ultimate generation. We see this in Fig. 15 and Fig. 16, which plots the attention contribution values between the last visual constraint token and the last token in the question across all the layers, using the methodology in Sec. 3.3. For LLaVa, the self-attention blocks in layers 16-17 are most active. This behaviour is similar in LLMs which also use mid-layer self-attention blocks to transfer information from (mid-layer) stored locations to the last token position.

4.4 Finding 4: Mid-layer self-attention contributions can be used to predict whether a MLLMwill generate a correct answer, but model confidence is a more reliable predictor

When a MLLM generates a correct answer, we observe that the self-attention contributions in its middle layers are higher to when it generates an incorrect answer (see Fig. 15). For LLaVa, this is specifically the attention contributions between the last constraint token and the last prompt token in the 16th and 17th layer. This holds potential to detect when a model will answer correctly without running a full inference pass, therefore enabling 'early' failure mode detection. To inves-

Introducing Long-Tailed Knowledge

Fixing Error Cases

Editing Efficacy

Generalization

Specificity

Editing Efficacy

Generalization

Specificity

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

Metric

Metric

Finetune

Finetune

Finetune (Constrained)

Finetune (Constrained)

MultEdit (Ours)

MultEdit (Ours)

tigate this, we compute the scalar average of the attention contributions from these two layers as a 16 c last ,c constraint + a 17 c last ,c constraint 2 and find that it can classify a correctly generated answer with an AUROC of 0.63 on the Known dataset in VQA-Constraints (see Fig. 16). We find, however, that the model's confidence - the probability of the generated output token at the final layer - is a slightly stronger predictor, with an AUROC of 0.76 (see Fig. 17). This parallel's previous work in LLMs [37] which used the attention contributions from all the LLM's layers (via a linear model) to predict the generated answer's correctness. In comparison, our results suggest that a subset of MLLM's middle layers alone can be leveraged as a coarse 'early' failure mode detector.

Takeaway. MLLMs behave differently in terms of information retrieval from their parametric memory however quite similarly to LLMs in terms of information transfer to the final token.

5 Correcting and Inserting Long-Tailed Information in MLLMs

Previous works have shown that counterfactual information can be inserted into LLMs by editing their causal layers [23, 24]. In this section, we verify if a similar approach can be used to edit a MLLM. Specifically, we introduce MULTEDIT, which applies a closed-form update to the early causal MLP layers we identified in Sec. 4. We show that our approach can effectively both (i) fix erroneous answers, and (ii) insert new long-tailed information in LLaVa in a VQA task.

5.1 MULTEDIT

Given an image-question ( x , y ) , we denote a MLLM's generated answer as O . MULTEDIT updates a few parameters in the model such that it generates a new answer O ∗ . Similar to [23, 13], we update the W ℓ proj matrix at specific causal MLP layers such that the probability P ( O ∗ ) increases. Specifically, we view W ℓ proj as a linear-associated memory (where the matrix's input are treated as keys and its output as values) and uses a closed-form update to map the original keys to new (correct) values. Below, we outline the process for acquiring both the keys and values.

Obtaining keys. Let { v i } N i =1 be the visual token embeddings and { e i } i M =1 the text token embeddings. Given a causal layer ℓ and the last token of a constraint c , we refer to the input of the layer's W ℓ proj matrix as its keys. Specifically, we define the key k c,ℓ to be the input embedding to the W ℓ proj matrix corresponding to the last token of the constraint. This can be obtained with a simple forward pass with the visual and text token embeddings.

Obtaining values. Given a causal layer ℓ , we refer to the output of the layer's W ℓ proj matrix as its values. Specifically, we define z c,ℓ as the output embedding corresponding to the last token of the constraint. We optimize z c,ℓ such that the probability of the correct answer P ( O ∗ ) increases as:

where L ( z c,ℓ ) is the standard next-token prediction loss used to train LLMs:

MULTEDIT modifies the W ℓ proj matrix such that the old keys k c,ℓ are mapped to the new optimized values z ∗ c,ℓ which increase P ( O ∗ ) . We define MULTEDIT's editing objective as:

The second regularization term ensures that W ℓ ′ proj , the weights before the edit, do not deviate too much from W ℓ proj . This helps to preserve performance on unrelated VQA pairs.

5.2 Experimental details

We experimentally validate MULTEDIT in two real-world model-editing applications:

Fixing incorrect answers to common questions. We test MULTEDIT on a set of ∼ 450 visual questions which LLaVa answers incorrectly (detected using incorrect VQA accuracy) from the multi-modal Known dataset in VQA-Constraints . These are generally questions about well-known places, persons and brands and companies.

Inserting long-tailed VQA knowledge. We test MULTEDIT on a set of visual questions from the Encyclopedia-VQA dataset [25]. These query fine-grained knowledge about rare landmarks around the world, which MLLMs have been shown to struggle on [25].

For both settings, we compare MULTEDIT to the fine-tuning baselines:(i) fine-tuning from [23] which fine-tunes all the layers using the language modeling objective, and (ii) fine-tuning with constraints from [38] which fine-tunes the layers in a language model with a constraint on the weights to ensure local loss continuity.

We measure the success of each edit operation using the following metrics: (i) Editing Efficacy , which uses P ( O ∗ ) to measure the edited model's ability to generate the correct answer for imagequestion ( x , y ). (ii) Generalization , which uses P ( O ∗ ) to measure the edited model's ability to generate the correct answer for the question y paraphrased using a language model (see Appendix for details), and (iii) Specificity , which uses VQA accuracy [1] to measure the edited model's performance on unrelated VQA questions. We consider unrelated questions to be those from the OK-VQA and Movies datasets in VQA-Constraints (see further details in Appendix G).

5.3 Results

Overall, our results show that updating the projection matrix using MULTEDIT at just a single early (causal) MLP layer can be a very effective approach for both correcting incorrect answers and inserting new knowledge in MLLMs.

Fixing incorrect answers. In Fig. 6 (left), we show that MULTEDIT is able to successfully fix the generations for all questions with wrong answers. In editing efficacy, the average probability of the correct answer improves from 0.07 to 0.82 after the edit. We also observe strong generalization, with a probability of 0.76 for the correct answer even when the question is paraphrased. Although we see a small drop of 1 . 5% accuracy on unrelated image-question pairs, we note that MULTEDIT outperforms fine-tuning with and without constraints on all the metrics.

Inserting long-tailed information. In Fig. 6 (right), we show that MULTEDIT is able to reliably insert new long-tailed knowledge in the model with an editing efficacy of 0.83. We see similar strong generalization when paraphrasing questions with an efficacy of 0.79. Similar to above, MULTEDIT incurs a small drop of 1.4 % on unrelated questions but is less affected than other methods.

In Fig. 14, we also provide ablations showing that editing the early causal MLP layers leads to better editing efficacies than editing the middle or the later MLP layers.

6 Conclusion

Our paper takes a closer look at how MLLMs process multi-modal information. We contribute a novel multi-modal causing tracing methodology and test-bed, VQA-Constraints , as well as a range of novel insights on how MLLMs retrieve and transfer information. We also introduce a novel model-editing algorithm, MULTEDIT, which can effectively fix errors or introduce long-tailed knowledge in MLLMs using a simple closed-form update which targets the early causal MLPs. Overall, this work deepens our scientific understanding of recent MLLM architectures, and enables future work in this direction.

Author Contributions

Samyadeep Basu conceived the idea, implemented the algorithms, curated the dataset, ran experiments and wrote the paper. Daniela Massiceti provided critical feedback at every stage of the project (algorithms, dataset) and worked with Samyadeep to write the paper. Besmira Nushi, Soheil Feizi and Cecily Morrison provided critical feedback on the project direction, paper writing and presentation. Martin Grayson helped the team with experiments on curating the probe dataset.

References

A. Agrawal, J. Lu, S. Antol, M. Mitchell, C. L. Zitnick, D. Batra, and D. Parikh. VQA: Visual Question Answering, 2016.

A. Agrawal, I. Kaji´ c, E. Bugliarello, E. Davoodi, A. Gergely, P. Blunsom, and A. Nematzadeh. Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-ofDistribution Generalization, 2023.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, and M. Reynolds. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems , volume 35, pages 23716-23736, 2022.

S. Basu, N. Zhao, V. Morariu, S. Feizi, and V. Manjunatha. Localizing and Editing Knowledge in Text-to-Image Generative Models, 2023.

S. Basu, K. Rezaei, P. Kattakinda, R. Rossi, C. Zhao, V. Morariu, V. Manjunatha, and S. Feizi. On Mechanistic Knowledge Localization in Text-to-Image Generative Models, 2024.

S. Cheng, B. Tian, Q. Liu, X. Chen, Y. Wang, H. Chen, and N. Zhang. Can We Edit Multimodal Large Language Models?, 2024.

W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems , volume 36, 2024.

N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021.

M. Geva, J. Bastings, K. Filippova, and A. Globerson. Dissecting Recall of Factual Associations in Auto-Regressive Language Models, 2023.

P. Hase, M. Bansal, B. Kim, and A. Ghandeharioun. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models, 2023.

M. He, Y. Liu, B. Wu, J. Yuan, Y. Wang, T. Huang, and B. Zhao. Efficient Multimodal Learning from Data-centric Perspective. arXiv preprint arXiv:2402.11530 , 2 2024.

Kevin Meng and David Bau and Alex Andonian and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems , 35:1735917372, 2022.

T. Kohonen. Correlation Matrix Memories. IEEE Transactions on Computers , C-21:353-359, 1972.

C. Lawless, J. Schoeffer, L. Le, K. Rowan, S. Sen, C. S. Hill, J. Suh, and B. Sarrafzadeh. "I Want It That Way": Enabling Interactive Decision Support Using Large Language Models and Constraint Programming. arXiv preprint arXiv:2312.06908 , 2023.

J. Li, D. Li, C. Xiong, and S. Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation, 2022.

J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning , pages 19730-19742. PMLR, 2023. ISBN 2640-3498.

Y. Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and Y. T. Lee. Textbooks Are All You Need II: phi-1.5 technical report, 2023.

H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved Baselines with Visual Instruction Tuning. arXiv preprint arXiv:2310.03744 , 10 2023.

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in Neural Information Processing Systems , 36, 2024.

A. Lu, H. Zhang, Y. Zhang, X. Wang, and D. Yang. Bounding the capabilities of large language models in open text generation with prompt constraints. arXiv preprint arXiv:2302.09185 , 2023.

K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. CoRR , abs/1906.00067, 2019. URL http://arxiv. org/abs/1906.00067 .

K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3195-3204, 2019.

K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt, 2023.

K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau. Mass-Editing Memory in a Transformer, 2023.

T. Mensink, J. Uijlings, L. Castrejon, A. Goel, F. Cadar, H. Zhou, F. Sha, A. Araujo, and V. Ferrari. Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories. In VQA , pages 3113-3124, 2023.

OpenAI. Gpt-4v(ision) system card, 9 2023.

OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y . Guo, C. Hallacy, J. Han, J. Harris, Y . He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, Łukasz Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Łukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung,

D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O'Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph. GPT-4 Technical Report, 2024.

J. Pearl. Direct and Indirect Effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence , UAI'01, page 411-420, 2001. ISBN 1558608001.

A. S. Sharma, D. Atkinson, and D. Bau. Locating and Editing Factual Associations in Mamba. arXiv preprint arXiv:2404.03646 , 2024.

G. B. M. Stan, R. Y. Rohekar, Y. Gurwicz, M. L. Olson, A. Bhiwandiwalla, E. Aflalo, C. Wu, N. Duan, S.-Y. Tseng, and V. Lal. LVLM-Intrepret: An Interpretability Tool for Large VisionLanguage Models, 2024.

S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, 2024.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. CoRR , abs/1706.03762, 2017. URL http://arxiv. org/abs/1706.03762 .

J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. M. Shieber. Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias. CoRR , abs/2004.12265, 2020. URL https://arxiv.org/abs/2004.12265 .

K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, 2022.

Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang. The dawn of LMMs: Preliminary explorations with GPT-4V (ision). arXiv preprint arXiv:2309.17421 , 9(1):1, 2023.

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. CoCa: Contrastive Captioners are Image-Text Foundation Models. arXiv preprint arXiv:2205.01917 , 5 2022.

M. Yuksekgonul, V. Chandrasekaran, E. Jones, S. Gunasekar, R. Naik, H. Palangi, E. Kamar, and B. Nushi. Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models. In International Conference on Learning Representations , 9 2023.

C. Zhu, A. S. Rawat, M. Zaheer, S. Bhojanapalli, D. Li, F. X. Yu, and S. Kumar. Modifying memories in transformer models. CoRR , abs/2012.00363, 2020. URL https://arxiv.org/ abs/2012.00363 .

A Limitations and Ethical Considerations

The findings of our work are limited to factual questions on natural images with short-form answers. Future work should investigate other types of questions (e.g. subjective), image domains (e.g. charts), and longer-form answers. Our work also does not map visual tokens to specific concepts in the input image - another important research direction. Our editing method, MULTEDIT, while effective for correcting and adding new information, could also be used to add false or harmful information into a MLLM. Work is needed on how to reliably detect misinformation in multi-modal settings.

B VQA-Constraints Details

In Sec. 3.1, we introduce the constraint framework for the task of factual VQA. In particular, there are two types of constraints: (i) Visual Constraint: A set of words in a question which refer to an entity in an image; (ii) Text Constraint: A set of words in a question which together with the visual constraint are responsible for answering a question. In our VQA-Constraints dataset there are two types of questions: (i) Single Constraint Questions: These questions comprise of only the visual constraint; (ii) Multi-Constraint Questions: These questions comprise of both the visual as well as the text constraint. In our paper, we primarily focus on the Single Constraint questions, therefore a majority of questions in our test bed of VQA-Constraints consist of only the visual constraints. Our VQA dataset VQA-Constraints which is annotated with constraints comprise of the following three parts:

OK-VQA : We annotate the questions in the test-set of OK-VQA with the constraints. In total, this part consists of 5k questions. We use the same images as those used in the original OK-VQA test-set.

Multimodal Movies : We use the text-only WikiMovies dataset from [37] and download images using the Bing API using the name of the directors. Our team filters and validates that the downloaded images are of the director itself and there's no noise in the downloaded images. We use a set of 1.5k questions in the final dataset.

Multimodal Known : For understanding knowledge storage in language models, [23] use a probe dataset (known.json) consisting of 1.2k questions. We first replace the subject in the question with a constraint and then download images from Bing API. Our team validates that the downloaded images are correct and uses it to create the Multimodal Known dataset.

Multi-Constraint Questions : The multi-constraint questions in VQA-Constraints are created from OK-VQA and Multimodal Movies. In particular, there are 150 multi-constraint questions for OKVQA and 500 multi-constraint questions from Multimodal Movies. Given that the primary focus of our paper is on Single Constraint questions, this set of Multi-Constraint questions is relatively small.

In total, VQA-Constraints consist of 9.7K VQA questions with Single Constraints and 650 MultiConstraint VQA questions which we use for interpretability analysis.

Description of Questions in VQA-Constraints

Dataset

Description

Example 1

Example 2

Example 3

OK-VQA

Single Constraint

What sport can you use

this

for?

Why might someone go to

this place

?

What flavor is

this pastry

?

OK-VQA

Single Constraint

Which airlines is this insignia?

What fruits come from

these trees

?

What is

this animal

mostly known for?

Multimodal Known

Single Constraint

This person

is a citizen of?

This newspaper

is written in?

The capital of

this city

is in?

Multimodal Known

Single Constraint

This venue

is owned by?

This building

is located in which city?

This show

debuted on?

Multimodal Movies

Single Constraint

Name a movie directed by

this director

?

Name a movie directed by

this director

?

Name a movie directed by

this director

?

Multimodal Movies

Single Constraint

Name a movie directed by

this director

?

Name a movie directed by

this director

?

Name a movie directed by

this director

?

OK-VQA

Multi-Constraint

Which is the most famous type of

this food

in

India

How does the population of

this city

compare to

Mumbai

?"

What is the relationship between

this person

and

?

Multimodal Movies

Multi-Constraint

Name a movie directed by

this director

in year

2006

?

Name a movie directed by

this director

in year

2010

?

Name a movie directed by

this director

in year

1994

?

Annotating and Validating the Visual Constraints. To automatically annotate the constraints in the visual questions from VQA-Constraints , we use a strong language model such as GPT-4. In particular, we annotate only the Multimodal Known and OK-VQA sub-parts of the VQA-Constraints dataset. First from each dataset, we annotate 100 examples which we use as in-context examples to the language model to annotate new questions. Given these annotations, our team then verified if the annotation is correct and if incorrect, modified the constraint annotation to make it correct. In total, the visual constraints are annotated for 9.7k VQA questions.

C Standard Causal Tracing Does Not Recover Causal States

In Fig. 7, we find that the standard causal tracing approach, which adds Gaussian noise to the span of visual tokens and the constraint in the text does not recover any causal states, even with a large window size of 10. This is true for recovering both the MLP as well as the self-attention causal layers.

No causal

states

No causal

states

Layers

Layers

Token Index

Token Index

0

0.5

0

0.5

5

0.4

5

0.4

0.3

0.3

0.2

0.2

0.1

20

0.1

0

10

30

0

10

30

D MULTIMODALCAUSALTRACE - Qualitative Plots

In this section, we present various qualitative plots corresponding to MULTIMODALCAUSALTRACE showing that our method is able to recover causal states in both the MLP as well as self-attention layers very early on in the model. We note that even using a small window size of 1 is sufficient to recover causal states. We highlight that earlier works [23, 24] show that for language models, a larger window size of 10 is required and also the middle layers are used to recover the relevant knowledge. However, we find that in the presence of visual prompts, relevant knowledge is retrieved from the early layers.

Very early causal layer

Very early causal layer

Very early causal layer

Very early causal layer

Very early causal layer

Very early causal layer

<S>

二

1.0

<S>

1.0

<5>

1.0

USER 3

This-

0.8

This

0.8

USER_

This

0.8

person

0.6

company.

owned

g

I5

T

0.6

0.6

Answer

fey

in

3

0.4

Answer

0.4

Answer

0.4

words

words

0.2

口

0.2

A

0.2

0

10

20

30

0

10

20

30

0

10

20

30

laptop

aThis

USER

<S>

0.8

0.6

1.0

langug

USER

<S>

the

0.8

0.6

1.0

bupiin

located

USER

iThis

<S>

3

0.8

0.6

1.0

Answer

&g

0.4

mixturg

0.4

Answer

0.4

words

0.2

0.2

0.2

0

10

20

30

0

10

20

30

0

10

20

30

Layers

Early Causal Layer

Early Causal Layer

Early Causal Layer

Early Causal Layer

Early Causal Layer

Early Causal Layer

<S>

1.0

<S>

1.0

<S>

1.0

persSok

USER

0.8

USER

This

0.8

company

USER

This

JTTTT

0.8

0.6

0.6

owned

0.6

g

g

0.4

Answer

0.4

Answer

0.4

0.2

0.2

0.2

0

10

20

30

0

10

20

30

0

10

20

30

<S>

1.0




3

1.0

<5>

1.0

USER

上

0.8

0.6

languR

mixturg

USER

the

3

F

0.6

0.8

Answer

USER

This

i9

L

3

TT

0.8

0.6

Answer

0.4

0.2

wofrys

A

0.2

0.4

words

2

0.2

0.4

0

10

20

30

0

10

20

30

0

10

20

30

E Multimodal Phi-2 Tracing Results

We use MULTIMODALCAUSALTRACE on multimodal phi-2, where the language model is phi-2 [17], a small language model capable of obtaining strong performances and often close to large language models for certain language tasks. Similar to our earlier observations for LLaVa, we find that even in multimodal phi-2, the early MLP and self-attention layers are causal. This shows the generalizability of our results for latest multimodal language models. We also note that a small window size of 1, is sufficient to recover causal states in multimodal language models.

Very early causal layer

Very early causal layer

Very early causal layer

Very early causal layer

Very early causal layer

Very early causal layer

Layers

USER

This

0.8

1.0

USER

This.

0.8

1.0

person

USER

This.

1.0

0.8

i

located

in

0.6

0.6

citizen

of

0.6

Answer

fo

Answer

words

0.4

0.2

Answer

0.4

0.2

A

0.2

0.4

ANT

0

10

20

30

0

10

20

30

0

10

20

30

1.0

1.0

1.0

company

owned

USER

This

IS

0.8

USER

aved

0.8

createa

Thia

the

0.8

Answer

0.6

m

0.6

Xnunos

fo

0.6

words

oa

in

0.4

Answer

0.4

Answer

0.4

0.2

0.2

0.2

0

10

20

30

0

10

20

30

0

10

20

30

Very early causal layer

Very early causal layer

Very early causal layer

Very early causal layer

Very early causal layer

No causal layer

Layers

1.0

1.0

1.0

Answer

persor

citizen

USER

This

0.4

0.8

0.6

company

Answer

USER

This

TT

C

0.8

0.6

0.4

Answer

owned

USER

This

g

is、

T

TTTT

0.8

0.6

0.4

s

words

words

0.2

上

0.2

0.2

0

10

20

30

0

10

20

30

0

10

20

30

1.0

1.0

1.0

USER

0.8

USER

0.8

USER

This"

0.8

person

0.6

0.6

0.6

fo

0.4

Answer

0.4

Answer

0.4

words

0.2

0.2

0.2

0

10

20

T

30

0

10

20

30

0

10

20

30

F Multi-constraint Questions Plots

In this section, we present qualitative plots for the multi-constraint questions from OK-VQA. These questions consist of a visual constraint as well as a text-constraint which in conjunction are used to answer a given question. Given that we earlier observe early causal layers corresponding to the visual constraint - in this section, we show results corresponding to the text-constraint. We have two primary observations: (i) A larger window size (at least a size of 6) is required to recover causal states, which highlights that a set of internal layers are used to retrieve relevant information; (ii) The causal layers span the early as well as the middle layers. This is true for both the MLP as well as self-attention layers.

Layers

USER

<S>

三

二

0.8

1.0

USER

<5>

0.8

1.0

USER

<S>

0.8

1.0

corteisg

0.6

0.6

pictf

0.6

direct

Answer

0.4

0.4

0.4

F0.2

0.2

wofeys

0.2

A

A晟

0

10

20

30

0

10

20

30

0

10

20

30

<S>

E

1.0

<S>

1.0

<5>

1.0

archfe

USER

0.8

USER

0.8

USER

0.8

farss

0.6

0.6

0.6

gipu

0.4

0.4

Answer

0.4

0.2

0.2

0.2

A

A

TTT

0

10

T

20

30

0

10

30

0

10

20

30

Layers

<S>

1.0

<S>

1.0

<S>

1.0

cort

USER

apple

0.8

0.6

picty

USER

0.8

0.6

farfs

Which

USER

th

0.8

0.6

m

0.4

0.4

0.4

woRas

0.2

0.2

0.2

0

10

20

30

0

10

20

30

0

10

20

30

G Model Editing Formulation and Further Results

As described in Sec. 5.1, MULTEDIT modifies the W ℓ proj matrix such that the old keys k c,ℓ are mapped to the new optimized values z ∗ c,ℓ such that the new values can drive the model towards increasing P ( O ∗ ) . We define the editing objective in MULTEDIT as below:

where W ℓ ′ proj denotes the weights of the W ℓ proj matrix before the edit optimization. The second term acts as a regularization which ensures that W ℓ ′ proj does not deviate too much from W ℓ proj .

From Eq. (6), we can observe that it can be solved using a simple closed form update as follows:

Given access to the keys and values, this closed-form optimization can be solved on a CPU. We note that this closed-form update for model editing has been recently used for text-to-image models [4, 5]. However to the best of our knowledge, ours is the first work to use such an update for MLLMs.

Hyperparameters. We use a learning rate of 0.1 to optimize for the values using Adam Optimizer. For the regularization factor λ , we use 0.01 after a grid search. Amongst the set of early causal layers between 0-5, we find editing Layer 2 to result in the best editing efficacy.

In Fig. 14, we show that editing the early causal MLP layers

Layers

Editing Efficacy

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

0

5

10

15

20

25

30

is better than the middle or the later layers. However, we find that editing the middle or the later layers also lead to a non-trivial editing accuracy, though it's lesser in terms of effectiveness than editing the early causal layers. Recent work [10] also observe this behaviour of non-trivial editing accuracy at locations which are not causal layers.

H Information Transfer and Early Failure Mode Detection - Qualitative Plots

In this section, we first show qualitative in Fig. 15 that the attention contributions from the constraint tokens to the last token is significantly more for correct answers than for incorrect answers. This is quantitatively highlighted and shown in Fig. 16 where we find that Layer 16 and Layer 17 to be the main orchestrator of this difference. A recent work [37] use attention contributions as a signal to detect hallucinations in language models. They show that the attention contribution values when used together with a linear model can be used to predict the correctness of the answer. In fact, they find this metric to be close to the confidence metric (which is the probability of the generated token). However, [37] necessitates the use of all self-attention layers which reduces it to be used as an 'early' failure mode detection metric. For multimodal language models, instead of using all the self-attention layers, we use the average attention contributions from Layer 16 and Layer 17 as an 'early' failure mode detection metric. From Fig. 17, we find that the average attention contributions indeed lead to a non-trivial AUROC of 0.63, but it lags behind the confidence metric which obtains AUROC of 0.76. Our early results show that using the model internals can be used to 'early' detect if the model is going towards a failure mode, but it lags behind the confidence although that requires the full forward pass.

Correct Answers

Layers

Layers

Layers

Layers

Layers

Layers

Incorrect Answers

0

5

0

0

10

4

4

10

4

3

3

3

20

2

2

20

2

1

1

1

30

30

0

0

1

2

3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

TextConstraintTokenIndex

TextConstraintTokenIndex

Text ConstraintTokenIndex

0

0

5

0

10

4

10

4

4

3

3

3

20

2

20

2

20

2

1

√

1

30

30

30

0.0

0.5

1.0

1.5

2.0

0

0.0

0.5

1.0

1.5

2.0

0

0.0

0.5

1.0

1.5

2.0

0

Text ConstraintTokenIndex

Text Constraint TokenIndex

VisualTokenIndex

Layers

Layers

Avg. Attention Contribution

Avg. Attention Contribution

Correct Answers

Incorrect Answers

Layers

Layers

Correct answers

have

higher

attention

contributions

in

mid-layers (16 th

and 17 th

layer)

Text-Constraint Tokens

Text-Constraint Tokens

Attention Contributions

Attention Contributions

1.75

1.50

1.50

1.25

1.25

1.00

0.75

0.75

0.50-

0251

0.00

5

10

25

30

0

5

20

25

30

5

3

3

2

2

1

30

30

0

Attention Contribution Metric

Count

Count

Norm of Attention Contributions

Probability

AUROC: 0.63

AUROC: 0.76

Correct

14

Correct

Incorrect

Incorrect

10

12

8

10

8

6

6

4

2

0.2

0.4

0.6

0.8

1.0

I Editing Dataset

The editing dataset consists of two parts: (i) VQA questions from Multimodal Known on which the LLaVa-7B gives incorrect answers; and (ii) VQA questions from Encyclopedia-VQA which tests the long-tailed knowledge of the model. The long-tailed questions are primarily about landmarks. Below we present some qualitative examples on the long-tailed VQA questions:

J Compute

All our experiments are performed on Nvidia-A6000 and A5000 GPUs.

In which country is this island located?

In which country is this hotel located?

In which country is this island located?

In which country is this house located?

In which country is this house located?

In which country is this lake located?

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction clearly state the claims made and they are backed by empirical results and plots in the subsequent sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our paper has a section on the Limitations of our work which also discusses ethical considerations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not have any theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper provides all the experimental details in the main paper as well as the Appendix. All the small implementation details mentioned in the paper can be used to reproduce our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will provide the final cleaned code with the camera-ready version of the paper. However, for the time being, we have provided all the experimental details in fine-grained details to reproduce our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the experimental details have been clearly stated in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All the variability in the experimental settings have been clearly stated in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or ./figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding ./figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our paper clearly states the GPU resources used in the experimentation.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have added a section on Ethics in the paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have added a section (Limitations and Societal Impact) in the paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We have described the safeguards for responsible release of the data or models in the Appendix.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have aptly credited and cited the original works and assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets in the paper are well documented and the documentation is provided in the main paper as well as the Appendix.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach to investigate the layer in which multi-modal large language models retrieve factual information and show several insights into their behavior. To investigate causal tracing, they propose replacing the input text tokens with different ones so that the model can respond differently from the correct answer with the replaced text tokens, then they copy activations from the clean layers until the model can reconstruct the correct answer. Through this investigation, their primary finding is that information to answer a visual question is mainly retrieved from early MLP and self-attention layers in MLLMs, which is different from the insight that LLMs retrieve factual information from middle layers. They also reveal several facts: only a subset of visual tokens are involved in sending information to the LLM's early layers, and mid-layer self-attention layers are involved in transferring information from the early causal layers to the question’s final token. They also demonstrate that an approach similar to editing factual knowledge of LLMs can be used to overwrite the knowledge of MLLMs.\n\nSTRENGTHS:\n1. Their main strength is providing an approach to locate the layers in which MLLM models are retrieving factual information. Since the research on MLLM is popular these days, the technique may be a good one to analyze the behavior of MLLM. \n\n2. They present a new dataset to study the issue. \n\n3. The insight obtained from their approach is interesting. Unlike LLMs which retrieve factual information from middle layers, MLLMs retrieve factual information from early layers. This fact can be new to the community. \n\n4. Other observations such as how visual tokens are used in MLLMs are insightful too. These facts might be useful to think about the design of MLLMs.\n\nWEAKNESSES:\n1. They offer an interesting observation that MLLM retrieves factual information from early layers, but they lack insight into why this happens. I think it is very important to give intuitions on why their behavior is different from LLMs. \n\n2. They present the insight from their approach using specific examples as in Fig. 3. However, if my understanding is correct, they do not provide numerical stats of which layers are responsible for retrieving factual information. Since they construct a new dataset, I guess it is easy to compute quantitative values to compare in which layers LLM and MLLM are retrieving information. \n\n3. The overall idea of their approach is not very novel. They borrow the idea from prior work and adapt it to MLLM. The novel part of their approach is replacing input text tokens so that the model makes an incorrect answer. \n\n4. I think section 5 does not improve the value of this submission. The section focuses on how to edit the knowledge of MLLM following prior work. The topic is related, yet different from their core contributions.\n\nQUESTIONS:\nAt this phase, I am on the borderline. I acknowledge the importance of analyzing the behavior of MLLMs and think the proposed approach is simple and reasonable. But, I also think the findings from this paper are not very novel. \n\n1. Related to 2 in weaknesses, is the value of Fig. 3 all from a specific example? or is it averaged over many examples? \n\n2. Please respond to all the weaknesses above since I might misunderstand some parts.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MULTIMODALCAUSALTRACE, an algorithm designed to identify hidden states in large language models (LLMs) that store factual information, specifically extending this capability to vision-language models where images are encoded as visual tokens. Building on previous work in information storage identification, MULTIMODALCAUSALTRACE constructs three models: a clean model, a corrupted model (where visual tokens are randomly replaced, perturbing the hidden states), and a restored model (where some corrupted hidden states are replaced with clean ones). By conducting mediation analysis on the causal relationship between state corruption and next-token prediction outcomes, the algorithm can pinpoint layers associated with different facts. The study reveals that multi-modal LLMs exhibit distinct behaviors in information storage and propagation, such as storing facts in middle layers rather than earlier transformer layers.\n\nAdditionally, the paper contributes the VQA-constraints dataset, derived from existing visual QA datasets and annotated with constraints.\n\nThe paper also proposes MULTEDIT, an algorithm for inserting or correcting factual knowledge in multi-modal LLMs. MULTEDIT optimizes the projection matrix in the MLP layers of transformers to minimize the mean squared difference between the projection output and the value vector that maximizes the prediction probability of the desired output. Empirical results demonstrate that MULTEDIT effectively corrects factual errors in VQA tasks.\n\nSTRENGTHS:\n+ The paper tackles an important task of identifying and correcting factual information in visual language model, a topic that received relatively less attention than in language modeling.\n\n+ The paper provided strong motivation to the problem and background it addresses, as well as clear visual and formal presentation of the methodology adapted. \n\n+ The proposed solution is comprehensive and encompassed various aspect of the information storage analysis problem. It delivers valuable insights to some practical challenges of adapting existing LLM-based methods to vision-language models such as the increased number of constraint tokens due to visual encoding.\n\nWEAKNESSES:\n+ The uniqueness and impact of multi-modal models are not thoroughly demonstrated. As stated in the paper, the proposed methods is largely based on the canonical Causal Trace(CT) and Rank-One Model Editing (ROME), with modifications in implementation allowing them to work for vision-language models. However, the choice (or absence) of most of such modifications are not well-motivated and no ablation study provided to understand their exact contribution. For example, the proposed MULTEDIT largely resembles the existing ROME method, with differences in how the key vector is found and how the optimal value vector is acquired. But it is not clear from the text why such changes are necessary and/or make the method works better for visual language model. \n\n+ The lack of discussion on multi-modal properties hinders the paper's novelty. The overall structure of the paper is similar to [23], and the main text does not substantially differentiate the methodology from prior work. \n\n+ Minor formatting issue in the reference: journal/conference names are missing in some items (such as [23] [24])\n\n(The reference indices of the original submission are used throughout this review)\n\nQUESTIONS:\n+ In vanilla ROME editing, the value vector is found by optimizing both the probability of the desired output and similarity to the essence of the original sentence. But in MULTEDIT the later term is dropped. What is the purpose of such change and how does the updated objective preserve the understanding of the overall prompt?\n\n+ In the creation of VQA-Constraints, does the authors manually correct all annotations generated by GPT-4? If so, what is the main advantage of starting with automated annotations?\n\nLIMITATIONS:\nThe authors included a limitation section in the appendix covering drawbacks of the proposed method and potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the information storage and transfer for the multi-modality large language model (MLLM). The author provides a comprehensive empirical study leveraging the causal tracking method, i.e., corrupting a clean model by perturbing the input prompt, to identify which layers are used to retrieve information relevant to the constraints in the prompt. The author also leverages the attention contributions to compute how much one set of input tokens influences a set of output tokens to track how information is transferred from visual tokens to the causal layers. To provide new insights, a new dataset called VQA-Constraints has been created to support the empirical study. The author provides many new insights that are different from the existing research in LLM that the MLLMs reply on MLP and self-attention block in much earlier layers for information storage and a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. The mechanism revealed in this study also inspires a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs.\n\nSTRENGTHS:\n1. Overall, this paper is very well written, and the main messages are conveyed smoothly. The findings and takeaway message have been delivered clearly.\n\n2. The paper's novelty stems not only from the research problem but also from the design of the exploration method, the novel insights, and the corresponding proposed model-editing method. To the best of my knowledge, this should be the first work that provides a comprehensive study of knowledge tracing on MLLM, providing a great foundation for future exploration.\n\n3. The research findings are very interesting and insightful. They are validated by different datasets and the newly proposed dataset, making the insights solid and sound.\n\n4. The proposed model-editing method is effective and can partially validate the mechanism identified in the paper.\n\nWEAKNESSES:\nThis is a strong paper in general, and the review does not realize the critical weakness of the present paper. The reviewer may be curious whether the code will be public upon the acceptance.\n\nQUESTIONS:\nPlease refer to the Weaknesses sections for more details.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscripts studies mechanistic interpretability in autoregressive vision-language models. Towards this, the authors propose `MultiModalCausalTrace`, an extension of the causal tracing technique for analyzing text-only LLMs, which is to perturb \"visual constraint\" tokens with a set of semantically coherent, but irrelevant tokens, and measure its change effects in model behavior. The authors observe that VLMs store and transfer information at early layers, while LLMs operate in early-to-mid layers. Based on this observation, the paper proposes `MultEdit`, a technique that injects long-tailed information in these causal layers of VLMs.\n\nSTRENGTHS:\n- The paper's presentation is very clear and tells a coherent story.\n- The main techniques (multimodal causal tracing) are reasonable; they are built upon well-tested frameworks for mechanistic interpretability of LLMs.\n- The findings are intriguing as the authors observe VLMs behave differently in terms of information storage and transferring, compared to LLMs.\n- `VQA_Constraint` is a valuable contribution to the interpretability community, and binding visual input to a natural language reference is a reasonable way to evaluate the knowledge for VQA, and is different from the many algorithmic tasks studied in prior mechanistic interpretability works on text-only LLMs.\n\nWEAKNESSES:\n- While `MultEdit` has demonstrated impressive editing efficiency and generalization performance, the technical novelty is somewhat limited  as it is an application of a well-tested technique for editing LLMs. I would also like to see its generalization performance not only on `VQA-Constraints`, but also on other standard VLM benchmarks, such as MMMU.\n- While the authors finding clearly indicates VLMs store and transfer information differently than text-only LLMs, I'm hoping that the authors could give a more detailed explanation to the cause of this. Since all tested VLMs are fine-tuned from LLMs, it is conceivable that they would operate in a similar fashion.\n- This work currently only examines multimodal fusion at the embedding level, while there are other popular approaches such as the Flamingo architecture. I'm curious to see whether the authors finding would still hold under these different architectural choices.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors have adequately addressed limitations and potential societal impacts of their work", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-23"}