Towards Neuron Attributions in Multi-Modal Large Language Models
Authors: Junfeng Fang
Zac Bi
Ruipeng Wang
Houcheng Jiang
Yuan Gao
Kun Wang
An Zhang
Jie Shi
Xiang Wang
Tat-Seng Chua
Venue: NeurIPS 2024 poster (2024)
Keywords: Neuron Attribution
Large Language Models
Multimodality
Primary area: interpretability_and_explainability
TL;DR: We propose a novel neuron attribution method tailored for MLLMs to reveal the modality-specific semantic knowledge learned by neurons.

Abstract: As Large Language Models (LLMs) demonstrate impressive capabilities, demystifying their internal mechanisms becomes increasingly vital. Neuron attribution, which attributes LLM outputs to specific neurons to reveal the semantic properties they learn, has emerged as a key interpretability approach. However, while neuron attribution has made significant progress in deciphering text-only LLMs, its application to Multimodal LLMs (MLLMs) remains less explored. To address this gap, we propose a novel Neuron Attribution method tailored for MLLMs, termed NAM. Specifically, NAM not only reveals the modality-specific semantic knowledge learned by neurons within MLLMs, but also highlights several intriguing properties of neurons, such as cross-modal invariance and semantic sensitivity. These properties collectively elucidate the inner workings mechanism of MLLMs, providing a deeper understanding of how MLLMs process and generate multi-modal content. Through theoretical analysis and empirical validation, we demonstrate the efficacy of NAM and the valuable insights it offers. Furthermore, leveraging NAM, we introduce a multi-modal knowledge editing paradigm, underscoring the practical significance of our approach for downstream applications of MLLMs.

Towards Neuron Attributions in Multimodal Large Language Models
Junfeng Fang,
Zongze Bi, Ruipeng Wang, Houcheng Jiang, Yuan Gao,
Kun Wang"
University of Science and Technology of China
zacb20a,, wwpp0021021111 janghc, gaoy, wk520529 y@mail wsw. edu.cn
An Zhang
Jie Shi Huawei
National University of Singapore
an mnn_zhang@nus.ede edu.
shi jiel @ehaweei.com
Xiang Wang"
Tat-Seng Chua
University of Science and Technology of China xi xiangwang 223@gmail....m..mmmmmmmm..mm...m
National University of Singapore dcscts@nus seretss@uuseedu...... edu.sg
Abstract
As Large Language Models (LLMs) demonstrate impressive capabilities, demys- tifying their internal mechanisms becomes increasingly vital. Neuron attribution, which attributes LLM outputs to specific neurons reveal the semantic properties they learn, has emerged as a key interpretability approach. However, while neuron attribution has made significant progress in deciphering text-only LLMs, its appli- cation to Multimodal LLMs (MLLMs) remains less explored. To address this gap, we propose a novel Neuron Attribution method tailored for MILMM, termed NAM. Specifically, NAM not only reveals the modality -ppecic semantic knowledge learned by neurons within MLLMs, but also highlights several intriguing properties of neurons, such as moosll invariance and semantic sensitivity. These proper- ties collectively elucidate the inner workings mechanism MILMss, providing a deeper understanding of how MI.LMss process and generate multi-modal content. Through theoretical analysis and empirical validation, we demonstrate the efficacy of NAM and the valuable insights it offers. Furthermore, leveraging NAM, we introduce a multi-modas knowledge editing paradigm, underscoring the practical significance of our approach for downstream applications of MLLMs. Our code is available at https//gttthb..om/llllitttttttt www.://ww..o.iiiieiltteeeienne..
Introduction
As Large Language Models (LLMs) demonstrate impressive capabilities [1, 3,4,, demystifying their internal mechanisms becomes increasingly vital, particularly in applications emphasizing fairness, trust, and ethical decision- maasiinnmmainggg [5, 6, 7]. To interpret LLMs. "neuron attribution" stands out as pivotal approach. This method involves attributing text outputs individual model components (e.g., neurons and hidden layers) to reveal the knowledge and linguistic properties they learn [5, 8, 9, 10, 11, 12]. Such insights not only facilitate tasks like model editing and pruning [13, 14, 15), but also offer deeper understanding of how LLMs internalize knowledge. For instance, leading neuron-attribution studies [16, 17, 13, 14, 18, 19] suggest that this capability of internalization may predominantly originate from their Feedforward Neural Networks (FFNs).
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Give image like:
Here is an image of Dog.
OPT (Base LLM)
GILL:
CLIP
Projector
Projector
Stable Diffusion
T-Neurons
T I-Nerron?
Here is image of Dog,
T-Neurons
Here is an image of Dog.
(Noise)
Lawn
Dogs
+ Dog
human's friends.
Lawn
FFN
(Noise)
I-Neurons
Text-only LLM
MLLM
MLLM
(b)
Figure 1: Illustration of neuron attribution methods for interpreting LLMs. (a) The paradigm of GILL; (b) current attribution methods tailored for text-only LLMs; (c) the challenges of extending current attribution methods MLLMs; (d) the paradigm of our NAM. Best viewed in color.
Recently, the rapid development of multi-modal large language models (MLLMs) [20, 21, 22, 23, 24] sparking the interest of interpretability [18, 19]. Here we focus on the MILMss that can perceive, generate texts and images simultaneously. Scrutinizing prior studies, we can summarize two common components beyond the LLM base: image encoding and generating modules. Specifically, the image encoding module projects the input image into the representation space of the base LLM; hereafter, the image generating module generates image outputs conditioned on the representations given by the base LLM. Take GILL [25] as example. As shown in Figure (a). it hires OPT [26] as the LLM base, CLIP Vit-L [27] with a cross-modal projector as the image encoding module, and Stable Diffusion [28] with another projector the image generating module.
While this expanded capacity endows GILL with versatility suitable for variety of downstream tasks, it concurrently presents challenges for interpretation, particularly concerning neuron attributions. Specifically, we outline these challenges through three progressive points:
Semantic Noise. As shown in Figure (b), for semantics like dog, current methods typically attribute the output neurons directly in text-only LLMs [16, 13]. However, in MLLMs, attributing the entire generated image to neurons directly might result in inaccuracies. As shown in Figure (c), when GILL is tasked with drawing dog, the generated image might contain other semantic elements like lawn, introducing noise and distorting attribution.
Source of Attribution:
Process of Attribution: Inefficiency. Leading attribution methods typically rely on gradients [16, 19] or causal effects [13, 14], requiring extensive forward/backward propagation processes, which are inherently mone-onnniingg and ccorage-intensinn The added complexity of encoding and generation modules in MLLMs further exacerbates this challenge.
Results of Attribution: Intermingled Neurons. In text-only LLMs, attributing the concept dog involves identifying neurons crucial for outputting the word "dog", termed T-neurons. In contrast, MLLMS also require identifying neurons crucial for image generation, called I-neurons. As illustrated in Figure 1 (c), the distribution of T-neurons and I-neurons differs for the same concept, leading to confficting results that complicate further analysis and applications.
In sight of this, we introduce new Neuron Attribution paradigm tailored for MLLMs, termed NAM. to reveal the modality-specific semantic properties learned by neurons within the FFN layers. Specifically, to address the above challenges, NAM undertakes the following efforts:
Image Segmentation for Semantic Noise: As shown in Figure (d), NAM employs the image segmentation model to distinguish regions containing the target semantics from other noisy semantic areas, and attributes these regions the neurons, rather than the entire image, ensure accuracy. Activation- Scores for Inefficiency: Drawing inspiration from prior studies on neuron activations [18, 5], NAM introduces new attribution score that relies on neuron activations, climinating the need for additional forward/backward propagation or gradient calculations. Modality Decoupling for Intermingled Neurons: NAM assigns -edaaliy--seeific attribution scores to neurons to prevent cross modal disturbances during attributions. This paradigm facilitates the decoupling analysis of T-neurons and I-neurons, as depicted in Figure (d).
Furthermore, based on the empirical results NAM, we reveal several intriguing neuron properties within MLLMs. These properties collectively elucidate the inner workings of MLLMs, enhancing

---

Give image like:
Attention
X
X
W
...Do..
h
W
A
W
h'
h
Image generation module
000
ki
p
A/
Win
A
Wou
Image generation module
W
A'
W
ab
a
ooo
ooo
Residng
ooo
Refin Aealoonnnne
ooo
oWor
W out
Stream
WWont
W ouu
h-
Importance of 1f -
Diffuser Interpreter
to "dog region
EVA02
Residual
W.
R 0.4
Wp Rp
Y1 Y/ yp Xin poooa
Stream
oooo C
Contribution of ok "dog region":
O
R
(b)
Vocabulary: "Dog"
Figure Illustration neuron attribution methods for interpreting L.Mss. (a) The paradigm of current attribution methods tailored for text-only LLMs. (b) The challenges of extending current attribution methods to MLLMs. (c) The paradigm of our NAM.
understanding of their capacity to process and generate multi-modal content. Interestingly, among these insights, a pivotal finding exhibits that when generating multi- content for the same semantics (e.g., the word "dog" & an image of dog), the erucial neurons (i.e. T-Neurons & I- Neurons) are typically not identical. This distinction underscores the complex nature of neurons within MILMs, and highlights the necessity of neuron attribution across modalities. Additionally, by applying NAM to enhance image editing tasks, we further underscore the significance and potential applications of NAM for MLLM community.
2 Preliminary
Transformer- Based LLMs. An autoregressive transformer language model G X + operates over the vocabulary V. It receives token sequence x E X' and generates probability distribution y = 1/,,/2,-.. 3/2 .... V E y to predict the next token [29, 13]. Each token is represented as a series of representations h' E IR in /-th layer, where h" is the embedding of the token in x. The model's final output, y W, (h²). is derived from the last representation h [h] h..,.., h/ in layer using the unembedding matrix W.. Figure (a) exhibits visualization how h' are computed within layer The representation in each layer results from the combination of the global attention a', the local MLP output m and the representation h from the previous layer. Formally,
(1) where Win E I and W... E HRd are the first and second linear layer in FFN with the dimensionality of the FFN's intermediate layer d: and y are rectifying and normalizing nonlinearity. For further background on transformers,
h' m h 1-1 +a', m' w out a (Wl. W. 7 (a' + h²-1))
refer [29]. Additionally, focusing the the k-th neuron u/. in the (-th FFN layers, we simplify the definition by considering its activation of as:
A' [01, a2, a.]
(w! our (a' h h1-1))
MILMs. Here, we take GILL as an example to illustrate a common paradigm MLLMs. As shown in Figure 2 (a), GILL incorporates the following modifications the above text-only LLMs: (1) In addition to the textual prompt, the input token sequence x also includes the encoding the input image produced by the image encoding module. (2) The representation of the last hidden layer is utilized input to the image generation module, facilitating conditional image generation.
3 Method
This section delineates the implementation of NAM within MILMs.. Specifically, Section and 3.2 introduce the attribution process for image and text outputs, respectively; In Section 3.3,

---

explore a significant application of NAM, i.e., editing images generated by MLLMs. To illustrate these processes, we utilize GILL [25]. a representative MLLM capable of image generation. For detailed introduction to GILL,, please refer to Appendix B.
3.1 Neuron Attribution for Image Generation
We first detail how to attribute the output images to the specific neurons within FFN layers. Retro- specting the challenges highlighted in Introduction, using the MLLM to generate images a specific concept (e.g., dog) often results in outputs that include extraneous, noisy elements (e.g., lawn). To mitigate the negative effects of semantic noise on neuron attribution, we propose a two-step approach to extract I-Neurons. Specifically, (1) the first step focuses attributing the output of the image generation module (i.e,, images) to the input of the image generation module (i.e., last representation h²), and (2) the second step endeavors attribute the input of the image generation module the specific neurons. Next, we provide detailed descriptions of these two steps.
3.1.1 STEP1: Attribution From Images to Representation h²
The purpose of this step is to attribute the image to h4 E R That is, to identify the contribution of each element in h for image generation. We define these contribution scores as R E TR As shown in Figure 2 (b), NAM acquires R by sequentially executing the following processes:
Given an image generated by prompting MLLM with the semantics dog, NAM first employs the leading segmentation model, EVA02 [30], to identify regions specifically related to dog. This is crucial for minimizing interference from extraneous semantics, such as lawn in the background: Subsequently, NAM utilizes the advanced attribution algorithm of the diffusion model, Diffuser- Interpreter [31], to access the relevance of each dimension the input of the image generation module to the dog region in the generated image.
Ultimately, by normalizing these relevance scores to (0, 1), we obtain the importance scores R [11, "2,,,,,,, rH] of h [h1, h2..... hH] w.r. the target semantics in the output image.
Due to the established applications of EVA02 and Diffuser- Interpeet, we provide detailed introductions in Appendix B. Furthermore, it is worth mentioning that NAM can be transferred to any other modality by utilizing the (1) semantic segmentation algorithms and (2) attribution algorithms of generation modules tailored for other modalities (e.g., audio and video). After obtaining R. by these advanced modality-sppecifi algorithms, the subsequent attribution steps are universal across all transformer- based MIL.Ms.
3.1.2 STEP2: Attribution From Representation h² to Neuron o/.
This step involves attributing the representation h" in the last layer to the specific neuron ut, within the FFNs of the base LLM. To this end, NAM aims to trace each neuron's 's contribution to h", and identify the neurons with significant contributions as I-Neurons for the semantic of interest. Hence, fair and efficient contribution scoring method is crucial.
Direct Contributions through Residual Stream. Current methods for scoring contributions often rely on gradients, such as the product of gradients and activations [19] or the integration of gradients [16]. However, these methods are computationally intensive, particularly for large-scale models with extensive parameters. In sight of this, drawing inspiration from prior studies on neuron activation [18, 19], we first introduce new attribution score that relies on the neuron activation at. Specifically, first try disassemble and deduce the generation procedure of h4 by expanding h- follows:
h = m 4hhh a
L-1
1=1
(3)
L a'
W A + h
Σ
ΣΣ
dg W dn
h +
Σ a
1=1
where (W! ) E R.H is the k-th column of the weight matrix W our corresponding to the index of neuron u/. as shown Figure 2 (b). Note that the first term of Equation (3) reflects the direct contribution of the neuron u/ to the last representation h4, i.e, the contribution through the residual stream [19] of the base LLM.

---

Hence, we employ of (W as the indicator for the neuron uy 's contribution to h
Furthermore, by integrating R in Section 3.1.1, can establish a complete attribution pipeline (i.e. targeted semantic region in generated image + representation h = neuron v4)) Recall that R has assigned the contribution for each dimension in h the neuron u/g 's contribution sy can be defined as of (W ) weighted by elements in R. That is, Sk = (V (a as illustrated in Figure 2 (b).
Contribution Score Considering Indirect Influence. While s/ quantifies the neuron n 's direct con- tribution to h through the residual stream, it does not account for all influential factors. Specifically, it overlooks the indirect contributions that neurons make through the attention mechanisms within subsequent FFN layers. Supporting evidence exhibited in Appendix B verifies that this oversight might lead to bias. To address this issue, and in line with our objective eschew complex computa- tions like gradient, we implement a heuristic optimization of the current indicator s Specifically, we employ the relative magnitude of neuron activation another indicator to identify neurons that may have a significant indirect contribution to h contribution which si. might overlook.
Furthermore, considering the computation of s already incorporates d'., the contributions reflected by these two indicators may overlap. To prevent redundancy from summing or multiplying these two metries, we utilize the maximum function in our final score design:
max
(4)
EL 2i=1 Ek=1 Eh eak
where the normalization operation ensures fair competition between s and af.- Note that our experi- ments in Section 4 have shown that this combined scoring approach is more effective than utilizing sy or al alone, verifying their complementary nature. By computing contributions following Equation (4) for various semantics across all layers, NAM identifies neurons that consistently demonstrate the highest contributions to the generated images. These neurons, distinguished by their significant roles, designated as I-neurons responsible for targeted semantics in MLLMs.
3.2 Neuron Attribution for Text Generation
We then focus on how to acquire the neuron's contribution to the text outputs. Similar to the derivation process of contribution score for image output, here we first focus on the contribution for output through the residual stream. Specifically, we expand h" follows:
L y = = (m- + h--1 + +a4) = Σ W.w. mu A + W.
(5)
According to commonly used assumption for analyzing the internal mechanisms of LLMs, represen- tations at any layer within the language models can be transformed into a distribution over the token vocabulary V using the output embeddings [18, 19, 32, 16, 13, 14]. Hence, W, W E [ can be considered as the new unembedding matrix at the end of the residual stream, and A' contributes to the model output distribution over the vocabulary through W. W A' as shown in Figure 2 (c).
Refined Contribution of Individual Neuron. We further disassemble Equation (5) to refine individ- ual neuron up 's contribution to the output word. Specifically, denoting p as the index of the word "dog" in the vocabulary V, we have:
yp = (Wu)p W ou A' + (W,,
aa) Σ (W.), (Wi.)) (W))
+ (W (h
aa), (6)
where (Wu)p E Ixd is the p-th row of W.. According Equation (6), the neuron ut, 's contribution chp to the p-th word "dog" on vocabulary can be obtained by ch a (Wa)pl (Wi )..)). as illustrated in Figure 2 (c). Furthermore, we would like to encourage the semantic specificity of the identified crucial neurons- that is, only preserving a single semantic concept with the maximum contribution, while discarding other semantics. Formally, for the semantics of p-th word on vocabulary, the neuron u/ 's contribution of can be defined as:
{
pº arg max of $/2 =
CKP 0
(7)
otherwise.
5

---

By substituting & to Equation (4), we can acquire the final attribution score of neuron ufe for text output. The neurons that consistently exhibit the highest contributions are then designated as T-neurons.
3.3 Image Editing Enhanced by NAM
Knowledge editing methods based on neuron attribution have already been explored for text outputs [13, 14, 33, 34]. Here, we focus on how to leverage attribution results to facilitate knowledge editing of images. This objective requires replacing some semantics (e.g., dog) with another semantics (e.g,, cat). To this end, we leverage the I-Neurons identified by NAM for image editing through straightforward, training-free approach. Specifically, we first construct the set of I-Neurons, U, for the semantics like dog. Then, we collect the positions (1, k)) of the neurons u. in U1 and construct the set of these position indices, I. For (1, k) E I. we add a perturbation A (W
out m to (Wi.), following:
W (A(W".)) oads for (1, E I}
A A(W out) arg min W
Σ (LR))T
A(W out (h² - -h4) + + Σ
A(W our 1 12,2, (8)
(I,))
where h and h is the last representation of the base LLM when generating the image of dog and cat, respectively; serves trade-off parameter. In Equation (8), the first term aims to facilitate shift in the image generation module's 's input from h to hi, while the second term is the & norm constraint for preventing drastic edits that might affect images containing other semantics. According this method, NAM can be utilized enable simple and efficient images editing, underscoring the significance and potential applications of the NAM for MLLMs
4 Experiment
In this section, we aim validate the effectiveness of NAM from three aspects:
What is the distribution of T/I-Neurons identified by NAM?
What properties do the T/I-neurons identified by NAM have? How to verify that the T/I-neurons identified by NAM are more critical compared to the neurons identified by baseline methods? Can the T/l-Neurons identified by NAM facilitate the image editing within MLLMs?
4.1 Investigation Setup
Target Models & Datasets. Our research focuses on GILL [25] and NEXT-GPT [35], two representa- tive MILMs with the capability of image generation. All experiments conducted the Common Objects in Context (COCO) [36], a large-scale object detection, segmentation, and captioning dataset including 80 object categories and five captions per image conduct our experiments. Due space limitations, we only present the experimental results on GILL in this section. The remains and the detailed implementations, such the setting of syyee--arameters. can be found in Appendix B.
Baselines. We collect five advanced neuron attribution methods across three categories (gradient- activation- and causality-bar -based attribution). Specifically, their abbreviations and the attribution scores they employ are: (1) AcT: neuron activation [5]; (2) AcU: The product of activation and the unembedding matrix, focused on the dimension corresponding to the output word [18]; (3) GraD: The gradient the output dimension corresponding to the output word w activation. (4) GraT: The
product of the gradients and activation [19]; (5) Gral: The integral of the gradients [16]: (6) CE: The
causal effect of activation on outputs [13, 14]. See detailed description in Appendix B. For NAM and baselines stand and their role in rich literature, please refer Appendix A (i.e., Related Work).
4.2
RQ1: Distribution of T/I-Neurons
We first focus on the distribution of T/I-neurons identified by NAM. Specifically, we randomly select 1000 images from the COCO dataset. Then, we feed each image GILL individually and instruct GILL to generate a similar image. The distributions of T/I-ncurons are exhibited in Figures 3 (a) and
The extension of NAM to broader scenarios will be detailed in Appendix C.

---

Table 1: The semantics of T/I-neurons with the highest attribution scores identified by different attribution methods, when the output of MLLMs containing the targeted semantics. (L1,Uk) denotes the k-th neuron at layer I. For each method, we report semantics with top-4 probabilities.
Output
Segmentation
Category
Method
Location
Semantics
"A girl is riding a horse."
Grad
L28. U4786
I'dog', "shark', 'cat", "bird") ('animuls' "unimal" "Animal", 'Animals']
T-Neurons
AcT
L30. 303818868
CE
L27. 277U21262
("vehicles" 'trucks'
"boats") |
NAM
L23.U5318
["horses' 'horse', 'Horses", "Horses"
Grad
228.714374
[ ("farming'' "farm,, "farms", farmsss.'ag) 'ag")
I-Neurons
AcT
CE
L26. 1129577
L.8.. U1208
('animal" 'animals" "veterin",
('Kinnikuman" "cfm"" 'Nanto' "Vaults"
"Nunto'""Valls" ('horses', 'horse', 'Horses", 'Horses")
NAM
L23.U5318
"A dog is rumning on the lawn."
Grad
L28. U12856
('child'. "Child", "children", 'male")
T-Neurons
AcT
L24. U12845
'Dog' "cunine" "those" "Those' 'that', "this"
'dogs' "dog" "Dogg"Do
CE
L25. U3655
NAM
L24.U10710
"dogs", "Dog', "Dogs', "pets")
Grad
L26. 35
['adopt., 'pet'. 'adopting", "apppiiio )
I-Neurons
AcT
L30. U13868
l'animals'. "unimal" 'Animal" 'Animals') ('weeds'. 'chickens", "compost", 'trash" ) ("dogs', "oogs'''oo''... 'dog' 'Dog', 'canine")
CE
L31. 51111
NAM
L24.U12845
"A small ship on the sca."
Grad
L26. U3972
('diving', "digging'. 'dred', "drilling") ) ('boat,, 'car
T-Neurons
AcT
CE
L30. U3335
U11438
('inar "Set" 'cum" (sship,, 'ships", 'Ship', 'shipping")
'vchicle")
NAM
L30. L25.3
Grad
L27. UB984
('bush", "tree", "brush', 'shr') I'swim" "swimming" "Swim" 'underwater" ('bronze', "sign". "box" 'SIGN")
I-Neurons
AcT
L25. U2539
CE
25.15113
NAM
L28.U18626
('ship", 'ships', 'sea', "ocean")
@a) I-Narrnns
(b) T-Neurons
c & I-Neurons
Number
4000
Top 100
4000
Top 100
6000
Neurons
Top
Top
1- Neurons
3000
Top
3000
Top
4500
2000
2000
3000
1000
1000
1500-
00
05
10
15
20
25
30
05
10
15
20
25
30
00
05
10
15
20
25
30
Layer
Layer
Layer
Figure 3: Distribution of (a) I-neurons, (b) T-neurons, and (c) intersection and subset of I-neurons
and T-neurons per layer identified by NAM, chosen by different number of neurons with top scores on average. Best viewed in color.
(b), while Figure 3 (c) illustrates the distribution of intersections of T- and I-neurons. These results demonstrate the following observations:
Observation 1: Within MILMM, the crucial neurons for the text and image output containing specific semantics predominantly occur the middle and high layers of the base LLM. Note that this observation is consistent with the previous works involving neuron attributions within LLMs [5. 13, 18]. Additionally, the similar distribution of T and I neurons suggests that the formation time of semantic concepts across different modalities in MLLMS may be consistent.
Observation 2: Figure (c) reveals a partial overlap between T and I neurons. However, it also pro- nounces distinctions between them. This finding substantiates the claim presented in the Introduction of this paper: even for the same semantics, critical neurons are modality-specifie within MLLMs.
4.3
RQ2: Properties of T/I-Neurons & Effectiveness of NAM
We then explore the following properties of T/I-neurons through comprehensive quantitative and qualitative experiments: Semantic Relevance, Cross-Sample Invariance, and Concept Specificity. Additionally, by comparing these properties with those of neurons identified by various baselines, we validate the effectiveness of our NAM. Note that we only present the results of the best-performing baseline for each class, and remains are shown in Appendix B.
4.3.1 Semantic Relevance
Following previous studies [19, 18), we treat the unembedding matrix and the second linear layer matrix in the FFN as a projection from neuron activation to the probability distributions of the vocabulary. Based on this, words with the average highest probability can be regarded the relevant
7

---

Table 2: Consistency between the neuron's semantics and the images/captions. Grad. Act. and Ca. denote gradient-, activation- and causality- -ased methods, respectively. and represent the CLIPSccre w.r.f input and output images. We use background highlights the best performance.
Class
T-Neurons
Method
Grad.
Act.
Ca.
CCLipccore
"CLipScore
BERTScore
MoveScore
BLEURT
CE
1..04
0.25110.00 10.022
0.273 029
0.25710.019
0.04010.005
Gral
0.239 20.018
0.244 020
0.276 032
0.2961.0.00
0.039
GraD
0.378
0.306
0.457 +02007
0,436+0.000
0.064
GraT
0. 0.425
0. 0,422 02.0
0. 0.4164a..13
0.477
0.072
AcT
0.556
0.594 046
1.62449.0557
0.653 054
0.13929.013
AcU
0.
0.624
0.618
0.609
0.135
NAM
0.502+0.054
0.637
0.640+0.030
1663740.045
0.148
Class
I-Neurons
Method
Grad.
Act.
Ca.
CCLipScore
*CLipScore
BERTScore
MoveScore
BLEURT
CE
0,228
0.21910 026
0.245 053
0.25020.022
0.044
Gral
/
0,230 #0.021
0.235
a.259+a..27
a.278
0.035
GraD
a.
0.377 1.377=a0.026
0.432
0.409
0.058
GraT
0. 0.43220.0m
0.3944:o 032
0.4533 0412
0.437 0.0.
0.068-0.008
AcT
0.547 20.044
0.580
0.597 2: 056
0.623
0.128
AcU
0.501 +0.061
0.601 +0.054
0.559 030000
0.548
0.137=20.013
NAM
0.558
0.613
0.611 048
0.630
0.1 0.144
GraD
Gral
GraT
NAM
Semantics dog
Ratios
AcT
AcT
AcU
10%
AcU
dog
beach
+50
house
45%
240
15%
woman
400
200
bost
160
airplanc
cat
500
20%
bus
40%
cat
Ratio of neuron overlap (%)
bike
apple
35%
25%
(a)
10
13
20
25
30
35
40
45
airplane
30%
Figure 4: Cross-sample invariance and semantic specificity of T/I-neurons. (a) exhibits invariance by
calculating the average ratio between T/l-neurons' subset and intersection across different text/image output; (b) quantifies specificity showing: 1. the number of neurons crucial for specific semantics solely and 2. the average number of neurons whose probability of being crucial to other semantics is lower than certain value. Best viewed in color.
semantics of neurons, as shown in Table 1. Note that the results of AcU are not exhibited since its attribution score is exactly this probability. Furthermore, explore the semantic relevance of the neurons quantitatively, we calculate the consistency between the semantics of neurons and the input/output images employing CLIPScore [37]. BERTScore [38], MoverScore [39], and BLEURT [40] are also employed to quantify their consistency with (1) input image's caption provided by the
dataset and (2) output image's is caption given by GPT [1]. Table exhibits the average quantified results. According to Table and 2, we have the following observation:
Observation 3:
The semantics of T/I-neurons identified by NAM align input/output images and their captions, while the other attribution methods typically identify the neurons that are hardly correlated with the targeted semantics. The quantitative results share similar tendency, confirming the high semantic relevance of T/I-neurons and the effectiveness of our NAM.
closely with the
4.3.2 Cross-sample Invariance
For different text/image outputs containing the same semantics, T/I- T/I-neurons identified by the attribution methods shall be consistent. To quantify this consistency, we instruct GILL to describe and generate images for the same semanties ten times, and collect the set of T/I-neurons each time. We then calculate the proportion of neurons that appeared in all ten sets as the quantification of

---

cross-sample invariance. The average invariance Figure 4 (a) presents the average invariance for different concepts. Specifically,
Observation 4; NAM outperforms all baselines by an average of 16.83% w.rl cross-sample invari- ance across all semantics. This demonstrates that NAM extracts the critical neurons for the targeted semantics across samples, effectively filtering out the neurons sensitive sample-specific noise.
Pre-editing
Post-editing Method Value (L)
"boy"
"girl"
CE
0.726
Gral
0.625
GraD
0.471
GraT
0.502
AcT
0.390
AcU
0.362
4.3.3 Semantic Specificity
NAM
0.350
"dog"
"cal"
CE
0.601
Neurons that are crucial for specific semantics should not be indiscriminately crucial across oth- ers. Therefore, we study the neuron's specificity in this part. We identify the Top-500 T/I-neurons for the specific semantics. Then, we show (1) the number of neurons that are crucial for specific se- mantics solely and (2) the average number of neu- rons whose probability of being crucial to other se- manties is lower than
Gral
0.540
GraD
0.458
GraT
0.321
AcT
0.259
AcU
0,288
NAM
0.255
"cantaloupe"
"appee"
CE 0.493
Gral
0.360
GraD
0.357
GraT
0.401
AcT
0.328
(10%, 15%.... 45%) in Figure 4 (b). The results of the three best- performing methods are exhibited here. These re- sults highlight that:
AcU 0.324
NAM
0.316
Table 3: Results of image editing. The Value rep- resents the &2 norm of the perturbation added to the I-neurons, demonstrating that NAM necessi- tates minimal perturbations for the editing.
Observation 5: The T/I-neurons identified by our NAM are specialized and not commonly sensitive across different semanties, verifying their speci- ficity across the semantics.
4.4 RQ3: Image Editing Enhanced by NAM
Lastly, we aim to edit the images generated by MLLMs through perturbing I-neurons identified by NAM, as outlined in Section 3.3. Table 3 exhibits the pre- and post-editing semantics, the selected images for collecting h and hi4, and the magnitude of the perturbations. Furthermore, for fair comparisons, we also perturb I-neurons identified by baselines to achieve similar editing results. According to Table we can find that:
Observation 6: NAM-enhanced editing methods can not only replace the original semantics with the target semantics precisely within the outputs of MLLMs, but also necessitate minimal perturbations. Specifically, the perturbation it added 40.2% less than the baselines on average, and nearly 15% less than the best baseline, underscoring its significance and potential applications for MLLMs.
Limitations & Future Work
This study provides new insights into interpreting MILMs, enhancing the understanding of their inner working mechanize. However, while our experiments thoroughly investigated the neuron properties within GILL and NEXTGPT, they did not extend broader range of models. Additionally, although the proposed attribution method can be transferred to any other modality, as demonstrated in Section 3.1.1, our experiments focused on text and image outputs solely. Looking forward, we plan to incorporate more MLLMs and modalities into our research, and streamline our attribution method to eliminate the reliance on external interpreters. By expanding the scope of our study and refining our method, we aim to uncover more valuable insights that will benefit the MLLM community.
6 Conclusion
We propose NAM, a novel neuron attribution method tailored for MLLMs. Specifically, NAM is tailored for multi-modal attribution, revealing the modality-speciffe semantie properties learned by neurons within the FFN layers.To To address the challenges of extending attribution methods from text-

---

only LLMs to MLLMs, NAM first employs a leading image segmentation model to remove the noisy semanties, then proposes a new attribution score eliminate the need for additional forward/backward propagation or gradient calculations. Based on NAM, we highlights several intriguing properties of neurons, elucidating the inner workings mechanism of MI.LMs.
Acknowledgments
This research is supported by the National Science and Technology Major Project (2023/D0121102), and National Natural Science Foundation of China (92270114).
References
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
[3] Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese. Jilin Chen, Emily Pitler, Timothy P. Lillicrap. Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Is- ard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïl White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CORR, abs/2312. 1185,, 2023.
[4] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[5] Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer- based language models. In EMNLP, pages 1 11132-11152. Association for Computational Linguistics, 2022.
[6] Xuansheng Wu, Haiyan Zhao, Yaochen Zhu, Yucheng Shi, Fan Yang. Tranming Liu, Xiaoming Zhai, Wenlin Yao, Jundong Li, Mengnan Du, and Ninghao Liu. Usable XAI: 10 strategies towards exploiting explainability in the LLM era. CoRR, abbs/2403.08944. 2024.
[7] Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep NLP models: A survey. Trans. Assoc. Comput. Linguistics, 1111285-1333. 2022.
[8] Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality- based localization vs. knowledge editing in language models. In NeurlPS, 2023.
[9] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In EMNLP,, pages 30-45, Association for Computational Linguistics, 2022.
10

---

[10] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitski,, and Dimitris Bert- simas. Finding neurons in haystack: Case studies with sparse probing. CoRR, abs/2305.01610, 2023.
[11] Niklas Stoehr, Mitchell Gordon, Chiyuan Zhang, and Owen Lewis. Localizing paragraph memorization in language models. CoRR, abs/2403. 19851, 2024.
[12] Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Zhou. Emergent modularity in pre-trained transformers. In ACL (Findings), pages 106664083,, Association for Computational Linguistics, 2023.
[13] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In NeurlPS, 2022.
[14] Kevin Meng. Arab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. In ICLR. OpenReview.net, 2023.
[15] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In EMNLP, pages 10222-10240. Association for Computational Linguistics, 2023.
[16] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In ACL (1), pages 8493-8502. Association for Computational Linguisties, 2022.
[17] Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In EMNLP, pages 12216-12235, Association for Computational Linguistics, 2023.
[18] Haowen Pan, Yixin Cao, Xiaozhi Wang, and Xun Yang. Finding and editing multi-modal neurons in pre-trained transformer. CoRR, 025/2311.074700 2023.
[19] Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multi- modal neurons in pretrained text-only transformers. In ICCV (Workshops), pages 2854-2859
IEEE, 2023.
[20] Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. CoRR, abs/2401.13601, 2024.
[21] Junnan Li. Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742, PMLR, 2023.
[22] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[23] Deyao Zha, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhaneing vision-language understanding with advanced large language models. CORR, abs/2304.10592, 2023.
[24] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. CoRR, abs/2308.01394 2023.
[25] Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. In NeurlPS, 2023.
[26] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.

---

[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI,, volume 139 of Proceedings of Machine Learning Research, pages 8748-8761 PMLR, 2021.
[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR. pages 10674-10685, IEEE, 2022.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998 -6088, 2017.
[30] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA-02: A visual representation for neon genesis. CoRR, abs/2303.11331, 2023.
[31] JoaoLages. diffusers-interpret.
ot ftussssdddduussssssterrrrre
2019.
[32] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In EMNLP (I), pages 5484-5484 5495. Association for Computational Linguistics, 2021.
[33] Junfeng Fang, Houcheng Jiang. Kun Wang, Yunshan Ma. Xiang Wang, Xiangnan He, and Tat seng Chua. Alphaedit: Null-space constrained knowledge editing for language models, 2024.
[34] Houcheng Jiang. Junfeng Fang, Tianyu Zhang, An Zhang, Ruipeng Wang, Tao Liang, and Xiang Wang. Neuron-level sequential editing for large language models, 2024.
[35] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal LLM. CoRR, abs/2309.05511 2023.
[36] Tsung-Ys Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV (5), volume 8693 of Lecture Notes Computer Science, pages 740-755. Springer, 2014.
[37] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP (1), pages 7514-7528. Association for Computational Linguistics, 2021.
[38] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In ICLR. OpenReview.net, 2020.
[39] Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. Mover- score: Text generation evaluating with contextualized embeddings and earth mover distance. In EMNLP/IJCNLP (1), pages 563-578. Association for Computational Linguistics, 2019.
[40] Thibault Sellam, Dipanjan Das, and Ankur P. Parikh, BLEURT: learning robust metrics for text generation. In ACI., pages 7881-7892 Association for Computational Linguistics, 2020.
[41] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. CoRR, abs/2306. 13549, 2023.
[42] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
[43] Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 2rX1::2301.12597 2024.
[44] W Dai, Li, D Li, AMH Tiong, Zhao, W Wang, B Li, P Fung and S Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint a755v:2305.06500.
12

---

[45] Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Bootstrapping vision- laiguaee learning with decoupled language pre-training In NewrIPS, 2023.
[46] Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. CoRR, abs/2306.1433 2023.
[47] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. CoRR, abs/2307.05222 2023.
[48] Kaizhi Zheng, Xuchai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language generation via generative vokens. CoRR, abs/2310.02239, 2023.
[49] Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Hervé Jégou, and Léon Bottou. Birth of transformer: A memory viewpoint. In NeurlPS, 2023.
[50] Cheongwoong Kang and Jaesik Choi. Impact of co-occurrence factual knowledge of large language models. In EMNLP (Findings), pages 7721-7735. Association for Computational Linguistics, 2023.
[51] Gaurav Verma, Minje Choi, Kartik Sharma, Jamelle Watson-Daniele Sejoon Oh, and Srijan Kumar. Mysterious projections: Multimodal IIms gain domain-specific visual capabilities without richer cross-modal projections. CoRR, 112/2442.118322 2024.
[52] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. CoRR, abs/2309.08600, 2023.
[53] Omer Antverg and Yonatan Belinkov. On the pitfalls of analyzing individual neurons in language models. In ICLR. OpenReview.net, 2022.
[54] Chandan Singh, Aliyah R. Hsu, Richard Antonello, Shailee Jain, Alexander G. Huth, Bin Yu, and Jianfeng Gao. Explaining black box text modules in natural language with language models. CoRR, abs/2305.098663 2023.
[55] Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. CoRR, 0is/2307.09455 2023.
[56] Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In ACI. (1), pages 16124-16170. Association for Computational Linguistics, 2023.
[57] Haiyan Zhao, Hanjie Chen, Fan Yang. Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang. Dawei Yin, and Mengnan Du Explainability for large language models: A survey. CoRR, fbs/2309.01029) 2023.
13

---

A Related Work
Multi-modal Large Language Models. In recent years, MLLMs have made significant progress, continually pushing the boundaries of performance in various downstream tasks [41]. The launch of models like GPT-4 (Visual) and Gemini, with their impressive multimodal (MM) understanding and generation capabilities, has sparked intense research interest in MM-LLMs [20]. In terms of understanding multimodal content, preliminary research often utilizes multimodal encoders like the ViT [42] and CLIP [27] VIT to capture representations, and projectors like Q-Former [43, 44] and P-Former [45] to align these representations with the embedding space of foundational LLMs. This approach covers tasks such as image-text understanding, with representative models including BLIP-2 [21], MiniGPT-4 [23], LLAVA [22], and OpenFlamingo [24].
Recently, the capabilities of MM-LLMS have expanded to support specific modal outputs. These methods align certain embeddings from the foundational LLM with the input space of well-trained multimodal generator through another projector. This extension includes tasks with image-text outputs demonstrated by models like GILL [25], Kosmos-2 [46], Emu [47], NExT-GPT [35] and MiniGPT-5 [48], which is the focus of this paper. Recent research endeavors have focused on mimicking human-like like mnytoo-any modality conversion, shedding light on the path to artificial general intelligence [20].
Interpretability of Pre-trained Transformers. Demystifying the internal mechanisms of LLMs becomes increasingly vital, particularly in applications emphasizing faimess, trust, and ethical decision-making [49, 50, 52, 53, 54, 55J. In recent years, many works have focused on explaining pre-trained transformers. For instance, [32] regards the FFN as unnormalized Key- Value Memories; [56] presents conceptual framework where all parameters are interpreted by projecting them into the embedding space; [9] analyzes the FFN updates in the vocabulary space, showing that each update can be decomposed to sub-updates; [17] studies how the model aggregates information about the subject and relation to predict the correct attribute; while [11] localizes the weights and mechanisms used by a language model to memorize and recite entire paragraphs of its training data.
Neuron Attribution. Neuron attribution aims to reveal the black box of pre- pre-trained transformers by answering the following questions [7, 57, 6]: (1) What concepts are learned within neurons of the network? (2) Are there neurons that specialize in learning particular concepts? (3) How localized/distributee and redundantly is the knowledge preserved within neurons of the network? To achieve these goals, many recent works have focused on exploring the properties of neurons in LLMs. Specifically, [16] present preliminary studies on how factual knowledge is stored in LLMs by introducing the concept of knowledge neurons; [5] finds the special ncurons whose activations on soft prompts are highly predictive of the task labels of inputs, and dub them skill neurons.
For MLLMs, [18] employs the neuron contribution within the residual stream to identify multi-moda neurons in Transformer-base multi- -ooall LLMs, while [19] claims that image prompts cast into the transformer embedding space do not encode interpretable semantics, and translation between modalities occurs inside the transformer. Additionally, despite the aforementioned methods are all based on activation and gradients to identify target neurons, some recent works focused on knowledge editing have discovered an alternative approach [15, 13]. This involves using causal effect methods to identify key neurons by perturbing neurons and observing changes in the output [14].
For the detailed contribution scores these works employed and their relationship with our NAM, please see Appendix B.I.
B
More Experiments
B.1 Experimental Settings
Baseline. Here we first detail the contribution score of the baseline methods used. We will use the formulas and symbols from the paper to provide formal explanation. Specifically, for the p-th word in Vocabulary V and the k-th neuron un in layer I, we have:
* AcT takes the neuron activation as the contribution score following:
$ of
(9)
14

---

AcU takes the product of the neuron activation and the unembedding matrix as the contribution score following:
= - ( (Wu)p owt))
(10)
GraD takes the gradient of the output dimension corresponding to the output word w.r/ activation as the contribution score following:
Oyp
(11)
Dat
GraT takes the product of the gradients and activation as the contribution score following:
k OUp
di daf
(12)
Gral takes the product of the integral of the gradients and the activation as the contribution score following:
$/ = of [
Oy dak
da,
(13)
where vi is the p-th dimension of the output when we fix the activation of neuron u/ as of Note that perturbing and performing forward propagation for each neuron multiple times is time- consuming. Therefore, we add the same coefficient a to multiple neurons simultaneously and calculate their integrals to obtain their respective contribution scores.
CE takes the causal effect of activation on outputs, quantified by the output change when activation is perturbed, as the contribution score following:
0yp "
(14)
daf
where v/ is the p-th dimension of the output y when we add the activation of neuron un with a Gaussian Noise. We set the mean of the Gaussian noise to 0 and the variance to the total variance of the neuron activation values the respective layer. Similar to Gral, for each individual neuron, performing forward propagation for each neuron multiple times is costly. Therefore, we add perturbations to multiple neurons simultancously and evenly distribute the resulting output changes among them their respective contribution scores.
Furthermore, is important to note that the above baselines all attribute text outputs neurons. Therefore, we have adapted all these baselines to image outputs for comprehensive validation. For example, the attribution score of the baseline Grad for text output the gradient of activation when the output is the word "dog". When transferred to adapt the image output, its attribution score is the gradient of activation when the output an image of a dog. Specifically, since the last representation h is the input of the image generation module, we have:
Öh
(15)
daf
Hyperparameter Configuration. For our experiments, we sourced the training and testing data for the COCO dataset directly from its website". Similarly, we obtained the source code for the target models GILL* and NEXT-GPT', the image segmentation model E and the attribution algorithm Diffuser Interpreter used for stable diffusion from the links cited in their respective publications
All hyperparameter settings, such as the division of training and testing datasets, learning rate, and optimizer, consistent with the original configurations of the above link unless otherwise stated. Additionally, it is important to note that, unless explicitly mentioned, the samples used in the experiments were 500 images randomly selected from the COCO dataset.
Furthermore, we use Quadro RTX6000 GPUs with 24GB of memory as a representative example of consumer-level GPUs; 40GB A100s and 80GB H100s to provide datacenter-level benchmarks.
PCOCO: http://images.cccoaatasst
'GILL: https::/githu..com/o/oooo wowj.oogooooooigggggglllllll
'NEXT-GPT: http://gttubb.oo/AEEETTGPP
"EVA-02: https///ghhhbbooomooooog//////te
15

---

Table 4: The semantics of T/I-neurons with the highest attribution scores identified by different attribution methods, when the output of MLLMs containing the targeted semantics. (L1,Uk) denotes the k-th neuron at layer I. For each method, we report semantics with top-4 probabilities.
Output
Segmentation
Category
Method
Location
Semantics
Grad
L38.U18188
"card" "screen" "Card")
T-Neurons
AcT
L26.
"collee "tea" "hrew"
L27. U8520
("CLSID" "Nanto" "Kinnikuman"
NAM
L27. 29810
I'glass', "Glass", "bud")
Grud
L28. U3335
I ("spirits' 'spirit' 'run') "bit" "lot' 'few"
I-Neurons
AcT
L30. US844
CE
13193
("order' 'Order', 'orders") ('drink', 'drinking', 'drinks')
NAM
L27.U413
Grad
L30. U4516
['roll', 'rolls". 'Roll") ''eat' "cating" "eaten" ("llexible" 'britle", "bend" ("taste', 'lasted', "tasting"
T-Neurons
AcT
L28. U6623
CE
L25, U1183
NAM
L30.U8704
Grad
L30. 44888
('fruit' 'rose'
I-Neurons
AcT
L27. U6888
("snap'. 'taken", 'snapped") ["ood' 'negative' "develops' ('cooiies, "pancakes' "baked")
CE
L25, 414616
NAM
02615
Grad
L31. U484
("vaults' "70710;, '20439"] d'rrierr 'derivers' "vehicle")
T-Neurons
AcT
CE
L24. U15338
'few 'few."oo llo"" 'bit" I (bbike, 'bikes', 'cycle)
NAM
L25.U3913
Grad
L28. 128.81208
('20439', 'Kinnikuman', 'Nanto"} (CLSiD "gun" 'car', "bike"]
I-Neurons
AcT
CE
L25. U4343
L27. U6437
('CLSID "shapeshifter, 'couple') ("bike', 'bikes", 'cycle")
NAM
L25.U3911
Ablation Study. Our attribution score design primarily consists of two components: activation and its mapped values on the target semantic dimension in the vocabulary. We aim to leverage the complementary effects both components, using them as indicators of indirect and direct contributions, respectively. The results of baseline methods AcT and AcU can be approximated as the attribution effects when using each component separately. Therefore, comparing our method with AcT and AcU can also be considered as ablation study. This comparison also reflects that using only the mapped values (AcU) can introduce certain biases, leading to poorer performance across various metrics.
External Model Algorithm. Then, we introduce the semantic segmentation model EVA02 and attribution algorithm for the stable diffusion model Diffuser Interpret. Specifically, EVA02 [30] comprises a series of robustly optimized plain Vision Transformers (ViT's of moderate model sizes, featuring transferable bidirectional visual representations learned from powerful CLIP vision encoder via masked image modeling (MIM) pre-training. Compared to current leading vision models with billions of parameters, the EVA-02 variants necessitate significantly fewer computational resources, enabling a more in in-depth exploration of often-overlooked aspects. Furthermore, Diffuser Interpret [31] attributes the pixel values of the generated image the input embedding of the stable diffusion using gradient information. We also apply this strategy to the attribution of the projector used by GILL.
B.2 More Experimental Results
Results Semantic Relevance. To better illustrate the comparison between our attribution method and the baseline attribution methods, we present additional qualitative experimental results to provide more intuitive comparison, shown in Table The effects demonstrated by these methods similar to those shown in the experimental results in the main text, indicating that the neurons identified by our NAM method are semantically closer to the target semantics.
Results on NEXTGPT. In the main body of the text, we have completed testing the effectiveness of various neuron attribution algorithms with GILL as the target model. In this section, expand our experiments to include NEXTGPT as the target model. Below, we present the key neurons identified in NEXTGPT and their semantic correlations with images and captions. The results can be found in Table 5.
Consistency Between Neuron's Semanties and Caption Given by GPT. The main text only presents the average consistency the T/I-neurons identified by various attribution methods with the captions of the input/output images. To refine comparison, provide the specific consistency
16

---

Table 5: Consistency between the neuron's semantics and the images/captions. Grad. Act. and Ca. denote gradient-, activation- and causality-basee methods, respectively. and represent the CLIPScore w.r. input and output images. We use background highlights the best performance.
Class
T-Neurons
Method
Grad.
Act.
Ca.
CCLipccore
"CLipScore
BERTScore
MoveScore
BLEURT
CE
0.20810.017
0.244
0.257 032
0.261
0.03810.000
Gral
0.228 20.018
0.233 022
0.268 034
0.290 0d5
0.04220.006
GraD
0.309 0.309 #0.061
0.387 42.
4.454.000
0.402
0.059
GraT
0.406ga..oz
0. 0.409
0,485 026
0. 0,480 20.00a
0.068g-a.00s
AcT
0.607=20.036 230006
1.58720.044
0.608 2.0. 065
0..0220.0555
0.120
AcU
0.593
0.615 053
0.592
0.611
0.141 00.00
NAM
0.614
0.620
0.616+0.042
0.63340.096
0.145+a.021
Class
I-Neurons
Method
Grad. Act.
Ca.
CCLipScore
*CLipScore
BERTScore
MoveScore
BLEURT
CE
0,217
0.210
0.238 036
0,254
0.041
Gral
0.226
0.242 aa.088
0.246
0.265
0.033
GraD
0.362
0.369 24.944.00z
0.418
0.388
0.055
GraT
0.41720.045
0.386 030
0.4666 039
0.421 0415
0.06920.000
AcT
0..03
0.564
0.568 062
0.608
0.11920.015
AcU
0.59220.005
0.583
0.582 042
0.535
0.130
NAM
0.608
0.585
0.592
0.635
0.142
scores w.z/ the captions generated by GPT for the output images. The detailed results are shown in Table 6 and 7. These results also demonstrate the highest consistency scores of our attribution method compared to the baselines.
C Broader Impact
In this paper, we present a novel neuron attribution method NAM to interpret the MLLMs. Based on NAM, we reveal several intriguing neuron properties within ML..Ms These properties collectively elucidate the inner workings of neurons within MI.LMs, enhancing our understanding of their capacity to process and generate multi-modal content. This approach can contribute to wide range of applications of MLLMs, boosting the MLLMs across various downstream tasks such as knowledge editing [15, 13, 14]. We believe that the neuron properties drawn from NAM can shed light for future research on MLLM community, and inspire further exploration into understanding neurons within other pre-trained transformers.
17

---

Table 6: Consistency between the neuron's semantics within GILL and the captions given by GPT. Grad., Act. and Ca. denote gradient-, activation- and causality-basee methods, respectively. i and represent the CLIPScore w.zf imput and output images. We use background highlights the best performance.
Class
T-Neurons (GIIL)
Method
Grad.
Act.
Ca.
BERTScore
MoveScore
BLEURT
CE
0.25440.008
0.263
0.036
Gral
0.262
0.278 20.022
0.04010.008
GraD
0.452 00.027
0.392 1.392200001
0.062
GraT
0.476 029
0.475
0.066+
AcT
0.599
0.623
0.113+0.010
AcU
0. 0.584
0.615
0.1344..020
NAM
0.601 =0000000000 037
:.601=0.000
0.618
0.142+0.019
Class
I-Ncurons (GILL)
Method
Grad.
Act.
Ca.
BERTScore
MoveScore
BLEURT
CE
0.24 1.241:0..00 037
0.248
0.038 009
Gral
0.237 03.033
0.267 0.267 10.003
0.027
GraD
0.409
0.382 10.036
0.053
GraT
0.470 0..70.0..035
0.408 10.041
0.070
AcT
0.554 050
0.602 #0.062
0.117
AcU
0.571 00.037
0.528 062
0.124+0.014
NAM
0.584 00.39
0.622
0.13710.021
Table Consistency between the neuron's semantics within NExTGPT and captions given by GPT, Grad., Act. and Ca. denote gradient-, activation- and causality-based methods, respectively. $ and represent the CLIPScore wet input and output images. We use background highlights the best performance.
Class
T-Neurons (NEXTGPT)
Method
Grad.
Act.
Ca.
BERTScore
MoveScore
BLEURT
CE
0.263
0.259 020021
0.036+20007
Gral
0.264.0.007
0.288 10.05
0.040
GraD
0.455
0.393
0.060
GraT
0.482++.024
0.472 .039
0.070+0.010
AcT
0.602
0.624 0043
0.11440.007
AcU
0.585
0.623 (6.2310.02
0.135
NAM
0.613 10.034
0.624
0.140
Class
I-Neurons (NExTGPT)
Method
Grad.
Act.
Ca.
BERTScore
MoveScore
BLEURT
CE
0. 0.23440.022
0.243
0.037
Grai
0.239 +0.028
0.271 025
0.036+0.000
GraD
0.417 =0.000
0.382 03307
0.054
GraT
0.458 10.034
0.406.1n.04
0.070
AcT
0.570++..00 065
0.597 10.97+0.054 05.0
0.11440.008
AcU
0.579 039
0.528 054
0.126=20018
NAM
0.588
0.627
0.138+0.020
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, we introduce the neuron attribution method for MILMs aimed enhancing the transparency and reliability of MLLMs (See Abstraction and Introduction Section).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: In this work, we systematically discuss the limitations of our research and outline directions for future work (See Introduction Section).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include experimental results related to theoretical aspects. Guidelines:
The answer NA means that the paper does not include theoretical results.
19

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide the code necessary for replicating the studies described in this paper via an anonymous link, and we detail the experimental setup for the replication in the article itself (See Abstraction Section).
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken
to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: For the datasets disclosed in the article, we have provided information regarding their sources and origins (See Experimental Section).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
20

---

Please see the NeurIPS code and data submission guidelines (https://nips..
/ttegguuid///ooeeuummmm ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht tps:
/ //nips. cc/publcpeeeccccccccc ttliii//uuieeeee / cotuubiissionss ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new reproducible, they
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: we have specified all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail
that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Ys]]
Justification: In this paper, we have reported error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If emror bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
21

---

8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: In this paper, we provide detailed information about the experimental resources, including GPU configurations used in our studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neerraa http:://weerrpps.ccppiiceeeeeeee wwtpp//bbi/ttthtcieuuieeeeeeeeee iineessleelee?e Answer:
[Yes]
Justification: The study presented in this paper conforms to the NeurIPS Code of Ethics. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: We have provided the societal impacts of the work. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
22

---

Answer: [NA]
Justification:
This paper does address issues related to this aspect.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with
necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All creators and original owners of the assets used in our paper, such as code, data, and models, have been properly credited. We have explicitly mentioned the licenses and terms of for each asset and have ensured full compliance with these terms throughout our research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are relcased, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The research presented in this paper is not concerned with new assets. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve experiments or research related to human subjects. Guidelines:
23

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not address potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "ULjMO8YgJG", "reviewer": "Reviewer_kLxz", "summary": "Summary:\nThis paper introduces NAM (Neuron Attribution Method), a novel approach for attributing neurons to specific semantic concepts in multimodal large language models (MLLMs). The key contributions are: (1) A method to identify modality-specific neurons (text or image) that are crucial for particular semantic concepts. (2) Analysis of neuron properties like cross-modal invariance and semantic sensitivity. (3) A framework for multimodal knowledge editing based on the identified neurons.\n\nThe authors evaluate NAM on GILL and NExTGPT models, comparing it to several baseline attribution methods. They demonstrate NAM's effectiveness in identifying semantically relevant neurons, its cross-sample invariance, and its utility for targeted image editing tasks.", "questions": "None.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an important gap in the interpretability of MLLMs, extending neuron attribution techniques from text-only models to multimodal systems.\n- The authors conduct extensive experiments to validate NAM, including comparisons with multiple baselines and analysis of various neuron properties.\n- The method is well-motivated, building on existing work in LLM interpretability while addressing the unique challenges of multimodal systems.\n- The paper provides a detailed description of the NAM algorithm, making it potentially reproducible by other researchers.", "weaknesses": "- While the authors acknowledge this limitation, the evaluation is conducted on only two MLLMs (GILL and NExTGPT). Testing on a broader range of models would strengthen the generalizability claims.\n- The experiments focus solely on text and image modalities. Exploring other modalities (e.g., audio, video) would provide a more comprehensive evaluation of NAM's capabilities.\n- The method depends on external image segmentation models to remove noisy semantics. This introduces a potential source of bias and may limit the method's applicability to scenarios where such models are unavailable or unsuitable.\n- The paper would benefit from ablation studies to isolate the impact of different components of the NAM algorithm, particularly the novel attribution score calculation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer kLxz:\n\nThank you very much for your comments. We sincerely appreciate the time and effort you have dedicated to reviewing our work! Below, we meticulously provide responses to each of your comments and outline the modifications based on your suggestions. Hope that our responses could address your concerns!\n\n>***W1. Testing on a broader range of models would strengthen the generalizability claims.***\n\nThank you for your insightful suggestion. Following your feedback, in addition to the GILL and NextGPT, we conducted experiments on two additional multimodal LLMs, EMU and DreamLLM. The results, summarized in the table below, demonstrate that our explanation method achieves optimal performance across all metrics, thus further validating the generalizability of our approach.\n||||||\n|:-:|:-:|:-:|:-:|:-:|\n|Method|Model|CLipScore|BERTScore|MoveScore|\n|GraD||0.376|0.461|0.387|\n|AcT|EMU|0.469|0.582|0.595|\n|NAM||**0.499**|**0.625**|**0.634**|\n||||||\n|GraD||0.382|0.449|0.362|\n|AcT|DreamLLM|0.510|0.502|0.568|\n|NAM||**0.538**|**0.521**|**0.593**|\n||||||\n\nWe have incorporated these additional results into the revised version of the paper, along with a more detailed analysis. Due to the limited time available during the rebuttal phase, we were able to include only these two sets of experiments. We hope for your understanding in this matter.\n\nFurther experiments with additional multimodal LLMs are currently underway, and we plan to include them in the experimental section of the paper to enhance the robustness and comprehensiveness of our study. We hope that our response and the additional experiments meet your expectations.\n\nThank you once again for your valuable feedback!\n\n>***W2. Exploring other modalities would provide a more comprehensive evaluation of NAM's capabilities.***\n\nThanks for your valuable suggestion regarding the exploration of other modalities. Following your recommendation, we have extended our analysis beyond text and image modalities to include audio modality on NExT-GPT, which is capable of generating sound, and the following results validate its cross-modal generalizability.\n|||||\n|:-:|:-:|:-:|:-:|\n|Method|Specificity|Relevance|Invariance|\n|CE|0.217|0.403|0.274|\n|GraI|0.264|0.500|0.337|\n|GraD|0.263|0.509|0.355|\n|GraT|0.317|0.496|0.347|\n|AcT|0.346|0.561|0.406|\n|AcU|0.349|0.552|0.396|\n|NAM|**0.361**|**0.596**|**0.428**|\n|||||\n\nDue to the time-consuming nature of generating other modalities, such as video, we were unable to complete all experiments within the short rebuttal period. However, these experiments are currently in progress, and we plan to include them in the updated version of the paper once completed. We appreciate your understanding on this matter.\n\nHope that our response and the additional experiments could meet your expectations!\n\n\n>***W3. The method depends on external models (image segmentation models), which introduces a potential source of bias.***\n\nThanks for raising this important concern. We acknowledge that our method depends on external models, which would introduce potential bias. However, we believe that this dependency is an acceptable aspect within the study of explainability in multimodal LLMs for several reasons.\n\n1.\t**External Models in Multimodal LLMs:**\nLet's first consider the current landscape of multimodal LLMs (*e.g.*, EMU, DreamLLM, Unified-IO 2, RPG, CM3Leon, and GILL). These models rely on external models (*e.g.*, diffusion models) for multimodal generations. While these external models may introduce bias similar to the models our method employ, they have not hindered the remarkable success of multimodal LLMs. On the contrary, these well-developed models have become significant assets for multimodal research. Hence, we believe that the advantages of using external models far outweigh the potential bias introduced.\n\n2.\t**Optimal Performance Across Metrics:**\nAccording to our experimental results, our method has achieved optimal performance across various metrics. This indicates that the additional bias introduced by external models has a minimal impact on the model's accuracy, and does not impede our method from achieving state-of-the-art results. Moving forward, as segmentation models advance, the upper bound of our method's capabilities will continue to improve.\n\n3.\t**Early Stage of Multimodal Explainability:**\nThe field of multimodal explainability is still in its early stages. Utilizing well-established external models from a mature field, such as image segmentation models, to aid initial exploration is inevitable. Furthermore, we believe that as the field advances, the dependency on external models will gradually decrease, and domain-specific models will emerge to further progress in research. We hope to contribute meaningfully toward achieving this goal.\n\nTo address your concern, we have added the above discussion into the revised version to enhance the rigor of the paper.\n\nHope our response could address your concern!\n\n>***W4. The paper would benefit from ablation studies to isolate the impact of different components.***\n\nThank you for your concern. Following your suggestion, we have included ablation studies in the revised version, which focuses on two main components: our proposed attribution score (AS) and the denoising module (DM). Here are the results:\n||||||\n|:-:|:-:|:-:|:-:|:-:|\n|Model|Method|Specificity|Relevance|Invariance|\n||w/o AS|0.625|0.752|0.613|\n|GILL|w/o DM|0.660|0.759|0.635|\n||NAM|0.674|0.782|0.644|\n||||||\n||w/o AS|0.598|0.643|0.527|\n|NExTGPT|w/o DM|0.611|0.661|0.532|\n||NAM|0.624|0.689|0.546|\n||||||\n\nThese results demonstrate that each component of our design enhances the performance of our method. We hope these additional experiments could improve the overall integrity of our experiments.\n\n*Once again, we are deeply appreciative of the time and expertise you have shared with us. We are more than happy to add clarifications to address any additional recommendations and reviews from you!*\n\n*Best,*\n\n*Authors*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CxqyGc4w7I", "reviewer": "Reviewer_VhwX", "summary": "The work introduces a novel Neuron Attribution Method (NAM) tailored for MLLM. The NAM approach aims to reveal the modality-specific semantic knowledge learned by neurons within MLLMs, addressing the interpretability challenges posed by these models. The method highlights neuron properties such as cross-modal invariance and semantic sensitivity, offering insights into the inner workings of MLLMs.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. NAM effectively differentiates between modality-specific neurons, ensuring accurate attribution of text and image outputs to the relevant neurons. This granularity aids in understanding how MLLMs process multi-modal content.\n\n2. The observation  \"..T/I-neurons identified by our NAM are specialized, showing specificity across different semantics. They are not generally sensitive to multiple semantics, confirming their targeted functionality... \" and the application of NAM to image edit are interesting.", "weaknesses": "The NAM method relies on advanced segmentation models (like EVA02) and attribution algorithms (like Diffuser-Interpreter) tailored for specific generation modules. This dependency could limit the method's applicability and flexibility, especially when dealing with different or emerging MLLMs and generation technologies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer VhwX:\n\nThank you very much for your comments. We sincerely appreciate the time and effort you have dedicated to reviewing our work! Below, we meticulously provide responses to each of your comments and outline the optimizations made to the manuscript based on your suggestions. Hope that our responses could address your concerns!\n\n\n>***W1: The NAM method relies on external models (e.g., image segmentation models), which could limit the method's applicability and flexibility.***\n\nThank you for raising this important concern. We acknowledge that reliance on external models may introduce limitations in terms of applicability and flexibility. However, we believe that this dependence is acceptable in the context of multimodal LLMs explainability for several reasons. Specifically,\n\n1. **Employment of External Models in Multimodal LLMs:** Let's first consider the current landscape of multimodal LLMs (*e.g.*, EMU [1], Unified-IO 2 [2], DreamLLM [3], RPG [4], CM3Leon [5], GILL [6], and NextGPT [7]). These models often incorporate external models (*e.g.*, diffusion models) for multimodal understanding and generation. While these external models may introduce limitations in applicability and flexibility, they have not hindered the success and rapid development of multimodal LLMs. On the contrary, these mature models from well-established domains have become significant assets for multimodal research. **Hence, we believe that the benefits of utilizing external models in this domain** (*****i.e.***, multimodal research) far outweigh the potential limitations they might introduce.**\n\n2.\t**Optimal Performance Across Diverse Task Configurations:** According to our experimental results, our method achieves optimal performance across various modalities, datasets, and metrics. This impressive outcome, achieved through the integration of external models, demonstrates the substantial advantages of cross-domain collaboration. It indicates that the limitations introduced by external models (1) have a minimal impact on our method's application to mainstream downstream tasks, and (2) do not impede our model from achieving state-of-the-art results across various tasks. Moreover, another potential advantage of cross-domain collaboration is that as the external models continue to advance, the upper bounds of our method's capabilities will also improve, expanding the range of problems it can address. **Therefore, we believe that cross-domain collaboration should not be constrained by the potential limitations introduced by external models**.\n\n3.\t**Early Stage of Multimodal Explainability:** The field of multimodal LLM Explainability is still in its early stages. Leveraging well-established external models to aid initial exploration is a common practice. We believe that as this field advances, the dependency on external models will gradually decrease, and domain-specific models will emerge to further propel research. We hope to contribute meaningfully toward achieving this goal.\n\nAdditionally, to address your concern, **we have emphasized the potential limitations introduced by external models in the Limitations section of the revised version, and included the above points of discussion in the Appendix section**, hoping to inspire further thought among researchers in the field of LLMs explainability.\n\nHope that our response and the revisions made to the manuscript could address your concern!\n\n\n*Once again, we are deeply appreciative of the time and expertise you have shared with us. Your insightful suggestions have significantly enhanced the rigor of our experimental design and enriched the content of our manuscript, and we are more than happy to add clarifications to address any additional recommendations and reviews from you!*\n\n*Best,*\n\n*Authors*\n\n[1] Generative Pretraining in Multimodality. ICLR 2024\n\n[2] DreamLLM: Synergistic Multimodal Comprehension and Creation. ICLR 2024\n\n[3] Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action. CVPR 2024\n\n[4] Next-GPT: Any-to-any multimodal LLM. Arxiv 2023\n\n[5] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs. ICML 2024 \n\n[6] CM3Leon Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning. Arxiv 2024\n\n[7] Generating Images With Multimodal Language Models. NeurIPS 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZDM3MKPRf", "reviewer": "Reviewer_VfTi", "summary": "This paper proposes a method to attribute the multimodal output to neurons that are influential in the generation process. The approach considers image-text multimodal LLM where 1) the base LLM's internal representation is directly used to generate text, and 2) also passed to an image generation model. These this two tiered model architecture, the proposed approach breaks down the problem into attribution for the base model and attribution for the image generation model, which can be combined to cover all neurons in the considered architecture. The proposed approach is evaluated with baselines and shown to outperform in several tests including cross sample invariance.", "questions": "- Q1: Are the bars in Fig 3(c) stacked or overlayed? Is the green region T \\cup I - T \\cap I?\n- Q2: How do we know the activations of the neurons are signals. Why not lack of activations?\n- Q3: Related work should be moved to the main text. What's the difference to AcU [2]? Is it the same as the activations through the residual stream in your approach (Sec 3.1.2.)?\n- Q4: How do you know the neuron semantics and how they are correct? How do you know they are monosemantic? Why \"dog\", \"shark\" are wrong for L28.U4786?\n\nOverall, I find the paper has quite a bit of room to improve. However, I think this paper attacks an important problem, and shows promising results. Although not all evaluation methods are convincing, this is common for a growing subfield. At least some of the evaluation results can be useful to particular use cases, and I find this work worth dissemination. I wouldn't be upset if someone want this paper to be more refined and required another full review round, given there are some obvious presentation issues.", "limitations": "They mentioned the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- S1: The proposed approach is efficient as it does not require backpropagation.\n- S2: The proposed approach outperforms the baselines in several evaluation tasks including cross-sample invariance.\n- S3: The evaluation shows additional examples showing the semantics of the neurons and image editing.", "weaknesses": "- W1: It is unclear how to get to the Eq 1. From the GILL paper, it is using a regular OPT/GPT architecture. How did you get the linear addition of the attention vector instead of the typical multihead self-attention? Also, normalizations should be applied after a self-attention block/linear layer is applied, making linear relation from an intermediate layer to the last layer / penultimate embedding not possible.\n- W2. In the semantic relevance evaluation, relevant words are extracted using the same signal paths considered in the paper, while the signals can flow other ways for the baselines. Therefore, it is unclear if we can consider these words represent the semantics of the neurons identified by other methods. Moreover, the artificial neurons are known to be multisemantic, and neuron groups form a certain semantic instead of individual neurons in isolation. Thus, the metric is deemed to emphasize what the proposed approach is doing by leveraging its definition but might not be suitable for evaluating and comparing with the baselines.\n- W3: The presentation can be improved in many places (e.g., Line 110, 209-210, 217-218, 260, the definition of specificity).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer VfTi:\n\nThank you very much for your comments. We sincerely appreciate the time and effort you have dedicated to reviewing our work! Below, we meticulously provide responses to your comments and outline the optimizations made to the manuscript based on your suggestions. Hope that our responses could address your concerns!\n\n>***W1: How did you get the linear addition of the attention vector in Eq1?***\n\nThanks for your question. As mentioned in Preliminary, vector $a$ in Eq 1 is the output of the attention block that has undergone normalization. Therefore, performing linear addition on $a$ is consistent with the structure of the transformer. Moreover, this linear addition is also supported by existing works on various downstream tasks, such as LLMs editing [1] and explanation [2]. Due to space limitations, we cannot elaborate further here, but we recommend reviewing these works for additional insights.\n\nAdditionally, following your feedback, we have enhanced the Preliminary in the revised version, particularly the descriptions related to Eq1. We believe this can strengthen the rigor and readability of the theoretical parts of our manuscript.\n\nHope our response could address your concern!\n\n[1] Locating and Editing Factual Associations in GPT. NeurIPS 2022\n\n[2] Multi-modal Neurons in Pretrained Text-Only Transformers. ICCV 2023\n\n>***W2: The semantic relevance is deemed to emphasize what the proposed approach is doing, but might not be suitable for comparing with the baselines.***\n\nThanks for your concern. We acknowledge that using semantic relevance for comparing different methods carries potential bias, as its theoretical foundation is similar to our approach, all based on residual connections.\n\nTherefore, following your suggestion, we have revised the experimental section in the new version. Specifically, while other metrics (*e.g.*, invariance and specificity) are retained, semantic relevance is only employed to validate the theoretical soundness of our method. We believe this adjustment will enhance the rigor of our manuscript.\n\nWe hope our response could address your concern!\n\n>***W3: Presentation can be improved in Line 110, 209-210, 217-218, 260 and the definition of specificity.***\n\nWe apologize for the oversight that resulted in some grammatical errors in the original text. Following your feedback, we have rechecked our paper and corrected grammatical errors and unclear expressions. Here are some examples of the corrections made:\n\n-\t*Line 110: To identify the contribution, we define these contribution scores as R.*\n-\t*Line 209-210: The remaining results can be found in Appendix B.*\n-\t*Line 217-218: For details on NAM and the baselines, please refer to Appendix A.*\n-\t*Line 260：Figure 4 (a) presents the average invariance of concepts.*\n-\t*Definition of specificity: Specificity is defined as the proportion of  neurons that are crucial for a single concept solely.*\n\nWe believe these revisions enhance the overall quality and readability of the manuscript. Thanks once again for your valuable comment!\n\n>***Q1: Are the bars in Fig 3(c) stacked or overlayed? Is the green region T \\cup I - T \\cap I?***\n\nWe apologize for any confusion caused. The bars in Fig 3 (c) are **overlayed**, and the green region represents T \\cap I. To avoid this confusion, we have added this clarification to the reversion.\n\n>***Q2: How do we know the activations of the neurons are signals. Why not lack of activations?***\n\nThanks for your concern. Our work aims to explore which neurons play a critical role in generating multimodal content. Intuitively, neurons that are not activated do not seem to contribute to the output. Therefore, we naturally focus on the activated neurons solely.\n\nHowever, as you mentioned, it is possible that non-activated neurons could contribute to the output in a subtle or indirect way. Hence,  we have included this insight in the Discussion. In the future, we will endeavor to explore this point, and we look forward to contributing to this area.\n\nWe hope our response could address your query!\n\n>***Q3: Related work should be moved to the main text. Is AcU the same as the residual stream in your approach?***\n\nThanks for your suggestion. In response, we have moved the Related Work into the main text. Regarding the comparison with AcU, our method differs in that we consider both the activations and the residual stream. In contrast, AcU only considers the latter.\n\n>***Q4: How do you know the neuron semantics and how they are correct? How do you know they are monosemantic? Why dog and shark are wrong for L28.U4786?***\n\nWe apologize for any confusion caused. Here are our clarifications:\n\n1.\t**Neuron Semantics:** As mentioned in Line 240-243, the unembedding matrix and the second linear of the FFN are treated as the projection from neurons to the vocabulary words. In this case, the word with the highest probability can be regarded as the neuron semantics. We cannot guarantee that these semantics are entirely accurate, but we consider it acceptable to use them as one of the references to assist in analyzing neuron characteristics.\n\n2.\t**Polysemantic:** Many existing works have demonstrated the polysemantic nature of neurons. Our paper does not deny polysemy; instead, it aims to identify the most dominant of the multiple semantics. Thus, it does not conflict with polysemy but rather builds upon it. \n\n3.\t**L28.U4786:** If L28.U4786 is selected as the explanation for \"cat\" while its semantics are \"dog\" and \"shark\", the explanation method is inaccurate. This is the point conveyed in Table 1.\n\nFollowing your feedback, we have incorporated these clarifications into the appropriate sections to improve clarity and readability. Hope our response could address your concerns!\n\n*Once again, we are deeply appreciative of the time and expertise you have shared with us, and we are more than happy to add clarifications to address any additional recommendations and reviews from you!*\n\n*Best,*\n\n*Authors*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sM7PXTFJeD", "reviewer": "Reviewer_gq3m", "summary": "The paper describes a neuron attribution method for multimodal LLMs that the authors term \"NAM\". The method is broken down into two steps, a first step that uses image segmentation and a pretrained attribution algorithm, Diffusers-Interpret, to assign relevance scores to the final hidden state of the model. The second step involves attributing the final hidden state to neurons within the feed forward network of the base LLM.", "questions": "It is not clear how NAM performs in cases where one might want to perform attribution for semantic concepts that are not discrete objects which can be easily segmented out. A fundamental assumption here is that the semantics of the concept that one would like to explain have to be visually segmentable. I would like to see some discussion or clarification on whether the method is limited only to attribution of concrete, grounded concepts. For example, in the sentence \"A school of fish is chased by a shark.\", how would NAM attribute the word \"chased\"?\n\nA clarifying question: An underlying assumption here also seems to be that it is the FFN neurons that are primarily the ones that predominantly encode semantic knowledge. The authors give some citations in the introduction to support this assumption. Is there some basic intuition for why this cannot be encoded in the self-attention block instead?\n\nThe authors state that one of the primary motivations of this method over others is that their neuron attribution method does not require intensive backwards-and-forwards passes. How much of a problem is this actually? It would be nice to have a comparison of computational costs, especially since this seems to be something that the authors argue.\n\nI wonder if this method would be applicable to the more challenging problem of sound object identification and attribution in multimodal audio LLMs. For example, given an image of a spectrogram of a duck quacking and a person talking, could you use this technique (perhaps replacing some of the constitent pieces here) to identify the components of the spectrogram that correspond to each sound object? To be clear, I am not requesting an additional experiment here, but do want to hear the author's thoughts on the matter and whether there are any limitations to applying these ideas here.", "limitations": "See questions for potential concerns about limitations. No obvious negative societal impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The method seems to be effective at what it sets out to achieve, namely attributing model semantics to individual FFN neurons in the base model, and separating out the relative modality-specific semantic properties into I-Neurons and T-Neurons. The cross-sample invariance and semantic selectivity are compelling and convincing of the method's efficacy. The editing application gives an example of an especially relevant use case of the method.", "weaknesses": "The paper is at times difficult to read and needlessly obtuse, reflecting an unfortunate general trend in these sorts of papers. The paper would benefit from being less notationally dense and being less of a chore to read, especially since the underlying ideas are not nearly as complex as the notation suggests. Figure 2 is especially confusing and there are way too many moving parts given the relatively brief caption. There are some assumptions and potential limitations that may need to be justified (see questions for details).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer gq3m:\n\nThank you very much for your comments. We sincerely appreciate the time and effort you have dedicated to reviewing our work! Below, we meticulously provide responses to each of your comments and outline the modifications made to the manuscript based on your suggestions. Hope that our responses could address your concerns!\n\n>***W1: The paper would benefit from being less notationally dense.***\n\nThank you for your valuable feedback. Following your suggestions, we have revised our paper to improve clarity and readability. Specifically, we have streamlined the presentation by reducing notational density and simplifying definitions wherever possible (*e.g.*, Equation 4~6). \n\nMoving forward, we will be mindful of your suggestions and strive to maintain simplicity in both our descriptions and equations. We hope to contribute to curbing the trend of unnecessary complexity in academic writing.\n\nThank you again for your thoughtful comments! \n\n>***Q1: I would like to see some discussion on whether the method is limited to attribution of concrete concepts.***\n\nThank you for your insightful suggestion. To address your concern, we have expanded our discussion on the limitations of our method. Here are the key points:\n\n-\tFirstly, our method relies on external segmentation models which can only segment concrete concepts. Hence, we acknowledge that this dependency makes our method challenging to attribute abstract concepts.\n-\tDespite this limitation, we believe that **this dependency** (*i.e.*, leveraging established external models from mature fields) **can be beneficial**, especially during the early stages of developing a new domain. Our experiments support this approach, demonstrating the advantages of cross-domain collaboration. Moreover, as segmentation models continue to advance, the upper bound of our method's capabilities will improve.\n-\tFinally, we find it pertinent to mention our ongoing project that builds on this work and aims to attribute abstract concepts within multimodal LLMs. Specifically, it seeks to capture commonalities in the hidden layer when LLMs output abstract concepts, thereby circumventing the limitations of segmentation models. We plan to release it soon after this paper is accepted.\n\nFurthermore, following your suggestion, we have added a subsection in the revised version, which elaborates on these points. We hope this addition enhances the rigor of the paper and provides readers with valuable insights.\n\n>***Q2: The authors give citations to support that FFN predominantly encodes semantic knowledge. Is there some basic intuition for why this cannot be encoded in the self-attention block instead?***\n\nThank you for raising this important question. The prevailing view — or basic intuition — suggests that the self-attention block is primarily responsible for gathering global features, while the FFN extracts contextual information based on these features, concretizing complex features into stable semantic concepts. Hence, the FFN potentially stores information that requires abstraction and synthesis beyond the immediate context.\n\nFurthermore, from an experimental standpoint, several studies in recent years have supported this view. One of the most influential works is the ‘‘causal tracing’’ [1]. It employed neuron activation interventions and observed their impact on the output, achieving causal attribution of different blocks. We highly recommend reading this fascinating study for more detailed insights.\n\nFinally, if you feel it would be beneficial, we are prepared to **expand the Preliminary section in the revised version to include a more thorough discussion** of both the theoretical and experimental evidence supporting this hypothesis. \n\nHope our response could address your concerns!\n\n[1] Locating and Editing Factual Associations in GPT. NeurIPS 2022\n\n>***Q3: It would be nice to have a comparison of computational costs, especially since the author claims that their neuron attribution method does not require time-consuming backwards-and-forward passes.***\n\nThanks for your suggestion. Based on your feedback, we have added an analysis of computational costs in the revised version. The table below presents the average time (in seconds) required to attribute a concept. This demonstrates that avoiding the traditional backwards-and-forward pass significantly reduces computational overhead, effectively supporting our claims.\n\n|||||\n|:-:|:--:|:-:|:-:|\n|Method|Backwards|GILL|NExTGPT|\n|GraI|Yes|712.7|914.9|\n|GraD|Yes|279.3|299.7|\n|GraT|Yes|303.2|418.0|\n|NAM|No|**46.4**|**71.5**|\n|||||\n\nWe hope these additional experiments provide a clearer understanding of the efficiency gains offered by our approach. \nYour input has been invaluable in enhancing the rigor and clarity of our paper. Thank you once again for your insightful feedback!\n\n\n>***Q4: I wonder if this method would be applicable to audio modalities.***\n\nThank you for your insightful question. Our method can indeed be applied to other modalities such as audio, since the new attribution function we propose is modality-agnostic and can generalize across different types of data.\n\nTo better illustrate this point, we conducted experiments with audio outputs using NExT-GPT. Here are the results:\n|||||\n|:-:|:-:|:-:|:-:|\n|Method|Specificity|Relevance|Invariance|\n|CE|0.217|0.403|0.274|\n|GraI|0.264|0.500|0.337|\n|GraD|0.263|0.509|0.355|\n|AcU|0.349|0.552|0.396|\n|NAM|**0.361**|**0.596**|**0.428**|\n|||||\n\nDue to time constraints, we were only able to complete this single set. We hope for your understanding in this matter. Additional experiments are currently underway, and we plan to include them in the paper after acceptance.\n\n*Once again, we are deeply appreciative of the time and expertise you have shared with us. Your insightful suggestions have significantly enhanced our manuscript, and we are more than happy to add clarifications to address any additional recommendations and reviews from you!*\n\n*Best,*\n\n*Authors*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers:

We gratefully thank you for your valuable comments! We are truly encouraged by the reviewers' recognition of that our work has **addressed an important gap in MLLMs** (by all Reviewers), **provided interesting and instructive observations** (Reviewer VhwX and VfTi), and **conducted convincing and comprehensive experiments** (Reviewers kLxz and gq3m).

Here we meticulously give point-by-point responses to your comments, and further revise our manuscript following your suggestions. Hope that our responses could address all your concerns and meet the expectations of the conference committee.

Once again, we sincerely appreciate your time and effort in reviewing our paper. Your constructive criticism has been invaluable in refining our work, and **we are more than happy to add clarifications to address any additional recommendations and reviews from you**!

Best,

Authors

Meta-review: This paper presents NAM, a new technique for attributing neurons to specific semantic concepts in multimodal large language models (MLLMs). The authors show NAM outperforms GILL and NExTGPT models in various tests. NAM reveals neuron properties such as cross-modal invariance and semantic sensitivity, making it useful for targeted image editing tasks.

Strength: All reviewers point out that the paper addresses a crucial point in the interpretability of MLLMs by expanding neuron attribution techniques from text-only models to multimodal systems. NAM demonstrates its effectiveness in distinguishing between modality-specific neurons and surpasses baseline methods in multiple evaluation tasks, including cross-sample invariance. The editing application showcases a particularly relevant use case for the method.

Weakness: Several reviewers point out the limitations including 1) the method depends on external powerful segmentation models and attribution algorithms tailored for specific generation modules. This dependency introduces a potential source of bias and may limit the method's applicability and flexibility to scenarios when dealing with different or emerging MLLMs and generation technologies. 2) The presentation could be improved in many places.

After careful consideration, despite the limitations, given that many of the reviewers' raised concerns have been carefully addressed in authors' revision and comments, I'm leaning toward an accept.
Decision: Accept (poster)