Towards Neuron Attributions in Multi-Modal Large Language Models

Authors: junfeng  fang
zongze  bi
ruipeng  wang
houcheng  jiang
yuan  gao
kun  wang
an  zhang
jie  shi
xiang  wang
tat-seng  chua
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:19:44

Abstract: As Large Language Models (LLMs) demonstrate impressive capabilities, demystifying their internal mechanisms becomes increasingly vital. Neuron attribution, which attributes LLM outputs to specific neurons to reveal the semantic properties they learn, has emerged as a key interpretability approach. However, while neuron attribution has made significant progress in deciphering text-only LLMs, its application to Multimodal LLMs (MLLMs) remains less explored. To address this gap, we propose a novel Neuron Attribution method tailored for MLLMs, termed NAM. Specifically, NAM not only reveals the modality-specific semantic knowledge learned by neurons within MLLMs, but also highlights several intriguing properties of neurons, such as cross-modal invariance and semantic sensitivity. These properties collectively elucidate the inner workings mechanism of MLLMs, providing a deeper understanding of how MLLMs process and generate multi-modal content. Through theoretical analysis and empirical validation, we demonstrate the efficacy of NAM and the valuable insights it offers. Furthermore, leveraging NAM, we introduce a multi-modal knowledge editing paradigm, underscoring the practical significance of our approach for downstream applications of MLLMs.

Main paper: 
 introduction: 
 As Large Language Models (LLMs) demonstrate impressive capabilities [1, 2, 3, 4] , demystifying their internal mechanisms becomes increasingly vital, particularly in applications emphasizing fairness, trust, and ethical decision-making [5, 6, 7] . To interpret LLMs, "neuron attribution" stands out as a pivotal approach. This method involves attributing text outputs to individual model components (e.g., neurons and hidden layers) to reveal the knowledge and linguistic properties they learn [5, 8, 9, 10, 11, 12] . Such insights not only facilitate tasks like model editing and pruning [13, 14, 15] , but also offer a deeper understanding of how LLMs internalize knowledge. For instance, leading neuron-attribution studies [16, 17, 13, 14, 18, 19] suggest that this capability of internalization may predominantly originate from their Feedforward Neural Networks (FFNs). Recently, the rapid development of multi-modal large language models (MLLMs) [20, 21, 22, 23, 24] is sparking the interest of interpretability [18, 19] . Here we focus on the MLLMs that can perceive, generate texts and images simultaneously. Scrutinizing prior studies, we can summarize two common components beyond the LLM base: image encoding and generating modules. Specifically, the image encoding module projects the input image into the representation space of the base LLM; hereafter, the image generating module generates image outputs conditioned on the representations given by the base LLM. Take GILL [25] as an example. As shown in Figure 1 (a), it hires OPT [26] as the LLM base, CLIP Vit-L [27] with a cross-modal projector as the image encoding module, and Stable Diffusion [28] with another projector as the image generating module.While this expanded capacity endows GILL with versatility suitable for a variety of downstream tasks, it concurrently presents challenges for interpretation, particularly concerning neuron attributions. Specifically, we outline these challenges through three progressive points:• Source of Attribution: Semantic Noise. As shown in Figure 1 (b), for semantics like dog, current methods typically attribute the output to neurons directly in text-only LLMs [16, 5, 13] . However, in MLLMs, attributing the entire generated image to neurons directly might result in inaccuracies.As shown in Figure 1 (c), when GILL is tasked with drawing a dog, the generated image might contain other semantic elements like lawn, introducing noise and distorting attribution. • Process of Attribution: Inefficiency. Leading attribution methods typically rely on gradients [16, 19] or causal effects [13, 14] , requiring extensive forward/backward propagation processes, which are inherently time-consuming and storage-intensive. The added complexity of encoding and generation modules in MLLMs further exacerbates this challenge. • Results of Attribution: Intermingled Neurons. In text-only LLMs, attributing the concept dog involves identifying neurons crucial for outputting the word "dog", termed T-neurons. In contrast, MLLMs also require identifying neurons crucial for image generation, called I-neurons. As illustrated in Figure 1 (c), the distribution of T-neurons and I-neurons differs for the same concept, leading to conflicting results that complicate further analysis and applications.In sight of this, we introduce a new Neuron Attribution paradigm tailored for MLLMs, termed NAM, to reveal the modality-specific semantic properties learned by neurons within the FFN layers. Specifically, to address the above challenges, NAM undertakes the following efforts:• Image Segmentation for Semantic Noise: As shown in Figure 1 (d), NAM employs the image segmentation model to distinguish regions containing the target semantics from other noisy semantic areas, and attributes these regions to the neurons, rather than the entire image, to ensure accuracy. • Activation-based Scores for Inefficiency: Drawing inspiration from prior studies on neuron activations [18, 5] , NAM introduces a new attribution score that relies on neuron activations, eliminating the need for additional forward/backward propagation or gradient calculations. • Modality Decoupling for Intermingled Neurons: NAM assigns modality-specific attribution scores to neurons to prevent cross-modal disturbances during attributions. This paradigm facilitates the decoupling analysis of T-neurons and I-neurons, as depicted in Figure 1 (d) .Furthermore, based on the empirical results of NAM, we reveal several intriguing neuron properties within MLLMs. These properties collectively elucidate the inner workings of MLLMs, enhancing our Contribution of 𝑎 𝑘 𝑙 to "dog region":. 𝐑 𝐓 understanding of their capacity to process and generate multi-modal content. Interestingly, among these insights, a pivotal finding exhibits that when generating multi-model content for the same semantics (e.g., the word "dog" & an image of a dog), the crucial neurons (i.e., T-Neurons & I-Neurons) are typically not identical. This distinction underscores the complex nature of neurons within MLLMs, and highlights the necessity of neuron attribution across modalities. Additionally, by applying NAM to enhance image editing tasks, we further underscore the significance and potential applications of our NAM for MLLM community. 
preliminary: 
 Transformer-Based LLMs. An autoregressive transformer language model G : X → Y operates over the vocabulary V . It receives a token sequence x ∈ X and generates a probability distribution y = [y 1 , y 2 , ..., y |V | ] ∈ Y to predict the next token [29, 13] . Each token is represented as a series of representations h l ∈ R H in l-th layer, where h 0 is the embedding of the token in x. The model's final output, y = W u h L , is derived from the last representationh L = h L 1 , h L 2 , ..., h L H ⊤ in layerL using the unembedding matrix W u . Figure 2 (a) exhibits a visualization of how h l are computed within layer l. The representation in each layer results from the combination of the global attention a l , the local MLP output m l , and the representation h l-1 from the previous layer. Formally,EQUATIONwhere W in ∈ R d×H and W out ∈ R H×d are the first and second linear layer in FFN with the dimensionality of the FFN's intermediate layer d; σ and γ are rectifying and normalizing nonlinearity.For further background on transformers, we refer to [29] . Additionally, focusing on the the k-th neuron u l k in the l-th FFN layers, we simplify the definition by considering its activation a l k as:EQUATIONMLLMs. Here, we take GILL as an example to illustrate a common paradigm of MLLMs. As shown in Figure 2 (a), GILL incorporates the following modifications on the above text-only LLMs: (1) In addition to the textual prompt, the input token sequence x also includes the encoding of the input image produced by the image encoding module. (2) The representation h L of the last hidden layer is utilized as input to the image generation module, facilitating conditional image generation. 
method: 
 This section delineates the implementation of NAM within MLLMs. Specifically, Section 3.1 and 3.2 introduce the attribution process for image and text outputs, respectively; In Section 3.3, we explore a significant application of NAM, i.e., editing images generated by MLLMs. To illustrate these processes, we utilize GILL [25] , a representative MLLM capable of image generation. For a detailed introduction to GILL, please refer to Appendix B.We first detail how to attribute the output images to the specific neurons within FFN layers. Retrospecting the challenges highlighted in Introduction, using the MLLM to generate images of a specific concept (e.g., dog) often results in outputs that include extraneous, noisy elements (e.g., lawn). To mitigate the negative effects of semantic noise on neuron attribution, we propose a two-step approach to extract I-Neurons. Specifically, (1) the first step focuses on attributing the output of the image generation module (i.e., images) to the input of the image generation module (i.e., last representation h L ), and (2) the second step endeavors to attribute the input of the image generation module to the specific neurons. Next, we provide detailed descriptions of these two steps.The purpose of this step is to attribute the image to h L ∈ R H . That is, to identify the contribution of each element in h L for image generation. We define these contribution scores as R ∈ R H . As shown in Figure 2 (b), NAM acquires R by sequentially executing the following processes:• Given an image generated by prompting MLLM with the semantics dog, NAM first employs the leading segmentation model, EVA02 [30] , to identify regions specifically related to dog. This is crucial for minimizing interference from extraneous semantics, such as lawn in the background; • Subsequently, NAM utilizes the advanced attribution algorithm of the diffusion model, Diffuser-Interpreter [31] , to access the relevance of each dimension in the input of the image generation module to the dog region in the generated image. • Ultimately, by normalizing these relevance scores to (0, 1), we obtain the importance scoresR = [r 1 , r 2 , ..., r H ] ⊤ of h L = [h 1 , h 2 , ..., h H ] ⊤ w.r.t the target semantics in the output image.Due to the established applications of EVA02 and Diffuser-Interpret, we provide detailed introductions in Appendix B. Furthermore, it is worth mentioning that NAM can be transferred to any other modality by utilizing the (1) semantic segmentation algorithms and (2) attribution algorithms of generation modules tailored for other modalities (e.g., audio and video). After obtaining R by these advanced modality-specific algorithms, the subsequent attribution steps are universal across all transformerbased MLLMs.From Representation h L to Neuron u l kThis step involves attributing the representation h L in the last layer to the specific neuron u l k within the FFNs of the base LLM. To this end, NAM aims to trace each neuron's contribution to h L , and identify the neurons with significant contributions as I-Neurons for the semantic of interest. Hence, a fair and efficient contribution scoring method is crucial.Direct Contributions through Residual Stream. Current methods for scoring contributions often rely on gradients, such as the product of gradients and activations [19] or the integration of gradients [16] . However, these methods are computationally intensive, particularly for large-scale models with extensive parameters. In sight of this, drawing inspiration from prior studies on neuron activation [18, 19] , we first introduce a new attribution score that relies on the neuron activation a l k . Specifically, we first try to disassemble and deduce the generation procedure of h L by expanding h L as follows:EQUATIONwhere (W l out ) k ∈ R H is the k-th column of the weight matrix W l out corresponding to the index of neuron u l k , as shown in Figure 2 (b). Note that the first term of Equation (3) reflects the direct contribution of the neuron u l k to the last representation h L , i.e., the contribution through the residual stream [19] of the base LLM.Hence, we employ a l k (W l out ) k as the indicator for the neuron u l k 's contribution to h L . Furthermore, by integrating R in Section 3.1.1, we can establish a complete attribution pipeline (i.e., targeted semantic region in generated image ⇒ representation h L ⇒ neuron u l k ). Recall that R has assigned the contribution for each dimension in h L , the neuron u l k 's contribution s l k can be defined as a l k (W l out ) k weighted by elements in R. That is, s l k = a l k (W l out ) k R ⊤ , as illustrated in Figure 2 (b). Contribution Score Considering Indirect Influence. While s l k quantifies the neuron u l k 's direct contribution to h L through the residual stream, it does not account for all influential factors. Specifically, it overlooks the indirect contributions that neurons make through the attention mechanisms within subsequent FFN layers. Supporting evidence exhibited in Appendix B verifies that this oversight might lead to a bias. To address this issue, and in line with our objective to eschew complex computations like gradient, we implement a heuristic optimization of the current indicator s l k . Specifically, we employ the relative magnitude of neuron activation as another indicator to identify neurons that may have a significant indirect contribution to h L -contribution which s l k might overlook. Furthermore, considering the computation of s l k already incorporates a l k , the contributions reflected by these two indicators may overlap. To prevent redundancy from summing or multiplying these two metrics, we utilize the maximum function in our final score design:EQUATIONwhere the normalization operation ensures fair competition between s l k and a l k . Note that our experiments in Section 4 have shown that this combined scoring approach is more effective than utilizing s l k or a l k alone, verifying their complementary nature. By computing contributions following Equation (4) for various semantics across all layers, NAM identifies neurons that consistently demonstrate the highest contributions to the generated images. These neurons, distinguished by their significant roles, are designated as I-neurons responsible for targeted semantics in MLLMs.We then focus on how to acquire the neuron's contribution to the text outputs. Similar to the derivation process of contribution score for image output, here we first focus on the contribution for output y through the residual stream. Specifically, we expand h L as follows:EQUATIONAccording to a commonly used assumption for analyzing the internal mechanisms of LLMs, representations at any layer within the language models can be transformed into a distribution over the token vocabulary V using the output embeddings [18, 19, 32, 16, 13, 14] . Hence, W u W l out ∈ R |V |×H can be considered as the new unembedding matrix at the end of the residual stream, and A l contributes to the model output distribution y over the vocabulary through W u W l out A l , as shown in Figure 2 (c ). Refined Contribution of Individual Neuron. We further disassemble Equation ( 5) to refine individual neuron u l k 's contribution to the output word. Specifically, denoting p as the index of the word "dog" in the vocabulary V , we have:EQUATIONwhere (W u ) p ∈ R 1×d is the p-th row of W u . According to Equation ( 6), the neuron u l k 's contribution c l k,p to the p-th word "dog" on vocabulary can be obtained by c l k,p = a l k (W u ) p (W l out ) k , as illustrated in Figure 2 (c ). Furthermore, we would like to encourage the semantic specificity of the identified crucial neurons -that is, only preserving a single semantic concept with the maximum contribution, while discarding other semantics. Formally, for the semantics of p-th word on vocabulary, the neuron u l k 's contribution s l k can be defined as:EQUATION)By substituting s l k to Equation ( 4), we can acquire the final attribution score of neuron u l k for text output. The neurons that consistently exhibit the highest contributions are then designated as T-neurons.Knowledge editing methods based on neuron attribution have already been explored for text outputs [13, 14, 33, 34] . Here, we focus on how to leverage attribution results to facilitate knowledge editing of images. This objective requires replacing some semantics (e.g., dog) with another semantics (e.g., cat). To this end, we leverage the I-Neurons identified by NAM for image editing through a straightforward, training-free approach. Specifically, we first construct the set of I-Neurons, U, for the semantics like dog. Then, we collect the positions (l, k) of the neurons u l k in U, and construct the set of these position indices, I. For (l, k) ∈ I, we add a perturbationEQUATION)where h L and ĥL is the last representation of the base LLM when generating the image of dog and cat, respectively; τ serves as a trade-off parameter. In Equation ( 8), the first term aims to facilitate a shift in the image generation module's input from h L to ĥL , while the second term is the ℓ 2 norm constraint for preventing drastic edits that might affect images containing other semantics. According to this method, NAM can be utilized to enable simple and efficient images editing, underscoring the significance and potential applications of the NAM for MLLMs2 . 
experiment: 
 In this section, we aim to validate the effectiveness of NAM from three aspects:• What is the distribution of T/I-Neurons identified by NAM?• What properties do the T/I-neurons identified by NAM have? How to verify that the T/I-neurons identified by NAM are more critical compared to the neurons identified by baseline methods? • Can the T/I-Neurons identified by NAM facilitate the image editing within MLLMs?Target Models & Datasets. Our research focuses on GILL [25] and NExT-GPT [35] , two representative MLLMs with the capability of image generation. All experiments are conducted on the Common Objects in Context (COCO) [36] , a large-scale object detection, segmentation, and captioning dataset including 80 object categories and five captions per image to conduct our experiments. Due to space limitations, we only present the experimental results on GILL in this section. The remains and the detailed implementations, such as the setting of hyper-parameters, can be found in Appendix B.Baselines. We collect five advanced neuron attribution methods across three categories (gradient-, activation-, and causality-based attribution). Specifically, their abbreviations and the attribution scores they employ are: (1) AcT: neuron activation [5] ; (2) AcU: The product of activation and the unembedding matrix, focused on the dimension corresponding to the output word [18] ; (3) GraD: The gradient of the output dimension corresponding to the output word w.r.t activation. (4) GraT: The product of the gradients and activation [19] ; (5) GraI: The integral of the gradients [16] ; (6) CE: The causal effect of activation on outputs [13, 14] . See detailed description in Appendix B.1. For NAM and baselines stand and their role in rich literature, please refer to Appendix A (i.e., Related Work).We first focus on the distribution of T/I-neurons identified by NAM. Specifically, we randomly select 1000 images from the COCO dataset. Then, we feed each image to GILL individually and instruct GILL to generate a similar image. The distributions of T/I-neurons are exhibited in Figures 3 (a) and Table 1 : The semantics of T/I-neurons with the highest attribution scores identified by different attribution methods, when the output of MLLMs containing the targeted semantics. (Ll,Uk) denotes the k-th neuron at layer l. For each method, we report semantics with top-4 probabilities. Observation 1: Within MLLMs, the crucial neurons for the text and image output containing specific semantics predominantly occur in the middle and high layers of the base LLM. Note that this observation is consistent with the previous works involving neuron attributions within LLMs [5, 13, 18] . Additionally, the similar distribution of T and I neurons suggests that the formation time of semantic concepts across different modalities in MLLMs may be consistent.Observation 2: Figure 3 (c ) reveals a partial overlap between T and I neurons. However, it also pronounces distinctions between them. This finding substantiates the claim presented in the Introduction of this paper: even for the same semantics, critical neurons are modality-specific within MLLMs.We then explore the following properties of T/I-neurons through comprehensive quantitative and qualitative experiments: Semantic Relevance, Cross-Sample Invariance, and Concept Specificity. Additionally, by comparing these properties with those of neurons identified by various baselines, we validate the effectiveness of our NAM. Note that we only present the results of the best-performing baseline for each class, and remains are shown in Appendix B.Following previous studies [19, 18] , we treat the unembedding matrix and the second linear layer matrix in the FFN as a projection from neuron activation to the probability distributions of the vocabulary. Based on this, words with the average highest probability can be regarded as the relevant semantics of neurons, as shown in Table 1 . Note that the results of AcU are not exhibited since its attribution score is exactly this probability. Furthermore, to explore the semantic relevance of the neurons quantitatively, we calculate the consistency between the semantics of neurons and the input/output images employing CLIPScore [37] . BERTScore [38] , MoverScore [39] , and BLEURT [40] are also employed to quantify their consistency with (1) input image's caption provided by the dataset and (2) output image's caption given by GPT [1] . Table 2 exhibits the average quantified results. According to Table 1 and 2, we have the following observation:Observation 3: The semantics of T/I-neurons identified by NAM align more closely with the input/output images and their captions, while the other attribution methods typically identify the neurons that are hardly correlated with the targeted semantics. The quantitative results share a similar tendency, confirming the high semantic relevance of T/I-neurons and the effectiveness of our NAM.For different text/image outputs containing the same semantics, the T/I-neurons identified by the attribution methods shall be consistent. To quantify this consistency, we instruct GILL to describe and generate images for the same semantics ten times, and collect the set of T/I-neurons each time.We then calculate the proportion of neurons that appeared in all ten sets as the quantification of cross-sample invariance. The average invariance Figure 4 (a) presents the average invariance for different concepts. Specifically, Observation 4: NAM outperforms all baselines by an average of 16.83% w.r.t cross-sample invariance across all semantics. This demonstrates that NAM extracts the critical neurons for the targeted semantics across samples, effectively filtering out the neurons sensitive to sample-specific noise.Neurons that are crucial for specific semantics should not be indiscriminately crucial across others. Therefore, we study the neuron's specificity in this part. We identify the Top-500 T/I-neurons for the specific semantics. Then, we show (1) the number of neurons that are crucial for specific semantics solely and (2) the average number of neurons whose probability of being crucial to other semantics is lower than κ ∈ {10%, 15%, . . . , 45%} in Figure 4 (b). The results of the three bestperforming methods are exhibited here. These results highlight that:Observation 5: The T/I-neurons identified by our NAM are specialized and not commonly sensitive across different semantics, verifying their specificity across the semantics.Lastly, we aim to edit the images generated by MLLMs through perturbing I-neurons identified by NAM, as outlined in Section 3.3. Table 3 exhibits the pre-and post-editing semantics, the selected images for collecting h L and ĥL , and the magnitude of the perturbations. Furthermore, for fair comparisons, we also perturb I-neurons identified by baselines to achieve similar editing results.According to Table 3 we can find that:Observation 6: NAM-enhanced editing methods can not only replace the original semantics with the target semantics precisely within the outputs of MLLMs, but also necessitate minimal perturbations. Specifically, the perturbation it added is 40.2% less than the baselines on average, and nearly 15% less than the best baseline, underscoring its significance and potential applications for MLLMs.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide the code necessary for replicating the studies described in this paper via an anonymous link, and we detail the experimental setup for the replication in the article itself (See Abstraction Section). Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
limitations & future work: 
 This study provides new insights into interpreting MLLMs, enhancing the understanding of their inner working mechanize. However, while our experiments thoroughly investigated the neuron properties within GILL and NExTGPT, they did not extend to a broader range of models. Additionally, although the proposed attribution method can be transferred to any other modality, as demonstrated in Section 3.1.1, our experiments focused on text and image outputs solely. Looking forward, we plan to incorporate more MLLMs and modalities into our research, and streamline our attribution method to eliminate the reliance on external interpreters. By expanding the scope of our study and refining our method, we aim to uncover more valuable insights that will benefit the MLLM community.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: For the datasets disclosed in the article, we have provided information regarding their sources and origins (See Experimental Section). Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 We propose NAM, a novel neuron attribution method tailored for MLLMs. Specifically, NAM is tailored for multi-modal attribution, revealing the modality-specific semantic properties learned by neurons within the FFN layers.To address the challenges of extending attribution methods from text-only LLMs to MLLMs, NAM first employs a leading image segmentation model to remove the noisy semantics, then proposes a new attribution score to eliminate the need for additional forward/backward propagation or gradient calculations. Based on NAM, we highlights several intriguing properties of neurons, elucidating the inner workings mechanism of MLLMs. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Multi-modal Large Language Models. In recent years, MLLMs have made significant progress, continually pushing the boundaries of performance in various downstream tasks [41] . The launch of models like GPT-4 (Visual) and Gemini, with their impressive multimodal (MM) understanding and generation capabilities, has sparked intense research interest in MM-LLMs [20] . In terms of understanding multimodal content, preliminary research often utilizes multimodal encoders like the ViT [42] and CLIP [27] ViT to capture representations, and projectors like Q-Former [43, 44] and P-Former [45] to align these representations with the embedding space of foundational LLMs. This approach covers tasks such as image-text understanding, with representative models including BLIP-2 [21] , MiniGPT-4 [23] , LLAVA [22] , and OpenFlamingo [24] .Recently, the capabilities of MM-LLMs have expanded to support specific modal outputs. These methods align certain embeddings from the foundational LLM with the input space of a well-trained multimodal generator through another projector. This extension includes tasks with image-text outputs as demonstrated by models like GILL [25] , Kosmos-2 [46] , Emu [47] , NExT-GPT [35] and MiniGPT-5 [48] , which is the focus of this paper. Recent research endeavors have focused on mimicking human-like any-to-any modality conversion, shedding light on the path to artificial general intelligence [20] .Interpretability of Pre-trained Transformers. Demystifying the internal mechanisms of LLMs becomes increasingly vital, particularly in applications emphasizing fairness, trust, and ethical decision-making [49, 50, 51, 52, 53, 54, 55] . In recent years, many works have focused on explaining pre-trained transformers. For instance, [32] regards the FFN as unnormalized Key-Value Memories; [56] presents a conceptual framework where all parameters are interpreted by projecting them into the embedding space; [9] analyzes the FFN updates in the vocabulary space, showing that each update can be decomposed to sub-updates; [17] studies how the model aggregates information about the subject and relation to predict the correct attribute; while [11] localizes the weights and mechanisms used by a language model to memorize and recite entire paragraphs of its training data.Neuron Attribution. Neuron attribution aims to reveal the black box of pre-trained transformers by answering the following questions [7, 57, 6] : (1) What concepts are learned within neurons of the network? (2) Are there neurons that specialize in learning particular concepts? (3) How localized/distributed and redundantly is the knowledge preserved within neurons of the network? To achieve these goals, many recent works have focused on exploring the properties of neurons in LLMs. Specifically, [16] present preliminary studies on how factual knowledge is stored in LLMs by introducing the concept of knowledge neurons; [5] finds the special neurons whose activations on soft prompts are highly predictive of the task labels of inputs, and dub them skill neurons.For MLLMs, [18] employs the neuron contribution within the residual stream to identify multi-modal neurons in Transformer-based multi-modal LLMs, while [19] claims that image prompts cast into the transformer embedding space do not encode interpretable semantics, and translation between modalities occurs inside the transformer. Additionally, despite the aforementioned methods are all based on activation and gradients to identify target neurons, some recent works focused on knowledge editing have discovered an alternative approach [15, 13] . This involves using causal effect methods to identify key neurons by perturbing neurons and observing changes in the output [14] .For the detailed contribution scores these works employed and their relationship with our NAM, please see Appendix B.1.Baseline. Here we first detail the contribution score of the baseline methods used. We will use the formulas and symbols from the paper to provide a formal explanation. Specifically, for the p-th word in Vocabulary V and the k-th neuron u l k in layer l, we have: • AcT takes the neuron activation as the contribution score following:EQUATION• AcU takes the product of the neuron activation and the unembedding matrix as the contribution score following:EQUATION) • GraD takes the gradient of the output dimension corresponding to the output word w.r.t activation as the contribution score following:EQUATION• GraT takes the product of the gradients and activation as the contribution score following:EQUATION)• GraI takes the product of the integral of the gradients and the activation as the contribution score following:EQUATIONwhere y ′ p is the p-th dimension of the output y when we fix the activation of neuron u l k as α • a k l . Note that perturbing and performing forward propagation for each neuron multiple times is timeconsuming. Therefore, we add the same coefficient α to multiple neurons simultaneously and calculate their integrals to obtain their respective contribution scores.• CE takes the causal effect of activation on outputs, quantified by the output change when activation is perturbed, as the contribution score following:EQUATIONwhere y ′′ p is the p-th dimension of the output y when we add the activation of neuron u l k with a Gaussian Noise. We set the mean of the Gaussian noise to 0 and the variance to the total variance of the neuron activation values in the respective layer. Similar to GraI, for each individual neuron, performing forward propagation for each neuron multiple times is costly. Therefore, we add perturbations to multiple neurons simultaneously and evenly distribute the resulting output changes among them as their respective contribution scores. Furthermore, it is important to note that the above baselines all attribute text outputs to neurons. Therefore, we have adapted all these baselines to image outputs for comprehensive validation. For example, the attribution score of the baseline Grad for text output is the gradient of activation when the output is the word "dog". When transferred to adapt the image output, its attribution score is the gradient of activation when the output is an image of a dog. Specifically, since the last representation h is the input of the image generation module, we have:EQUATION)Hyperparameter Configuration. For our experiments, we sourced the training and testing data for the COCO dataset directly from its website 3 . Similarly, we obtained the source code for the target models GILL 4 and NExT-GPT 5 , the image segmentation model EVA02 6 , and the attribution algorithm Diffuser Interpreter7 used for stable diffusion from the links cited in their respective publications All hyperparameter settings, such as the division of training and testing datasets, learning rate, and optimizer, are consistent with the original configurations of the above link unless otherwise stated. Additionally, it is important to note that, unless explicitly mentioned, the samples used in the experiments were 500 images randomly selected from the COCO dataset.Furthermore, we use Quadro RTX6000 GPUs with 24GB of memory as a representative example of consumer-level GPUs; 40GB A100s and 80GB H100s to provide datacenter-level benchmarks. Ablation Study. Our attribution score design primarily consists of two components: activation and its mapped values on the target semantic dimension in the vocabulary. We aim to leverage the complementary effects of both components, using them as indicators of indirect and direct contributions, respectively. The results of baseline methods AcT and AcU can be approximated as the attribution effects when using each component separately. Therefore, comparing our method with AcT and AcU can also be considered as an ablation study. This comparison also reflects that using only the mapped values (AcU) can introduce certain biases, leading to poorer performance across various metrics.External Model & Algorithm. Then, we introduce the semantic segmentation model EVA02 and attribution algorithm for the stable diffusion model Diffuser Interpret. Specifically, EVA02 [30] comprises a series of robustly optimized plain Vision Transformers (ViTs) of moderate model sizes, featuring transferable bidirectional visual representations learned from a powerful CLIP vision encoder via masked image modeling (MIM) pre-training. Compared to current leading vision models with billions of parameters, the EVA-02 variants necessitate significantly fewer computational resources, enabling a more in-depth exploration of often-overlooked aspects. Furthermore, Diffuser Interpret [31] attributes the pixel values of the generated image to the input embedding of the stable diffusion using gradient information. We also apply this strategy to the attribution of the projector used by GILL.Results on Semantic Relevance. To better illustrate the comparison between our attribution method and the baseline attribution methods, we present additional qualitative experimental results to provide a more intuitive comparison, as shown in Table 4 . The effects demonstrated by these methods are similar to those shown in the experimental results in the main text, indicating that the neurons identified by our NAM method are semantically closer to the target semantics.Results on NExTGPT. In the main body of the text, we have completed testing the effectiveness of various neuron attribution algorithms with GILL as the target model. In this section, we expand our experiments to include NExTGPT as the target model. Below, we present the key neurons identified in NExTGPT and their semantic correlations with images and captions. The results can be found in Table 5 .Consistency Between Neuron's Semantics and Caption Given by GPT. The main text only presents the average consistency of the T/I-neurons identified by various attribution methods with the captions of the input/output images. To refine our comparison, we provide the specific consistency scores w.r.t the captions generated by GPT for the output images. The detailed results are shown in Table 6 and 7 . These results also demonstrate the highest consistency scores of our attribution method compared to the baselines.In this paper, we present a novel neuron attribution method NAM to interpret the MLLMs. Based on NAM, we reveal several intriguing neuron properties within MLLMs. These properties collectively elucidate the inner workings of neurons within MLLMs, enhancing our understanding of their capacity to process and generate multi-modal content. This approach can contribute to a wide range of applications of MLLMs, boosting the MLLMs across various downstream tasks such as knowledge editing [15, 13, 14] . We believe that the neuron properties drawn from NAM can shed light for future research on MLLM community, and inspire further exploration into understanding neurons within other pre-trained transformers. • All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.The extension of NAM to broader scenarios will be detailed in Appendix C.COCO: http://images.cocodataset.orgGILL: https://github.com/kohjingyu/gillNExT-GPT: https://github.com/NExT-GPTEVA-02: https://github.com/baaivision/EVA/Diffuser-Interpreter: https://github.com/JoaoLages/diffusers-interpret 


Decision: This paper presents NAM, a new technique for attributing neurons to specific semantic concepts in multimodal large language models (MLLMs). The authors show NAM outperforms GILL and NExTGPT models in various tests. NAM reveals neuron properties such as cross-modal invariance and semantic sensitivity, making it useful for targeted image editing tasks.

Strength: All reviewers point out that the paper addresses a crucial point in the interpretability of MLLMs by expanding neuron attribution techniques from text-only models to multimodal systems. NAM demonstrates its effectiveness in distinguishing between modality-specific neurons and surpasses baseline methods in multiple evaluation tasks, including cross-sample invariance. The editing application showcases a particularly relevant use case for the method.

Weakness: Several reviewers point out the limitations including 1) the method depends on external powerful segmentation models and attribution algorithms tailored for specific generation modules. This dependency introduces a potential source of bias and may limit the method's applicability and flexibility to scenarios when dealing with different or emerging MLLMs and generation technologies. 2) The presentation could be improved in many places.

After careful consideration, despite the limitations, given that many of the reviewers' raised concerns have been carefully addressed in authors' revision and comments, I'm leaning toward an accept.

Reviews: review: paper_summary: paper_summary: Summary:
This paper introduces NAM (Neuron Attribution Method), a novel approach for attributing neurons to specific semantic concepts in multimodal large language models (MLLMs). The key contributions are: (1) A method to identify modality-specific neurons (text or image) that are crucial for particular semantic concepts. (2) Analysis of neuron properties like cross-modal invariance and semantic sensitivity. (3) A framework for multimodal knowledge editing based on the identified neurons.

The authors evaluate NAM on GILL and NExTGPT models, comparing it to several baseline attribution methods. They demonstrate NAM's effectiveness in identifying semantically relevant neurons, its cross-sample invariance, and its utility for targeted image editing tasks.
strength_weakness: strength_weakness: - While the authors acknowledge this limitation, the evaluation is conducted on only two MLLMs (GILL and NExTGPT). Testing on a broader range of models would strengthen the generalizability claims.
- The experiments focus solely on text and image modalities. Exploring other modalities (e.g., audio, video) would provide a more comprehensive evaluation of NAM's capabilities.
- The method depends on external image segmentation models to remove noisy semantics. This introduces a potential source of bias and may limit the method's applicability to scenarios where such models are unavailable or unsuitable.
- The paper would benefit from ablation studies to isolate the impact of different components of the NAM algorithm, particularly the novel attribution score calculation.
questions: questions: None.
limitations: limitations: Yes.
paper_summary: paper_summary: The work introduces a novel Neuron Attribution Method (NAM) tailored for MLLM. The NAM approach aims to reveal the modality-specific semantic knowledge learned by neurons within MLLMs, addressing the interpretability challenges posed by these models. The method highlights neuron properties such as cross-modal invariance and semantic sensitivity, offering insights into the inner workings of MLLMs.
strength_weakness: strength_weakness: The NAM method relies on advanced segmentation models (like EVA02) and attribution algorithms (like Diffuser-Interpreter) tailored for specific generation modules. This dependency could limit the method's applicability and flexibility, especially when dealing with different or emerging MLLMs and generation technologies.
questions: questions: NA
limitations: limitations: NA
paper_summary: paper_summary: This paper proposes a method to attribute the multimodal output to neurons that are influential in the generation process. The approach considers image-text multimodal LLM where 1) the base LLM's internal representation is directly used to generate text, and 2) also passed to an image generation model. These this two tiered model architecture, the proposed approach breaks down the problem into attribution for the base model and attribution for the image generation model, which can be combined to cover all neurons in the considered architecture. The proposed approach is evaluated with baselines and shown to outperform in several tests including cross sample invariance.
strength_weakness: strength_weakness: - W1: It is unclear how to get to the Eq 1. From the GILL paper, it is using a regular OPT/GPT architecture. How did you get the linear addition of the attention vector instead of the typical multihead self-attention? Also, normalizations should be applied after a self-attention block/linear layer is applied, making linear relation from an intermediate layer to the last layer / penultimate embedding not possible.
- W2. In the semantic relevance evaluation, relevant words are extracted using the same signal paths considered in the paper, while the signals can flow other ways for the baselines. Therefore, it is unclear if we can consider these words represent the semantics of the neurons identified by other methods. Moreover, the artificial neurons are known to be multisemantic, and neuron groups form a certain semantic instead of individual neurons in isolation. Thus, the metric is deemed to emphasize what the proposed approach is doing by leveraging its definition but might not be suitable for evaluating and comparing with the baselines.
- W3: The presentation can be improved in many places (e.g., Line 110, 209-210, 217-218, 260, the definition of specificity).
questions: questions: - Q1: Are the bars in Fig 3(c) stacked or overlayed? Is the green region T \cup I - T \cap I?
- Q2: How do we know the activations of the neurons are signals. Why not lack of activations?
- Q3: Related work should be moved to the main text. What's the difference to AcU [2]? Is it the same as the activations through the residual stream in your approach (Sec 3.1.2.)?
- Q4: How do you know the neuron semantics and how they are correct? How do you know they are monosemantic? Why "dog", "shark" are wrong for L28.U4786?

Overall, I find the paper has quite a bit of room to improve. However, I think this paper attacks an important problem, and shows promising results. Although not all evaluation methods are convincing, this is common for a growing subfield. At least some of the evaluation results can be useful to particular use cases, and I find this work worth dissemination. I wouldn't be upset if someone want this paper to be more refined and required another full review round, given there are some obvious presentation issues.
limitations: limitations: They mentioned the limitations.
paper_summary: paper_summary: The paper describes a neuron attribution method for multimodal LLMs that the authors term "NAM". The method is broken down into two steps, a first step that uses image segmentation and a pretrained attribution algorithm, Diffusers-Interpret, to assign relevance scores to the final hidden state of the model. The second step involves attributing the final hidden state to neurons within the feed forward network of the base LLM.
strength_weakness: strength_weakness: The paper is at times difficult to read and needlessly obtuse, reflecting an unfortunate general trend in these sorts of papers. The paper would benefit from being less notationally dense and being less of a chore to read, especially since the underlying ideas are not nearly as complex as the notation suggests. Figure 2 is especially confusing and there are way too many moving parts given the relatively brief caption. There are some assumptions and potential limitations that may need to be justified (see questions for details).
questions: questions: It is not clear how NAM performs in cases where one might want to perform attribution for semantic concepts that are not discrete objects which can be easily segmented out. A fundamental assumption here is that the semantics of the concept that one would like to explain have to be visually segmentable. I would like to see some discussion or clarification on whether the method is limited only to attribution of concrete, grounded concepts. For example, in the sentence "A school of fish is chased by a shark.", how would NAM attribute the word "chased"?

A clarifying question: An underlying assumption here also seems to be that it is the FFN neurons that are primarily the ones that predominantly encode semantic knowledge. The authors give some citations in the introduction to support this assumption. Is there some basic intuition for why this cannot be encoded in the self-attention block instead?

The authors state that one of the primary motivations of this method over others is that their neuron attribution method does not require intensive backwards-and-forwards passes. How much of a problem is this actually? It would be nice to have a comparison of computational costs, especially since this seems to be something that the authors argue.

I wonder if this method would be applicable to the more challenging problem of sound object identification and attribution in multimodal audio LLMs. For example, given an image of a spectrogram of a duck quacking and a person talking, could you use this technique (perhaps replacing some of the constitent pieces here) to identify the components of the spectrogram that correspond to each sound object? To be clear, I am not requesting an additional experiment here, but do want to hear the author's thoughts on the matter and whether there are any limitations to applying these ideas here.
limitations: limitations: See questions for potential concerns about limitations. No obvious negative societal impact.

Comments: title: Gratitude and Seeking for Final Review Consideration
Gratitude and Seeking for Final Review Consideration
Gratitude and Seeking for Final Review Consideration
Heartfelt Thanks for Your Encouraging Feedback
Gratitude to Reviewer VfTi and Respectful Seeking for Re-evaluation
Thank you for your support!
Additional Clarification on Reviewer VfTi’s Concern
Thanks
Gratitude to Reviewer gq3m for Feedback
Revised Score
comment: Dear Reviewer kLxz,

We are extremely grateful for the insightful comments you provided and for the recognition you have shown toward our work. Your suggestion to include additional experiments has significantly strengthened the rigor of our approach, making our contribution even more valuable to the community.

**As our current score still hovering around the borderline, we humbly ask if you might consider the possibility of raising your score. Your support at this stage is immensely important to us!** If you believe there are still areas in our paper that could be further improved, we would be more than happy to engage in any additional discussion to address any remaining concerns, particularly as the rebuttal period is about to conclude in just a few hours.

We are fully committed to advancing the field of explainable LLMs and contributing meaningfully to the community. Your feedback and support are invaluable to us in achieving this goal.

Thank you once again for your thoughtful engagement.

Warm regards,

Authors
Dear Reviewer VfTi,

We are extremely grateful for the insightful comments and the recognition you have shown toward our work. Your suggestion for optimizing the paper's presentation and experiments has significantly strengthened the rigor of our approach, making our contribution even more valuable to the community.

**As our current score still hovering around the borderline, we humbly ask if you might consider the possibility of raising your score. Your support at this stage is immensely important to us!** If you believe there are still areas in our paper that could be further improved, we would be more than happy to engage in any additional discussion to address any remaining concerns, particularly as the rebuttal period is about to conclude in just a few hours.

We are fully committed to advancing the field of explainable LLMs and contributing meaningfully to the community. Your feedback and support are invaluable to us in achieving this goal.

Thank you once again for your thoughtful engagement.

Warm regards,

Authors
Dear Reviewer VhwX,

We are extremely grateful for the insightful comments you provided and for the recognition you have shown toward our work. Your suggestion to include additional discussion on the introduction of external modules has significantly strengthened the rigor of our approach, making our contribution even more valuable to the community.

**As our current score still hovering around the borderline, we humbly ask if you might consider the possibility of raising your score. Your support at this stage is immensely important to us!** If you believe there are still areas in our paper that could be further improved, we would be more than happy to engage in any additional discussion to address any remaining concerns, particularly as the rebuttal period is about to conclude in just a few hours.

We are fully committed to advancing the field of explainable LLMs and contributing meaningfully to the community. Your feedback and support are invaluable to us in achieving this goal.

Thank you once again for your thoughtful engagement.

Warm regards,

Authors
Dear Reviewer kLxz,

Thank you very much for your timely feedback and for the positive adjustment to your score. We are truly delighted to see that our responses have successfully addressed your concerns. Your encouragement is incredibly valuable to our team.

As you are aware, the field of explainable LLMs is still in its infancy. **Your insightful comments and suggestions throughout the review process have not only improved our work but also motivated us deeply. We are committed to continuing our research in this emerging area, striving to contribute further to its development.**

Thank you once again for your essential role in refining our work and for your inspiring support.

Warm regards,

Authors
Thanks for your response and additional experiments. This addresses some of my concerns and I have raised my score.
Dear Reviewer VfTi,

We would like to express our sincere gratitude for the valuable insights and feedback you have provided throughout the review process. As the discussion period is drawing to a close in about one day, we would really appreciate it if you could kindly raise your score considering our follow-up responses. In the following, we would like to summarize the contributions and responses of this paper again.

**Contributions:**

- Innovative Framework: Our work introduces a novel NAM framework which pioneers the explanation of MLLMs and ***addresses a crucial gap in MLLMs*** (Reviewers `gq3m`, `kLxz`, `VhwX`, `VfTi`).
- Robust Experiments: Our experimental design is rigorous and  ***provides convincing evidence and comprehensive results***  (Reviewers `gq3m`,`VhwX`).
- Valuable Insights: Our study ***offers interesting and instructive observations for the development of MLLMs*** (Reviewers `VhwX`, `VfTi`).

**Responses to Your Concerns:**

- Neurons Semantics: We have elaborated on the theoretical reasoning, cited sources, intuitive understanding, and practical applications of our approach to neuron semantics in two rounds of responses. 
- Evaluation: We have adjusted the evaluation metrics to treat semantics as a visualization for noval insights of neuron interpretability.
- Validation of Linear Addition: We provided detailed derivations and justifications for the linear addition approach, confirming its correctness.
- Enhanced Clarity and Presentation: We have refined the presentation at several key points you have mentioned (*e.g.*, Figure 3(c), Related Work, Line 110, 209-210, 217-218, 260) to enhance readability.
- Neuronal Activations as Signals: We clarified the strong relevance of this choice to the issues explored in our paper, explaining why alternatives were not suitable for signaling in our context.

We sincerely hope that our clarifications and detailed discussions address your concerns and that you might consider supporting our submission during this final phase of the review process. Should you have any further questions or need additional clarification, please do not hesitate to contact us. We are more than willing to provide further information promptly. Hope you a nice day and thanks again.

Warm regards,

The Authors of Submission 10306
Dear Reviewer,

Thank you sincerely for your valuable and positive feedback.   We deeply appreciate your recognition of our efforts in addressing the concerns from the reviews.

Your acknowledgment of the improvements we have made is greatly encouraging, and we are grateful for your decision to maintain the positive score based on these enhancements.   We will continue to refine and develop our research to contribute meaningfully to the field.

Thank you once again for your careful consideration and support.

Best regards,
Authors!
Dear Reviewer VfTi,

Thank you very much for your active engagement and valuable feedback. We are pleased to hear that our previous response addressed most of your concerns, with the exception of Q4 (*''How do your know the semantics of neurons？''*). We appreciate the opportunity to provide further clarification. **Since this year’s NeurIPS does not allow submitting a new version during rebuttal, we will provide a detailed explanation here to address your concern as thoroughly as possible within the limited space.** We assure you that more detailed enhancements have been incorporated into the new version, and we sincerely hope for your understanding and trust in this process.

***1. Residual Streams***

To address your question, let’s first revisit the concept of residual streams. The output of the $l$-th layer in the LLM can be computed as follows:$$\mathbf{h}^l=\mathbf{m}^l+\mathbf{h}^{l-1}+\mathbf{a}^l$$

where $\mathbf{a}^l$ and $\mathbf{m}^l$ are the output of the attention and FFN block,  respectively. Specifically, we have $\mathbf{m}^l=W_{out} \mathbf{r}^l$
where $\mathbf{r}^l$ is the vector of the intermediate layer in FFN and $W_{out}^l$ is the output embedding matrix in FFN.

Based on these, the output of the final layer $\mathbf{h}^{L}$ can be recursively derived as:$$\mathbf{h}^L=\sum_{l=1}^L\mathbf{m}^l+\mathbf{h}^0+\sum_{l=1}^L\mathbf{a}^l$$

It indicates that the output of any FFN module can contribute directly through linear addition to $\mathbf{h}^L$. This linearly accumulated flow of information is referred to as the residual stream.

Given the complexity of the non-linear relationships within LLMs which are difficult to intuitively understand and control, many downstream tasks (*e.g.*, LLMs editing [1,2] and interpretability [3]) focus solely on the residual stream while reasonably neglecting other non-linear flows.

***2. Neuron Contributions via Residual Streams***

Revisiting the structure of LLMs, $\mathbf{h}^L$ is passed through the unembedding matrix $W_u$ to obtain a probability distribution over the vocabulary $\mathbf{y}$, with the word corresponding to the highest probability being the final output:$$\mathbf{y}=W_u\mathbf{h}^L$$

Since $\mathbf{m}^l$ contributes directly to the probability distribution through the residual stream, we define this contribution as $\mathbf{y}’$:$$\mathbf{y}’=W_u\mathbf{m}^l=W_uW^l_{out}\mathbf{r}^l$$

Here, $W_uW^l_{out}$ can be considered as a new unembedding matrix, and the probability distribution $\mathbf{y}’$ can be decomposed into contributions from each of the $k$ hidden neurons in $\mathbf{r}^l$:$$\mathbf{y}’=W_u\sum_{i=1}^k W^l_{\text{out}i}r_i$$

Thus, $W^{l}_{\text{out}i}{r}_i^l $ represents the distribution over the vocabulary for each hidden neuron. The word with the highest probability in this distribution is the one associated with that neuron. The intuition here is that **regardless of the magnitude of a neuron’s output, its contribution will always be most inclined toward its associated word**. For further details, please refer to [1].

***3. Using These Semantics as Evaluation Metrics***

As you pointed out, this method of deriving semantics could have limitations due to potential bias (since it only considers information within the residual stream). Therefore, as mentioned in our previous response, we **did not** use this as an evaluation metric in the reversion. However, given the success of focusing on residual stream in many fundamental LLM studies and downstream tasks [1,2,3], we believe these semantics still hold reference value. As such, we included this as an interesting experimental observation for visualizing neuron semantics, and **we believe this visualization has the potential to offer readers valuable insights and perspectives on the behavior of neurons within MLLMs**.

***4. Other Evaluation Metrics***

Although we removed this metric in the new version, **the other metrics in our paper are unaffected by the residual stream and are therefore unbiased**. The results from these metrics successfully validate the superiority of our method, ensuring the effectiveness of our experimental section.

Finally, we acknowledge that explainable MLLMs are still in their infancy, and there is currently no universally accepted metric. **This work not only aims to provide a new paradigm but also actively explores usable, fair, and open-source evaluation metrics to advance this field.** We humbly ask for your understanding and hope you can recognize the improvements we have made in the new version based on your feedback. 

*Thank you once again for participating in the discussion! We are ready to address any additional questions from you at any time and look forward to continuing our communication with you!*

*Best regards,*

*Authors*

[1] Locating and Editing Factual Associations in GPT. NeurIPS 2022

[2] Mass-Editing Memory in a Transformer. ICLR 2023

[3] Multi-modal neurons in pretrained text-only transformers. ICCV 2023
I appreciate the authors' response to the review, and that many points were clarified in the paper along with the bias and weaknesses of the metrics (although I cannot confirm the change since the new version is somehow not visible). I would further encourage the authors to clarify the assumptions mentioned in the response to Q4 and state the limitations of the method as well as the evaluation. Since my concerns were generally on the presentation, which is seemingly being improved, I'm keeping my recommendation to accept this paper despite the limitations and weaknesses which are always present in any methods.
Thanks for the response, I decide to keep the relative positive score.
Dear Reviewer gq3m:

Thank you very much for your positive feedback and for raising your score. We greatly appreciate your detailed comments and suggestions, which have significantly improved our paper. **Your recognition encourages us to continue refining our work.**

Your support and trust are invaluable to us. They not only validate our current efforts but also motivate us to push forward in the field of interpretable LLMs. We are committed to advancing our research and contributing meaningful insights and innovations to this rapidly evolving area.

**While we recognize that interpretable LLMs are still in their early stages and that there is a long road ahead, we firmly believe that with persistent effort and dedication, we will make significant progress.**

Thank you once again for your encouragement and support.

Best regards,

Authors
I believe my concerns here are reasonably satisfied. I am especially impressed by the turnaround on the audio model analysis, which I think is quite a nice addition to the paper. Conditional on that the results in this response are effectively integrated into the main paper, I am raising my score to an 8 to reflect these additional improvements, and would support acceptance.