MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models

## Abstract

Abstract In recent years, multimodal large language models (MLLMs) have significantly advanced, integrating more modalities into diverse applications. However, the lack of explainability remains a major barrier to their use in scenarios requiring decision transparency. Current neuron-level explanation paradigms mainly focus on knowledge localization or language- and domain-specific analyses, leaving the exploration of multimodality largely unaddressed. To tackle these challenges, we propose MINER , a transferable framework for mi ning modality-specific ne u r ons (MSNs) in MLLMs, which comprises four stages: ❶ modality separation, ❷ importance score calculation, ❸ importance score aggregation, ❹ modality-specific neuron selection. Extensive experiments across six benchmarks and two representative MLLMs show that (I) deactivating ONLY 2% of MSNs significantly reduces MLLMs performance ( 0.56 ∼ 0.24 ↓ similar-to 0.56 0.24 ↓ absent 0.56\sim 0.24\downarrow 0.56 ∼ 0.24 ↓ for Qwen2-VL, 0.69 ∼ 0.31 ↓ similar-to 0.69 0.31 ↓ absent 0.69\sim 0.31\downarrow 0.69 ∼ 0.31 ↓ for Qwen2-Audio), (II) different modalities mainly converge in the lower layers, (III) MSNs influence how key information from various modalities converges to the last token, (IV) two intriguing phenomena worth further investigation, i.e., semantic probing and semantic telomeres. The source code is available at this URL .

## 1 Introduction

**Table 1: X-specific neuron studies.**
|  | X-Specific Neurons |  |  |  |
| --- | --- | --- | --- | --- |
|  | Language<br>Tang et al. () | Domain<br>Huo et al. () | Modality<br>Ours |  |
| Granularity | Sample-level | ✔ | ✔ | ✗ |
| Token-level | ✗ | ✗ | ✔ |  |
| Importance<br>Metrics | Probability | ✔ | ✔ | ✔ |
| Mean | ✗ | ✗ | ✔ |  |
| Max | ✗ | ✗ | ✔ |  |
| Attention | ✗ | ✗ | ✔ |  |

Recently, multimodal language models (MLLMs) have made rapid advancements across various applications (Xu et al., ; Xiao et al., ; Yan et al., ), exemplified by models like GPT-4 (Achiam et al., ), LLaMA 3 (Dubey et al., ), Qwen-VL (Bai et al., ), and LLaVA-NEXT (Liu et al., ). However, their black-box nature presents challenges, particularly in fields like medical studies (González-Alday et al., ), where interpretability is essential. Understanding the decision-making process is vital, making explainability a central focus of ongoing research (Tjoa & Guan, ; Zhao et al., ).

Numerous studies have sought to understand how knowledge is stored in models (Sukhbaatar et al., ; Dai et al., ; Meng et al., ; Chen et al., ) and how this information influences decision-making (Geva et al., ; Petroni et al., ). For example, Dai et al. (); Geva et al. () investigate knowledge storage mechanisms, while Wendler et al. (); Zhang et al. () provide insights into layer-level explainability. Additionally, several works in the neuron-level explanation domain (Tang et al., ; Kojima et al., ; Huo et al., ) have identified language-specific or domain-specific neurons, referred to as X-specific neurons. However, these studies often neglect modality-level understanding, particularly how multimodal information is processed and its differences and similarities (Parekh et al., ; Rodis et al., ), as shown in table 1.

As shown in table 1 and fig. 1, recent X-specific neuron works (Tang et al., ; Huo et al., ) face two key limitations: First, they focus on sample-level neuron identification, assuming each sample belongs to a single language or domain, while multimodal samples often span multiple modalities. Second, they rely solely on activation probability as the importance metric, which is insufficiently comprehensive (See section 4.3 and Ob2 of section 5.4 for details).

This raises an intriguing question: Are there modality-specific neurons (MSNs) akin to those in multi-language or multi-domain settings? To address this, we must tackle the following challenges: ❶ How can we measure neuron importance for specific modalities? ❷ What mechanisms do these neurons use to influence the model? ❸ Can we identify some underlying patterns of MSNs?

We propose MINER, a transferable framework that develops new importance metrics for measuring neuron significance and introduces selection strategies for identifying key neurons for each modality. MINER tackles the challenges by: ❶ Decompose neuron importance for modalities into token importance from the top down, then restore neuron-modal importance through bottom-up aggregation. ❷ Analyze neuron behavior using feature dimensionality reduction plots and contribution scores of modality tokens to predictions. ❸ Our experiments revealed semantic probing and semantic telomeres. Through this design, we identified a set of important modality-related neurons and uncovered interesting phenomena during our experimental analysis, offering valuable insights for the research community. Our contributions can be summarized as follows:

- ☞
To our knowledge, we are the first to analyze modality-specific neurons (MSNs) in multimodal large language models (MLLMs).
- ☞
We introduce MINER, a transferable framework for selecting MSNs in both vision-based and audio-based MLLMs, capable of handling datasets with any combination of modalities uniformly.
- ☞
We provide a systematic problem definition and propose a novel token-level analysis pipeline that differs from existing sample-level methods.
- ☞
We validate the significance of MSNs through extensive experiments, uncover intriguing phenomena of semantic probing and semantic telomeres, and present new insights.

Figure: Figure 1: Comparison of Language-specific (a), Domain-specific (b), and our proposed Modality-specific Neuron detection and analysis framework, MINER (c).
Refer to caption: x1.png

## 2 Related Works

In this section, we provide a brief overview of knowledge location and neuron analysis work, while the related work for the remaining sections is included in section B.1.

Knowledge Localization.   Research has examined how factual and commonsense knowledge is represented in neural networks (Park et al., ; Hase et al., ; Zhu et al., ). For instance, Sukhbaatar et al. () demonstrated that persistent memory vectors can replace feed-forward network (FFN) layers in transformers without performance loss. Geva et al. () showed that FFN layers serve as key-value memories, linking training patterns to output vocabulary. Recently, Dai et al. () identified *knowledge neurons* in FFN layers of pretrained transformers, positively correlating their activation with factual knowledge expression. Meng et al. () and Chen et al. () further explored knowledge neurons in language models. In this context, our analysis aims to identify modality-specific neurons in the FFN layers of MLLMs.

Neuron Analysis.   Neuron analysis in pretrained models is an emerging research area in both computer vision and natural language processing. Recent studies have gone beyond explaining the concepts and knowledge represented by individual neurons (Bau et al., ; Oikarinen & Weng, ; Bills et al., ; Gao et al., ) to identify neurons that respond to specific patterns. For example, Schubert et al. () and Cammarata et al. () found visual neurons that detect high-frequency features or curves in images, while Tang et al. () and Kojima et al. () analyzed neurons uniquely activated by target languages, termed *language-specific neurons*, in large language models. In the multimodal domain, research has primarily focused on detecting neurons that respond to both textual and visual inputs (Goh et al., ; Schwettmann et al., ; Pan et al., ). Additionally, Huo et al. () adapted techniques from Tang et al. () to identify domain-specific neurons (e.g., in medicine and remote sensing) in MLLMs. However, the specialization of neurons in multimodal models is still underexplored. To the best of our knowledge, our work is the first to analyze modality-specific neurons in MLLMs.

## 3 Preliminaries

### 3.1 Definitions of Modality, Sample and Dataset

We start by clarifying two key terms: modality set and modality space, which will be referenced throughout the paper. Building on these definitions, we then define samples and datasets to enhance the understanding of our method.

Modality Set.   The classification of modalities in data is not singular. Here, we outline three potential methods for partitioning modality sets:

$$ $\displaystyle S_{\text{all}}=\{\textbf{{all}}\}$ $\displaystyle S_{M}=\{\textbf{{text}},\textbf{{special}},\textbf{{image}}, \textbf{{video}},\textbf{{audio}}\}=\{m_{1},\ldots,m_{M}\}$ $\displaystyle S_{\text{t+s}}=\{\textbf{{t+s}},\textbf{{image}},\textbf{{video} },\textbf{{audio}}\}$ $$

We employ $S_{M}$ in our method, while $S_{\text{all}}$ (treats all as one modality) and $S_{\text{t+s}}$ are used only in ablation studies. The special modality refers to elements not present in the raw data but introduced by certain MLLMs. For instance, in Qwen2-VL, this includes separators like [im-start] and [im-end] or placeholders such as [image-pad]. The only difference between $S_{M}$ and $S_{\text{t+s}}$ is whether we consider special and text as the same modality. If treated as the same, we derive $S_{\text{t+s}}$.

Modality Space.   Each modality $m\in S_{M}$ corresponds to a modality space $\mathcal{X}_{m}$, encompassing all data or features associated with that modality. This can be formally defined as:

$$ $\mathcal{X}_{m}=\{x_{m}\mid x_{m}\text{ is a feature corresponding to modality }m_{i}\text{ from a multimodal sample}\}$ (1) $$

Additionally, modality space can be viewed as a collection of exclusive information, where data from one space should not overlap with others. For example, the image modality space $\mathcal{X}_{\text{image}}$ includes raw images and features, encompassing all samples related to visual information.

Multimodal Space / Sample.   With the foundational definitions of modality established, we define the multimodal sample space as $\mathcal{X}$, where a sample is represented as:

$$ $\mathbf{x}=(x_{m}\mid m\in S_{M})\quad\text{where}\quad x_{m}=\begin{cases} \text{actual modality data}&\text{if modality }m\text{ exists}\\ \text{None}&\text{if modality }m\text{ does not exist}\end{cases}\vspace{-1mm}$ $$

where $x_{m}\sim\mathcal{X}_{m}$. For simplicity, we omit $x_{m}=\text{None}$, allowing a VQA sample to be expressed as $\mathbf{x}=(x_{\text{text}},x_{\text{image}})$. To perform a fine-grained analysis of multimodal samples, we define the modality function Mod to extract the modality of component $x_{m}$ as follows:

$$ $\text{Mod}(x_{m})=\begin{cases}m&\text{if }x_{m}\neq\text{None}\\ \varnothing&\text{otherwise}\end{cases}$ $$

The modality function can be generalized to extract a sample’s modalities, returning a set of modalities: $\text{Mod}(\mathbf{x})=\{\text{Mod}(x_{m})|m\in S_{M}\}$. For example, $\text{Mod}(\mathbf{x}=(x_{\text{text}},x_{\text{image}}))=\{\textbf{{text}}$. We divide the sample space $\mathcal{X}$ into two mutually exclusive subspaces based on the number of modalities: the uni-modality space $\mathcal{X}_{\text{uni}}=\{\mathbf{x}\mid\#\{x_{i}\mid x_{i}\neq\text{None}\}=1\}$ and the multi-modality space $\mathcal{X}_{\text{multi}}=\{\mathbf{x}\mid\#\{x_{i}\mid x_{i}\neq\text{None}
\}>1\}$.

Multimodal Dataset.   A dataset is a subset of $\mathcal{X}$ where all samples share common characteristics, such as modality or question type. For example, VQA datasets contain both text and image modalities, structured around answering questions. We define uni-modality datasets as $D^{\text{uni}}=\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots|\mathbf{x}_{i}\in
\mathcal{X}_{\text{uni}}\}$, and multi-modality datasets as $D^{\text{multi}}=\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots|\mathbf{x}_{i}\in
\mathcal{X}_{\text{multi}}\}$.

### 3.2 Relevant Transformer Concepts

MLLMs process $\mathbf{x}$ using modality-specific encoders and tokenizers (e.g., ViT (Dosovitskiy, ) for image), transforming it into a set of input tokens:

$$ $T_{\text{input}}=\bigcup_{m\in S_{M}}T_{m}=\{t_{1},t_{2},\ldots,t_{I}\}\quad \text{where}\quad T_{m}\cap T_{m^{\prime}}=\emptyset\quad\text{for }m\neq m^{\prime}$ (2) $$

By extending Mod to the token domain, we define $T_{m}=\{t_{i}\in T_{\text{input}}|\text{Mod}(t_{i})=m\}$ as the modality-specific token set, with size $|T_{m}|=I_{m}$. We also establish a mapping structure $\text{Ind}_{m}$ for each subset, enabling retrieval of the original index from the input token set. This allows us to trace any element in $T_{m}$ back to the original set using the relationship $T_{\text{input}}[\text{Ind}_{m}[i]]=T_{m}[i]$.

To enhance clarity in the subsequent definitions, we represent the token embedding corresponding to $t_{i}$ after being input into the LLM at layer $l$ as $t_{i}^{l}\in\mathbb{R}^{d}$, without distinguishing between the token and its embedding. We denote the embeddings at layer $l$ as $T^{l}_{\text{input}}=[t_{i}^{l}]_{i=1}^{I}\in\mathbb{R}^{I\times d}$ and the attention matrix after softmax as $\text{{A}}^{l}\in\mathbb{R}^{I\times I}$. Ignoring layer normalization, we denote the $i$-th value vector as $v_{i}^{l}$, allowing us to express the embedding update process as:

$$ $a_{i}^{l}=t_{i}^{l}+\sum_{j=1}^{I}\text{{A}}_{i,j}^{l}v_{j}^{l},\quad t_{i}^{l +1}=a_{i}^{l}+W^{l}_{\text{out}}\text{Act}(W^{l}_{\text{in}}a_{i}^{l})$ (3) $$

where $W_{\text{in}}\in\mathbb{R}^{d\times N}$ and $W_{\text{out}}\in\mathbb{R}^{N\times d}$ is the up-sampling and down-sampling layers, respectively. In the equation above, $a_{i}^{l}$ represents the output of the attention module, and $\textbf{H}^{l}\in\mathbb{R}^{I\times N}$ is the hidden activation vector ($\textbf{H}^{l}_{i,n}$ represents the activation value of $n$-th neuron for token $t_{i}$). Each neuron is denoted as $u_{l,n}$, resulting in a total of $L\times N$ neurons in the FFN modules, represented as the matrix $U_{L\times N}$. Then these embeddings pass through $L$ identical transformer blocks, generating output tokens sequentially in an autoregressive manner. We define the set of output tokens as $T_{\text{output}}=\{t_{I+1},t_{I+2},\ldots,t_{I+O}\}$.

## 4 Framework of MINER

### 4.1 Problem Analysis

Motivated by previous research on X-specific neurons, such as language-specific Tang et al. () and domain-specific neurons Huo et al. (), our work focuses on identifying a set of MSNs that are critical for processing multimodal samples in MLLMs. We briefly analyze how our approach differs from previous studies in fig. 1 and table 1. We focus on neurons in the FFN of MLLMs rather than other modules, such as modality-specific encoders or projection layers, for several reasons: (I) Previous studies indicate that FFN encode distinct, recoverable knowledge attributes (Geva et al., ; Dai et al., ; Meng et al., ; ), with some neurons representing the same concepts across different modalities (Schwettmann et al., ). (II) Research by Schwettmann et al. () identifies semantic alignment between images and text within the LLM, rather than in the projection layer. (III) Our goal is to identify neurons across modalities within the same set, so we exclude modality-specific encoders. Thus, our research focuses on the FFN module.

Figure: Figure 2: Four stages of MINER. ❶ Prompt tokens are divided into modality sets before being input into the LLM. ❷ Each neuron computes an importance score for the tokens of each modality. ❸ Aggregate these values to compute the Importance Score Matrix (ISM), reflecting the modality-level importance of each neuron. ❹ Various selection methods, as detailed in section 4.5, are employed to identify modality-specific neurons for each modality.
Refer to caption: x2.png

Our problem can be distilled into two steps: first, measure a neuron’s importance for a specific modality (Stages 1-3), and second, select the most important neurons based on this measure (Stage 4). Assessing the importance of $u_{l,n}$ in modality space $\mathcal{X}$ is complex due to the infinite number of potential samples. Therefore, we define the importance score between a neuron and samples as $\text{IS}_{s}(u_{l,n},\mathbf{x})$. If a neuron consistently demonstrates importance across many samples, we can conclude it is significant within the modality space. The four stages of MINER align with the sections in fig. 2, outlined as follows:

- ➠
Stage 1: Modality Separation.   Decomposing the information within the LLM by modality.
- ➠
Stage 2: Importance Score Calculation.   We decompose the importance of neurons to modalities into their token-level significance and define it accordingly.
- ➠
Stage 3: Importance Score Aggregation.   We aggregate the token-level importance of neurons to restore their modality-level significance.
- ➠
Stage 4: Modality-Specific Neuron Selection.   The aggregated scores and a selection strategy are used to identify the top-$K$ important MSNs.

### 4.2 Stage 1: Modality Separation

Figure: Figure 3: A VQA demo.
Refer to caption: x3.png

To identify neurons associated with specific modalities, we first need to separate the information by modality. While input tokens $T_{\text{input}}$ can be divided into distinct sets $\{T_{\text{image}},T_{\text{text}},\ldots\}$, the attention mechanism blends information across modalities, complicating complete separation. Inspired by the VQA demo in fig. 3, which shows that a small portion of license plate information suffices to answer the question, we propose the following hypothesis:

###### Hypothesis 1 :

As information passes through the LLM layers, most of the content in $T_{m_{i}}$ remains within its set, with only a small portion related to the question being transferred.

This hypothesis is supported by confirmatory experiments in section 5.5, prompting us to propose an approximate method for segmenting modality information: we assume the attention module functions only within distinct token sets of different modalities, preventing information exchange between subsets and enabling the partitioning of tokens into mutually exclusive modalities.

Our hypothesis generates a set of uni-modality datasets $\{D^{1}_{\text{uni}},D^{2}_{\text{uni}},\ldots\}$ at the token level by partially limiting information flow, aligning to some extent with Tang et al. (); Huo et al. (). We further refine the neuron-sample importance function $\text{IS}_{s}$ by defining it at a more granular level between neuron and token set as $\text{IS}_{t}(u_{l,n},T_{m})$. The next stage outlines how we utilize modality information and provides a detailed definition of the importance score.

###### Hypothesis 1 :

### 4.3 Stage 2: Importance Score Calculation

In the FFN, each neuron outputs an activation value for a token, which naturally serves as an importance score and is widely used (Tang et al., ; Huo et al., ). We use $\textbf{H}^{l}_{i,n}$, the activation value introduced in section 3.2, as the baseline for all other importance scores (sample-level $\text{IS}_{s}$, and token-level $\text{IS}_{t}$). As shown in fig. 12, the significant variation in token counts across modalities makes individual token-based importance unreliable. Therefore, we calculate importance over a token set, removing token count as a factor. We propose the following operations to aggregate activation values for a token set:

- ➢
Prob. The activation value reflects a neuron’s interest in a token, so we calculate the interest probability (activation value $>$ 0) for the tokens:
$\text{IS}_{t}^{\text{Prob}}(u_{l,n},T_{m})=\text{IS}_{t}^{\text{Prob}}(\textbf
{H}^{l}_{1,n},\ldots,\textbf{H}^{l}_{I_{m},n})=\frac{1}{I_{m}}\sum_{i=1}^{I_{m
}}\mathbb{I}(\textbf{H}^{l}_{i,n}>0)$
(4)
Here, $\mathbb{I}$ is the indicator function. A higher probability suggests greater engagement with the token set. However, high activation probability doesn’t necessarily imply importance. A neuron may have frequent low activations, while fewer high values could indicate a greater contribution. These limitations are confirmed in section 5.4).
- ➢
Mean. Taking the average is a common approach that captures the group’s overall characteristics while balancing the influence of individual tokens. We define this operation as follows:
$\text{IS}_{t}^{\text{Mean}}(u_{l,n},T_{m})=\text{IS}_{t}^{\text{Mean}}(\textbf
{H}^{l}_{1,n},\ldots,\textbf{H}^{l}_{I_{m},n})=\frac{1}{I_{m}}\sum_{i=1}^{I_{m
}}\textbf{H}^{l}_{i,n}$
(5)
- ➢
Max.
We use the maximum activation value as an importance measure. Unlike $\text{IS}_{t}^{\text{Mean}}$, which represents a uniform distribution, $\text{IS}_{t}^{\text{Max}}$ corresponds to a Dirac distribution.
- ➢
Attn-Q. We developed a method that uses attention values to assess a neuron’s importance for a token set, considering each $t_{i}\in T_{\text{input}}$:
$\text{IS}_{t}(u_{l,n},T_{m})=\frac{1}{T}\sum_{i=1}^{T}\sum_{j=1}^{I_{m}}w_{j}(
t_{i})\textbf{H}^{l}_{\text{Ind}_{m}[j],n}$
(6)
We use $\text{Ind}_{m}$ to map the index of $T_{m}$ back to $T_{\text{input}}$, retrieving the corresponding values from H. The $\text{IS}^{\text{Attn-Q}}_{t}$ employs the $i$-th row of the attention matrix $\mathbf{A}$, reflecting the attention scores for the $i$-th query across all keys, where $\mathbf{w}(t_{i})=[w_{j}(t_{i})]_{j=1}^{I_{m}}=\text{softmax}[\mathbf{A}_{i,
\text{Ind}_{m}[j]}]_{j=1}^{I_{m}}$.
- ➢
Attn-K. Like $\text{IS}^{\text{Attn-Q}}_{t}$, $\text{IS}^{\text{Attn-K}}_{t}$ uses the $i$-th column of the attention matrix, reflecting the attention scores for the $i$-th key across all queries, where $\mathbf{w}(t_{i})=\text{softmax}[\mathbf{A}_{\text{Ind}_{m}[j],i}]_{j=1}^{I_{m}}$.

As shown in table 1, previous studies on X-specific neuron analysis (Tang et al., ; Huo et al., ) rely solely on $\text{IS}^{\text{Prob}}$, which has inherent limitations. To address this, we define $\text{IS}_{t}$ as the weighted sum of the five metrics discussed earlier, combining both local and global perspectives. We also evaluate the effectiveness of each metric in section 5.4.

### 4.4 Stage 3: Importance Score Aggregation

We calculated the importance score for each neuron across all modality-specific token sets. Next, we aggregate these scores to define the sample-level importance score:

$$ $\text{IS}_{s}(u_{l,n},\mathbf{x})=\text{IS}_{s}(u_{l,n},T_{\text{input}})=[ \text{IS}_{t}(u_{l,n},T_{m})]_{m\in S_{M}}\in\mathbb{R}^{M}$ (7) $$

For each neuron, we compute a vector as described above, resulting in a sample-level Importance Score Matrix (ISM) defined as $\text{ISM}_{s}(\mathbf{x})=[\text{IS}_{s}(u_{l,n},\mathbf{x})]_{L,N}\in\mathbb
{R}^{M\times L\times N}$. We then aggregate the importance scores across samples from different datasets to obtain the modality-level ISM:

$$ $\text{ISM}(\mathcal{X})=\sum_{i}\sum_{j}^{|D_{i}^{\text{multi}}|}\text{ISM}_{s }(\mathbf{x}_{i,j})\in\mathbb{R}^{M\times L\times N},\text{ where }\mathbf{x}_ {i,j}\text{ is the }j\text{-th sample of }D_{i}^{\text{multi}}$ (8) $$

As the sample size grows, the importance matrix becomes more effective in assessing neuron significance across the entire modality space.

### 4.5 Stage 4: Modality-Specific Neuron Selection

The previous stage’s ISM$(\mathcal{X})$ evaluates the importance of all neurons in the MLLMs for each modality. We then designed four strategies to select the $K$ highest importance scores from the ISM matrix. However, because of potential neuron overlap, the final number of selected neurons may be less than $K$. The strategies are implemented as follows:

(I) Uniform selects $\lfloor\frac{K}{L\times M}\rfloor$ neurons for each modality in every layer (the strongest assumption).

(II) LA-MU (Layer-Adaptive & Modality-Uniform) selects $\lfloor\frac{K}{M}\rfloor$ neurons for each modality, allowing for adaptive quantities in each layer and relaxing certain constraints.

(III) LU-MA (Layer-Uniform & Modality-Adaptive) selects $\lfloor\frac{K}{L}\rfloor$ neurons for each layer.

(IV) Adaptive selects $K$ positions in the ISM matrix without constraints on modality or layer.

Figure: Figure 4: Selection strategies.
Refer to caption: x4.png

fig. 4 intuitively illustrates the segmentation methods for the four selection strategies. The assumptions range from strong to weak, beginning with a fixed number of neurons for each modality in every layer and progressively relaxing constraints until the fourth approach imposes no limitations. Together, these strategies offer a comprehensive set of selection criteria.

Using any selection strategy, we generate neuron positions as a Boolean mask $\mathbf{B}\in\{0,1\}^{L\times N}$ (with 1 indicating MSNs). As shown in eq. 8, different datasets yield distinct ISM, resulting in varied masks. Thus, with all other settings fixed, a specific dataset combination uniquely determines a neuron mask. For simplicity, we can define a mask function as $\mathbf{B}=\text{mask}(D_{1},D_{2},\ldots)$.

## 5 Experiment

We apply MINER in various settings to explore the existence and characteristics of MSNs. Our study aims to answer the following research questions:

- ➷
RQ1: Do the identified modality-specific neurons significantly contribute to multimodal models?
- ➷
RQ2: If RQ1 is validated, how do these neurons facilitate this contribution?
- ➷
RQ3: How do different hyperparameter settings influence the behavior of the MLLMs?
- ➷
RQ4: Can we uncover underlying patterns among modality-specific neurons?

### 5.1 Experiment Setup

Unless otherwise noted (e.g., in ablation studies), we adopt the following default settings. We define $S_{M}$ as the modality set, treating special and text as distinct modalities. We deactivate neurons across all modalities by setting their outputs to zero, then evaluate the impact on performance.

Models. We select Qwen2-VL (Wang et al., ) for visual tasks and Qwen2-Audio (Chu et al., ) for audio tasks to ensure a thorough exploration of the current model landscape. Qwen2-VL processes text, image, and video modalities, while Qwen2-Audio focuses on text and audio.

Datasets.   For text-only tasks, we chose MMLU (Hendrycks et al., ). For text-image tasks, we selected TextVQA (Singh et al., ), and for text-video tasks, we opted for MSVD-QA (Chen & Dolan, ). Since no datasets exist for image-only or audio-only tasks, we selected specific datasets with fixed prompts to minimize the impact of the text modality. For an approximate image-only dataset, we utilized the MS-COCO 2014 captioning benchmark (Lin et al., ), adopting the Karpathy split test set as per Li et al. (); Tang et al. (); Zhan et al. () by fixing the text tokens. For an approximate audio-only dataset, we employed LibriSpeech (Panayotov et al., ) and VocalSound (Gong et al., ) in a similar manner.

Table: Table 2: Main results of Qwen2-VL. We select MSNs (2%) using various dataset combinations (indicated by $\checkmark$) and then mask neurons across all modalities, recording the performance of the masked MLLMs. The minimum value in each column is marked in blue. We highlight any new values in gray when adding a new dataset improves the mask’s quality.

### 5.2 Main Results (RQ1)

To address RQ1, we generate neuron masks for each dataset combination to thoroughly evaluate our method. The main results for Qwen2-VL and Qwen2-Audio are shown in table 2 and table 3. Key experimental observations (Obs) include:

**Table 3: Qwen2-Audio results: Same format as table 2.**
| Datasets | Qwen2-Audio ($0.69\sim 0.31\downarrow$) |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MMLU |  | LibriSpeech |  | VocalSound |  | Average |  |  |  |  |
| MMLU | Libri | Vocal |  |  | Accuracy |  | WRR |  | Accuracy |  |  |
| - | - | - |  |  | 0.40 |  | 0.94 |  | 0.74 |  | 0.69 |
| ✓ |  |  |  |  | 0.00 |  | 0.53 |  | 0.41 |  | 0.31 |
|  | ✓ |  |  |  | 0.01 |  | 0.85 |  | 0.15 |  | 0.34 |
|  |  | ✓ |  |  | 0.01 |  | 0.87 |  | 0.19 |  | 0.36 |
| ✓ | ✓ |  |  |  | 0.00 |  | 0.85 |  | 0.34 |  | 0.40 |
| ✓ |  | ✓ |  |  | 0.01 |  | 0.86 |  | 0.37 |  | 0.39 |
|  | ✓ | ✓ |  |  | 0.01 |  | 0.81 |  | 0.35 |  | 0.39 |
| ✓ | ✓ | ✓ |  |  | 0.01 |  | 0.83 |  | 0.40 |  | 0.41 |

Ob1. Masking just 2% of neurons impacts performance.   The first row in both tables shows normal inference without masking. After masking the selected neurons, we observe performance drops, except for a slight increase in MSVD (analyzed later). Results in table 4 show that randomly masking 2% of neurons has no effect, confirming that the modality-specific neurons identified by our method significantly influence model performance.

Ob2. Neurons identified from diverse datasets are higher quality.   For MSVD-QA performance, apply mask(MSVD) drops performance to 0.32, while apply mask(MSVD, COCO) raises it to 0.52. This may be because the diverse text questions in MSVD-QA closely relate to video semantics, producing high-quality text-specific neurons. In contrast, the fixed text prompts in COCO Caption (see fig. 7) create a repetitive pattern, degrading neuron mask quality. MMLU’s diverse text data causes a larger performance drop (0.31) when combined with MSVD-QA.

Ob3. Adding more datasets can improve MSNs quality.   Many values in both tables are highlighted in Gray, showing that new datasets often enhance mask quality. However, some cases also demonstrate decreased effectiveness, possibly due to the diversity issues mentioned in Ob3 or conflicts among dataset characteristics that lead to incompatible ISM matrices. We will investigate these phenomena in future work.

### 5.3 Neuron Functionality Principles (RQ2)

This research question focuses on the intrinsic mechanisms of action of MSNs. Several key observations are as follows:

Ob1. The audio modality exhibits a “semantic probing” effect towards the text modality, indicating a potential trend toward aligning key information across modalities.   We present the feature distribution under three masking settings in fig. 5-(a). The complementary mask is defined as $1-\mathbf{B}$.
As the layer depth increases, audio embeddings extend “tentacles” toward the embeddings of other modalities, a phenomenon we call “semantic probing.” This suggests that the LLM aligns not the entire modal feature space, but specific key information.

Ob2. A “semantic telomeres” phenomenon occurs in special modality, anchoring at the edges of the text modality’s semantic space.
In fig. 5-(a), text embeddings initially form circular clusters that elongate into strips in deeper layers. Special tokens progressively align with and stabilize at the front of these clusters, a pattern we refer to as “semantic telomeres.”

Ob3. MSNs shape MLLMs behavior by directing how key information from different modality tokens converges into the last token.   We apply mask(COCO) to the COCO Caption dataset and calculate each modality’s contribution score to the final prediction based on the accumulated attention between its token set and the last token. As shown in fig. 5-(b), applying a mask for a specific modality results in $\Delta<0$, indicating reduced information flow to the last token.

Figure: Figure 5: (a) t-SNE plots for VocalSound, showcasing three masking settings: no masking, Mask(VocalSound), and complementary masking (from top to bottom). (b) Display the change ($\Delta$) in contribution scores between different token sets and the last token before and after masking, based on 100 samples (detail in section 5.3).
Refer to caption: x5.png

### 5.4 Ablation Studies (RQ3)

We present ablation studies on MINER’s components and highlight key observations:

**Table 4: Ablation results. Red (blue) indicates the minimum in each row (column), and green highlights values that are minimum in both.**
| Importance Metric |  | Selection Strategy |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Prob | Mean | Max | A-K | A-Q |  | Uniform | LU-MA | LA-MU | Adaptive | Random |
| 1 |  |  |  |  |  | 0.88 | 0.88 | 0.89 | 0.88 | 0.90 |
|  | 1 |  |  |  |  | 0.87 | 0.87 | 0.85 | 0.89 | 0.89 |
|  |  | 1 |  |  |  | 0.81 | 0.66 | 0.83 | 0.85 | 0.90 |
|  |  |  | 1 |  |  | 0.87 | 0.87 | 0.85 | 0.89 | 0.90 |
|  |  |  |  | 1 |  | 0.87 | 0.88 | 0.85 | 0.89 | 0.90 |
|  | 1/2 | 1/2 |  |  |  | 0.83 | 0.82 | 0.81 | 0.17 | 0.90 |
|  |  |  | 1/2 | 1/2 |  | 0.87 | 0.88 | 0.85 | 0.89 | 0.90 |
|  | 1/4 | 1/4 | 1/2 |  |  | 0.84 | 0.85 | 0.76 | 0.84 | 0.90 |
|  | 1/4 | 1/4 |  | 1/2 |  | 0.85 | 0.85 | 0.81 | 0.80 | 0.90 |
| 1/5 | 1/5 | 1/5 | 1/5 | 1/5 |  | 0.86 | 0.87 | 0.89 | 0.90 | 0.90 |

Ob1. Selection strategies with appropriate degrees of freedom yield higher-quality MSNs.   As shown in table 4, the uniform method, which imposes a fixed number of neurons per modality at each layer, performed the worst. In contrast, greater flexibility in selecting neurons across modalities and layers enabled a better approximation of important neuron distribution. Among these strategies, LA-MU proved most effective, as it maintained flexibility in neuron counts across layers while treating all modalities equally, thus avoiding imbalances.

Ob2. For the importance metrics, the activation probability was less effective than our other designs.   We evaluated the effectiveness of each of the five importance metrics individually and also assessed the impact of combining them. As shown in table 4, the activation probability performed poorly, aligning with the section 4.3 analysis. In contrast, our newly designed metrics, which account for both local and global perspectives, demonstrate improved effectiveness.

Ob3. The model’s performance significantly drops with an increase in masked MSNs.   As shown in fig. 6-(a), masking 1% of neurons has minimal impact, while 5% nearly collapses performance. Therefore, MINER selects 2% to strike a balance.

Ob4. The model’s performance is greatly influenced by the deactivation value settings.   Considering neuron $u_{l,n}$, we test three deactivation settings: fixing its output activation to 0, -0.1, and $\min(\mathbf{H}^{l})$. As shown in fig. 6-(a), a slightly negative activation value reduces model performance, indirectly highlighting the importance of the identified neurons for the modality, since randomly deactivated neurons below zero do not impact performance.

Ob5. Considering “special” and “text” as separate modalities leads to improved results.   We evaluate three modality sets: $S_{\text{all}}$, $S_{M}$ and $S_{\text{t+s}}$ defined in section 3.1, referred to as “all”, “t,s”, and “t+s” in fig. 6-(a). The poor performance of $S_{\text{all}}$ shows that our sample-level issue (all in one modality) cannot be resolved, highlighting the need for modality separation at the token level.

Ob6. Masking MSNs from different modalities affects performance, with a greater impact as more modalities are masked. We apply mask(COCO) to the COCO Caption dataset and normalize values within different metrics to a 0-1 range. Results are shown in Figure 1, with “t,” “s,” and “i” representing text, special, and image, respectively. We found that performance declines significantly with an increasing number of masked modalities.

Figure: Figure 6: (a) and (b) present the ablation results, while (c), (d), and (e) illustrate the neuron distribution across layers. (f) shows the information flow within and between token sets.
Refer to caption: x6.png

### 5.5 Identifying Patterns in Neurons (RQ4)

In this section, we investigate potential MSNs patterns through experiments to offer valuable insights. Our key observations are as follows:

Ob1. Most MSNs are concentrated in the shallow layers, with only a small portion in the final layers.   As shown in fig. 6, figures (c), (d), and (e) visualize the distribution of modality-specific neurons across layers. This trend aligns with findings in Zhang et al. (), indicating that cross-modal perception primarily occurs in the early layers (detailed in section B.2). Thus, we believe that different modalities primarily converge in the lower layers.

Ob2. Most modality information stays within the token set, with only a small amount of key information transferring to other sets. To validate Hypothesis 1 from section 4.2, we measure information flow using attention values, calculating the cumulative in-set attention and cross-set attention (the diagonal blocks of $\mathbf{A}$). We apply mask(MSVD-QA) to the MSVD-QA dataset, with results shown in fig. 6-(f). Our findings indicate that in-set information flow significantly surpasses cross-set flow, supporting our hypothesis.

## 6 Conclusion

To our knowledge, this is the first study of modality-specific neurons (MSNs) in MLLMs. We select a small set of key neurons from the FFN that are crucial for processing multimodal data and design experiments to investigate their underlying mechanisms. We address this issue through the following steps: ❶ Define key concepts related to modalities, samples, and datasets. ❷ Conduct sample-level and token-level analyses to differentiate modality information at a finer granularity, evaluate the limitations of existing importance metrics, and propose new ones. ❸ Calculate importance scores for neurons associated with a set of tokens and aggregate these scores across samples and datasets for a global modality-level analysis. ❹ Define four selection strategies to extract the top-$K$ neurons with the highest importance scores as the output MSNs. In the experimental section, we validate our method by masking the MSNs and observing the resulting decline in model performance, while also identifying two intriguing phenomena: semantic probing and semantic telomeres.

### 6.1 Limitation and Future Work

Our work presents several areas for improvement: (I) We introduced the independence hypothesis of token sets to separate information by modality; however, key information exchanges between token sets occur and are difficult to capture. Future research could relax this assumption by incorporating cross-set information exchange for potentially better results. (II) Our study focused on vision-related Qwen2-VL and audio-related Qwen2-Audio models, limiting the range of modalities. Future work aims to include broader MLLMs, such as any-to-any models, and datasets with more diverse modalities. (III) We observed two intriguing phenomena—semantic probing and semantic telomeres—whose underlying causes remain unclear, presenting an opportunity for further exploration. (IV) The quality of datasets directly influences the quality of identified neurons, which may be linked to data diversity. Future research could investigate this further and design metrics to quantify dataset quality. (V) While this work emphasizes identifying important neurons, the next step is to explore how to leverage these neurons to enhance MLLMs performance on relevant modalities, potentially through neuron fine-tuning techniques.

## Appendix A Dataset Details

This section presents a detailed description of the datasets used in our evaluation, including data samples and the prompts provided to MLLMs.

### A.1 TextVQA

TextVQA (Singh et al., ) is a dataset for Visual Question Answering (VQA) that focuses on answering questions about text found in images. It features a variety of images, such as signs and labels, requiring models to combine visual understanding with Optical Character Recognition (OCR) to accurately respond to questions. TextVQA is a key benchmark for evaluating the integration of visual and textual reasoning in AI models.

### A.2 COCO caption

The COCO Caption dataset (Lin et al., ) is a large-scale resource containing over 1.5 million captions that describe more than 330,000 images, distributed across 80 diverse object categories. The images were selected to represent complex, real-world scenes, depicting everyday environments where common objects appear in their natural contexts. Each image is annotated with five captions, each of which was independently generated by human annotators. COCO Caption dataset has been a foundational benchmark for training and evaluating the image captioning performance of multimodal large language models.

### A.3 MMLU

The Massive Multitask Language Understanding (MMLU) (Hendrycks et al., ) dataset is a benchmark for evaluating language models across 57 subjects, ranging from elementary topics to advanced academic fields. It consists of multiple-choice questions testing a model’s knowledge and reasoning across diverse domains. MMLU is widely used to assess the generalization capabilities of large language models.

### A.4 MSVD-QA

The MSVD (Microsoft Research Video Description Corpus) (Chen & Dolan, ), also known as YouTube2Text, consists of 1,970 short videos, each ranging from 10 to 25 seconds with an average duration of 9 seconds. These videos depict a variety of subjects, including people, animals, actions, and different scenes. Each video is annotated with multiple sentences by different annotators, averaging around 41 sentences per clip, resulting in a total of 80,839 sentences. On average, each sentence contains 8 words, with approximately 16,000 unique words across the dataset.

### A.5 LibriSpeech

LibriSpeech (Panayotov et al., ) is a widely used dataset for automatic speech recognition (ASR) tasks. It contains approximately 1,000 hours of 16kHz English speech, sourced from the LibriVox audiobooks. The dataset is divided into several subsets, including “clean” and “other”, which distinguish recordings by noise levels. Each audio file is accompanied by an accurate transcription, making it ideal for training and evaluating ASR models. LibriSpeech’s diverse speaker base and detailed annotations also make it suitable for tasks like speaker identification and voice synthesis.

### A.6 VocalSound

VocalSound (Gong et al., ) is an open dataset containing 21,024 crowdsourced recordings of human vocalizations such as laughter, sighs, coughs, throat clearing, sneezes, and sniffing, from 3,365 individuals. Designed for classification tasks, it enables models to accurately identify various non-speech sounds. The dataset also includes metadata like speaker age, gender, native language, country, and health status, supporting research on how demographic factors affect vocal sounds. With its comprehensive scope and detailed annotations, VocalSound is a valuable resource for improving models in human vocalization classification.

### A.7 Prompts of Datasets

Figure: Figure 7: Prompt for COCO Caption.

Figure: Figure 8: Prompt for MMLU.

Figure: Figure 9: Prompt for LibriSpeech.

Figure: Figure 10: Prompt for Vocal Sound.

### A.8 Dataset Sample Statistics

Figure: Figure 11: Sample-level count statistics for different modalities. We provide a sample-level statistical analysis of all datasets used in this work. The inner ring represents different modalities, while the outer ring corresponds to the respective datasets, with sample counts labeled on both rings. For instance, a VQA sample containing text, special, and image modalities is counted under each modality it includes.
Refer to caption: x7.png

Figure: Figure 12: Token-level count statistics for different modalities. This image provides a token-level analysis. The inner ring represents different modalities, while the outer ring shows the corresponding datasets. We process each sample’s modality components through specific encoders and tokenizers to generate token sets, which are then categorized and summarized in the pie chart. Although the number of samples across datasets is balanced, the token count varies significantly by modality—video encoding produces far more tokens than text encoding, for instance. This underscores the need to compute importance scores within each modality’s token set and normalize by token count for fair cross-modality comparisons.
Refer to caption: x8.png

## Appendix B Additional Results

### B.1 Additional Related Works

Development of MLLMs.   Researchers have extensively investigated integrating additional modalities into foundational large language models (Liu et al., ). Notably, large vision-language models (Zhu et al., ) and audio-language models (Deshmukh et al., ) have gained significant attention by combining visual or audio inputs with text. For instance, (Liu et al., ) proposed aligning images with text by projecting visual embeddings from a pretrained vision encoder into word space through a single MLP layer, allowing LLMs to understand the post-projection tokens. Similarly, Chen et al. () and Lu et al. () used various projectors for this alignment. Recently, Qwen2-VL (Bai et al., ) introduced a universal vision encoder that processes both images and videos, integrating visual embeddings directly into the textual token stream. Parallelly, several studies have focused on integrating audio data into LLMs (Deshmukh et al., ; Wu et al., ), typically involving post-processing of auditory embeddings through additional modules like Q-Former (Tang et al., ) or downsampling layers (Das et al., ). Notably, Qwen2-Audio (Chu et al., ) has surpassed previous state-of-the-art models across various audio benchmarks without task-specific fine-tuning. In this research, we selected Qwen2-VL and Qwen2-Audio as our vision-language and audio-language baselines, both utilizing Qwen-7B (Bai et al., ; Yang et al., ) as the foundational LLM, with Vision Transformer (Dosovitskiy, ) and Whisper-large-v3 (Radford et al., ) serving as their respective vision and audio encoders.

### B.2 All distributions of neurons across layers

Figure: Figure 13: The distribution of modality-specific neurons across different layers, derived from all possible dataset combinations in Qwen2-VL.
Refer to caption: x9.png

Figure: Figure 14: The distribution of modality-specific neurons across different layers, derived from all possible dataset combinations in Qwen2-Audio.
Refer to caption: x10.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. Whether the analysis in this paper depends on specific models or not? If so, is it accurate to claim that it is the characteristic of MLLM? I wonder if the performance of FFN in different MLLMs plays a similar role.
soundness: 3
strengths: 1. The motivation is meaningful and interesting. With the increasing focus on MLLM, the underlying logic and pattern of each component of MLLM is important, and it is also under-explored in current research.
2. The proposed pipeline is clear and easy to follow. As shown in this paper, the framework mainly consists of 4 components or steps: separate modalities, calculate importance scores, aggregate importance scores, and select modality-specific neurons.
3. The experiments seem solid. Extent experiments are conducted to answer the four questions mentioned in this paper.
summary: This paper proposes a framework named MINER to understand the underlying pattern of modality-specific neurons in multimodal large language models (MLLM). The framework mainly consists of 4 components or steps: separate modalities, calculate importance scores, aggregate importance scores, and select modality-specific neurons. Experiments are conducted to analyze the existence and effectiveness of the modality-specific neurons.
weaknesses: 1. The writing of this paper is very confusing, even with very low-level mistakes. It seems the authors do not understand what is important and the meaning of the formulation. For instance, in the lines 203 and 204, what does H mean? There is even no H in the equation. Besides, there are some inconsistent formats, such as fig. xx, and Figure xxx, which makes this paper not well-prepared for submission.
2. Since this paper focuses on FFN of MLLM, it is better to show the structure of the focused FFN, making it clearer to understand what the purpose of this paper is.
3. The analysis in the experiments is confusing. For example, in line 403 and 404, MMLU (0.31)? Also, in line 407, mentioned in Ob3? I do not understand what the paper means.
4. Whether the analysis in this paper depends on specific models or not? If so, is it accurate to claim that it is the characteristic of MLLM? I wonder if the performance of FFN in MLLM plays a similar role.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: **Questions and Suggestions for the Authors:**

1. Could you add a definition for \( H_L \) and clarify what is meant by the “hidden activation function”? This addition would enhance the readability of the paper. Additionally, Table 3 currently appears a page before Table 2, which disrupts the logical flow.

2. Please clarify the implementation details for Stage I (modality separation) in your proposed method. Refer to the outlined weaknesses for specific questions and concerns. Additionally, is there supporting empirical evidence for this assumption in the existing literature?

3. Please consider adding a section detailing the computational requirements of your approach. Address Weakness 4 by including specifics such as GPU hours, memory demands, and a comparison with other commonly available explainability methods. This will provide the readers with a clearer understanding of the computational resources needed for your method.

4. Please conduct an ablation study using datasets of varying sizes to analyze whether the neurons are a subset of the original set.  Refer to Weakness 3.
soundness: 3
strengths: The proposed methodology has several notable strengths:

1. It explores the relatively unexplored area of identifying modality-specific neurons in Multimodal Large Language Models (MLLMs), offering insights into the internal workings of these complex models.

2. The authors provide comprehensive results across multiple MLLM architectures and datasets, accompanied by extensive ablation studies and in-depth discussions, demonstrating the robustness and applicability of their approach.
summary: The authors introduce a novel method, named MINER, to identify Modality-Specific Neurons in Multimodal Large Language Models (MLLMs). This approach comprises four main stages: 1. **Modality Separation**: The method begins by assuming that information within each modality's token set predominantly stays within that set, implying that cross-attention between modalities plays a minor role in MLLM tasks. 2. **Importance Score Calculation**: Instead of assessing the importance of individual activations for each sample, the method aggregates activations across tokens sets within a modality. The authors propose five distinct aggregation techniques for this step. 3. **Sample-Level Aggregation**: These aggregated importance scores are combined to generate a sample-level importance score. 4. **Neuron Selection**: Finally, neurons are ranked based on their importance scores, allowing for the selection of modality-specific neurons. This method offers a systematic approach to pinpoint neurons that are critical to individual modalities within MLLMs, advancing our understanding of how these models handle multimodal information.

The authors should the efficacy for their method on multiple tasks and multiple datasets.
weaknesses: In my view, there are several weaknesses in the paper's approach:

1. **Modality Separation Assumption**: The assumption of modality separation, as defined in the paper, is overly generalized and lacks robustness.

- **Dataset and Task Specificity**: Modality separation can only be reasonably argued for certain datasets and tasks. For example, it might be valid for specific questions in Visual Question Answering (VQA) but is unlikely to hold across all question types. Moreover, it seems especially problematic to assume modality separation for captioning tasks and datasets, where contextual understanding of the entire image is essential.

- **Dependence on Output Context**: Another critical consideration is whether modality separation should be treated as a function of the output. In captioning tasks, the entire image often correlates with the output, as generating captions requires broader contextual information. In contrast, the input (or prompt) for caption generation may hold less significance, further challenging the assumption of modality separation. Could you, if possible, conduct an ablation study to assess the significance of the output modality in the importance calculation?

- **Question on Importance Calculation**: This brings up a key question regarding the calculation of neuron importance: is the importance score determined solely based on the input, or does it also consider the model's output? If not, would including the model’s output in this calculation lead to different importance scores?

2. *Missing Definition*:  H_L is not defined in section 3.2. No reference is present in any of the equation 2 or 3 on the same page.


3. **Dataset Size Impact**: Table 2 illustrates that different datasets yield different neuron importance scores. However, it's not clear if varying the quantity of data within a single dataset also impacts these scores. For example, if using 100% of the dataset identifies important neurons \(N = \{n_1, n_2, n_3, n_4\}\), would using only 50% or 33% of the dataset result in a different set of important neurons, or would it merely be a subset of \(N\)?  If the importance neurons change, are they still importance neurons, or is there any underlying cause which we are not able to see.


4. **Computation Requirements**: The paper lacks an analysis of the computational resources needed to determine neuron importance. Since high-performance compute resources can be limited, an estimate of the required computational effort is essential to evaluate the feasibility of this explainability method.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: ### **Detailed Comments and Suggestions for Improvement**

**Major Comments**

1. **Insufficient Validation of Modality-Specific Neurons (MSNs)**  
   The deactivation experiments indicate that identified neurons are crucial for model performance, but this does not confirm they are modality-specific. If a neuron is equally essential across all modalities, its deactivation would similarly result in performance degradation without necessarily being MSN. Additional experiments should be introduced to verify the modality specificity of these neurons.

2. **Need for Cross-Modality MSN Overlap Analysis**  
   Given the inherent interactions between modalities, it would be valuable to analyze and present the extent of overlap among MSNs across different modalities. Showing the percentage of MSNs significant to multiple modalities and their distribution across layers would clarify the degree of cross-modality dependency.

3. **Lack of Practical Implications for MSN Discovery**  
   The discovery of MSNs is conceptually valuable, but the paper lacks a discussion on how these findings could practically benefit MLLM design or solve current challenges. Possible applications would increase the paper’s practical relevance.

4. **Potential Plagiarism in Figure 1**  
   Figure 1 (left) closely resembles Figure 1 of (Huo et al., 2024) but lacks proper citation. This raises concerns about originality.

5. **Explanation Gaps for Key Concepts**  
   Terms like **semantic probing** and **semantic telomeres** are introduced in the introduction without clear definitions, which can confuse readers. Providing definitions for these terms at the outset would improve clarity.

6. **Issues with Experiment Design and Metrics**  
   - **Typo**: There’s a typographical error on page 3, line 155 (“={text.”), which likely should be “={text, image}.”
   - **t-SNE vs. UMAP**: The paper utilizes t-SNE to compare embeddings across layers, but given t-SNE's stochasticity, UMAP could offer more consistent comparisons across layers, maintaining better global structure.
   - **Inconsistent Table Order**: Table 3 appears before Table 2, disrupting the flow and causing potential confusion.

**Minor Comments**

1. **Explanation for Confirmatory Experiment 5.5 (Hypothesis 1)**  
   Experiment 5.5 is referenced as supporting Hypothesis 1, but the link between the experiment’s findings and the hypothesis lacks clarity. A more detailed explanation would strengthen this connection.

2. **Typographical Issues in Figures and Captions**  
   - **Figure 6 Text Size**: The text in Figure 6 is too small, making it difficult to interpret. Larger font sizes would enhance readability.
   - **Axis Titles Missing in Figure 6(a)**: Figure 6(a) lacks x-axis titles, complicating interpretation. Clear labeling of axes would improve understanding.
   - **Overcrowded Plot in Figure 6(a)**: Multiple experimental results are displayed within a single figure, which is unconventional and may confuse readers. Separating these plots or including distinct legends for each result would improve clarity.
   - **Inadequate Captions for Figures 6(c), 6(d), and 6(e)**: These figures lack proper captions, making it challenging to understand the displayed results without referring back to the main text.

3. **Potential Typographical Error in Table 4**  
   In Table 4, the result for “Adaptive” under “Mean 1/2 and Max 1/2” is listed as 0.17, which appears unusually low and could be a typo. Clarifying this result would help in understanding the significance of the presented values.

4. **Experiment Suggestion: Varying MSN Deactivation Levels**  
   It would be informative to present results for different levels of MSN deactivation (from 0.5% to 5% at 0.5% intervals). This could reveal the relationship between neuron deactivation levels and performance, providing further insights into the impact of MSNs.
soundness: 2
strengths: - **Clarity and Organization**: The paper is well-structured, with a clear flow that makes complex concepts accessible.
- **Novelty**: The introduction of a framework specifically for modality-specific neuron detection in multimodal models is innovative.
- **Interesting Observations**: The identified phenomena, particularly semantic probing and semantic telomeres, add valuable insights to MLLM research.
- **Thorough Ablation studies**: The ablation studies showcase the impact of different strategies when various importance metrics are employed.
summary: The paper presents **MINER**, a novel framework designed to identify Modality-Specific Neurons (MSNs) within multimodal large language models (MLLMs). Through a structured, four-stage process, MINER attempts to localize neurons specific to each modality, aiming to uncover patterns in how different modalities interact within the MLLM framework. Key claims include the framework’s ability to select MSNs that, when deactivated, result in substantial performance drops, suggesting their importance in maintaining multimodal functionality.
weaknesses: - **Validation of MSN Specificity**: The evidence provided to confirm that the identified neurons are genuinely modality-specific (i.e., MSNs) is insufficient.
- **Implications Remain Unclear**: The practical implications and potential applications of identifying MSNs are not well-discussed, limiting the paper’s impact.
- **Data Presentation and Clarity**: Several figures, particularly Figure 6, are challenging to interpret due to their small size and lack of clear labels.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - Could you expand some of the terminological definitions, especially "semantic probing" and "semantic telomeres”?
- Could you compare MINER to other neuron-selection frameworks (including from unimodal domains) and clarify where your method reduces perceived gaps in benchmarking or explainability?
- Would it be possible to improve the legibility of all figures (especially Figs 5 and 6)
soundness: 3
strengths: - The MINER framework is relatively new, although information theoretic (e.g., mutual information-based) methods for probing arbitrary multimodal networks already exists so this is not as fundamentally revolutionary as it is evolutionary.
- The experiments are extensive and detailed, with Qwen2-VL and Qwen2-Audio models and multiple datasets. 
- The identification of "semantic probing" is not entirely new but “semantic telomeres” may be, potentially opening new avenues in XAI methods and may be generalizable across different modalities. It is highly advised to rename ‘telomeres’, though.
summary: The paper proposes a method (MINER) for identifying modality-specific neurons (MSNs) in multimodal LLMs. There are four stages to this process, and it is meant to improve the explainability (or transparency) of these large models. As few as 2% of neurons seem to play a key role in the multimodal connections. The authors invoke telomeres (!) to explain how modality-specific neurons interact across layers.
weaknesses: - The model should apply to modalities other than vision and audio, but the evaluation does not extend beyond these modalities. This is relatively minor, of course, since a paper that introduces a new method is not required to exhaust all empirical possibilities.
- The "modality separation" approach assumes minimal cross-modal information flow, which is a major oversimplification. It is not clear from an initial review to what extent removing this brick from the base of the theoretical structure collapses the rest. To a large extent this weakness is touched on in the first limitation in Sec 6.1, but it is not _addressed_. The natural interdependence between modalities, throughout the layers, should be better explained or explored. This is a more major limitation.
- It would be preferable to connect the concepts of ‘semantic telomeres’, for example, to real-world problems. What is the practical impact of this work?
- Other interpretability techniques for multimodal LLMs are not evaluated whatsoever.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. This paper is the first to investigate the concept of modality-specific neurons (MSNs) in MLLMs.  This is a significant and novel contribution that opens up a new avenue of research in the field of interpretability and explainability for multimodal models.    

2. This approach is shown to be effective in identifying MSNs that significantly impact model performance.  For example, deactivating just 2% of the MSNs identified by MINER significantly reduce MLLMs performance (0.56 ∼ 0.24 ↓ for Qwen2-VL, 0.69 ∼ 0.31 ↓ for Qwen2-Audio).

3. The authors uncover some impactful trends such as finding that different modalities tend to converge in the lower layers of the MLLM.  They also identify two phenomena, "semantic probing" and "semantic telomeres", which describe how audio and special modality tokens, respectively, interact with text modality tokens.
summary: This paper introduces a novel concept of modality-specific neurons (MSNs) within multimodal large language models (MLLMs).  These MSNs are specialized neurons that play a crucial role in processing information from a particular modality, such as image or text. The authors propose a framework named MINER to identify and analyze these MSNs, utilizing a unique token-level analysis pipeline to gauge the importance of neurons for specific modalities. This approach diverges from traditional sample-level methods and offers a more fine-grained understanding of how MLLMs process multimodal information.    

Through extensive experimentation on a variety of MLLMs and multimodal benchmarks, the authors demonstrate the significant impact of MSNs on model performance. Notably, they show that even a small reduction in MSNs can lead to noticeable performance drops, highlighting the critical role these neurons play. The paper also reveals intriguing phenomena such as "semantic probing," where audio modalities seem to reach out to text modalities, and "semantic telomeres," where special tokens anchor themselves to text modalities.
weaknesses: 1. A potential weakness of the paper lies in its approach to modality separation. The authors acknowledge the challenge of completely separating modalities due to the attention mechanism blending information across modalities. They hypothesize that information largely remains within its modality set as it passes through the LLM layers, with minimal transfer between modalities. They assume the attention mechanism functions only within distinct token sets, preventing information exchange between modalities. However, this assumption oversimplifies the attention mechanism, which is designed to capture dependencies regardless of modality. By limiting information flow, the authors may artificially constrain the model.

2. The authors restrict their analysis to neurons in the FFN module of the MLLMs, based on prior work highlighting the role of FFN in knowledge encoding. However, other modules, such as the attention layers, might also contain modality-specific neurons or contribute to modality-specific processing in ways not captured by this analysis.

3. The experiments primarily focus on image and text or audio and text combinations. Exploring more diverse combinations, such as image and audio or all three modalities together, could reveal a more complete picture of MSN interactions.

Meta-review: The paper proposes MINER, a framework for identifying modality-specific neurons. The paper is well-motivated and has an interesting concept with a novel direction. However, key shortcomings include unclear design choices for the experiments as well as unanswered questions regarding some of the analysis and findings. Additionally, the writing and presentation of the work require further improvement to make things more clear and easier to follow.

Decision: Reject