Capability Localization: Capabilities Can be Localized rather than Individual Knowledge

Authors: Xiusheng Huang, Jiaxiang Liu, Yequan Wang, Jun Zhao, Kang Liu
Primary area: interpretability and explainable AI
Keywords: Capability Localization; Knowledge Localization

Abstract:
Large scale language models have achieved superior performance in tasks related to natural language processing, however, it is still unclear how model parameters affect performance improvement. Previous studies assumed that individual knowledge is stored in local parameters, and the storage form of individual knowledge is dispersed parameters, parameter layers, or parameter chains, which are not unified. We found through fidelity and reliability evaluation experiments that individual knowledge cannot be localized. Afterwards, we constructed a dataset for decoupling experiments and discovered the potential for localizing data commonalities. To further reveal this phenomenon, this paper proposes a **C**ommonality **N**euron **L**ocalization (**CNL**) method, which successfully locates commonality neurons and achieves a neuron overlap rate of 96.42% on the GSM8K dataset. Finally, we have demonstrated through cross data experiments that commonality neurons are a collection of capability neurons that possess the capability to enhance performance. Our code is available at https://github.com/nlpkeg/Capability-Neuron-Localization.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730289692979

thread_type:
official_review

<thread_object>
creator: reviewer_wrao

<content>
summary:
The authors verify the assumption that knowledge is localized in the model parameter space by evaluating existing methods on a synthetically generated dataset. They falsify this assumption and instead argue that capabilities are localized in model weights and layers instead. Subsequently, the authors propose the Commonality Neuron localization method to find neurons that have commonalities within a variety of datasets. By manipulating these neurons, they causally show that the identified neurons that make up only 0.15% of the parameters significantly affects model performance.

strengths:
-- assessed 3 existing methods to verify knowledge representation in neural networks
-- developed a new method to categorize neurons that are selective to similar information to influence performance
-- The decoupling dataset and experiments are informative.
-- The authors used a wide variety of datasets (math, program, language) to evaluate their proposed CNL method.
-- the authors demonstrated that fine-tuning commonality neuron improved performance within and across datasets, demonstrating their casual role in performance.

weaknesses:
-- for Section 3.1 and 3.2, the authors used GPT4o to generate samples with the same semantic and attempted to identify localized representations in GPTJ. It will be informative to generate samples using GPTJ and verify localization methods on the same model i.e. GPTJ only. If this similarly shows low localization, it would further support the authors' claim. 
-- The authors used GPT4o to rewrite 1000 factual statements to populate the factual dataset. How did they verify that the new samples have the same semantic meaning?
-- Why did the authors not evaluate the CNL method on the same dataset i.e. Table 1, that was used to evaluate previous methods?
-- Authors should include the baseline performance of the model without any fine-tuning in Table 2 as in Table 3 for comparison. Is there a claim the authors are making with respect to Table 2?

questions:
-- what is the difference between knowledge and capabilities? 
-- Fig 2 is missing its x axis label. I am guessing it is the layer number?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582498243

thread_type:
official_review

<thread_object>
creator: reviewer_4n9w

<content>
summary:
This work introduces a Commonality Neuron Localization method as an alternative approach to overcome weaknesses of current methodologies in identifying and locating individual factual knowledge in large language models (LLMs) parameters.

After showing that current algorithms that localize individual knowledge do not respect fidelity and reliability requirements, the authors propose a metric to identify capability neurons for a given dataset based on a gradient attribution score of model parameters.

For some datasets, the authors show that Commonality Localization can be related to the performance of the model by showing that enhancing or erasing their role affects the results of the model on the dataset downstream task.

strengths:
The authors propose a change of perspective to address the current limitations of techniques that attempt to localize individual knowledge in LLMs parameters. They propose to focus on a coarser localization task that focuses on common traits specific to a dataset and task.

weaknesses:
The conceptual differences and limitations that incur when replacing individual knowledge location with "commonality neurons" are not sufficiently discussed. These are radically different procedures, with extremely different potential applications. The authors should thoroughly justify why it is relevant to localize phenomena at a much coarser level.

A more thorough discussion of the definition of attribution score would be beneficial, accompanied by a comparison to existing choices for attribution score appeared in the literature.

In the current form, the mapping between commonality neurons and capability is conjectural (see line 525-526). This is not coherent with the current main statement of the work "the located neurons embody the collection of capabilities".

questions:
It is not clear from the results if "Existing technology cannot localize individual model parameters" (line 71) or if "Individual knowledge cannot achieve parameter localization" in general. Clarifying this would be beneficial.

In order to support the claim that "the located neurons embody the collection of capabilities" it should be at least clarified explicitly which capabilities the authors are referring to. In particular, it should be stressed which of the quantitative results in Section 6 support the existence of the specific "common capabilities".

The results of Subsection 5.4 (in particular "Commonality across datasets") should be explained with more care. 

The paper could be largely improved in terms of clarity and quality of the presentation:
- extended and more precise Figure captions would help clarify results;
- the role of section 4.2 is not clear given the current form of the paper;
- in formula (8), line 329, the parameter sets $P_s$ should be clearly defined;
- in several circumstances, the authors use terminology incoherent with scientific rigor (e.g. "believes" or "ideologically" referring to existing literature);
- there are several typos and incomplete sentences in the current version.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680428091

thread_type:
official_review

<thread_object>
creator: reviewer_skue

<content>
summary:
This  paper explores the idea of localising “capabilities” within LLMs, rather than “individual knowledge”. The argument is that (existing methods for localizing) individual knowledge is unreliable. For instance, the authors show that current methods like parameter layers and chins are perfectly faithful to individual knowledge. For example, rewriting semantically similar prompts did not lead to consistent localisation results, and manipulating identified parameters did not reliably produce expected changes in model behaviour.

Instead the paper forwards a new framework for localizing “commonality neurons”, the neurons that represent shared attributes or capabilities across datasets. They introduce the **Commonality Neuron Locating (CNL) method** which identifies neurons that consistently activate for similar tasks or data types. The paper concludes that capabilities can be localised within LLMs, potentially leading to new ways of understanding and manipulating LLM functionality

strengths:
1. The paper is clear and well motivated. It clearly showcases the existing methods for understanding LLMs and their limitations.
2. The paper also addresses an important question in LLM research. How LLM parameters relate to their performance is topical. The paper challenges the existing methods providing compelling evidence. 
3. The paper also introduces a way of plugging those gaps. The CNL approach is a clear structured approach and inspired by well-grounded methods like integrated gradients from CV.

weaknesses:
1. The usage of arbitrary thresholds (magic numbers): The study uses a threshold of σ = 6 for identifying capability neurons without sufficient justification or exploration of alternative values. 
2. The decoupling experiment  to investigate the localisation of commonalities, primarily focuses on mathematical tasks. This narrow scope limits the generalisability of the findings regarding the potential for localising data commonalities. Expanding the decoupling experiment to include tasks from other domains, such as emotion recognition or language understanding, is crucial for supporting broader claims about the relationship between data commonalities and parameter localisation. 
3. The study shows that enhancing localised neurons leads to performance improvements, but the statistical significance of these improvements is not adequately assessed. Merely stating that the located neurons are "most sensitive" to performance improvement without providing statistical evidence leaves the strength of this claim open to question. 
4. The biggest weakness though is the glaring lack of common evaluation metrics. The study evaluates different knowledge localisation methods (ROME, KN, KC, and CNL) using distinct metrics tailored to each method's specific assumptions. Establishing a shared metric that can be applied consistently across all methods would enable a more objective and informative evaluation. Such a metric could consider factors like the accuracy of knowledge localisation, the granularity of identified parameters, and the impact on model performance when manipulating identified parameters.

questions:
1. The authors should provide justification for the specific hyperparameters chosen and clarify the methodology in more detail. For instance, in Section 5.1, line 352, why is S=19? Was this value selected through a hyperparameter sweep? If so, please include these details in the appendix. Similarly, in Section 5.1, line 358, why is the attribution score threshold set at σ=6? Offering detailed steps for identifying commonality neurons is essential to ensure that the results are not cherry-picked. This transparency will also help reproduce the findings independently.
2. For the random-neuron conditions in the “enhance” and “erase” experiments ([Table 2] and [Table 3]), did you evaluate across multiple random samples to ensure statistical robustness? If so, please indicate the number of samples tested; without this, the reliability of comparisons with targeted neurons remains unclear.
3. We need some more control conditions to convincingly demonstrate the impact of commonality neurons. For instance, a comparison with multiple sets of randomly selected neurons, averaged over several trials, would help determine if performance changes are indeed specific to commonality neurons rather than general effects from a neuron subset. Testing different thresholds for neuron activation and varying the size of neuron groups would also clarify whether the observed effects are specific to the identified commonality neurons. Such controls would strengthen the claim that these neurons contribute uniquely to model capabilities rather than reflecting random network fluctuations.
4. By far the biggest thing missing from the paper is a common metric for comparing different methods. It is crucial to benchmark CNL against established knowledge and capability localization methods, such as distributed parameters, parameter layers, and parameter chains. This comparison should involve measuring CNL’s accuracy, consistency, and computational efficiency relative to these approaches in reproducible way such that improvements can be tracked carefully. 

5a. Another major concern is the scalability of CNL. There are at least 3 issues with model scalability and the authors need to discuss (or even better perform new tests) the scalability strategies of CNL.  A) As new models grow larger (e.g., from billions to hundreds of billions of parameters), the CNL method’s neuron-level analysis could become computationally prohibitive. 

5B. Larger models might need different threshold and hyperparameter sweeps to maintain CNL’s accuracy which demand additional tuning efforts. 

5C. CNL relies on extensive data from various domains to identify commonality neurons. As model sizes increase, the amount of data needed for localization will also grow. 

6. The authors should clarify how capability localization could enhance model interpretability in practical applications. It remains unclear whether the CNL approach addresses the theoretical question of how information is localized within the model or if it offers actionable changes for real-world LLM applications. Specifically, the authors should explain how identifying capability neurons could be leveraged to improve model performance, robustness, or transparency in practical scenarios. Without clear guidance the potential of using CNL in real-world settings is ambiguous.
7. While the paper includes extensive mathematical notation and text, the figures are under-explained and hard to follow. For instance, it is unclear what Figure 1 is intended to convey, the purpose of the red boxes in Figure 3, or the significance of the colors in Figure 4. The authors should enhance these figure descriptions to make the visuals more accessible and informative for readers, clearly explaining all elements, symbols, and color schemes. This will improve readability and ensure that the figures effectively support the paper’s findings.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698331065

thread_type:
official_review

<thread_object>
creator: reviewer_dq76

<content>
summary:
The manuscript investigated the storage of individual knowledge in model parameters, and proposed a Commonality Neuron Localization method that locates commonality neurons.

strengths:
1. The manuscript conducts experiments of previous knowledge localization methods and demonstrates those methods lack reliability and fidelity. The finding is insightful to the community.
2. The manuscript proposed a method that can effectively identify those neurons that embody the model capability, which help the understanding of large-scale neural networks.

weaknesses:
1. It is unclear to me what the author means by "commonality". And what is its relation to knowledge localization?
2. Figure 2 is blurry.
3. In line 182, $e_i$ seems to be undefined.
4. Missing reference in line 216.

questions:
1. line 344, "we expand the KN method", what is the key improvement of CNL compared to KN?
2. What is the conclusion of Table 2?
3. The method successfully identifies those commonality neurons that embody the model capability. However, why is finding these neurons useful? For example, do we have any insights from analyzing the characteristics of the commonality neurons?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732023258156

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank the reviewers for their thoughtful feedback. We are glad the reviewers find that:
* Our motivation is innovation and has great significance
    * "The finding is insightful to the community." - dq76
    * "The paper is clear and well motivated." - Skue
* Our approach presents a new perspective on the research question
  * "It clearly showcases the existing methods for understanding LLMs and their limitations." - Skue
  * "The authors propose a change of perspective to address the current limitations of techniques.“ - 4N9w
* Our approach is effective and has a wide range of application in real-world scenarios..
  * "which help the understanding of large-scale neural networks." - dq76
  * "Developed a new method to categorize neurons." - wrao

**[Main Motivation]**

Large-scale language models have achieved superior performance, however, it is still unclear how model parameters affect performance improvement. Despite recent interesting hypotheses suggesting that individual knowledge is stored on local parameters, extensive experimental evidence has shown that previous hypotheses are unreliable. Through decoupling and capability neuron localization experiments, we have demonstrated that capabilities can be localized. Fine-tuning 0.15% of capability neurons will significantly improve model performance, this proposes a new norm of "Localization + Fine-tuning".  As highlighted by reviewer dp76, this work holds considerable significance.

**[Supplementary Experiments]**

* **1. Accuracy experiment.** We performed localization experiments on more settings:

**1.1** A larger model -> Llama2-13B

|ratio|GSM8K|Emotion|Code25K|Meta_Math|Imdb|
|:-:|:-:|:-:|:-:|:-:|:-:|
|overlap|94.26|94.68|91.10|95.37|98.32|
|IoU|88.92|89.79|83.62|91.10|96.68|
|neuron|0.10|0.19|0.11|0.09|0.08|

**1.2** A different model -> GPTj-6B
|ratio|GSM8K|Emotion|Code25K|Meta_Math|Imdb|
|:-:|:-:|:-:|:-:|:-:|:-:|
|overlap|95.66|87.62|91.25|83.62|98.27|
|IoU|91.63|77.96|83.89|71.77|96.60|
|neuron|0.28|0.19|0.11|0.27|0.26|

**1.3** Different $\sigma$ with llama2-7B:

$\sigma = 3$
|ratio|GSM8K|Emotion|Code25K|Meta_Math|Imdb|
|:-:|:-:|:-:|:-:|:-:|:-:|
|overlap|97.21|97.35|89.88|93.47|98.76|
|IoU|94.49|94.84|81.61|87.56|97.55|
|neuron|0.46|0.62|0.30|0.44|0.36|

$\sigma = 12$
|ratio|GSM8K|Emotion|Code25K|Meta_Math|Imdb|
|:-:|:-:|:-:|:-:|:-:|:-:|
|overlap|97.53|98.03|95.49|93.15|97.92|
|IoU|95.16|96.12|91.34|87.10|95.92|
|neuron|0.05|0.06|0.03|0.05|0.04|

* **2. Enhance experiment.** We extended our enhancement experiments (For convenience, we only list the results under the last epoch here.). The best results are in **bold** and `highlighted` means
the suboptimal.

**2.1** A different $\sigma$ with llama2-7B:
$\sigma = 3$
|Method|GSM8K|Emotion|Code25K|$Avg.$|
|:-:|:-:|:-:|:-:|:-:|
|random|23.75|`26.79`|`53.47`|`34.67`|
|w/o located|`25.19`|19.29|42.77|29.08|
|located|**26.31**|**51.62**|**56.02**|**44.65**|

**2.2** A different model -> GPTj-6B:
|Method|GSM8K|Emotion|Code25K|$Avg.$|
|:-:|:-:|:-:|:-:|:-:|
|random|25.69|28.54|`51.07`|35.10|
|w/o located|**32.00**|**38.71**|48.50|`39.73`|
|located|`27.38`|**48.58**|**52.53**|**42.83**|

* **3. Erase experiment.**  We simultaneously perform erase experiments on Llama2-13B:
$\sigma=6$  the specific experimental results are in our updated Table3.

|Dataset|GSM8K|Emotion|Code25K|$Avg.$|
|:-:|:-:|:-:|:-:|:-:|
|Base acc.|0.38|31.96|45.95|26.10|
|random|0.31 ($\downarrow 0.07$)|33.75 ($\uparrow 1.79$)|45.81 ($\downarrow 0.14$)|26.62 ($\uparrow 0.52$)|
|locate|0.00 ($\downarrow 0.38$)|5.38 ($\downarrow 26.58$)|18.06 ($\downarrow 27.89$)|7.81 ($\downarrow 18.29$)|

* **4. Verification experiment.**   We evaluated the image of enhanced neurons on other tasks that are not involved
in training. The following represents the performance improvement of the model in other tasks:

|Model|random|w/o located|located|
|:-:|:-:|:-:|:-:|
|LLama2-7B $\sigma=3$|5.20|-5.58|9.43|
|LLama2-7B $\sigma=6$|3.20|-3.31|11.18|
|GPTJ-6B $\sigma=6$|9.00|14.73|17.66|

We find that more accurate localization methods cause less damage or can even benefit downstream tasks.

* **5. Convergence experiment.** We validated the relationship between different data volumes and model sizes.

Llama2-13B&Emotion[1]:

|Num_data|100|200|300|400|500|600|700|800|900|1000|1100|1200|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|overlap|89.51|93.60|87.56|83.94|95.56|95.37|94.21|95.72|91.76|95.75|94.01|94.68|
|IoU|80.51|87.94|76.35|69.30|91.48|91.11|88.97|91.66|84.22|91.84|88.63|89.79|

GPTj-6B&Emotion[1]:

|Num_data|100|200|300|400|500|600|700|800|900|1000|1100|1200|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|overlap|45.67|79.04|83.85|65.92|77.81|85.85|80.99|77.68|87.82|84.28|91.73|87.62|
|IoU|29.46|65.26|72.18|49.11|63.64|75.20|68.04|63.43|78.26|72.82|84.70|77.96|

**The Rebuttal revision provides more detailed experimental results.**

*Ref*

[1] Context based emotion recognition using emotic dataset. IEEE Trans 2019.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733032048777

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thank you for your previous suggestions! As the discussion period is coming to an end, do you have any additional questions? We welcome your further inquiries.

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper makes an interesting claim that existing forms of individual knowledge storage are all inaccurate. Instead, the author presents a new framework for localizing “commonality neurons”, the neurons that represent shared attributes or capabilities across datasets. They introduce the Commonality Neuron Locating (CNL) method which identifies neurons that consistently activate for similar tasks or data types. The paper concludes that capabilities can be localized within LLMs, potentially leading to new ways of understanding and manipulating LLM functionality.

The majority (3/4) reviewers rated favorably towards an acceptance, however all marginally. I recommend acceptance for its rather novel insights, and strong rebuttal.