Capability Localization: Capabilities Can be Localized rather than Individual Knowledge
Authors: Xiusheng Huang
Jiaxiang Liu
Yequan Wang
Jun Zhao
Kang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Capability Localization; Knowledge Localization
Primary area: interpretability and explainable AI

Abstract: Large scale language models have achieved superior performance in tasks related to natural language processing, however, it is still unclear how model parameters affect performance improvement. Previous studies assumed that individual knowledge is stored in local parameters, and the storage form of individual knowledge is dispersed parameters, parameter layers, or parameter chains, which are not unified. We found through fidelity and reliability evaluation experiments that individual knowledge cannot be localized. Afterwards, we constructed a dataset for decoupling experiments and discovered the potential for localizing data commonalities. To further reveal this phenomenon, this paper proposes a **C**ommonality **N**euron **L**ocalization (**CNL**) method, which successfully locates commonality neurons and achieves a neuron overlap rate of 96.42% on the GSM8K dataset. Finally, we have demonstrated through cross data experiments that commonality neurons are a collection of capability neurons that possess the capability to enhance performance. Our code is available at https://github.com/nlpkeg/Capability-Neuron-Localization.

Published as a conference paper at ICLR 2025
CAPABILITY LOCALIZATION: CAPABILITIES CAN BE LOCALIZED RATHER THAN INDIVIDUAL KNOWLEDGE
Xiusheng Huang" 1,2,3 Jiaxiang Liu" 1,2, Yequan Wang Jun Zhao and Kang Liu
'The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences
"School of Artificial Intelligence, University of Chinese Academy of Sciences Beijing Academy of Artificial Intelligence, Beijing, China
huangxiushheng222200 ac. cn, liujiaxiangee @mailmmlle 1s ucas. ac cn, Eshwangyequandgmaai com, {jzhao, k kliujen pr.eenppr..e pr. ia. ac. on
ABSTRACT
Large scale language models have achieved superior performance in tasks related to natural language processing, however, it is still unclear how model parameters affect performance improvement. Previous studies assumed that individual knowl- edge is stored in local parameters, and the storage form of individual knowledge is dispersed parameters, parameter layers, or parameter chains, which are not unified. We found through fidelity and reliability evaluation experiments that individual knowledge cannot be localized. Afterwards, we constructed a dataset for decou- pling experiments and discovered the potential for localizing data commonalities. To further reveal this phenomenon, this paper proposes a Commonality Neuron Localization (CNL. method, which successfully locates commonality neurons and achieves a neuron overlap rate of 96.42% on the GSM8K dataset. Finally, we have demonstrated through cross data experiments that commonality neurons are a collection of capability neurons that possess the capability to enhance performance. Our code available
Nenuuron-Locallzttionn
INTRODUCTION
Large scale language models (LLMs) have received widespread attention due to their superior performance in the field of natural language processing (Zhao et al. [023] Yao et al. 2023). Although LLMs have demonstrated extraordinary capabilities, humans are still unclear about the relationship between model parameters and superior performance (MacAskill & Kittler 2010). More and more
research is focusing on the security (Bonaldi et al.) 2024) Sun et al. 2024), ethics (Yan et al. 2024 Haltaufderheide & Ranisch 2024), and potential performance of models, with the black box nature of models being the main limitation of these studies (Guidotti et al. 2018; Huang et al. 2025).
Therefore, establishing a mapping relationship between the internal parameters and capabilitiess of the model is becoming increasingly important (Ding et al. 2023 Huang et al. 2024).
Recent research has mainly focused on the correspondence between individual knowledge and parameters (Ledeen et al. [976). Specifically, KN (Dai et | believes that individual knowledge stored on distributed parameters, and the validity of the conclusions verified by increasing or zero the activation. ROME (Meng et al. 2022a) believes that individual knowledge is stored on the entire parameter layer, and then utilizes knowledge editing techniques to prove that modifying the parameters of the entire layer modify individual knowledge. Finally, the KC (Yao et al.) 2024) assumes that individual knowledge is stored in a parameter chain and utilizes the entire parameter chain to recall the knowledge. Different jobs believe that individual knowledge has different storage forms, which has caused the following difficulties for researchers: (A) What is the storage form of individual knowledge? (B) If the existing locating methods inaccurate, can individual knowledge really be parameter localized?
"Equal contribution.
Corresponning authors.

---

Published as a conference paper at ICLR 2025
To address the above challenges, we design experiments for evaluating fidelity and reliability (Veh et al. 1995). Specifically, for the fidelity experiment, we assume that utilizing the rewritten individual knowledge prompt, the similarity of the located neurons should have a higher degree of overlap compared to the original (Trimmer 2013). Therefore, we constructed a knowledge rewriting dataset to evaluate the fidelity of previous knowledge localization methods individual knowledge. The experimental results show that the coincidence degree of causal tracing (Meng et al. 2022a) in ROME the highest, reaching 37.3%, while the coincidence degrees of KN (Dai al. 2021) and KC (Yao) et al. 2024) are 27.5% and 72,, respectively. This indicates that previous knowledge localization methods were not faithful to the same individual knowledge. The answer to question A is: The
existing forms of individual knowledge storage are all inaccurate.
Previous knowledge localization methods have proposed corresponding validation methods, reliability experiments will evaluate the reliability of these methods. Firstly, to verify that individual knowledge stored on distributed parameter, KN (Dai et al. (02T) verifies the correspondence between the individual knowledge and the parameters by increasing zeroing the activation. Our reliability experiments indicate that increasing or zeroing activation does not necessarily lead to an enhancement or weakening of corresponding knowledge. At the same time, we utilize corresponding models to demonstrate that there is no strong correlation between the parameters and knowledge. Secondly, to
verify that individual knowledge is stored the parameter layer, ROME (Meng et al. [2022a] edits the knowledge by updating the parameters of the entire layer (Peng et al. [0224] Liu et al. 2024).
However, reliability experiments can still achieve knowledge editing by editing other layers (not localized layers). Finally, to verify that individual knowledge is stored on the parameter chains, KC (Yao et al., 2024) utilizes the localized parameter chains recall the knowledge. The reliability experiment proves that the accuracy of using the parameter chain recall knowledge for localization (top k=1) is 12.3%. In addition, the entire parameters chain occupies 2.6% of the overall model parameters, and the granularity of knowledge and parameters does not match, which does not indicate the correspondence between parameters and knowledge. Using reliability experiments, the answer to question B is: Existing technologies cannot localize individual knowledge parameters.
To further reveal the form of knowledge storage, we designed 1000 comparative samples and conducted decoupling experiments. For example, the comparative samples include subsample I "providing the correct answer for 1+1=?" and subsample "programming as Python code for 1+1=?"' This comparative sample contains the same main part "1+1=?", which also tests the mathematical and programming capabilities of the model. Utilizing gradient response analysis (Hinterstoisser al. () it is found that the coincidence rate of response parameters between subsample 1 and subsample 2 was 15.6%. Interestingly, we find that the coincidence rate of response parameters for all subsample 1 was 7.3%, which is 8.6% for all subsample 2. Individual knowledge cannot achieve parameter localization, can the commonality of data be achieved?
To verify this hypothesis, we propose a Commonality Neurons Locating (CNL) method. We utilize samples of the same type to obtain commonalities in the data and successfully located commonality neurons. Furthermore, we demonstrate through cross datasets experiments that commonality neurons are collection of capability neurons that possess the capability to enhance performance.
To our knowledge, we are the first to prove the unreliability of existing individual knowledge localization conclusions and point out that the capability can achieve local parameterization, which will help reveal the utility of internal parameters in the model. Our contributions can be summarized as follows:
In order clarify the storage form of individual knowledge, we conduct fidelity and reliability evaluation experiments. The experimental results indicate that existing localization methods are not faithful to individual knowledge. At the same time, existing validation methods cannot support parameter localization with individual knowledge.
To further reveal the form of knowledge storage, we conduct decoupling experiments. The experiment find that the coincidence rate of response parameters between subsamples with the same main part is only 15.6%, while the coincidence rate of response parameters between samples that test the same capability of the model is 8.6%, which means that the commonality of data corresponds to parameters.
We propose a Commonality Neurons Locating method and successfully located commonality neurons. Furthermore, we demonstrate through cross datasets experiments that commonality
2

---

Published as a conference paper at ICLR 2025
neurons collection of capability neurons, and establish mapping relationship between capabilities and parameters.
BACKGROUND
Recent research has mainly focused on the correspondence between individual knowledge and parameters, and suggests that the storage forms of individual knowledge distributed parameters. parameter layers, or parameter chains. Next, we will introduce three locating methods and their corresponding validation experiments.
2.1 DISTRIBUTED PARAMETERS
Locating Method. Given input prompt [01,-' ,xx], first define the model output
Pawy (ux) as the probability of the correct answer predicted by pretrained model:
Pa (w))) = ply" (20, w [x]] w(x
(I)
where y" denotes the correct answer, wix is defined in the Appendix B and is given constant that the output w [x]] is assigned to. In order to calculate the attribution score of neuron Attr(w/d), calculated by the pretrained model, and we gradually change w [ar] from 0 its original value w [ac] meanwhile integrate the gradients:
f a=0
OP.. (owt)(x)) ow [x]
Attr(w w
da
(2)
where 8P... (ow)((()) calculates the gradient of the model output with regard to why, As changes 8w
from 0 to 1, by integrating the gradients, (w!") accumulates the output probability change caused by the change w [x:] If the neuron has a great influence on the expression a fact, the gradient will be salient, which in turn has large integration values. Therefore, the attribution score measure the contribution of the neuron wht to the factual expressions.
Verification Experiment. Given a relational fact, we manipulate its knowledge neurons in two ways: (1) suppressing knowledge neurons by setting their activations to O; (2) amplifying knowledge neurons by doubling their activations.
2.2 PARAMETER LAYERS
Locating Method. Similar the causal tracing (Meng al. 2022b). , clean run that predicts the fact, corrupted run where the prediction is damaged, and a capability of a single state to restore prediction. More details are displayed in Appendix C
with-restoration run that tests
Verification Experiment. This method assumes that individual knowledge is stored in the parameter layer and validated using knowledge editing techniques. Specifically, by updating the entire parameter layer, the original knowledge of the model is changed and assumed to be stored in that layer.
2.3 PARAMETER CHAINS
Locating Method. KC Yao et al. 2024 believes that individual knowledge is stored parameter chain and utilizes the entire parameter chain to recall knowledge. More details are shown in Appendix
D
the score S(e;) is less than the predefined threshold, the e; refers to calculating the edges in graph G, they consider the edge to be non-critical and remove it from the computation graph, updating the

---

Published as a conference paper at ICLR 2025
Fidelity Evaluation Experiments
ger (HETT
of Conelee
Reliability Evaluation Experiments
Edit
Q
Q
Fig. 1: Overall framework diagram of fidelity and reliability evaluation experiments. Experiment A represents the visualization results of samples with the same semantic localization, while experiment B represents the impact of operating localization neurons on performance
temporary circuit. They first sort the graph by topological rank and traverse all edges in this manner, they derive circuit C2 that contributes to representing the knowledge necessary to answer the factual question:
== NR, Fa. >
where Cj is the circuit for the knowledge triplet k, consisting of the nodes N₂ and edges Eg.
Verification Experiment. To verify that individual knowledge is stored on the knowledge chain, KC (Yao al.] 2024) utilizes the parameter chain of localization recall the knowledge. They measure the rank of the target entity among the top 10 predicted tokens.
3 EVALUATE INDIVIDUAL KNOWLEDGE LOCALIZATION METHODS
We will introduce fidelity and reliability evaluation experiments this section.
3.1 FIDELITY EVALUATION EXPERIMENTS
As show in Fig the fidelity experiment is mainly aimed verifying the fidelity of existing individual knowledge localization methods to individual knowledge. We observe the overlap score of each knowledge localization result by rewriting individual knowledge prompt 5 times. The following experiments are uniformly set as follows: base model is GPTJ (Achiam et al. 2023) model.
Rewritten Dataset, We first randomly select 1000 factual samples from the COUNTERFACT (Meng et al. (022a) dataset, and rewrite the samples 4 times using GPT4o (Achiam al. 2023). We will obtain samples (D) Day. E [1,2, 1000) E
with the same semantic meaning.
Distributed Parameters. Kn DDi et al. 2021 believes that individual knowledge is stored
few parameters, with an average of 15.3 localized parameters found per sample. We will match the localization results of 5 samples ( j E 2.......)) with the same semantic meaning pairwise. As a result, is found that the overlap of localization parameters for samples with the same semantic meaning was only 37.3%. which clearly does not meet the expectation of locating individual knowledge.
Parameter Layers. ROME (Meng et al. 2022a) believes that individual knowledge is stored on the parameter layer, and we select the top k=3 parameter layer as the localization result. We will match the localization results of 5 samples with the same semantic meaning pairwise. As a result, it is found that the overlap of localization parameters for samples with the same semantic meaning was only 32.7% %. At the same time, the parameter layer accounted for a large proportion the entire model, and the granularity of individual knowledge and parameters did not match.
4

---

Published as a conference paper at ICLR 2025
Parameter Chains. KC Yao et al., 2024) believes that individual knowledge is stored on parameter chain, and we compare the parameter chains of sample localization with the same semantic meaning. Determine the fidelity of the locating method based on whether the parameter chains the same. The experimental results show that the coincidence degree of the positioning parameter chain obtained using KC (Yao et al. 2024) is only 7.2%. In addition, the average proportion of parameters in the parameter chain to the entire model is 1.6 %, and the granularity of individual knowledge and parameters does not match.
Three fidelity experiments indicate that existing single knowledge localization methods are not faithful to knowledge. Samples with the same semantic meaning cannot obtain identical or highly similar localization results using localization knowledge.
3.2 RELIABILITY EVALUATION EXPERIMENTS
As show in Fig I the existing single knowledge localization methods all have separate validation methods. Below we will introduce reliability experiments for locating methods. The following experiments are uniformly set as follows: base model GPTJ (Achiam al. 2023) model, dataset factual dataset zsRE (Levy et al. 2017).
Distributed Parameters. KN Dai et al. 2021) verifies the effectiveness the localization param- eters by setting the activation value to zero or increasing it. We randomly select equal amounts of parameters, set them to zero or increase the activation value, and compare the experimental results with the located parameters.
The experimental results show that when we double the localized activation values, 85.1% % of samples increase their target probabilities, while 14.9% decrease. For random activation values, 72.3% of samples see an increase, and 27.7% decrease. Additionally, when we set localized activation values to zero, 27.4% of samples increase their target probabilities, while 72.4% decrease. For random activation values, 17.6% % of samples increase, and 82.6% decrease. The experimental results are similar to the random activation values, which cannot fully demonstrate the effectiveness of the verification method.
Parameter Layers. ROME (Meng al.. 2022a utilizes knowledge editing techniques verify the effectiveness of the parameter layer. ROME chooses to edit the parameters of the 5- layer for validation, and utilizes the changes in the model's output basis to prove that a single knowledge exists single layer. In Fig 2 randomly edited all layers and found that editing other layers resulted similar per- formance, indicating that the parameter layer validation method ineffective.
The impact of editing different layers(ROME)
Nerformenc
0.6
rephrase acc neighborhcod acc
Parameter Chains. KC Yao al. 2024 uti- lizes the parameter chain recall knowledge after localization verify the effectiveness of local- ization. We utilize the parameter chain recall knowledge after localization select the candidate items with rank=l that are equal the target sam- ples successful samples. The experimental results showed that the success rate of recall knowledge was only 12.3%. At the same time, we find that the samples with successful recall contains large number of parameters, accounting for an average of 2.6 % of the entire model parameters.
The performance of editing at different layers, the horizontal axis is layer number.
Fig. 2:
4 DECOUPLING EXPERIMENT
By evaluating previous knowledge localization methods, we find that none of the existing methods convincing. In order further reveal the form of knowledge storage, design decoupling experiments to decouple different factors of knowledge and explore the corresponding relationships between parameters and indicators.
5

---

Published as a conference paper at ICLR 2025
Decoupling Datasets
Subsammle
Subsample
Subsample1:
"provides the correct anawer for 1+1=?"
Subsample2:
"programs Python code for 1+1=?"
part
provides
programs Python code
Replaceable parts
1+1 =?
2-1 =
3
9/3=
Overall framework diagram of decoupling experiments. The upper half represents the overlapping neurons for obtaining the localization of subsamplel and subsample2, while the lower half represents the composition of the subsample, consisting of the main and replaceable parts.
Fig. 3:
4.1 DECOUPLING DATASETS
We design 1000 comparative samples and conduct decoupling experiments. Each comparison sample contains two sub samples, whose replaceable parts remain consistent but differ in main parts. This helps better analyze the correspondence between the parameters and indicators within the model. Specifically, as show in 3ig we utilize mathematical calculation formulas as the replaceable parts, with task requirements for direct computation and code programming. For example, subsample "provides the correct answer for 1+1=?" and subsample 2 "programs Python code for 1+1=?",
4.2 POSITIONING DECOUPLING DATA
Given an comparative prompt {=t ) we first define the model output Pai.ut (wx.) as the probabil- ity of the correct answer predicted by a pretrained model:
Petn (wx) p(vi
plyil w [x+]] = w
where vi denote the correct answer, w/ denotes the j-th intermediate neuron in the 1-th FFN; x given constant that w'X' [xt]] is assigned to. In order to calculate the attribution score of neuron (ar(())))))) gradually change w// [x]]] from 0 its original value w/y [xt]] calculated by the pretrained model, and meanwhile integrate the gradients:
OP. Poi (w)x)) ow 1J
We [xr]]
do
(5)
Jamo
[x1]
where OP
calculates the gradient of the model output with regard to wht As changes
D [=]]]
from to 1, by integrating the gradients, Attr(w! (af)) accumulates the output probability change caused by the change of w [ar]]. If the neuron has great influence the expression of fact, the gradient will be salient, which in turn has large integration values. Therefore, the attribution score can measure the contribution of the neuron w the factual expressions.
Obtain the coincidence rate CF: of the parameter set P, by comparing the parameters of the sample localization
En= (Pa Att ( (P.jAtrooo)))) t t= t,t" E {1,2} (6)
W.
C.J Ni (tt
where the P, refers to the located parameters. The coincidence rate C of response parame- ters between subsample and subsample 2 is found to be 15.6 %. Interestingly, find that the

---

Published as a conference paper at ICLR 2025
coincidence rate of response parameters for all subsamplel was 7.3%, which is 8.6% for all subsam- ple2. Individual knowledge cannot achieve parameter localization, can the commonality of data be localized?
IDENTIFYING CAPABILITY NEURONS
In this section, we propose a Commonality Neurons Locating (CNL) method. By utilizing the parameters of localization, significantly enhance the corresponding capabilities of the model.
5.1 CNL METHOD
To find out the capability neurons under the dataset D {2 [r1,.... ,xx,, ... ..ry D}. we expand the KN method (Dai et al., 2021 Yao et al. 2024 to compute the contribution of the neuron as:
0P: Mmm (5w %m [Zml] (. 7m [=m]
Score(w (e,y)D
Zm 3m
YS
Ow
2m + 90.m-1
where ① means splice of two text. The above equation 7 simulates the expectation of gradient different outputs of the neuron under the dataset D. In our experiment let step S 19. To identify the commonality neuron, we take the M matrix:
Maski =
{ 1
|Score(w!i) mean(Score(w))) else
vva((Scor(w))
(8)
where mean(-) denotes the mean value of all scores and var(.) indicates the variance the neurons. is the threshold guiding to find the commonality neurons. In the absence of any special instructions follow, we view the neurons with scores outside a as commonality neurons.
5.2 EXPERIMENT
Our experiment mainly has the following two findings: 1) find that set of data has certain commonalities, which reflected in the manner of neurons. 2) This commonality crosses over datasets, reflecting the capabilities of the model.
Datasets.
In the experiment, utilize three types of datasets that reflect the commonality on the
data.
Math: (1) GSM8K (Cobbe et al. 2021) contains approximately 8,000 elementary math problems with detailed solutions, designed to train mathematical reasoning models; (2) Meta_Math (Yu al. (203) focuses on meta-learning for math problems, aims enhancing the model's adaptive learning and reasoning capabilities. In order to solve this task, need the model to be mathematically competent, as well as have some multiple choice and language comprehension skills.
Program: Code25K (Beguš) 2021) contains around 25,000 code snippets, supporting tasks like code generation and completion. order solve this task, we need the model have program capability along with comprehension.
Language: (1) Emotion (Kosti et al. 2019) with text data labeled with various emotions, suitable for sentiment analysis tasks, including social media posts and comments; (2) Imdb (Tripathi et al. 2020) contains movie reviews and ratings, widely utilized for sentiment analysis and recommendation system research. In order to accomplish this task, we need the model to be capable of sentiment analysis, as well as some multiple choice and language comprehension.
7

---

Published as a conference paper at ICLR 2025
5.3 COMMONALITY NEURON EXPERIMENT
Commonality Neuron Locating. We utilize 1600 GSM8K, 2400 Emotion, 1200 Code25K, 700 Meta_Math and 800 Imdb data for commonality neuron locating. When locating neurons, we divide the entire dataset into two subsets, and b, and calculate the overlap and LoU. The indicator neuron refers to the proportion localized parameters to MLP parameters.
landl
lanbl [6]
a n
overlap
(a]
TOU
lau
E E/=1 Maskij
neuron
L.J
The results obtained are shown Table [
We find that both overlap and IoU have high values (like 96.42% and 95.95%), indicating set of data has certain commonalities, which is reflected by the neurons. At the same time, the low number of neuron indicates granularity matching parameters and commonalities.
Model
ratio overlap
GSM8K
Emotion
Code25K
Meta Math
Imdb
96.42
97.93
90.14
94.33
95.81
Ilama2-7B
ToU
93.08
95.95
82.05
89.15
91.96
neuron
0.14
0.19
0.11
0.14
0.19
overlap
94.26
94.68
91.10
95.37
98.32
Llama2-13B
IoU
88.92
89.79
83.62
91.10
96.68
neuron
0.10
0.19
0.11
0.09
0.08
overlap
95.66
87.62
91.25
83.62
98.27
GPTJ-6B
ToU 91.63
77.96
83.89
71.77
96.60
0.28
0.19
0.11
0.27
0.26
Table Neurons overlap ratio and the proportion of targeted neurons.
The experimental results above indicate that the neurons we find can reflect commonality in the data. This commonality is not only confined to single data, but reflects shared attribute of dataset. is independent of our partitioning method of the dataset. And the Appendix E illustrates the change in localization accuracy with the variation in the amount of data.
5.4 CAPABILITY NEURON EXPERIMENT
In the following experiments, prove that the commonality neurons related to the perfor- mance of the model and validate the potential of CNL.
Proving I: Enhance Experiment. The above experiment demonstrates that have successfully located neurons with commonality. To verify the effectiveness the commonality neurons, we conduct enhance experiments. Specifically, fine-tune the training set and evaluate the validation set. Adam (Kingma (014)) is selected as the optimizer algorithm with Ir=le-5, and the optimized parameters are follows:
Random: We randomly select neurons that are consistent with the number of the located neurons, the proportion of occupying the overall parameters of the neurons is 0.15%.
W/o located: We mask the located neurons (set their parameter to 0) and fine-tune all other neurons, the proportion of occupying the overall parameters of the neurons is 99.85%.
Located: We fine-tune the located neurons, the proportion of occupying the overall parame-
ters of the neurons is O. 15%.
8

---

Published as a conference paper at ICLR 2025
Model
Metoo
CSSHK
Enccion
spach
Code25K
Asg-
GSMWK
Enction
spach -5
Code25K
Ars.
GSMHK
Emotion
epoch
Code25K
10
.Amg-
Raniom
0.00
14.62
50.79
22.47
0.00
14.62
52.90
22.51
5:25
14,99
53.05
24.43
Llumal 7B
Wio located
25.35
44.43
28.55
39.93
26.67
46.48
50.51
Located
23.52
23,57
54.28
34.12
24.79
32.33
55.57
37.96
25.75
55.68
41.12
14.114
52.85
24.31
22.35
25.75
26.70
36.37
18.29
19.20
GPT1-68 (-
Located
located
23.31
24.75
30,00
23.00
51.48
36.74
26.38
33.67
31.50
52.42
36.77
35.26
27.18
48.58
38.71
$2.53
39.73
Table Enhancement of different sets of neurons. the suboptimal.
The best results are in bold and underline means
As shown Table 2, the enhancement experiment is validated on two models, Ilama2 and GPTJ. Compared random and w/o located, our located has achieved superior performance in multiple datasets. For example, the llama2-7B model achieves 5.17% % improvement in the Aug metric for epoch 1, while the GPTJ-6B model achieves 9.87% improvement the Emotion dataset for epoch 10. We can find that we only need to update a small portion of parameters to effectively improve the performance of the current task. We also plot the loss curve for Llama2-7B
in Appendix F]
Proving II: Erase Experiment. To further verify the correlation between commonality neurons and model performance, we conduct crasure experiments. Specifically, we choose different thresholds (a E [3, 6, 12)) compare the performance changes.
Model
Llama2 7B
Llamaz B3B
Emotion
Codr25K
Cudk25K
Hosc accuracy
4030
19.28
11.96
45.5
28
Kondom
DAD [, 0.30)
17.38:002
2139 0211
0.132
3006E1 1.001
45,721J 0.23)
25.89 0221)
3
Locate
0.00 (1 0.00)
0.00 (1 17.03)
25.86 () 14.94)
8.62
10.86)
0.00 0.381
0.33
31.60)
21.98
24.00)
7.42 18.88)
33.75
3,38
11
12
Random
Locale
0.00 (L 0.90)
0.06 (1 0.06)
17,54
10.38 (L 7:25)
0.00))
40.79
34.11 6.660)
(1 D.00
14 4.63)
(9.44 0.00
0.31 ( 0.07)
0.28 (1
32.04 0.08)
2.50 28.46)
20.48
43.80
25-47)
0.15)
7,76 (L 18:34)
26.07 0.05)
Table 3: Erase of different sets neurons. The values represent the comparison between the performance of the current model and the base model after erasing the located neurons.
As shown in the Table g compared to random, erasing the neurons we locate will significantly impair the performance of the model. For example, the Llama2-13B model shows 18.68% % decrease in performance when a single dataset, which indicates that commonality reflects the capabilities of the model. However, capability be attribute that can span datasets. Furthermore, we will explore whether the neurons we located can perform across datasets.
3. This means that method can locate the capability of the model
Layer
GSM8K
(b) Emotion
(c) Code25K
Fig. 4: Visualisation of commonality neurons. For ease of observation, we choose the neuron with the highest absolute value among 100 neighboring neurons. The horizontal axis shows neuron IDs, the vertical axis shows model layer IDs, and darker squares indicate more prominent located neurons.
Commonality Across Datasets. Firstly, we visualize the neurons we locate. From Fig that neurons under different datasets have overlaps, which means that the commonality have cross
4 it seems

---

Published as a conference paper at ICLR 2025
dataset characteristics. To further substantiate this viewpoint, enhance and erase the neurons located on GSM8K, Emotion, and Code25K, and test the performance changes the model on other datasets.
Enthance data mnden
Erase data
Erata datt
-0
Test data
Test data
Test data
(a) Neurons enhancement
(b) Neurons erasing
(c) Overlap ratio across dataset
Fig. 5: Effects of capacity neurons on other dataset. The vertical axis represents the enhanced or erased dataset, and the horizontal axis represents the tested dataset. The number represents the performance difference between the enhanced or zeroed model and the base model.
The Fig shows that the located neurons have cross-dataset characteristics. Experimental results on additional models are presented in Appendix G For example, Fig enhancing the neurons for GSM8K localization will significantly improve the performance of mathematical related datasets (meta math) by 22% with little impact on other datasets. In the Appendix H. also calculate the average performance improvement on other datasets not involved in training. The results indicate that, on average, enhancing commonality neurons can bring about more generalized capability improvement. And the detailed results are shown in Appendix | These results show that the neurons we have located can reflect the cross-data capability of the model. An interesting point that the GSM8K and Emotion's neurons have higher overlap rate, which is because there is greater overlap in capacity required for GSM8K (math, multiple choice, comprehension, etc.) and Emotion (sentiment analysis, multiple choice, comprehension, etc.). Therefore, we can assert that the located neurons embody the collection of capabilities. Additionally, to demonstrate the potential of CNL, conducte comparison of existing parameters localization methods.
Comparison of Different Localization Meth- ods. We construct three unified evaluation in- dicators, including the consistency of knowledge localization (Overlap). the granularity of iden- tified parameters (Neuron). and the Impact model Performance when manipulating iden- tified Parameters (IPP) for comparison. The calculation formula for this IPP is: Perfor- mance (parameters for operation localization) periiaaa-prffrmmcce (random parameters for multiple sampling). The result from Table 4 proves that previous research (individual knowledge localization) unreasonable, which also brings new perspectives subsequent research: focusing the localization of model capabilities.
Method KN
Overlap (f)
Neuron
IPP
37.3
0.2
11.4
ROME
32.7
16.7
0.0
KC
7.2
10.5
CNL(Ours)
96.4
20.0
Table 4: Comprehensive analysis of different meth- ods. The superior indicators highlighted in bold.
CONCLUSION
In this Paper, we aim to clarify whether existing individual knowledge storage is correct and clarify that capabilities can be localized. Firstly, we demonstrate through fidelity and reliability experiments that the existing knowledge localization methods are unreasonable. In order to further reveal the form of knowledge storage, we found through decoupling experiments that individual knowledge cannot be localized, and the commonality of data has the potential to be localized by parameters. Finally, we propose a commonality neuron localization method that utilizes samples of the same type to obtain commonalities in data and successfully locates commonality neurons. Furthermore, we have demonstrated through cross data experiments that commonality neurons are a collection of capability neurons that possess the capability enhance performance other datasets.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Key R&D Program of China (No. 20222D0116311 Beijing Natural Science Foundation (L243006) and the National Natural Science Foundation of China (No. 62106249).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303.0877 2023.
Gašper Beguš. Ciwgan and fiwgan: Encoding information in acoustic data model lexical learning with generative adversarial networks. Neural Networks, 139:305-325, 2021.
Helena Bonaldi, Yi-Ling Chung, Gavin Abercrombie, and Marco Guerini. Nlp for counterspeech against hate: A survey and how-to guide. arXiv preprint arXhv:2403.20100, 2024.
Marco Bonvini and Simone Marzani. Four-loop splitting functions at small Journal of High Energy Physics, 2018(6):1-38. 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Damai Dai, Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in 2021.
pretrained transformers. arXiv preprint
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, al. parameter-effcienn fine-tuning of large-scale preitrained language models. Nature Machine Intelligence, 5(3):220-255 20200223,,200 2023.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1-42 2018.
Joschka Haltaufderheide and Robert Ranisch. The ethics of chatgpt in medicine and healthcare: systematic review on large language models (Ilms). NPJ Digital Medicine, 7(1):183, 2024.
Stefan Hinterstoisser, Cedric Cagniart, Slobodan Ilic, Peter Sturm, Nassir Navab, Pascal Fua, and Vincent Lepetit. Gradient response maps for real-time detection of textureless objects. IEEE transactions on pattern analysis and machine intelligence, 34(5):876-885 2011.
Xiusheng Huang, Yubo Chen, Kang Liu, Yuantao Xie, Weijian Sun, and Jun Zhao. Nsrl: Named entity recognition with noisy labels via selective review learning. In Knowledge Graph and Semantic Computing: Knowledge Graph Empowers New Infrastructure Construction: 6th China Conference, CCKS 2021, Guangzhou, China, November 4-7, 2021, Proceedings 6, pp. 157-170. Springer, 2021a.
Xiusheng Huang, Yubo Chen, Shun Wu, Jun Zhao, Yuantao Xie, and Weijian Sun. Named entity recognition via noise aware training mechanism with data filter. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4791-4803, 2021b.
Xiusheng Huang. Bin Li, Fei Xia, and Yixuan Weng. A novel initial reminder framework for acronym extraction. In SDU@ AAAI, 2022a.
Xiusheng Huang, Hang Yang, Yubo Chen, Jun Zhao, Kang Liu, Weijian Sun, and Zuyu Zhao. Document-level relation extraction via pair-aware and entity-enhanced representation learning. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 2418-2428, 2022b.
Xiusheng Huang. Yequan Wang, Jun Zhao, and Kang Liu. Commonsense knowledge editing based on free-text in llm.. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14870-14880, 2024.
11

---

Published as a conference paper at ICLR 2025
Xiusheng Huang, Jiaxiang Liu, Yequan Wang, and Kang Liu. Reasons and solutions for the decline in model performance after editing. Advances in Neural Information Processing Systems, 37: 68833-68853, 68853, 2025.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412::441266980 2014.
Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. Context based emotion recognition using emotic dataset. IEEE transactions on pattern analysis and machine intelligence, 42(11):2755-2766, 2019.
RW Ledeen, JA Skrivanek, LJ Tirri, RK Margolis, and RU Margolis. Gangliosides of the neuron: localization and origin. In Ganglioside Function: Biochemical and Pharnacological Implications, pp. 83-103, Springer, 1976.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv: 00110016.11,,. 2017.
Bin Li, Fei Xia, Yixuan Weng, Xiusheng Huang, and Bin Sun. Simelad: A simple framework for contrastive learning of acronym disambiguation. arXiv preprint arXiv:2111. 141066, 2021a.
Bin Li, Fei Xia, Yixuan Weng, Xiusheng Huang, Bin Sun, and Shutao Li. Psg: prompt-based sequence generation for acronym extraction. arXiv preprint arXiv:2111.1430 2021b.
Jiateng Liu, Pengfei Yu, Yuji Zhang, Sha Li, Zixuan Zhang, and Heng Ji. Evedit: Event-based
knowledge editing with deductive editing boundaries. arXiv preprint arXiv:2402.1 1324, 2024.
Andrew F MacAskill and Josef T Kittler. Control of mitochondrial transport and localization in neurons. Trends cell biology, 2022):10--112, 2010.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances Neural Information Processing Systems, 05517799-17722 2022a.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory transformer. arXiv preprint arXiv:2210.07229, 2022b.
Hao Peng, Xiaozhi Wang, Chunyang Li, Kaisheng Zeng, Jiangshan Duo, Yixin Cao, Lei Hou, and Juanzi Li. Event-level knowledge editing, arXiv preprint arXiv:2402. 13093, 2024.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang. Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness large language models. arXiv preprint arXiv:2401.05566 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXin:2307.09288 2023.
James S Trimmer. Subcellular localization of k+ channels in mammalian brain neurons: remarkable precision in the midst of extraordinary complexity. Neuron, 85(2):238-25 2263, 2015.
Sandesh Tripathi, Ritu Mehrotra, Vidushi Bansal, and Shweta Upadhyay. Analyzing sentiment using imdb dataset. In 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN), pp. 30-33. IEEE, 2020.
Rüdiger W Veh, Ralf Lichtinghagen, Sabine Sewing, Frank Wunder, Isabella M Grumbach, and Olaf Pongs. Immunohistochemical localization of five members of the kvl channel subunits: contrasting subcellular locations and neuron-specific co-localizations rat brain. European Journal of Neuroscience, 7(11):219922) 1995.
Yixuan Weng, Fei Xia, Bin Li, Xiusheng Huang, and Shizhu He. Adbemm: acronym disambiguation by building counterfactuals and multilingual mixing, arXiv preprint arXiv:2112.08991, 2021.
Fei Xia, Bin Li, X Huang, Y Weng, and Shizhu He. Anaconda: Adversarial training with in-trust loss in acronym disambiguation. SDU@ AAA1-22, 2022.
12

---

Published as a conference paper at ICLR 2025
Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonade Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević. Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology, 55 (1):90-112, 2024.
Yiqun Yao, Xiusheng Huang, Xuezhi Fang, Xiang Li, Ziyi Ni, Xin Jiang, Xuying Meng, Peng Han, Shuo Shang, Kang Liu, al. nanolm: an affordable 11m pre-training benchmark via accurate loss prediction across scales. arXiv preprint arXiv:2304.06875. 2023.
Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. Knowledge circuits pretrained transformers. arXiv preprint urXiv:2405. 17969, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303. 20t.:23333.8 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX / LIMITATIONS
We note a few limitations of the experiments conducted in this paper:
(I) In the experimental section, utilize the GPT2 model as the base model and did not attempt to evaluate its performance on other models (like Llama2 (Touvron et al. 2023)
(2) The maximum parameter size of the tested model is only 7B, and is possible to attempt capability localization on larger models.
B APPENDIX / ATTENTION MECHANISM
Attention mechanism. Given a sequence of text x = [x1. 1 ,x]. the transformer's hidden state h at the layer I and the token i is calculated:
h| [x]
h 1-1 [2]] + att) [x] + m[[x]]
att [x] attention (h1-
[=],.... [x), ... h/ [xx)
(10)
m [a] = w W (w][))) w|[l] = [ate]]:.n7ntttte (a)t
where y indicate layer norm and a means a non-linear function function(Huang et al., 2021b 2022b m [a] could be considered the memory from LLM because feed-forward layers act key-val value mechanism Dai et al. 2021 Meng et al. 2022b). In order to generate m the input ky == (att)) (x)) activates the corresponding neurons with inner production:
w/ [a]] W k [W : :...
WLN k.. [Whh ky.
W KJT
(11)
with w/ [x] wio k standing for the origin output of neuron W Meanwhile Wh is row vector of W.n. For brevity, we utilize Who to denote W and w 1 [x] means the output of W at tt token.
in
C APPENDIX/ THE LOCATING METHOD OF PARAMETER LAYERS
In the clean run, they pass a prompt I [......, x into model Fg and collect all hidden activations {héli E [L, X]. E [1, L]}. L represents the number of hidden layers in the model.
In the corrupted run, they consider the text before the relationship as the subject, and the text after the relationship as the object. The subject is obfuscated from Fo before the network runs. Concretely, immediately after x is embedded as [Aq. ho..... h6], we hς h + 8 for all indices that correspond to the subject entity, where 8 E (0, 2²). Fa
a is then allowed to continue normally, giving
us a set of corrupted activations [h E X], I E [1, L).. Because Fg loses some information about the subject, it will likely return an incorrect answer et 2021a Huang et al. 2022a).
In the corrupted- wieeddd-wt----etrtteee run, they have the Fg run calculations on noise embeddings, except some tokens Zy and layers l'. Afterwards, we hook Fg and forced to output clean state hi continue without intervention. Afterwards, The capability of a few clean states to restore correct facts afterwards indicates their importance the calculation graph (Huang et al. 2021a Weng al. 2021
Future caleulations
The probability value Py of restoring the target answer will be utilized as the contribution of this layer I' common sense knowledge. The larger Py, the greater the probability that individual knowledge is stored in this layer(Li et al. 2021b Xia et al. 2022).
D APPENDIX/ THE LOCATING METHOD OF PARAMETER CHAINS
They concentrate on the task answering factual open-domain questions, where the goal to predict target entity given subject-relation pair (s, r). Individual knowledge triplet & (s, o) is often presented to the model. To identify the circuit that are critical for predicting the target entity o for a given subject-relation pair (s, r), ablate each special edge e (nzz Ny) in the computation graph
14

---

Published as a conference paper at ICLR 2025
G. They then measure the impact of ablating the edge (zero ablation in implementation) on the model's 's performance using the MatchNLL (Bonvini & Marzani 2018) loss for the target o:
G
S(e:) log(
<<(o((sr)) + log(C(ol(s,r))
(12)
E APPENDIX/ LOCATING ACCURACY CONVERGENCE
As show in Fig 6 we gradually scale up the amount of data for the localisation neurons and found that the consistency of the localisation converges as the data size increases.
ratio
ratio
per data
per data
per 50 data
(a) Llama2-7B-GGSMBK
(b) -llama--B--Eootiiin
(c) Llama2-7B-Code22)
-undaa
ratl
per 100 data
per 100 data
per 100 data
(d) Llama2-13B--SAAA
(e) 1aetaa2--3---otttte
(f) Llama2- 13B- 13B-Code254
rat0
per 100 data
100 data
per 100 data
GSM8K
(h) GBPJJ-6--Eootion
Fig. 6: The relationship between ratio and data. The horizontal axis represents the amount of data utilized for localization, the vertical axis represents locating ratio, which means location accuracy.
The Fig demonstrate that, the location ratio will gradually converge with increasing of data. This suggests that we only need a subset of the dataset to effectively identify the commonality the entire dataset. Additionally, and GPTJ-6B. As shown in Fig d-i, for models of different size and architecture, the locational consistency of commonality neurons progressively improves with the increase of data.
concurrently conduct localization experiments under both Llama2-13B
15

---

Published as a conference paper at ICLR 2025
F APPENDIX/ ENHANCEMENT LOSS WITH DIFFERENT SETS OF NEURONS
1035
(a) GSM8K
(b) Emotion
(c) Code25K
Fig. 7: Enhancement loss with different sets of neurons.
G
APPENDIX/ CROSS-DAVE RESULT ON OTHER MODEL
50
gandd
Erase data unddon
Erase data
10
10
pindk matt amaton
coda23k
jandi
misumalh
unedon
mab
cook23c
Test data
Test data
(a) GPTJ-6B enhancemen
(b) Llama2-13B erasing
Fig. 8: Effects of task neurons on other dataset.
H APPENDIX/ CHANGES IN MODEL PERFORMANCE ON OTHER DATASETS
We evaluated the image of enhanced neurons other datasets that are not involved in training. The following represents the performance improvement of the model in other datasets:
Model
random
w/o located
located
LLama2-7B 6
3.20
-3.31
11.18
LLama2-7B σ = 3
5.20
-5.58
9.43
GPTJ-6B o 6
9.00
14.73
17.66
Table The changes in the average performance on other datasets that involved in training.
16

---

Published as a conference paper at ICLR 2025
1
APPENDIX/ CROSS-DATA RESULT ON OTHER METHODS
pente
pentt
34
-30
13
12
-3
53
Enhance data vredle
Enhance data senden
10
-25
conte rea.oth mrdies
cent: nea mith endtes
Feds
Test data
Test data
(a) Llama2-7B random
(b) Llama2-7B w/o located
Enhance pirk expie
Enhance p>*k espie
prntic
pintic rut
Test data
Test data
(c) GPTJ-6B random
(d) GPTJ-6B w/o located
Enhance data emode
Enhance data emdde
Test data
Test data
(c) Llama2-7B KN
(f) GPTJ-6B KN
Fig. 9: Effects of task neurons on other methods.
17


Reviews:
[{"review_id": "KPkCbzihaM", "reviewer": "Reviewer_dq76", "summary": "The manuscript investigated the storage of individual knowledge in model parameters, and proposed a Commonality Neuron Localization method that locates commonality neurons.", "questions": "1. line 344, \"we expand the KN method\", what is the key improvement of CNL compared to KN?\n2. What is the conclusion of Table 2?\n3. The method successfully identifies those commonality neurons that embody the model capability. However, why is finding these neurons useful? For example, do we have any insights from analyzing the characteristics of the commonality neurons?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. The manuscript conducts experiments of previous knowledge localization methods and demonstrates those methods lack reliability and fidelity. The finding is insightful to the community.\n2. The manuscript proposed a method that can effectively identify those neurons that embody the model capability, which help the understanding of large-scale neural networks.", "weaknesses": "1. It is unclear to me what the author means by \"commonality\". And what is its relation to knowledge localization?\n2. Figure 2 is blurry.\n3. In line 182, $e_i$ seems to be undefined.\n4. Missing reference in line 216.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z51fmnqVnZ", "reviewer": "Reviewer_Skue", "summary": "This  paper explores the idea of localising “capabilities” within LLMs, rather than “individual knowledge”. The argument is that (existing methods for localizing) individual knowledge is unreliable. For instance, the authors show that current methods like parameter layers and chins are perfectly faithful to individual knowledge. For example, rewriting semantically similar prompts did not lead to consistent localisation results, and manipulating identified parameters did not reliably produce expected changes in model behaviour.\n\nInstead the paper forwards a new framework for localizing “commonality neurons”, the neurons that represent shared attributes or capabilities across datasets. They introduce the **Commonality Neuron Locating (CNL) method** which identifies neurons that consistently activate for similar tasks or data types. The paper concludes that capabilities can be localised within LLMs, potentially leading to new ways of understanding and manipulating LLM functionality", "questions": "1. The authors should provide justification for the specific hyperparameters chosen and clarify the methodology in more detail. For instance, in Section 5.1, line 352, why is S=19? Was this value selected through a hyperparameter sweep? If so, please include these details in the appendix. Similarly, in Section 5.1, line 358, why is the attribution score threshold set at σ=6? Offering detailed steps for identifying commonality neurons is essential to ensure that the results are not cherry-picked. This transparency will also help reproduce the findings independently.\n2. For the random-neuron conditions in the “enhance” and “erase” experiments ([Table 2] and [Table 3]), did you evaluate across multiple random samples to ensure statistical robustness? If so, please indicate the number of samples tested; without this, the reliability of comparisons with targeted neurons remains unclear.\n3. We need some more control conditions to convincingly demonstrate the impact of commonality neurons. For instance, a comparison with multiple sets of randomly selected neurons, averaged over several trials, would help determine if performance changes are indeed specific to commonality neurons rather than general effects from a neuron subset. Testing different thresholds for neuron activation and varying the size of neuron groups would also clarify whether the observed effects are specific to the identified commonality neurons. Such controls would strengthen the claim that these neurons contribute uniquely to model capabilities rather than reflecting random network fluctuations.\n4. By far the biggest thing missing from the paper is a common metric for comparing different methods. It is crucial to benchmark CNL against established knowledge and capability localization methods, such as distributed parameters, parameter layers, and parameter chains. This comparison should involve measuring CNL’s accuracy, consistency, and computational efficiency relative to these approaches in reproducible way such that improvements can be tracked carefully. \n\n5a. Another major concern is the scalability of CNL. There are at least 3 issues with model scalability and the authors need to discuss (or even better perform new tests) the scalability strategies of CNL.  A) As new models grow larger (e.g., from billions to hundreds of billions of parameters), the CNL method’s neuron-level analysis could become computationally prohibitive. \n\n5B. Larger models might need different threshold and hyperparameter sweeps to maintain CNL’s accuracy which demand additional tuning efforts. \n\n5C. CNL relies on extensive data from various domains to identify commonality neurons. As model sizes increase, the amount of data needed for localization will also grow. \n\n6. The authors should clarify how capability localization could enhance model interpretability in practical applications. It remains unclear whether the CNL approach addresses the theoretical question of how information is localized within the model or if it offers actionable changes for real-world LLM applications. Specifically, the authors should explain how identifying capability neurons could be leveraged to improve model performance, robustness, or transparency in practical scenarios. Without clear guidance the potential of using CNL in real-world settings is ambiguous.\n7. While the paper includes extensive mathematical notation and text, the figures are under-explained and hard to follow. For instance, it is unclear what Figure 1 is intended to convey, the purpose of the red boxes in Figure 3, or the significance of the colors in Figure 4. The authors should enhance these figure descriptions to make the visuals more accessible and informative for readers, clearly explaining all elements, symbols, and color schemes. This will improve readability and ensure that the figures effectively support the paper’s findings.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper is clear and well motivated. It clearly showcases the existing methods for understanding LLMs and their limitations.\n2. The paper also addresses an important question in LLM research. How LLM parameters relate to their performance is topical. The paper challenges the existing methods providing compelling evidence. \n3. The paper also introduces a way of plugging those gaps. The CNL approach is a clear structured approach and inspired by well-grounded methods like integrated gradients from CV.", "weaknesses": "1. The usage of arbitrary thresholds (magic numbers): The study uses a threshold of σ = 6 for identifying capability neurons without sufficient justification or exploration of alternative values. \n2. The decoupling experiment  to investigate the localisation of commonalities, primarily focuses on mathematical tasks. This narrow scope limits the generalisability of the findings regarding the potential for localising data commonalities. Expanding the decoupling experiment to include tasks from other domains, such as emotion recognition or language understanding, is crucial for supporting broader claims about the relationship between data commonalities and parameter localisation. \n3. The study shows that enhancing localised neurons leads to performance improvements, but the statistical significance of these improvements is not adequately assessed. Merely stating that the located neurons are \"most sensitive\" to performance improvement without providing statistical evidence leaves the strength of this claim open to question. \n4. The biggest weakness though is the glaring lack of common evaluation metrics. The study evaluates different knowledge localisation methods (ROME, KN, KC, and CNL) using distinct metrics tailored to each method's specific assumptions. Establishing a shared metric that can be applied consistently across all methods would enable a more objective and informative evaluation. Such a metric could consider factors like the accuracy of knowledge localisation, the granularity of identified parameters, and the impact on model performance when manipulating identified parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g2vKtDmDbQ", "reviewer": "Reviewer_4N9w", "summary": "This work introduces a Commonality Neuron Localization method as an alternative approach to overcome weaknesses of current methodologies in identifying and locating individual factual knowledge in large language models (LLMs) parameters.\n\nAfter showing that current algorithms that localize individual knowledge do not respect fidelity and reliability requirements, the authors propose a metric to identify capability neurons for a given dataset based on a gradient attribution score of model parameters.\n\nFor some datasets, the authors show that Commonality Localization can be related to the performance of the model by showing that enhancing or erasing their role affects the results of the model on the dataset downstream task.", "questions": "It is not clear from the results if \"Existing technology cannot localize individual model parameters\" (line 71) or if \"Individual knowledge cannot achieve parameter localization\" in general. Clarifying this would be beneficial.\n\nIn order to support the claim that \"the located neurons embody the collection of capabilities\" it should be at least clarified explicitly which capabilities the authors are referring to. In particular, it should be stressed which of the quantitative results in Section 6 support the existence of the specific \"common capabilities\".\n\nThe results of Subsection 5.4 (in particular \"Commonality across datasets\") should be explained with more care. \n\nThe paper could be largely improved in terms of clarity and quality of the presentation:\n- extended and more precise Figure captions would help clarify results;\n- the role of section 4.2 is not clear given the current form of the paper;\n- in formula (8), line 329, the parameter sets $P_s$ should be clearly defined;\n- in several circumstances, the authors use terminology incoherent with scientific rigor (e.g. \"believes\" or \"ideologically\" referring to existing literature);\n- there are several typos and incomplete sentences in the current version.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The authors propose a change of perspective to address the current limitations of techniques that attempt to localize individual knowledge in LLMs parameters. They propose to focus on a coarser localization task that focuses on common traits specific to a dataset and task.", "weaknesses": "The conceptual differences and limitations that incur when replacing individual knowledge location with \"commonality neurons\" are not sufficiently discussed. These are radically different procedures, with extremely different potential applications. The authors should thoroughly justify why it is relevant to localize phenomena at a much coarser level.\n\nA more thorough discussion of the definition of attribution score would be beneficial, accompanied by a comparison to existing choices for attribution score appeared in the literature.\n\nIn the current form, the mapping between commonality neurons and capability is conjectural (see line 525-526). This is not coherent with the current main statement of the work \"the located neurons embody the collection of capabilities\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nJQ6MqYl5w", "reviewer": "Reviewer_wrao", "summary": "The authors verify the assumption that knowledge is localized in the model parameter space by evaluating existing methods on a synthetically generated dataset. They falsify this assumption and instead argue that capabilities are localized in model weights and layers instead. Subsequently, the authors propose the Commonality Neuron localization method to find neurons that have commonalities within a variety of datasets. By manipulating these neurons, they causally show that the identified neurons that make up only 0.15% of the parameters significantly affects model performance.", "questions": "-- what is the difference between knowledge and capabilities? \n-- Fig 2 is missing its x axis label. I am guessing it is the layer number?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- assessed 3 existing methods to verify knowledge representation in neural networks\n-- developed a new method to categorize neurons that are selective to similar information to influence performance\n-- The decoupling dataset and experiments are informative.\n-- The authors used a wide variety of datasets (math, program, language) to evaluate their proposed CNL method.\n-- the authors demonstrated that fine-tuning commonality neuron improved performance within and across datasets, demonstrating their casual role in performance.", "weaknesses": "-- for Section 3.1 and 3.2, the authors used GPT4o to generate samples with the same semantic and attempted to identify localized representations in GPTJ. It will be informative to generate samples using GPTJ and verify localization methods on the same model i.e. GPTJ only. If this similarly shows low localization, it would further support the authors' claim. \n-- The authors used GPT4o to rewrite 1000 factual statements to populate the factual dataset. How did they verify that the new samples have the same semantic meaning?\n-- Why did the authors not evaluate the CNL method on the same dataset i.e. Table 1, that was used to evaluate previous methods?\n-- Authors should include the baseline performance of the model without any fine-tuning in Table 2 as in Table 3 for comparison. Is there a claim the authors are making with respect to Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)