The Rise of Parameter Specialization for Knowledge Storage in Large Language Models
Authors: Yihuai Hong
Yiran Zhao
Wei Tang
Yang Deng
Yu Rong
Wenxuan Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Interpretability
Knowledge Storation in LLM
Primary area: deep_learning

Abstract: Over time, a growing wave of large language models from various series has been introduced to the community. Researchers are striving to maximize the performance of language models with constrained parameter sizes. However, from a microscopic perspective, there has been limited research on how to better store knowledge in model parameters, particularly within MLPs, to enable more effective utilization of this knowledge by the model. In this work, we analyze twenty publicly available open-source large language models to investigate the relationship between their strong performance and the way knowledge is stored in their corresponding MLP parameters. Our findings reveal that as language models become more advanced and demonstrate stronger knowledge capabilities, their parameters exhibit increased specialization. Specifically, parameters in the MLPs tend to be more focused on encoding similar types of knowledge. We experimentally validate that this specialized distribution of knowledge contributes to improving the efficiency of knowledge utilization in these models. Furthermore, by conducting causal training experiments, we confirm that this specialized knowledge distribution plays a critical role in improving the model's efficiency in leveraging stored knowledge.

The Rise of Parameter Specialization for Knowledge Storage in Large Language Models
Yihuai Hong 1.2+
Yiran Zhao
Wei Tang
Yang Deng
Yu Rong'
Wenxuan Zhang
Alibaba DAMO Academy NNew York University
National University of Singapore
4 SSinapore Management University 'Singapore University of Technology and Design
yihuaihongenyu, edu, wxchangSsutd. edu..
Abstract
Over time, a growing wave of large language models from various series has been introduced the community. Researchers striving maximize the performance of language models with constrained parameter sizes. However, from a microscopic perspective, there has been limited research on how better store knowledge model parameters, particularly within MLPs, to enable more effective utilization of this knowledge by the model. this work, we analyze twenty publicly available open-source large language models to investigate the relationship between their strong performance and the way knowledge is stored in their corresponding MLP parameters. Our findings reveal that as language models become more advanced and demonstrate stronger knowledge capabilities, their parameters exhibit increased specialization. Specifically, parameters in the MLPs tend to be more focused on encoding similar types of knowledge. We experimentally validate that this specialized distribution of knowledge contributes to improving the efficiency of knowledge utilization in these models. Furthermore, by conducting causal training experiments, confirm that this specialized knowledge distribution plays critical role in improving the model's efficiency leveraging stored knowledge.
Introduction
An increasing number of powerful large language models (LLMs) have emerged in recent years (Touvron et al.) 2023a Achiam et al. 2023 Groeneveld et al. 2024 Bai et al. 2023 Team 2025). often demonstrating remarkable capabilities across various benchmarks and tests (Hendrycks et al.)
2021a Chen et al. 2021a| Cobbe et al. 2021). Thanks to the large parameter space, they have shown an exceptional ability encode vast amounts of knowledge within their parameters, enabling superior performance on knowledge-intenste tasks (Hendrycks et al.) 2021a Zhang et al.) 2023).
To understand the internal mechanism of knowledge storage, many studies have been conducted. For example, Geva et al. (202Tb) interprets the MIP layers of the transformer architecture (Vaswani et 2017) as key-value memories, where the factual knowledge encoded in the weights retrieved and transmitted to the output layer during inference (Geva et al. 2023 Meng et al. 2022 Yu et al. 2024). Furthermore, researchers have observed that, in the final layer of the MLP, each vector in that value matrix can act as a fundamental unit knowledge storage (Geva et al.) 2022a (b)). However, there has been limited research how better store and compress knowledge within constrained model parameters to enable more effective utilization of that knowledge by the model.
WWor done during an internship at Alibaba Group, before joining New York University. Conrespondig author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we investigate the relationship between language models' knowledge storage patterns and their performance. To identify parameters associated with specific knowledge concepts, we analyze consistently activated parameters in MLP layers when the model processes questions related to the particular knowledge concept. Building on the key-value interpretation of the MI.P by Geva et al. (2021b), which treats the up-projection matrix as the key and the down-projection matrix as the value (i.e., stored knowledge), we extract the intermediate representations between these two matrices and treat their absolute value as the activation of corresponding parameters. To support empirical analysis, we construct a new encyclopedic knowledge benchmark based on Wikipedia, covering knowledge concepts with varying frequencies. We then apply the knowledge parameter identification method to 20 open-source LLMs across a wide range of model families, enabling us to explore correlations between knowledge storage patterns and overall model performance.
Our extensive empirical analysis re- veals that stronger models exhibit higher parameter specialization for distinet knowledge, whereas weaker models distribute knowledge more diffusely across parameters. Conse- quently, significantly more parameters required to store individual knowl- edge weaker models. As illustrated advancing model capabil- ity correlated with improved parame- ter specialization for encoding knowl- edge: fewer parameters are allocated per knowledge concept, while each pa- rameter governs a narrower subset of concepts.
Evolution of Knowladge Distribution in Parameter Vectors during Model Iteration
When querying
Mario:
Super
Harry Fotter
LLaMAz MLP
LLaMAO MLP
Rerack Obarsa
in Figure [
Figure 1: Evolution of knowledge distribution in model pa- rameters during three iterations of LLaMA models. Each parameter vector corresponds to column in the value matrix of the MLP module, as indicated by the dashed rectangles.
Motivated by this observation, we fur- ther conduct four sets of controlled experiments, each involving continued training on the Llama2-7B 7B (Touvron et al., 2023a) and Qwen2-7B (Yang al. 2024) models with knowledge respectively, validate the strong causal relationship between improved parameter specialization and enhanced performance of the models on knowledge tasks. Overall, the experiments reveal that encoding similar knowledge into the same parameter vectors better aligns with the model's internal knowledge retrieval mechanism. This approach helps the model utilize knowledge more efficiently, improves knowledge compression, and reduces hallucination generation.
Our contributions can be summarized as follows:
To the best of our knowledge, this is the first attempt to quantify and compare the degree of parameter specialization for knowledge storage across different LLMs.
We investigate the relationship between parameter specialization and model performance in LLMs, constructing a dedicated probing dataset for an in-depth analysis on 20 open-source LLMs. Our findings indicate that more capable LLMs exhibit greater parameter specialization.
Through controlled training experiments, we provide empirical evidence of causal link between increased parameter specialization and improved performance on knowledge-intensiee tasks.
2
Related Work
Knowledge Storage in LLMs
Studying how knowledge is stored and utilized in LLMs has been
an important area in the research of LLM interpretability (Meng et al., 2022 Geva al. 2021b
Sukhbaatar et al.) 2013 Geva et al.] 2023). 3. Recent studies have shown that MLPs are the primary and crucial components for storing factual knowledge and associations in transformer-basse
language models (Geva et al. 20226 Dar et al. 2023). They can be conceptualized as key-value memories (Geva et al.) 202 where the factual knowledge encoded in the MLP weights is recalled and transmitted to the output layer during inference (Geva et al., 2023 Meng et al. 2022 Yu et al.) 2024). Additionally, researchers have found that in the final layer of the ML.P, each vector in the value matrix serve fundamental unit for storing knowledge (Geva et al. 2022a b). They have also verified

---

that by directly manipulating or disrupting these parameter vectors, specific knowledge can be edited or unlearned (Hong al. 2024a b Meng et al. 2022). leading to changes the model's 's responses.
Knowledge Superposition in LLM Elhage al. 2022 Olah (2023) propose the concept of Knowledge Superposition. It refers to an inevitable phenomenon in neural network models, especially large language models, during training and data memorization: since the number of data features greatly exceeds the number of parameters in the model, each parameter does not have a simple one-to-one mapping with the data features or knowledge. Neurons are often involved with multiple data features simultaneously. In our work, we treat each vector in the last layer of MLP as a basic unit for storing knowledge and investigate the superposition of knowledge within these vectors.
3 Parameter Specialization Analysis for Knowledge Storage
3.1 Preliminary
In transformer-based language models, the MLP is crucial component for storing the model's factual knowledge, and its sub-layers can be viewed as key-value memories (Geva et al. 2021b). To be specifie, the first laye MLP sublayers can be viewed (k), .g...., ,.. k, J. used to capture a set of patterns in the input sequence, and ultimately outputting the coefficient scores. The second layer be viewed with each value vector containing the corresponding factual knowledge.
matrix W formed by key vectors
matrix Wy formed by value vectors
{V1, .., vn
Formally, the output of the MLP in the transformer's 4-th layer, given an input hidden state x5, can be defined as:
M = f(Wk 7(x + W/ =m'm= m'we.
(1)
where W W. IRaxd, The function f and y represent non-linearity layer normalization, respectively. In the transformer's ε-th layer, m E R denotes the coefficient scores, and A represents the output of the attention component. 'The hidden state dimension is d, while the intermediate MI.P has dimension of n. Then, by denoting vý as the j-th column (which will be called the value vector or parameter vector in the following sections) of W and m as the j-th element in the coefficients produced by the first layer of the MLP, we can view MLP's output M as a linear combination of the value vectors in Wi,, with their corresponding coefficients m²:
M
Finally, the hidden states the l-th layer of the language model can be defined as:
x x M A
(3)
where X Mc and A represent the hidden states, MLP's output, and the attention component's output in the transformer's f-th layer, respectively. In this work, we focus on studying the impact of the MLP on the knowledge output of the hidden states.
3.2 Knowledge Vectors Masking Procedure
Referring to Eq. 2 if we aim to ablate the impact of the knowledge contained in the vectors for particular subset S of indices in E-th layer, can directly set the corresponding m² values for
j E St zero. Hence, we have:
M Muaskedd = -'i myve
+ Σ
j=1 0 jest
Σn
Σ
j=m m j
jESt
"In most decoder-onl only models, such as GPT-2 Radford et al 2019 and GPT-J Chen et al. 2021b the MLP component consists of two layers, whereas in LLaMA (Touvron al. 2023b) it comprises three layers. However, we can still regard LLaMA's s first two layers collectively as the key matrices, with their output representing cnefficient scores. For brevity, the bias term is omitted.
3

---

Therefore, given a concept, when we aim to identify which specific value vectors in the model's s MLPs are most closely related to the knowledge contained in that concept-wwile avoiding the masking of vectors associated with the model's general capabilities (Meng et al., 2022 Geva et al. 2023), i.e., determining the appropriate subset S at each layer of the model for this concept-we will run & concept-related questions and t* irrelevant questions on the selected model. Then we will compute the corresponding coefficients m and m" which are the averages of the coefficients for the concept-related questions and irrelevant questions, respectively, at each layer of the model. For details on the generation of concept-related and irrelevant questions, as well as the selection of and t", please refer to $3.4 After obtaining m and md at each layer, we perform the computation using the following formula:
S = { m;' lissi n, mÿ E em, m5e em²)
(5)
Next, we will sort S in descending order and select the value vectors corresponding to the indices of the top & elements, which will be used as the subset S for the masking operation. This allows us to observe and analyze the impact of masking these vectors on the model's knowledge output for certain concepts.
3.3 The Definition of Parameter Specialization
After obtaining the subset of value vectors St that exhibit specificity to a given concept at each layer of the model, described in 43.2, we apply the masking operation to these value vectors, as shown in Eq. (4. We then analyze its impact on the model's final outputs for the t concept-related questions and t" irrelevant questions. By comparing the model's responses after masking with the ground truth answers, we compute the accuracy on concept-related questions, referred to as the Concept Specific Score after surgery. and the accuracy on irrelevant questions, referred to as the General Score after surgery. To quantify the degrees of specialization of the model's value vectors with respect to the concept-related knowledge, we define the Parameter Specialization Score (PSS):
PSS A
|General Score after surgery Concept Specitic Score after surgery General Score before surgery
(6)
which is obtained by taking the absolute difference between the General Score and the Concept Specific Score after surgery, and then dividing by the model's accuracy on the entire dataset before surgery. A higher PSS indicates that the parameter vectors in the model's MLP layers exhibit a higher degree of specialization towards specific knowledge. Conversely, lower PSS suggests more severe knowledge superposition phenomena within the parameter vectors, resulting in a lower degree of specialization.
3.4 Dataset Construction
To thoroughly investigate the parameter specialization of knowledge with different frequencies in the parameter vectors of LLMs' MLP, we introduce a dataset named Spec Wiki. includes 525 concepts selected from Wikipedia [ widely recognized high-quality corpus for LLM training. These concepts are categorized based on their frequency levels ensure a diverse distribution. We then design two distinct question formats-mulnn multiple-choice questions and open-ended generation prompts to facilitate a thorough examination of the models' knowledge storage.
Concept Selection We treat each Wikipedia item as a defining concept. typically represented by an article focused on a specific subject, indicated by its title. We focus on specific entity concepts, such as historical figures, events and locations. We began by randomly sampling 2,400 pages (a 0.01% % rate) from the 2019 version of Wikipedia. Subsequently, we performed manual filtering to remove overly commonsensical or abstract concepts (such as the letter 'S' and the word 'Freedom'), ambiguous concepts (like 'Apple"), ), and those associated with pages under 1,000 words. Ultimately, this resulted in 525 high-quality concepts spanning specific topics like people, arts, and events.
'The term "general ability" refers to the model's fundamental skills, such as processing text inputs correctly and generating coherent outputs, rather than encoding knowledge specific to a particular concept. bhttps//en.wikipedia...m wikipedia.org/

---

Given that the frequency of knowledge in training datasets significantly influences a model's ability retain and comprehend it (Allen-Zhu & Li 2023 Meng al., 2022] Mallen al., ) we utilize Wikipedia page views as proxy for knowledge frequency in the models' pre-training datasets] [ To this end, we calculated the page views for each concept on Wikipedia between January 1, 2010, and December 31, 2019 Based on these statistics, concepts are categorized by page view frequency into three equal tiers: low-frequency (bottom 33% of the distribution), medium-frequenge (middle 33%), and nigh-frequency (top 33%). A more detailed distribution of the categories and the corresponding example data of Spec Wiki dataset are provided in Table 4 and Table respectively, in the Appendix. This approximation helps estimate the likelihood of concept's presence in the models' pre- pre-training datasets and allows us to explore how knowledge at different frequency levels is stored in models and provides a more comprehensive evaluation.
Question Generation To more precisely assess the retention of knowledge within the model, we design two sets of question formats.
Multi- Chotie Questions. Drawing inspiration from the widely used Massive Multitask Language Understanding (MMLU) dataset (Hendrycks et al. 2021b), which evaluates general knowledge across models, we similarly designed ten multiple-choice questions for each concept, ensuring the knowledge and answers could be directly found in the relevant Wikipedia articles. Specifically, we provided GPT-4o (OpenAI et al.) 2024) with the appropriate Wikipedia article for each concept and instructed it to extract ten questions without overlap, along with the correct answers derived from the article's s text. Next, it was instructed to generate three additional incorrect answers, aside from the golden answer, ensuring that none of them overlapped with the correct answer to avoid confusion. The detailed prompt is available in A We also include sample multiple-choice questions and results of manual verification of the generated data in Appendix AA.2]
Open-ended Generation. To more effectively assess the model's 's ability generate knowledge text freely, and to overcome the randomness and lack of depth inherent in the Multi-Choice Question evaluation method, we also up series of Open-ended Generation questions. For each question related to a concept, we prompted the model to generate an answer of up to 1 150 tokens directly and used GPT-4o as an evaluator to evaluate whether the generated response correctly matched the golden answer.
4 Experiment
4.1 Experimental Setup
Evaluated Models
To provide more comprehensive evaluation of how the degree of parameter
specialization evolves across large language models, we assessed 20 open-source models from various families and sizes in the community. Specifically, we evaluated LLaMA series Touvrrnn
et al. 2023a b. Grattafiori et al., 2024), Qwen series (Bai et al. 2023 Yang et al. 2024 Team 2025)
Gemma series (Tem al. 2024a b), OLMo series (Groeneveld al. 2024 OLMo 2025) ) Yi
series (Al et al. 2025), Mistral series (Jiang et 2023), GPT-j-6b (Wang & Komatsuzaki (022).
Pythia-6.95 9b (Biderman al. (022). Falcon-7b (Almazrouei et al. 2023) and Mpt-7b (Databricks
2023). Refer to Appendix (B. for the implementation details of these models.
Knowledge Vectors Masking Setup Based on the descriptions in 3.2 in order to obtain m' and m for each concept in SpecWiki at each layer of the model, we set the number of concept-related questions t to 10. Additionally, we randomly select irrelevant concepts with no knowledge overlap from the benchmark, and gather the corresponding questions associated with these irrelevant concepts, resulting in t* = 50 irrelevant questions. These collected questions will also be directly utilized in the computation of both the Concept-Specine Score and the General Score.
"To better support this point, we include experiments in of Appendix that validate the strong correlation between concept popularity and their frequency in the pretraining data.
'The earliest release date of the evaluated models, such as GPT-J, 2019. Therefore, their pretraining datasets could not include knowledge or concepts that emerged after this period. To ensure a fair evaluation across all models, any knowledge introduced post-2019, including the COVID-19 19 pandemic, was excluded from the benchmark.
5

---

SpecWiki Performance Parameter Specialization Score
(Parsen 1,0 U.M.
0.14
Parameter Specializaton ScoreiPSS:
0.10
Nulcase 1Ade
0.06
0.04
0.00
63
SpecWiki Performence (Multi-Choice Questions)
Figure 2: Correlation between the performance Spec and parameter specialization score (PSS) in 20 language models. We use a color gradient to distinguish the release times of the models, with cooler colors indicating earlier release dates and warmer colors representing later releases. Additionally, the size of each circle reflects the model's ' performance on MMLU, with larger circles indicating better performance. The blue trendline, obtained through linear regression fitting of the data points, suggests a strong correlation between a model's performance on Spec and its degree of Parameter Specialization.
Regarding the selection of model layers for masking, since the initial layers of model typically handle fundamental capabilities like basic text processing (Meng et al., 2022 Geva et al. (023). masking these layers could severely impair the model's basic text generation abilities. Therefore, for all models in our study, we preserve the first 5 layers without masking and only apply vector masking operations to subsequent layers.
For the PSS computation of each model, we selected five different fixed k values- 10%, 20%, 30%, 40%, and 50%-which represent the proportion of value vectors the model's MLP layers that were masked. For each k, we calculated the corresponding PSS following 6 and then averaged the results obtain the final PSS score for each model. This criterion was applied consistently across both the Multiple-Choice Questions (MCQ) and Open-ended Generation (OEG) tasks.
4.2
Main Results
The main results for the Multiple-Choier Questions setting can be seen in Figure 2 We observe strong correlation between the degree of Parameter Specialization (measured by PSS) and model performance on SpecWiki across 20 models, with Pearson and Spearman coefficients of 0.92 and 0.93, respectively. Models achieving better performance on SpecWiki exhibit higher Parameter Specialization Scores. Furthermore, models with higher PSS often those released more recently (warmer color) and exhibit stronger general abilities, as measured by their MMLU performance (larger circle). The corresponding results for the Open-ended Generation setting can be found in Figure 6 in B.2 which exhibit similar patterns and trends.
To better analyze the variations in Parameter Specialization across models within the same family, we selected eight models from four model families: ILaMA, Qwen, Mistral, and Gemma. We examined how the difference between the General Score, which represents the model's ability to handle irrelevant knowledge, and the Concept Specific Score, which reflects the model's ability to handle task-specific knowledge, changes under different masking ratios of parameter vectors. The results are shown in Figure B
From the figure, we observe very similar pattern across models from the four families:
1. Among models within the same family, more advanced models tend to achieve higher peaks in the General Score Concept Specific Score difference. This indicates that more advanced models generally exhibit higher levels of Parameter Specialization.

---

LLAMA
Owen1.577
Mistral 42
Gemma
LLaMA3-88
0wen2.5-78
Mistral-7Bo33
Gemmmal-00 98
Store Diteense (atners Contect Speatrr)
Scire Diferrs Suurers Sorusstppulr
Difeeente Verners Speatrd
414 Series Model
Deven Series Modelss
Ristral Series Models
Gemma Serbes Modek
0.23
0:00
0.20
6.88
0.30
0:10
5.15
0.05
0.25
6.55
0.00
0.00
2.00
-.05
C
0.
0.
4::5
022
-005
6.2
Vactor Masking Rata
Vecter Masking Rate
Vector Masking Patu
Vector Masking Rata
Figure 3: Analysis of Parameter Specialization variations across models within the same family. We selected eight models from four model families: LLaMA, Qwen, Mistral, and Gemma. The figure shows how the difference between the General Score (representing the model's ability to handle irrelevant knowledge) and the Concept Specific Score (representing the model's ability to handle task-specific knowledge) changes under different masking ratios of parameter vectors.
As the masking ratio of parameter vectors increases, from approximately 5% to 20%, the difference between the General Score and the Concept Specific Score gradually increases to a peak. This indicates that we are removing parameter vectors that are highly specific the target knowledge. After reaching the peak, as the masking ratio continues to increase, the difference gradually decreases to zero. This suggests that parameter vectors with lower activation are often those that have a higher degree of knowledge superposition and are less specialized in the target knowledge.
Additionally, we uncxpectedly found that when small proportion of concept-related vectors (ranging from 5% 10%) were masked, the performance of the masked models on unrelated questions even surpassed that of the original models. This observation is consistent across various models and indicates the positive impact of reducing irrelevant information interference the model's representation, leading to improved performance.
In S we will further validate the causal relationship between the degree of model parameter specialization and its ability to better utilize target knowledge through the finetuning experiments on additional data.
4.3 Impact of Model Scale on Parameter Specialization
In this section, to better explore the dif- ferences in the degree of parameter spe- cialization across models different sizes, conducted Knowledge Vectors Mask- ing experiments on five Qwen1 1.5 models of varying sizes (0.5B, 1.8B, 4B, 7B, and 14B) and on 2 Gemma2 models (2B and 9B). The results are shown in Table We observe that in both the Qwen and Gemma model families, as the model size increases, the corresponding Parameter Specializa- tion Score also increases. This trend is accompanied by improved performance on SpecWiki. This suggests that in larger- scale models, the degree of superposition for specific knowledge decreases and it tends to be distinctly represented across designated parameter vectors.
Model
Accuracy 1
PSS 1
5500.BB
0.61 (+0.2)
0.019 (+00001)
Qwen1.5-1.SP 1.5BB.BB
0.61
(+0.3)
0.044 (+0.02)
Qwen .5-4B
0.73
+0.2
0.075 (+0.02)
Qwenl 1.5-BB
0.75
+0.2)
0.121 (+0.04
Qwen owenl.5-144 155
0.82
(+0.2)
0.184
(+0.03)
Gemma2-2B Gemma2-99
0.72 (+0.3)
0.057
(+.00
0.86 (+0.1)
0.138
(+0.03)
Table 1: Performance comparison of language models with varying sizes on Multiple-Choice Question and Parameter Specialization Score. Both the Qwen1.5 and Gemma2 series models show improved Parameter Spe- cialization as the model scale increases, accompanied by better performance on the MCQ testing in SpceWiki.
4.4 Evolution of Parameter Specialization During Pretraining
To better investigate the development of Parameter Specialization in the model from the perspective of model training dynamies, we analyzed 10 checkpoints from the 01-4--2-1124-78 (OLMo et al.. 2025) pretraining process by using our Wiki The results are shown in Figure below.
7

---

nater Scecis tralicn ScomiPS51 GJMc2's Checopaim
Innanter Spcce intanSoonn
Spectcll Fedurran (HOO))
Recur
30-
163
6.8 C.t 6.5 637
433 o3x aie (151
Specikretii Scara FS5)
924-0-122-18 hanng Choceenn
Figure 5: Relationship between concept popu- larity, model accuracy on MCQ, and Parameter Specialization Score in LLaMA2-7B and Qwen2- 7B models.
Figure 4: Development Parameter Specializa-
tion in 1LM--2-1124-7B over the pretraining process.
Model
Accuracyurg
Accuracyor: 7
PSS f
Semantic Entropy 4
Local Intrinsic Dimension. J
LLaMA2-7B
[=0.2)
0.51 (20.1)
0.67 (=0.1)
0.67 (20.1)
11.23 (#2.1)
LLaMA2-7BPT-py
0.63 (0.3)
0.54 (+0.2)
0.65 (=0.2)
0.62 (=0.1)
11.12 (#1.4
11aMA2-7Bşr PV
(+0.3)
0.59 (+0.2)
(+-0.))
0,50 (+0.2)
7,89 (+2.1)
LLMMA--BPPT-0
0.62 (+0.1)
0.51 (+0.1)
0.63 (+0.2)
0.62 (=0.1)
11.12 (+1.4)
LJaMA2-7BpT RV
0.58 (+022)
0.49 (4.02)
0.65 (4:0.2)
0.65 (+.0.2
11.07(-2.7)
Qwen2-7B
(10.3)
0.63 (10.1)
0.124 (110.03)
0.56 (10.1)
9.78
Qwen2-7B
0.73 (.2))
0.67 (+0.2)
0.110 (=0002)
0.59 (20.2)
8.53 (+1.1)
Qwen2-7Be7-Pr
0.77 (0.1)
0.70 (+0.1)
0.133 (+0.03)
0.39 (=0.1)
6.92 (+1.3)
Owen2-7B,77 -CV
(002)
0.65 (+0.2)
0.114 (20.08)
0.55
(00.2
Qwen2-78Fr-Bn
0.71 (+02)
0.63 (+0.1)
0.122 (+0.02)
0.59
(=0.2)
9.65 (+1.4)
Table 2:
The performance both the original LLaMA2-7B-baa and Qwen2-7B-ban models, along with their FT-FV, FT-PV, FT-CV and FT-RV variants, was assessed selection of 10 high- frequency concepts from SpecWiki. Five metrics were used to evaluate their performance, including their general effectiveness, Parameter Specialization, and the degree of hallucination present in their output.
From the results, we observe that during the early training steps (step 10,000 to step 210,000), both the PSS and the accuracy Speci Wiki remain nearly unchanged and close to zero. In the subsequent phase (step 310,000 to step 510,000), although the model begins to show noticeable gains in accuracy on SpeciWiki. the PSS are still under 0. However, it is during the later training steps (step 610,000 to step 910,000) that parameter specialization begins to emerge, accompanied by a more substantial improvement in accuracy. These findings suggest that parameter specialization does not occur in the early stages of training, but rather emerges after a certain amount of data exposure. Furthermore, as training continues and the model sees more data, the degree of parameter specialization increases accordingly.
4.5 Parameter Specialization in Relation to Concept Popularity
In this section, analyze how the popularity of concepts themselves, which is roughly equivalent to their frequency in the pretraining data, will affect the level of parameter specialization for the corresponding knowledge. We follow the classification method for concepts outlined in $3.4 dividing them into high-frequency, mid-frequency, and low-frequency categories. The impact on their example models, LLLMA2-7B, and Qwen2.5-7B, which are shown in
PSS scores is measured Figure
From the figure, it is clear that in both the LLaMA2-7B and Qwen2-7B models, as the popularity of a concept decreases, the model's accuracy on that specific knowledge declines, accompanied by a lower Parameter Specialization Score. This suggests that the degree of Parameter Specialization for particular knowledge in the model's 's parameters is likely directly correlated with the frequency of that knowledge in the model's pretraining dataset. The higher the frequency, the greater the Parameter Specialization for that knowledge in the model.
5 Validation of Parameter Specialization Benefits for Knowledge Tasks
In this section, we conducted four sets of controlled training experiments, each involving continued fine-tuning on the Llama2-7B-ban -B-aee (Touvron et al. 2023a) and Qwen2-7B-base (Yang et al. 2024
8

---

models with additional knowledge data. These experiments aim to validate the causal relationship between improved parameter specialization and enhanced model performance on knowledge tasks.
5.1 Finetuning Setup
We randomly selected 10 high-freuency concepts from the SpecWiki benchmark. For each concept, we gathered relevant textual material from the top 10 most popular Google search results, including the corresponding Wikipedia article, and compiled this into an additional finetuning training dataset.
Next, we will validate whether the improvement in Parameter Specialization and the enhanced efficiency in the model's 's use of knowledge truly cxhibit causal relationship through four distinct finetuning experiments. The experimental setups are detailed below:
FT-FV((ull Vectors) Perform full finetuning (FT) on all parameter vectors of the MLPs across all layers in the model, while keeping the other parameters frozen.
FT-PP(PPrriall Vectors) Perform partial finetuning on a subset of the parameter vectors in the MLPs while keeping the other parameters frozen. Specifically, for each model, we apply finetuning (FT) to the top & most highly activated parameter vectors" For the selection of k, please refer to the description in $4.1
#T-CVVCooplemmentary Vectors) Perform finetuning only on the complementary set of parameter vectors, excluding the target vectors.
FT-RV(Random Vectors) Perform finetuning subset of parameter vectors randomly selected from the MLP, ensuring the same quantity as in the FT-PV setting.
5.2 Finetuning Results
In addition to evaluating the model's performance SpecWiki's Multi-choice Question and Open- ended Generation tests, as well as the Parameter Specialization scores, we also report two other metrics, Semantic Entropy (Kuhn et al. 2022) and Local Intrinsic Dimension (LID) ( (ii al.) 2024), for measuring the extent of hallucination the model's output. These metrics help evaluate whether training strategies that enhance Parameter Specialization- by aligning better with the can
model's 's knowledge retrieval mechanisms through a data-encoded strategy- can effectively reduce the unintended side effect of hallucination. For detailed introduction these two hallucination measurements, please refer to $B.3
The final results are presented in Table 2 From the results, can see that the FT-PV method, which finetunes only a small subset of the highly activated knowledge parameters, not only further enhances the model's Parameter Specialization compared to the three other finetuning setups, but also greatly improves the model's utilization of specific knowledge. As a result, it achieves the best performance the benchmark Multi- Multi-Choice questions and Open-ended generation tasks. Additionally, by reducing the influence of irrelevant information in the model's key parameter vectors, FT-PV helps to significantly reduce the level of hallucination in the generated text.
Although FT-FV and FT-CV does improve the model's performance on both the Multi-Choice questions and Open-ended generation tasks to some extent, compared to FT-PV, it does not lead to a better increase in Parameter Specialization. Additionally, the degree of hallucination in the generated text is not effectively reduced. FT-RV, serving as a counterpart to FT-PV, demonstrates that fine-tuning the same number of arbitrary value vectors in the model's 's MLP can not result in a desirable knowledge enhancement.
6 Conclusion
This study reveals that enhanced parameter ppecializaionn-heet related knowledge is encoded in focused parameter vectors vectors-ccrrelatte with superior performance in large language models. Analyz- ing 20 open-source models, we observed stronger models increasingly consolidate similar knowledge into fewer parameters, while weaker models distribute it diffusely. Controlled experiments confirmed
& à k "We experimented with 2' 4. g. and 18. and found that finetuning only & of the parameter vectors was sufficient to achieve excellent performance.
&

---

that optimizing this specialization improves task performance and reduces hallucination. These findings highlight the importance of aligning knowledge storage with models' retrieval mechanisms for efficiency and accuracy. Future work should explore dynamic knowledge updates and scalability, advancing both interpretability and performance L.LM design.
7 Limitations and Future Work
In our work, we have only examined and validated knowledge parameter specialization within the MLP, and this was done by treating vectors in the MLP as units of analysis. However, at least this remains one of the knowledge storage methods that has been extensively validated so far (Geaa et al. 2021a Meng al. 2022 Geva al. 2023). In fact, knowledge may also reside within the attention module of transformer models (Geva et al. 2023).
Additionally, due to GPU limitations, all the models we tested are smaller than or equal to 14B parameters, so we were unable to validate our conclusions on larger models, such as those with 35B parameters more.
In future work, we will progressively narrow the focus of our research individual neurons in language models, aiming to measure and validate more precise Parameter Specialization and Pa- rameter Superposition.
addition, we will extend this concept to the study of other related model
architectures, including Mixture of Experts (Fedus et al. 2022). which similarly enhances model performance by specializing expert parameters, as well as Sparse Auto-Encoders (Huben et al., (024). which help clarify the model's representations by leveraging larger parameter space and mitigating the superposition of these features.
Acknowledgment
This research is supported by the Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier grant, and funded through the SUTD Assistant Professorship Scheme (SAP 2025_001).
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt,
J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint XxXv:2303.08774, 2023.
AL ., :, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Wang. G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang, S., Yang, S., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P., Li, Y., Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, 7.,, Liu, Z., and Dai, Z. Yi: Open foundation models by O1.ai, 2025. WtR.hhtts:///rxii www.iii.ogg/abbbb.. www./////////...oo..////// 2203. 04652
Allen-Zhu, Z. and Li, Y. Physies of language models: Part 3.1, knowledge storage and extraction. 2023.
Almazrouei, F., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Étienne Goffinet, Hesslow, D., Launay, J., Malartic, Q., Mazzotta, D., Noune, B., Pannier, B., and Penedo, G. The falcon series of open language models, 2023. URL https:
org/abs////.o.ooooooooeo..e 22rg//ab//3111111 2211 16867
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang. F., Hui, B., Ji, L.. Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang. H., Yang, J., Yang. S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report, 2023. URL
https: //arxi: org/ aba/aa//////// 16609
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: suite for analyzing large language models across training and scaling, 2023. URL https // //arxiv. org/ abaaaba/2/0 2204. 01373
10

---

Chen, M., Tworek, J., Jun, H., Yuan, Q. de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N.. Brockman, G., Ray. A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P. Chan, B., Gray, S., Ryder, N., Paviov, M., Power, A., Kaiser, I.., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E.., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S.,
Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, L,, and Zaremba, W. Evaluating large language models trained on code, 2021a. httpst/s: /ttp:p:::///avvv..///. arxiv. org/ abs 2217 03374
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021b.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URL https: /ww.xvaaxvvvv///v///va//aaaaiiia org. hargg/bs/2111 abbs/210.... 14168
Dar, G., Geva, M., Gupta, A., and Berant, J. Analyzing transformers in embedding space. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 16124-16170, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. (0.118633/1120233 lon....... 893. URI hts.ttttpp.//acccatthol.... htt.://aaaaaaalloooor//2222..... acl- 893
Databricks. Introducing mpt-7b:: A new standard for open-source, commercially usable Ilms, May 2023.
http://www..aaaaii.oooo////o///a 7b Accessed: Jan 29, 2025.
Elazar, Y., Bhagia, A., Magnusson, I. H., Ravichander, A., Schwenk, D., Suhr, A., Walsh, E. P., Groeneveld, D., Soldaini, L., Singh, S., Hajishirzi, H., Smith, N. A., and Dodge. J. What's in my big data? In The Twelfth International Conference on Learning Representations, 2024. URL Sttts://openreet wtt.:/neeeeoooomoddddddtooo
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Saafield-Doddss Z., Lasenby, R., Drain, D., Chen, C., al. Toy models of superposition. arXiv preprint arXiv:2209.1065 2022.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. URL BRLhhttp:: /tta:///////ii//a/ org/
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A.. Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https: //tp///avvvv///////// orgo aaa/aaaaa222000 2ttp/22110000077
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021a. Association for Computational Linguistics. doi: 10.18653/w1/2221.cmm www.llemmmpppme.ee mainpp-min.446. https://:///caantttn http://aaaaaatooooocooo org/ 2021 emnlp- main. 446
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021b.
Geva, M., Caciularu, A., Dar, G., Roit, Sadde, S., Shlain, M., Tamir, B., and Goldberg, LM- debugger: An interactive tool for inspection and intervention in transformer-
language models. In Che, W. and Shutova, E. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 12-21, Abu Dhabi, UAF, December 2022a. Association for Computational Linguistics. doi: 10. 1655//11/20222220 http://222..mmmddmmmm...2.. URI
org/ org./r//002 2022. emnlp-demos.2

---

Geva, M., Caciularu, A.. Wang, K., and Goldberg, Y. Transformer forward layers build predictions by promoting concepts in the vocabulary space. In Goldberg. Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022b. Asso- ciation for Computational Linguistics. doi: 10. 1665://1/022...mmmmmmaaa //aclanthology. org 2022. emnlp-main.3
URL https
Geva, M., Bastings, J., Filippova, K., and Gloherson, A. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 12216-12235, 2023.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang. A., Fan, A., Goyal, A., Hartshorn, A., Yang. A., Mitra,
A., -Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodrigue7, A., Gregerson,
A., I.SSataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C.,, Marra, C., McConelll,, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song. D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D. Garcia-Olano, D., Perino. D., Hupkes, D., Lakomkin, E., AlBadawy. E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang, F, Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang. G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Zhang. J., Copet, J., Lee, J., Geffert, J.. Vranes, J., Park. J., Mahadeokar, J.. Shah, J.. van der Linde, J., Billock. J., Hong, J., Lee, J.. Fu, J., Chi. J., Huang. J., Liu, J.. Wang, J., Yu, J., Bitton, J., Spisak, J.. Park. J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K. Stone,, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala- Yeary, L., van der Maaten, L., Chen, L., Tan, L.,, Jenkins, L.., Martin, L.., Madaan, L.., Malo, L.., Blecher, L., Landzatt,, L., de Oliveira, L., Muzzi, M.. Pasupuleti, M.. Singh, M., Paluri, M., Kardas, M.. Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov. N., Bogoychev, N., Chatterji. N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P.,, Zhang, P., Li, P., Vasic, P., Weng. P., Bhargava, P., Dubal, P.. Krishnan, P., Koura, P. S., Xu. P.. He. Q. Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S.. Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R.. Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor. R., Silva, R., Hou, R., Wang, R., Hosseini. S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy. S., Shen, S., Wan, S., Bhosale, S., Zhang. S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom. T., Speckbacher, T. Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V. Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang. X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A.. Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang. C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichennhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D.,, Li, D., Adkins, D.,, Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D.,, Dowling, E., Jamil, F., Montgomery, E., Presani, F., Hahn, E., Wood, E., Le, E.- Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk. F., Tian, Kokkinos, F,, Ozgenel, F. Caggioni. F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G. Sizov, G., Guangyi. Zhang, Lakshminarayanan, G., Inan, H., Shojanaazzrri H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H. Zhan, H., Damlaj. I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E.. Gat, I.. Weissman, J.. Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang. J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J. McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A. L., Silva, L.. Bell, L., Zhang, L., Guo, L.,, Yu, L..
12

---

Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M.. Bhatt, M., Mankus. M., Hasson, M., Lennie, M., Reso, M.. Groshev, M., Naumov, M.. Lathi, M., Keneally, M., Liu, M., Seltzer. M. L.. Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P. Bontrager, P., Roux, P.. Dollar, P., Zvyagina, P.,, Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R.. Wang. R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S.,, Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S.. Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng. S., Lin, S., Zha, S. C., Patil, S., Shankar, S.. Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyighe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang. T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W.. Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z. The Ilama 3 herd of models, 2024. URL hRtpt:ts:///rrr //axiv. org/ abs/ 2407 21783
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang. Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Blazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V., Ravichander, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P. Lambert, N., Richardson, K., Zettlemoyer, L.,, Dodge, J., Lo, K. Soldaini, L., Smith, N. A., and Hajishirzi, H. Olmo: Accelerating the science of language models. arXiv preprint
200880088 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representa- fions, 2021a. URL httpps://peerrevie. httprrewwwwiwieeeefeeeeeeeeeeee net//oorumidddd77Bjjj
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Hong, Y., Yu, L., Yang, H., Ravfogel, S., and Geva, M. Intrinsic evaluation of unlearning using parametric knowledge traces. arXiv preprint arXiv:2406. 1614, 2024a.
Hong, Y., Zou, Y., Hu, L.,, Zeng, Z.., Wang, D., and Yang, H. Dissecting fine-tuning unlearning in large language models. In AI-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 3933-3941, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 18653vv11 2024.emnlp-main222 URL http:://///// peattps://aclaathooogg... org/2024. 2024. emnlp-main 228/
org/
Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL /ttp:ttttp////oeeeeeeeeee wttp://wwww..eeeeeoore/ddd//d//a
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F. Lengyel, G., Lample, G., Saulnier, I.., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL httpss
//arxiv.org ab..
Kuhn, L., Gal, Y.. and Farquhar, S. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2022.
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D.,, and Hajishirzi, H. When not to trust language models: Investigating effectiveness of parametric and nor-maraettric memories. Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61 Annual Meeting of the Association
13

---

for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/v1/2023 10tp:3/////22.....oooogg.... aal.-lon....54 URL
httpss//aclantbologgooooo org/ http:22223.ccc1111 acl-long. 546/
Meng, K., Bau, D., Andonian, A. J., and Belinkov, Y. Locating and editing factual associations in GPT. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL htttt:://opeereeeiiee http:///ppprreiiei..oo. oot.
Olah, C. Distributed representations: Composition & superposition.
https
(traasformer-cirrctooo ortaassooorrrrdrrttttttt./////// wwpprppstttooooooooooooooooooooo inw.diiiiixx.eic html 2023. Published May 4th, 2023.
OLMo, T. Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D.,, Brahman, F,, Clark, C., Dasigi, P., Dziri, N., Guerquin, M., Ivison, H.. Koh, P. W., Liu, J., Malik, S., Merrill, W., Miranda, L. J. V., Morrison, J. Murray, T., Nam, C., Pyatkin, V., Rangapur, A., Schmitz, M., Skjonsberg, S., Wadden, D., Wilhelm, C., Wilson, M., Zettlemoyer,, L.,, Farhadi, A., Smith, N. A., and Hajishirzi, H. olmo furious, 2025. URL httpttt/////axi..ooom htwp://gg///..////o///// 2501.00655.
OpenAI, Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., Madry, A., WhitcomD, A., Beutel, A., Borzunov, A., Carney, A., Chow, A., Kirillov, A., Nichol, A., Paino, A., Renzin, A., Passos, A. T., Kirillov, A., Christakis, A., Conneau, A., Kamali, A., Jabri, A., Moyer, A., Tam, A., Crookes, A., Tootoochian, A., Tootoonchian, A., Kumar, A., Vallone, A., Karpathy, A., Braunstein, A., Cann, A., Codispoti, A., Galu, A., Kondrich, A., Tulloch, A., Mishchenko, A., Baek, A., Jiang, A., Pelisse, A., Woodford, A., Gosalia, A., Dhar, A., Pantuliano, A., Nayak, A., Oliver, A., Zoph. B., Ghorbani, B., Leimberger, B., Rossen, B., Sokolowsky, B., Wang, B., Zweig. B., Hoover, B., Samic, B., McGrew, B., Spero, B., Giertler, B., Cheng, B., Lightcap, B., Walkin, B., Quinn. B., Guarraci, B., Hsu, B., Kellogg, B., Eastman, B., Lugaresi, C., Wainwright, C., Bassin, C., Hudson, C., Chu, C., Nelson, C., Li, C., Shern, C. J., Conger, C., Barette, C., Voss, C., Ding, C., Lu, C., Zhang, C., Beaumont, C., Hallacy, C., Koch. C., Gibson, C., Kim. C.. Choi, C., McLeavey, C., Hesse, C., Fischer, C., Winter, C., Czarnecki, C., Jarvis, C.. Wei, C., Koumouzelis. C., Sherburn. D., Kappler, D., Levin, D., Levy, D.. Carr. D., Farhi, D., Mely, D., Robinson, D., Sasaki, D., Jin, D., Valladares, D., Tsipras, D., Li, D.. Nguyen, D. P., Findlay, D., Oiwoh, E., Wong, E., Asdar, E., Proehl, E., Yang. E., Antonow, E., Kramer, E., Peterson, E., Sigler, E., Wallace, E., Brevdo, E., Mays, E., Khorasani, F., Such, F. P., Raso, F., Zhang. F., von Lohmann, F., Sulit, F., Goh, G., Oden, G., Salmon, G., Starace, G., Brockman, G., Salman, H., Bao, H. Hu, H., Wong. H., Wang. H. Schmidt, H., Whitney, H., Jun, H., Kirchner, H., de Oliveira Pinto, H. P., Ren, H., Chang. H., Chung, H. W., Kivlichan, I., O'Connell, I., O'Connell, I., Osband, I., Silber, I., Sohl, I., Okuyucu, I., Lan, I., Kostrikov, I., Sutskever, I., Kanitscheider, I., Gulrajani, I., Coxon, J., Menick, J., Pachocki, J., Aung. J., Betker, J.. Crooks, J., Lennon, J., Kiros, J., Leike. J., Park. J., Kwon, J.. Phang. J.. Teplitz, J., Wei, J., Wolfe, J., Chen, J.. Harris, J., Varavva, J., Lee, J. G., Shich, J., Lin, J.. Yu, J., Weng, J., Tang, J., Yu, J., Jang.. J.. Candela, J. Q.. Beutler, J.. Landers, J., Parish. J., Heidecke, J.. Schulman. J., Lachman, J.. McKay, J., Uesato, J., Ward, J., Kim, J. W., Huizinga, J., Sitkin, J., Kraaijeveld, J., Gross, J., Kaplan, J., Snyder, J., Achiam, J., Jiao, J., Lee, J., Zhuang, J., Harriman. J., Fricke, K., Hayashi, K., Jinginhll, K., Shi, K., Karthik, K., Wood, K., Rimbach, K., Hsu, K., Nguyen, K., Gu-Lemberg, K., Button, K., Liu, K., Howe, K. Muthukumar, K., Luther, K., Ahmad, L.,, Kai, L.. Itow, L... Workman, L.. Pathak, L., Chen, L., Jing, L., Guy, L.. Fedus, L., Zhou, L.., Mamitsuka, L., Weng, L., McCallum, L.,, Held, L.,, Ouyang, I.., Feuvrier, L.,, Zhang, I.., Kondraciuk, I.,, Kaiser, L., Hewitt L., Metz, I.., Doshi, I.., Aflak, M., Simens, M., Boyd, M., Thompson, M., Dukhan, M., Chen, M., Gray, M., Hudnall, M., Zhang. M. Aljubeh, M.. Litwin, M., Zeng, M.. Johnson, M., Shetty, M., Gupta, M., Shah, M., Yatbaz, M., Yang. M. J., Zhong, M., Glaese, M., Chen, M., Janner, M., Lampe, M., Petrov, M., Wu, M., Wang, M., Fradin, M., Pokrass, M., Castro, M., de Castro. M. O. T., Pavlov, M., -Brundage, M., Wang, M., Khan, M., Murati, M., Bavarian, M., Lin, M., Yesildal, M. Soto, N., Gimelshein, N., Cone, N., Staudacher, N., Summers, N., LaFontaine, N., Chowdhury, N., Ryder, N., Stathas, N., Turley, N., Tezak, N., Felix, N., Kudige, N., Keskar, N., Deutsch, N., Bundick, N., Puckett, N., Nachum, O., Okelola, O., Boiko, O., Murk, O., Jaffe, O., Watkins, O., Godement, O., Canpbell-Moore, O., Chao, P., McMillan, P., Belov, P., Su, P., Bak, P., Bakkum, P., Deng, P., Dolan, P., Hoeschele, P., Welinder, P., Tillet, P., Pronin, P., Tillet, P., Dhariwal, P., Yuan, Q., Dias, R., Lim, R., Arora, R., Troll, R., Lin, R., Lopes, R. G., Puri, R., Miyara, R., Leike, R., Gaubert, R.,
14

---

Zamani, R., Wang, R.. Donnelly, R., Honsby, R., Smith, R., Sahai, R., Ramchandani, R., Huet, R., Carmichael, R., Zellers, R., Chen, R., Chen, R., Nigmatullin. R., Cheu, R., Jain. S., Altman, S., Schoenholz, S., Toizer, S., Miserendino, S. Agarwal, S., Culver, S., Ethersmith, S., Gray, S., Grove, S., Metzger, S., Hermani, S., Jain, S., Zhao, S., Wu, S., Jomoto, S., Wu, S., Shuaiqi, Xia, Phene, S., Papay, S., Narayanan, S., Coffey, S., Lee, S., Hall, S., Balaji, S., Broda, T., Stramer, T. Xu, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Cunninghman, T., Degry, T., Dimson, T., Raoux, T., Shadwell, T., Zheng, T., Underwood, T., Markov, T., Sherbakov, T., Rubin, T., Stasi, T., Kaftan, T., Heywood, T., Peterson, T., Walters, T., Eloundou, T., Qi, V.. Moeller, V., Monaco, V.. Kuo, V., Fomenko, V., Chang, W., Zheng, W., Zhou, W., Manassra, W., Sheu, W., Zaremba, W., Patil, Y. Qian, Y., Kim, Y., Cheng, Y., Zhang, Y., He, Y., Zhang, Y., Jin, Y., Dai, Y., and Malkov, Y. Gpt-4o system card, 2024. halLhhtt:://// /arxiv.xvvv/. org/ abs abs/2410.... 21276
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Sukhbuatar, S., Weston, J., Fergus, R., et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L.,, Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, I.., Sessa, P. G., Chowdhery, A., Roberts, A., Barua, A. Botev, A., Castro-Ross A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan, C. L., Choquette-Chin
C. A., Crepy, C., Cer, D., Ippolito. D., Reid, D., Buchatskaya, E., Ni, E., Noland, E., Yan, G., Tucker, G., Muraru, G.-C., Rozhdestvenskly, G. Michalewski, H., Tenney, I., Grishchenko, I., Austin, J., Keeling, J., Labanowski, J., Lespiau, J.B.., Stanway, J., Brennan, J., Chen, J., Ferret, J., Chiu, J., Mao-Jones, J., Lee, K., Yu, K., Millican, K., Sjoesund, L. L., Lee, L., Dixon, L., Reid, M., Mikula, M., Wirth, M., Sharman, M., Chinaev, N., Thain, N., Bachem, O., Chang, O., Wahltinez, O., Bailey, P., Michel, P., Yotov, P., Chaabouni, R., Comanescu, R., Jana, R., Anil, R., Mcllroy, R., Liu, R., Mullins, R., Smith, S. L., Borgeaud, S., Girgin, S., Douglas, S., Pandya, S., Shakeri, S., De, S., Klimenko, T., Hennigan, T., Feinberg, V., Stokowiec, W., hui Chen, Y., Ahmed, Z.,, Gong, 7.,, Warkentin, T., Peran, I.., Giang, M., Farabet, C., Vinyals. O., Dean, J., Kavukcuoglu, K., Hassabis. D., Ghahramani, Z., Eck, D., Barral, J., Pereira, F., Collins, E., Joulin, A., Fiedel, N., Senter, E., Andreev, A., and Kenealy, K. Gemma: Open models based on gemini research and technology, 2024a. URL
https: /ttps://arxivv org/ abs///////330 08295
Team, G., Riviere M., Pathak, S., Sessa, G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momehey, N., Hoffman,
M., Thakoor, S., Grill, J.-B., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu. B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Choquette-Chone C. A., Sinopalnikov, D., Weinberger, D., Vijaykumar, D.. Rogozińska, D., Herbison, D., Bandy, E., Wang, E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H., Klimezak-Plucisske H., Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J.. Gordon, J., Lipschultz, J., Newlan, J., yeong Ji, J., Mohamed, K., Badola, K., Black, K., Millican, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjoesund, L. L., Usui, L.,, Sifre, L., Heuermann, L., Lago, L., McNealus, L.,, Soares, L. B., Kilpatrick, I.., Dixon, L., Martins, I.., Reid, M., Singh, M., Iverson, M., Görner, M., Velloso, M., Wirth, M., Davidow, M., Miller, M., Rahtz, M., Watson, M., Risdal, M.. Kazemi, M., Moynihan, M., Zhang. M.. Kahng, M., Park, M.. Rahman, M.. Khatwani, M.. Dao, N., Bardoliwalla, N.. Devanathan, N., Dumai, N., Chauhan, N., Wahltinez, O., Botarda, P., Barnes, P., Barham, P., Michel, P., Jin, P., Georgiev, P., Culliton, P., Kuppala. P., Comanescu, R., Merhej, R., Jana, R., Rokni, R. A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S. M., Cogan, S., Perrin, S., Arnold, S. M. R., Krause, S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T., Yu, T., Eccles, T., Hennigan, T., Kocisky, T., Doshi, T., Jain, V., Yadav, V., Meshram, V., Dharmadhikari, V., Barkley, W., Wei, W., Ye, W., Han, W., Kwon, W., Xu, X., Shen, Z., Gong, Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang, M., Peran, L., Warkentin, T., Collins, E., Barral, J., Ghahramani, Z.,, Hadsell, R., Sculley, D., Banks, J., Dragan, A., Petrov, S., Vinyals, O., Dean, J., Hassabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya, E.,, Borgeaud, S., Fiedel, N., Joulin,
15

---

A., Kenealy, K., Dadashi, R., and Andreev, A. Gemma 2: Improving open language models at practical size, 2024b.
wwwp://...gbbbb/....ooo 00121
Team, Q. Qwen3: Think deeper, act faster. https://qwenlm.giie htt.hhuu..bbblo///ggg/gg/go / 2025.
Accessed: 2025-04-29.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023a. URL httph:ttt//aat wtwpvv/aaxvvvvv. org/ abs/ 2302. 13971
Touvron, H., Martin, L., Stone, R., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S.. Bikel, D., Blecher, L.,, Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I. M., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Mao, Martinet, X.. Mihaylov, T., Mishra, P. Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X., Tang, B., Taylor, R., Williams, A., Kuan, X.X., Xu, P., Yan, 7., 7arov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, I.lama Open foundation and fine-tuned chat models. arXiv preprint wrXiv:2307..09220 2023b.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, Attention is all you need. Advances in neural information processing systems, 30, 2017.
Wang, B. and Komatsuzaki, A. Gpt-j-6b: A 6 billion parameter autoregressive language model, May 2021. URL hwt.hhttts///e thub. httph//..w///.g.ooooooo///aaaaaa jax Accessed: Jan 29, 2025.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C.,, Liu, D., Huang, F., Dong, G. Wei, H., Lin, H., Tang. J., Wang. J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang. J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng. R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, 7.,, Guo, 7.,, and Fan, 7. Qwen2 technical report, 2024. URL htt.https:.. /atp/vxxxvvv////////////////aa org/ ab////////// 2200 10671
Yin. F. Srinivasa, J., and Chang, K.-W. Characterizing truthfulness in large language model generations with local intrinsic dimension. In ICML,, 2024. URL httpss:://opeeeevieee net/ forum?id=7DbIy((fffl
Yu, L., Cao, M., Cheung, J. C. K., and Dong, Y. Mechanisms of non-factual hallucinations in language models. arXiv preprint arXiv:2403.18166 2024.
Zhang, W. Aljunied, M., Gao, C., Chia, K., and Bing, M3exam: multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Neural Information Processing Systems, 36:5484-5505, 2023.
16

---

A Details of Dataset
A.1 Dataset Construction Prompts
Below is our prompt for querying GPT-4o to generate the options for Julti-Choice questions of cach concept:
Please provide four answer options (A, B, C, D) for the following question, and indicate the correct answer. Example: Question: was Costa Coffee founded?' Options: A) 1971 1985 c) 1992 2000
'When
Correct Answer: A) 1971
Now, please answer the following question: Options:
Question:
Question
Below is our prompt for querying the model to generate the answers for Multi-choice questions in three-shot setting:
*Question:*** ** What is the capital city of France? **Options:** A.
Berlin B. Madrid C. Paris D. Rome **Answer:** C
**Question*** What is the largest planet in our solar system?
**Options:** A. Earth B. Jupiter C. Mars D. Venus **Answer*** ** B
**Question:** Which element has the chemical symbol "0"? **Options:**
A. Oxygen B. Gold C. Silver D. Iron **Answer:
**Question** question **Options:** A. option a B. option b C. option D. option
Below is our prompt for collecting the coefficients in model when querying about concept-reated knowledge:
Question:
question Answer:
answer:
A.2 Manual Verification
Here, describe the manual verification process used in constructing SpecWiki, including the validation of model-generated data:
Specifically, we analyze subset of 524 (10%) questions from SpecWiki, by sampling 50% of the concepts and randomly selecting 2 questions per concept. Then, we manually verify that the questions are about the given concept and that they are simple and reasonable. In addition, we review all the generated questions for 200 sampled concepts and verify they are not repetitive. We find that all analyzed questions were about the given concept and that 522 (99%) of them are reasonable simple questions. Moreover, we observe that questions are generally diverse, with only out of 20 concepts having 2 (out of 10) similar questions. This shows that our data generation process produces valid and diverse instances for evaluation.
A.3 Dataset Categories and Examples
Here, we provided more detailed distribution of the categories and the corresponding example data of Spec dataset in Table 4 and Table respectively.
A.4 Validation of Populartty-Frequency Correlation
We included simple experiment to validate the strong correlation between the popularity of concepts and their frequency in the pretraining data. To be specific, we used The Pile(Gao et al., (020)). which currently serves as significant portion of the pretraining dataset for most large language models Touvron et al.) 2023ai Groeneveld et al. 2024 Team al., 2024a), as an example of pretraining corpus. We then counted the frequency with which each of the 525 concepts from SpecWiki appeared in all text segments of The Pile dataset via the Flasticsearch API(Flazar et 2024). Subsequently, compared these frequencies with the popularity metrics for each concept and
17

---

SpecWiki Performance Parameter Specialization Score
PParsen 5/1te00
0.14
Parameter Specialization ScoreiPSS:
0.32
0.08
0.06
0.02
Preuess
0.6
SpecWiki Performance (Open-ended Generations)
Figure 6: Performance across 20 models Parameter Specialization Scores on Open-endes Genera- tion Setting.
computed the corresponding Spearman's 's rank correlation coefficient. The result is 0.814, indicating strong correlation.
Additionally, presents the top 3 most popular and the bottom 3 least popular concept examples WpecWiki, along with their occurrence counts The Pile dataset and their corresponding popularity scores.
Top 3 High Popularity Example Concept
Frequency
Popularity
Wikipedia
91 1708
1414686
Barack Obama
984586 3839241
1128538
India
1024513
Bottom Low Popularity Example Concept
Frequency
Popularity
Dark Souls (video game)
49
27
Culture of Latin America
251
90
Array (data structure)
1164
94
Table 3: Top and bottom 3 concepts ranked by popularity and their corresponding frequencies.
B Details of Experiments
B.1 The implementation of the models
For all models, the inference performed a text completion/generait mode, without the addition of any instruction tokens, to better assess the knowledge present in the model. For the Multi-Choice Questions task, we use three-shot setup for each model and search for the answer within the next 30 tokens generated by the model. For the open-ended generation task, prompt the model in zero-shot setting to produce an answer no longer than 150 tokens.
All the experiments in this work were conducted four 80GB NVIDIA A800 GPUs.
B.2 Parameter Specialization Scores on OEG Setting
In Figure 6 we provided the performance across models on Parameter Specialization Scores on Open-ended Generation Setting.
18

---

High Frequency Concepus 183)
Medium Froguicc (Number
Law Frequncyy
(Nummer
Concepts 1913
(Nummber
Concepes: 153)
Country
18:95 156
Technology
2,0%
Tochnology
19.9%
Matheroniss
4,4%
Perwen
996
6.3%
Culture
9.5%
BrundPPoduset
7.6%
Art and Enierninment
11.1%
Pollics
4,45
Hatwoy
10.6%
Medical
5.5%
Locaton
8.65
Person
6.7%
Natual Sciences
10.5%
Location
4,45
Enteriatiment
8.6%
Cuztuee
2.5
Histocy
8,0%
Medical
6.7%
Medical/Biclogy
7.7%
Country
3.9
CompanggOrpanizztooo
7.3%
Others
2.3%
Sparts
7.65
Eateriatinnnal
6.7%
Caltuee
CompaayyQigaiicaann
3,350
Natand Sciunees
2.05
Table 4: Ten most frequent concept categories of Spec in high frequency, medium frequency, and low frequeney levels.
Conrept The Lood the Rings
Prequenry
Foomple
Open-ended
High Montlly Views 1775a)
Art and Exter 13t0te
"Wo is persgomii of TThe Lord the Rige Option: Fredo Beggles B. Geaccllf Croy". C. "Aingon", TTgiiaa"
Dastion:
Question: "Whe is oothor "hh Lord Rings Inlogs
Matum
Biolegy
Quettor Whht demncooss connrms chten remeztee"
Queaiee: "Whrr trm conammp- by
deorrpaing
semaal feem" "Sinlight waur". D. "morganic misera and mouis.
Anewer "Sarmoolippaa"""
Atweer
Malin
Pience
Quntion "In stichilty Mikena anaet
Queotoe: "Whar Madoor Shamnt Anawer: "Doe
Moaitly Vinced 3352
Ditioent A "Ragesr, de "Sal" C: "Mavilh" Dr "Crrggra""
Auswer
Table 5: Example data from the Spec dataset.
B.3 Hallucination Metric Descriptions
In this experiment, we additionally incorporate two metrics, Semantic Entropy (Kuhn et al. 2022) and Local Intrinsic Dimension (LID) (Yin et al. 2024), to assess hallucination. This helps evaluate whether the finetuning methods that enhance Parameter Specialization also effectively mitigate the unintended side effect of hallucination in the model's output.
Semantic Entropy Semantic entropy is defined as a measure of uncertainty based on the distri- bution of semantically equivalent outputs. In this method, the outputs grouped into clusters of semantically similar responses, and the entropy is calculated among these groups. Formally, it is expressed as:
1 |C
Semantic Entropy
log ((C]2z)
where Cj represents the summed likelihood of outputs in the i-th group, and |C| the total number of such groups. The measure captures the uncertainty not in individual responses but within clusters of semantically similar outputs. This approach accounts for semantic equivalence among different responses, providing more robust evaluation of entropy in generative tasks.
Local Intrinsic Dimension The Local Intrinsic Dimension (LID) method detects hallucinations in Large Language Models by measuring the discrepancy in the local intrinsic dimension of model activations. This approach is grounded in the principle that LID represents the minimal number of activations required to characterize a data point, with truthful outputs exhibiting lower LID values due their closer alignment with natural language structure, while hallucinated outputs tend to show higher LID values due to mixing human prompt and model distributions. Technically, the method employs Maximum Likelihood Estimation (MLE) using a Poisson process to approximate the count of neighbors surrounding sample points, computed through the formula m m(X,) = (1/(T-
1) (loo((Qr/Qp) (Q,))) where T represents the number of nearest neighbors and Q, denotes the Euclidean distance to the j-th nearest neighbor. For more details about the Local Intrinsic Dimension metric, please refer to the work (Yin et al.) 2024).
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and the introduction in the paper clearly state the claims, contri- butions and scope of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
Our work does not include any theoretical analysis or formal results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide sufficient details in $3.4 and $ to ensure that our datasets and experiments fully reproducible.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We release the code and datasets in the supplementary materials submitted alongside the main paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.cc/public/giee guwde//ddee/// Cot//ooodeeesiseeeeeeeeee wwwpibmmsssioooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed specifications of all training and evaluation settings in and $B.T
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include error analysis to support the main experiments in our paper, specifically in $4.3 and $5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient details on the replication of the experiments in B.. in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't It make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httngttts///////ppppccccchhhhhtt
Answer: [Yes]
Justification:
We affirm that our research fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our research focuses on the interpretability of internal model parameters and does not have direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For the usage of other assets, we clearly cite their original sources and indicate the corresponding versions and licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include the complete dataset in the supplementary materials, and provide detailed information about its construction in $3.4
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
25

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We only used GPT-4o to assist in generating evaluation questions for the dataset, and included manually verified results in $A.2 of Appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standarr components.
Please refer to our LLM policy for what should or should be described.
26


Reviews:
[{"review_id": "5DclVprpNv", "reviewer": "Reviewer_vpb2", "summary": "This paper analyzes knowledge storage in LLMs from a novel perspective: the specialization of knowledge storage. A higher specialization means that the knowledge of similar concepts is stored in a smaller range of neurons, instead of being spread out across many neurons. This paper leverages the idea that knowledge is stored as key-value pairs in the MLP layers. It identifies the specific values stored in the MLP layers related to specific concepts, which could indicate the degree of knowledge specialization. The experiments discover that increased knowledge specialization indicates stronger model performance.  Then, the paper utilizes the findings to improve fine-tuning. It finds that utilizing the specialized distribution of knowledge contributes to improving the efficiency of knowledge utilization.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n(1) The motivation is novel and meaningful. It is meaningful to study knowledge storage in LLMs, but previous papers usually study knowledge storage locations, knowledge editing, etc. This paper takes a novel perspective: the knowledge specialization. The findings might benefit future applications in knowledge updating, LLM fine-tuning, etc.\n\n(2) The proposed methods are reasonable. This paper mainly relies on the idea that the MLP layers act as key-value pairs to store knowledge. It identifies the “values” in the MLP layers that are related to specific concepts. Then, we can mask out these value vectors to evaluate the influence. It is a good method.\n\n(3) The paper covers a thorough study cycle, from finding the positive relationship between knowledge specialization and model performance, to analyzing other related factors such as how parameter specialization develops during pretraining, to applying the findings to improve the fine-tuning.\n\n(4) This paper conducts experiments on 20 models, which is very extensive. It also conducts extensive ablation studies.\n\n(5) The experiment results well support the findings.\n\n(6) The writing is clear and easy to follow.\n\n### Weaknesses:\n\n(1) The masking method uses zero-ablation, which means to set the weight of the masked value vector to zero. It might change the output scale, which might cause other interference. It might be better to adjust other value vectors’ weights to ensure the total weight is not changed.\n\n(2) Lines 213-214 mention that the fixed k values are 10%, 20%, 30%, 40%, and 50%. I am not sure, but it might be possible that some concepts are only stored in very few neurons. If we take at least 10% of neurons, it might be still too many for those concepts.\n\n(3) It would be better if the authors could provide some insights into how the findings can benefit real-world applications. For example, how to more effectively train a LLM, or some recommendations on knowledge updates, etc.", "was_revised": true, "final_justification": "The author's rebuttal addressed most of my concerns. I think the paper is generally solid and novel, and it has interesting findings which might benefit future research. It has some minor points to improve, such as the ablation method I mentioned, or the concern of the PSS score mentioned by other reviewers, but I think these points are not big issues, and this paper's contributions worth a positive score. So I decide to maintain my score (5).", "rebuttal": "We sincerely thank Reviewer vpb2 for the strong and encouraging recognition of our work — including the recognition of the novelty and significance of our motivation and ideas, the rigor and thoroughness of our methodology, the completeness and strength of our experimental results in supporting our conclusions, as well as the clarity of our writing and structure!\n\nWe also greatly appreciate the valuable suggestions and insightful questions raised. We address these points in detail below and would be happy to respond to any further questions or concerns the reviewer may have.\n\n> **W1:** The masking method uses zero-ablation, which means to set the weight of the masked value vector to zero. It might change the output scale, which might cause other interference. It might be better to adjust other value vectors’ weights to ensure the total weight is not changed.\n\nThank you for pointing this.\n\nHowever, we believe that keeping the total weight unchanged by scaling up the coefficients of unrelated value vectors in the MLP—those not selected—**could push the model's representations in unintended directions and introduce spurious importance**. Given this concern, we argue that maintaining zero-ablation, which isolates a single variable, is a more accurate and reliable choice for measuring the influence of the located parameters and assessing their degree of parameter specialization.\n\n> **W2**: Lines 213-214 mention that the fixed k values are 10%, 20%, 30%, 40%, and 50%. I am not sure, but it might be possible that some concepts are only stored in very few neurons. If we take at least 10% of neurons, it might be still too many for those concepts.\nThank you for pointing these out.\n\nIn our experiments, we observed that even though parameter specialization does occur for the vast majority of concepts, the overall knowledge distribution remains relatively spread out. For example, on our SpecWiki dataset, **we have tested all concepts** and found that ablating only 10% of the value vectors in the MLPs (by setting their corresponding coefficient values to zero) is generally insufficient to reduce performance to less than half of its original level.\n\n\n> **W3:** It would be better if the authors could provide some insights into how the findings can benefit real-world applications. For example, how to more effectively train a LLM, or some recommendations on knowledge updates, etc.\n\nIn this work, we have demonstrated that the trend of parameter specialization in knowledge contributes positively to a model’s ability to utilize target knowledge (as shown in the experiments in Section 5). Specifically, it enables more effective retrieval of the target knowledge, reduces interference from irrelevant information, and ultimately contributes to enhanced accuracy and a decrease in hallucinations.\n\nThese findings offer important insights into a key research question: Given the constraints of a fixed-size pre-training dataset, how can we better organize the distribution of these knowledge data across the model's internal parameters? \n\nHere, we propose two actionable research directions:\n\n1. **Sparse Finetuning in the Post-Training Phase**\n\n   As demonstrated in Section 5, if the goal is to selectively enhance certain model capabilities—such as mathematical reasoning or coding proficiency—or to improve the model’s grasp of specific domains of knowledge, it is sufficient to apply sparse finetuning [1, 2] to only those parameters responsible for the targeted capability, while freezing the remaining paramters of the model. This approach can effectively improve task-specific performance while minimizing negative interference with unrelated capabilities. Future work should explore more precise methods for localizing knowledge within model parameters, as well as more efficient and robust techniques for sparse finetuning.\n\n\n2. **Knowledge Clustering in the Pre-training Corpus**\n\n   During pretraining, organizing knowledge-related data fragments close together in the corpus—such as grouping Harry Potter novels, related articles, and corresponding Wikipedia entries—may promote greater parameter specialization. Based on the findings in [1, 2], we hypothesize that this approach facilitates the backpropagation of gradients associated with shared knowledge into a concentrated set of parameters, thereby enhancing the overall degree of parameter specialization.\n\n\nOur work opens up a new interpretability perspective for understanding knowledge storage in LLMs—an area that remains underexplored and calls for further investigation by the research community. In the revised version of the paper, we will include a dedicated subsection discussing these future work in more detail.\n\n---\n\n**References**：\n\n[1] Backward Lens: Projecting Language Model Gradients into the Vocabulary Space\n\n[2] How Do Large Language Models Acquire Factual Knowledge During Pretraining?"}, {"review_id": "gDNpY4daYd", "reviewer": "Reviewer_8bPy", "summary": "The authors build on previous work showing that MLP layers in LLMs seem to function as key-value stores for persistent memories. They analyze the distribution of these memories over 20 LLMs, with a specific interest in the specialization of parameters for particular concepts. They find that stronger models exhibit higher parameter specialization, and weaker models distribute knowledge more diffusely across parameters.\n\nThis is performed by masking out keys relating to particular concepts based on the difference in weights for concept-related and irrelevant questions (the irrelevant questions prevents masking out “general abilities” like processing text for question answering). They measure specialization as “score on general tasks after surgery” minus “score of concept-specific tasks after surgery”, normalized by “score on general tasks before surgery”", "questions": "I would appreciate if the analysis were repeated with a \"parameter specialization\" score that more purely reflects specialization (as discussed in \"weaknesses\" above).", "limitations": "yes, excepting the question raised above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This is a very interesting finding, and shines some light on the question of trying explain mechanistically what happens within models as they get stronger. This is a compelling set of results indicating that parameter specialization in MLPs is one factor in model performance.\n- I appreciate the novelty (at least to me) of the methods, e.g. applying the same analysis to different models within the same model family, or their method for applying concept-related and irrelevant questions to the model.\n\nWeaknesses\n- This is not a causal analysis – it is very possible that the strength of models does not \\*arise\\* from parameter specialization, but rather that parameter specialization is just a side effect of how stronger models are trained.\n- I believe that the PSS score conflates two effects: the specialization of the parameters for a given task, and also how distributed a particular task is across multiple neurons. For example, you could have neurons that are each highly specialized for task X (they do not represent any task), but the task representation could be highly redundant such that even ablating the top few neurons for task X does not significantly affect performance on task X", "was_revised": true, "final_justification": "Concerns were suitably addressed by the authors. Score has been raised from 4 to 5.", "rebuttal": "We sincerely thank Reviewer 8bPy for the thoughtful and encouraging feedback. We truly appreciate your recognition of the novelty of our methodology and the significance of our findings regarding parameter specialization. We have addressed each of the valuable questions you raised in the responses below, and we would be happy to respond to any further questions or concerns the reviewer may have.\n\n> **W1:** This is not a causal analysis – it is very possible that the strength of models does not *arise* from parameter specialization, but rather that parameter specialization is just a side effect of how stronger models are trained.\n\nThank you for the question.\n\nWe acknowledge that the degree of parameter specialization in a model does not have a perfect one-to-one correspondence with overall model performance. Model performance is influenced by many other factors, such as architecture, the quality and quantity of the pre-training data, and more.\n\nHowever, as shown in the multiple controlled and contrastive fine-tuning experiments in Section 5—where we injected an equal amount of knowledge into the same LLM by fine-tuning different groups of model parameters—we found that the approach of fine-tuning only the parameters responsible for the target knowledge, while freezing all unrelated parameters (i.e., increasing the degree of parameter specialization), consistently led to more efficient knowledge embedding. This includes better performance, reduced hallucination, and improved utilization of the target knowledge (see Table 2). In contrast, other approaches that did not further enhance parameter specialization also failed to achieve better model performance.\n\nTherefore, we provide evidence that increasing the degree of parameter specialization **causally improves** the model's ability to utilize target knowledge more effectively.\n\n\n> **W2&Q1:** I believe that the PSS score conflates two effects: the specialization of the parameters for a given task, and also how distributed a particular task is across multiple neurons. For example, you could have neurons that are each highly specialized for task X (they do not represent any task), but the task representation could be highly redundant such that even ablating the top few neurons for task X does not significantly affect performance on task X\n\nThank you for the great question!\n\nWe acknowledge the possibility that, in addition to a small subset of highly specialized parameters that contain the target knowledge, there may also exist other parameters that redundantly store this knowledge in a more dispersed manner. However, because our ablations in Section 4.1 are conducted over significant portions of the model’s parameters (10%, 20%, 30%, etc.), **we are ultimately investigating parameter specialization at the model-wide level**—not just at the level of individual specialized parameters, with the goal of comparing the knowledge storage patterns across different models."}, {"review_id": "nbYFLT4V94", "reviewer": "Reviewer_BzMR", "summary": "This paper investigates how large language models store and utilize knowledge within their MLP parameters, revealing that stronger models increasingly specialize their parameters to encode similar types of knowledge more compactly, while weaker models distribute knowledge more diffusely. By analyzing 20 open-source LLMs with a new benchmark, and validating through controlled fine-tuning experiments, the authors demonstrate that higher parameter specialization improves knowledge retrieval efficiency, reduces hallucinations, and boosts performance on knowledge-intensive tasks. These findings highlight the importance of aligning parameter structures with knowledge retrieval mechanisms to achieve both greater interpretability and better performance in LLMs.", "questions": "1.This formula 6 can measure parameter specialization, but why is it normalized in this way?\n\n2.In Figure 3, why is the vertical axis (the score difference between the two) zero when the mask rating is 0? Does that mean their accuracies are equal?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1.This work systematically quantify and compare the degree of parameter specialization across different large language models, proposing the Parameter Specialization Score (PSS) as a metric and validating its effectiveness through multiple experimental methods.\n\n2.The paper conduct large-scale and systematic experiment, covering 20 open-source large language models with a carefully constructed benchmark, as well as its comprehensive parameter-masking and fine-tuning experiments. These extensive experiments strongly support and validate the paper’s conclusions about parameter specialization, ensuring their correctness and demonstrating the practical relevance of the findings for understanding and improving knowledge storage in LLMs.\n\nWeaknesses\n\nThe conclusion is relatively narrow. although the paper conducts extensive experiments, its findings mainly describe observed model phenomena, offering limited actionable insights.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' rebuttal and my concerns are mostly addressed in the rebuttal. The score will remain positive.", "rebuttal": "We sincerely thank Reviewer BzMR for the detailed and encouraging feedback. We greatly appreciate your recognition of our efforts to systematically quantify parameter specialization across LLMs, the introduction of the Parameter Specialization Score (PSS), and the validation of our findings through comprehensive experiments. We have addressed each of the valuable questions you raised in the responses below, and we would be happy to respond to any further questions or concerns you may have.\n\n> **W1:** The conclusion is relatively narrow. although the paper conducts extensive experiments, its findings mainly describe observed model phenomena, offering limited actionable insights.\n\nThank you very much for raising this great point.\n\nOur work is the first to propose and investigate the concept of **Parameter Specialization**, and we quantify how model parameters specialize in specific knowledge across different series and scales of LLMs. We not only show that this trend contributes positively to a model’s ability to utilize target knowledge (as demonstrated in the experiments in Section 5), but also highlight that this direction holds substantial potential for further research. It deserves more attention and deeper exploration from the research community — for example: *Given the constraints of a fixed-size pre-training dataset, how can we better organize the distribution of knowledge data across a model's internal parameters to improve knowledge utilization efficiency?*\n\nHere, we propose two actionable research directions:\n\n1. As demonstrated in Section 5, through extensive controlled and contrastive fine-tuning experiments, we have shown that fine-tuning only the parameters responsible for the target knowledge—while freezing other unrelated parameters—can help the model better acquire the target knowledge, while minimizing negative interference with unrelated capabilities. Therefore, future work could explore more precise methods for localizing model parameters that are responsible for specific target knowledge, as well as develop more efficient and robust techniques for sparse fine-tuning.\n\n2. During pretraining, organizing knowledge-related data fragments close together in the corpus—such as grouping Harry Potter novels, related articles, and corresponding Wikipedia entries—may promote greater parameter specialization. Based on the findings in [1, 2], we hypothesize that this approach facilitates the backpropagation of gradients associated with shared knowledge into a concentrated set of parameters, thereby enhancing the overall degree of parameter specialization.\n\n\nOur work opens up a new interpretability perspective for understanding knowledge storage in LLMs—an area that remains underexplored and calls for further investigation by the research community. In the revised version of the paper, we will include a dedicated subsection discussing these potential future work in more detail.\n\n\n> **Q1:** This formula 6 can measure parameter specialization, but why is it normalized in this way?\n\nThank you for the good questions.\n\nWe define this formula to quantify the extent of overlap and superposition between the concept-related knowledge embedded in the target parameters, and the other unrelated knowledge.\n\nSpecifically, we first calculate the difference between the model’s average performance on the unrelated questions set and its average performance on the concept-related dataset, after ablating the located target parameters. **This difference represents how much of the knowledge contained in the located target parameters is both parameter-specific and concept-related.** Since each model performs differently on the SpecWiki dataset, we use its original (unintervened) performance as the denominator to serve as a proxy for the total knowledge encoded in the model. **This allows us to finally estimate what proportion of the concept-related, parameter-specific knowledge accounts for the model's overall knowledge.**\n\nIf the PSS (Parameter Specialization Score) is closer to 1, it indicates a higher proportion of concept-related knowledge and a lower proportion of unrelated knowledge in the target parameters. Conversely, a lower score suggests that the target parameters contain less concept-related knowledge and more unrelated knowledge.\n\n> **Q2:** In Figure 3, why is the vertical axis (the score difference between the two) zero when the mask rating is 0? Does that mean their accuracies are equal?\n\nYes.\n\nSince our benchmark evaluation runs through all the concepts in the dataset and averages the model’s performance across them, to compute the Concept-Specific score (y-axis), a masking rating of 0 means that we do not intervene in the model’s internal inference in any way during this process. \n\nIn this case, the model’s performance on concept-specific questions across all concepts is identical to its average accuracy on general questions (which are also evaluated across all concepts in the entire benchmark).\n\n\n\n---\n\n**References**：\n\n[1] Backward Lens: Projecting Language Model Gradients into the Vocabulary Space\n\n[2] How Do Large Language Models Acquire Factual Knowledge During Pretraining?"}, {"review_id": "uFLafJvZDA", "reviewer": "Reviewer_7hsq", "summary": "This paper investigates how large language models store knowledge within their MLP parameters, analyzing twenty open-source models. The findings indicate that more advanced models with stronger knowledge capabilities exhibit increased specialization in their MLP parameters, meaning parameters are more focused on encoding similar types of knowledge. This specialized knowledge distribution is shown to improve the efficiency of knowledge utilization.", "questions": "Why is 0 used in Formula 4 instead of random noise?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths：\nThe paper conducts extensive experiments to validate the relationship between parameter size and knowledge.\nWeaknesses：\nThe paper doesn't propose a new method to find the relationship between knowledge and parameters; it directly uses existing methods. Furthermore, the perspective for examining parameters is singular, relying solely on Causal Tracing. Other methods like Gradient Attribution or statistical methods [1] were not considered.\nThe paper's conclusion lacks robustness, as there's insufficient evidence to support the SpecWiki Performance range of 0.4-0.7. This makes it difficult to establish a consistent pattern.\nThe data construction carries a risk of noise. Randomly selecting knowledge from Wikipedia pages over just one year means it could have been modified later, leading to discrepancies between this knowledge and the real world or the model's understanding.\nThe paper's statements are inaccurate. It only considers MLP layer parameters, so it cannot be generalized as a relationship between model size and knowledge.\n[1] Knowledge neurons in pretrained transformers", "was_revised": true, "final_justification": "The rebuttal addresses most of my concerns, and I will raise the score from 2 to 3.", "rebuttal": "We sincerely thank Reviewer 7hsq for the thoughtful feedback and for acknowledging the strength of our experimental validation. We believe, however, that some concerns stem from misunderstandings about the scope and goals of our work. In the following response, we clarify these points in detail and aim to further highlight the core contributions of our paper. We would be happy to respond to any further questions or concerns the reviewer may have.\n\n\n> **W1:** The paper doesn't propose a new method to find the relationship between knowledge and parameters; it directly uses existing methods. \n\nThe key contribution of our work lies in introducing a novel perspective and metric for understanding the storage pattern of knowledge within LLM parameters—specifically, the degree to which certain parameters are **specialized** or **dedicated** to a particular knowledge. Our goal is not to propose a new intervention, attribution, or visualization method for measuring the relationship between parameters and knowledge. The existing interpretability methods we used are sufficient to support the new perspective proposed in this work.\n\n> **W2:** Furthermore, the perspective for examining parameters is singular, relying solely on Causal Tracing. Other methods like Gradient Attribution or statistical methods were not considered. \n\nAs we mentioned in W1, the interpretability methods used in this work are not the focus of innovation or contributions. Instead, our aim is to apply established and mature interpretability techniques to uncover new patterns and phenomena between model parameters and knowledge, and to demonstrate that these phenomena help the model better leverage the target knowledge (Section 5).\n\nMoreover, causal tracing has already become a widely adopted and mainstream interpretability method for verifying the correlation between model parameters and target knowledge [1, 2, 3], and our use of it follows these prior studies.\n\n> **W3:** The paper's conclusion lacks robustness, as there's insufficient evidence to support the SpecWiki Performance range of 0.4-0.7. This makes it difficult to establish a consistent pattern. \n\nSpecWiki is a dataset we propose, consisting of 525 Wikipedia concept entries with varying frequencies. For each concept, we design 10 distinct questions in two different formats. The full construction process of SpecWiki has been described in detail in Section 3; it is both well-justified and fully reproducible. And we have already released the complete dataset in the Supplementary Material. Moreover, anyone can follow our methodology to construct a similar dataset in their own domain, ensuring that parameter specialization across models remains consistent and generalizable.\n\nPlease feel free to raise any further questions or points of confusion. We sincerely appreciate your engagement in the discussion and would be glad to know if our response has addressed your concerns and clarified the contributions of our work.\n\n\n> **W4:** The data construction carries a risk of noise. Randomly selecting knowledge from Wikipedia pages over just one year means it could have been modified later, leading to discrepancies between this knowledge and the real world or the model's understanding. \n\n1. **It is important to note that we did not simply choose the year \"2019.\"** In fact, the 2019 version of Wikipedia does not only include concepts newly introduced in that year, **but also encompasses all the knowledge entries accumulated over the previous years.** As such, its content can be seen as an approximation of the majority of knowledge concepts available to human society up to 2019. Based on this vast distribution of concepts, we performed random sampling and ultimately obtained 525 entries with varying frequencies, which constitute our SpecWiki dataset.\n\n2. Secondly, 2019 is already considered relatively recent for the pretraining corpora of most models. As described in Section 3.4, for example, the pretraining dataset of GPT-J only includes data up to and including 2019. Therefore, even though some knowledge has been updated in the years since, the proportion of such updates in the data would be minimal.\n\n> **W5:** The paper's statements are inaccurate. It only considers MLP layer parameters, so it cannot be generalized as a relationship between model size and knowledge.\n\nWe adopt the prevailing perspective in the field of LLM interpretability [1, 2, 4], which posits that the MLP components are the primary parameters responsible for storing knowledge in the model, while the attention components are more focused on transporting that knowledge, and other components serve additional distinct roles. These points have been discussed in our Limitations section — Section A in the Appendix. Furthermore, the view that value vectors within MLPs act as units of knowledge storage [2, 4] provides us with a more practical way for investigating and quantifying parameter specialization across models.\n\nConsequently, in our study of knowledge storage patterns, we focus our analysis on the MLP components, which we believe is sufficient for current efforts in quantifying factual knowledge in LLMs. In future work, we will expand our analysis to include a broader range of parameters from other components within the transformers.\n\n\n> **Q1:** Why is 0 used in Formula 4 instead of random noise?\n\nLet us consider a hypothetical scenario: suppose there are ten value vectors, each contributing equally to the representation of Harry Potter domain knowledge, with their feature directions aligned and coefficients all set to 0.1 (see Equation 2 for the definition of coefficients). If we reduce the influence of one of these value vectors (i.e., set its coefficient to zero), the representation of Harry Potter knowledge remains unaffected due to the presence of nine redundant carriers of the same information. This aligns with the realistic expectation of robustness in knowledge representation.\n\nHowever, if we instead add a noise vector of the same magnitude but in a random direction (which is likely orthogonal to the original target direction [1]) to that value vector—even with a scale as small as 0.1 (a commonly used scale in recent intervention studies [2, 3])—the resulting direction can deviate significantly. This may lead to a notable distortion in the overall representation, falsely suggesting a change in the importance of the targeted knowledge.\n\nWe therefore believe that **ablation (or “knock-out”)** provides a more accurate means of assessing the contribution of specific parameters to a given piece of knowledge, compared to **noise-based interventions**.\n\n\n—\n\n**References:**\n\n[1] Locating and Editing Factual Associations in GPT\n\n[2] Transformer Feed-Forward Layers Are Key-Value Memories\n\n[3] Open Problems in Mechanistic Interpretability\n\n[4] Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space"}, {"review_id": "Y59avmoovi", "reviewer": "Reviewer_kanx", "summary": "The authors analyze the MLP layers of 20 open-source LLMs to determine the relationship between performance and the way knowledge is stored. They build on the key-value interpretation by Geva et al. (2021) and inspect the absolute values of the intermediate representations between the up- and down-projection matrix in the MLP layers. In their emprical analysis, they authors use a newly developed knowledge benchmark (SpecWiki), based on Wikipedia, which covers knowledge concepts with varying frequencies. They define a new metric, called the Parameter Specialization Score (PSS) which measures the absolute difference between the model’s post-masking Concept-Specific Score (accuracy on questions about the target concept) and its General Score (accuracy on unrelated control questions) divided by the model’s original overall accuracy. The authors find that as language models become more advanced and demonstrate stronger knowledge capabilities, their parameters exhibit increased specialization, and the specialized distribution of knowledge contributes to improving the efficiency of knowledge utilization in LLMs.", "questions": "- Could you share a sensitivity sweep showing how PSS changes with different masking ratios (and any alternative denominator choices) so we can confirm the size–quality correlation isn’t just a scaling artefact?\n- The authors discuss the concept of Knowledge Superposition in LLMs. Maybe it would be interesting to discuss how your metric relates, or may suffer, from the concepts of mono- and polysemanticity?\n\nBricken, et al., \"Towards Monosemanticity: Decomposing Language Models With Dictionary Learning\", Transformer Circuits Thread, 2023. (https://transformer-circuits.pub/2023/monosemantic-features/index.html)", "limitations": "The authors clearly describe the most important limitations of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenghts:\n- The authors introduce the Parameter Specialization Score (PSS) and a new benchmark (SpecWiki) which is a reproducible and clear metric to quantify how narrowly facts are tied to individual MLP value vectors.\n- The authors include 20 open source LLMs in their analysis which shows the positive relationship between factual accuracy and higher PSS.\n- Showing that fine-tuning only the high-PSS vectors improves recall and lowers hallucinations is a lightweight way to regularize models.\n\nWeaknesses:\n- PSS divides the post-mask accuracy gap by the pre-mask control accuracy, so baseline model strength distorts the score. Apart from one diagnostic sweep in Figure 3, the paper reports only the aggregated PSS values which makes it impossible to inspect the underlying pre- and post-mask accuracies that would reveal which term really drives the correlation. We can’t know whether the size–quality trend is genuine or a scaling artefact.\n- As the authors acknowledge in their limitations section: prior work shows attention heads, router experts and even layer-norm scales also store factual info (e.g., Geva et al., 2023).", "was_revised": true, "final_justification": "The authors have responded to some of my concerns and overall I believe this paper could be a valuable contribution to NeurIPS. However, the limited originality compared to earlier work makes me keep the score of 4 (borderline accept).", "rebuttal": "We sincerely thank Reviewer kanx for the thoughtful and encouraging feedback. We truly appreciate your recognition of our proposed Parameter Specialization Score metric, the design of the SpecWiki benchmark, and the significance of our large-scale analysis across 20 open-source LLMs. Given the valuable questions you raised, we have provided detailed responses below and would be happy to address any further questions or concerns the reviewer may have.\n\n> **W1 & Q1:** PSS divides the post-mask accuracy gap by the pre-mask control accuracy, so baseline model strength distorts the score. Apart from one diagnostic sweep in Figure 3, the paper reports only the aggregated PSS values which makes it impossible to inspect the underlying pre- and post-mask accuracies that would reveal which term really drives the correlation. We can’t know whether the size–quality trend is genuine or a scaling artefact... Could you share a sensitivity sweep showing how PSS changes with different masking ratios (and any alternative denominator choices) so we can confirm the size–quality correlation isn’t just a scaling artefact?\n\n\nThank you for the great questions.\n\nTo better address your concerns, we would like to further clarify and explain the formulation of the Parameter Specialization Score (PSS) we proposed.\n\n1. Specifically, we first calculate the difference between the model’s average performance on the unrelated questions set and its average performance on the concept-related dataset, after ablating the located target parameters. **This difference represents how much of the knowledge contained in the located target parameters is both parameter-specific and concept-related.** Since each model performs differently on the SpecWiki dataset, we use its original (unintervened) performance as the denominator to serve as a proxy for the total knowledge encoded in the model. **This allows us to finally calculate what proportion of the concept-related, parameter-specific knowledge accounts for the model's overall knowledge.**\n\n2. Suppose we do not include the denominator in the PSS formula. Consider two 7B models—one with stronger performance (e.g., achieving 80% accuracy on SpecWiki) and another with weaker performance (e.g., 40% accuracy). If we ablate the same 10% of MLP parameters, and both models lose the same proportion of parameter-specific, concept-related knowledge relative to their total knowledge (i.e., the drop in the numerator accounts for 20% of the denominator in both cases), the stronger model will show a larger absolute performance drop in the numerator (0.8 × 20% vs. 0.4 × 20%). This would make it appear to have a higher degree of parameter specialization, but in reality, it simply contains more target knowledge to begin with, as evidenced by its stronger original performance.\n\nMoreover, the results in Figure 2 show that stronger-performing models tend to exhibit higher Parameter Specialization Scores. If we remove the denominator—which currently offsets the advantage of stronger models—the trend in Figure 2 would appear even steeper and more pronounced.\n\n\n> **W2:** As the authors acknowledge in their limitations section: prior work shows attention heads, router experts and even layer-norm scales also store factual info (e.g., Geva et al., 2023).\n\nAlthough subsequent studies have claimed that other components within LLMs may also store factual knowledge (as we discuss in our Limitations section), we follow the prevailing perspective in the field of LLM interpretability, which posits that the MLP components are the primary parameters responsible for storing knowledge in the model. Furthermore, the view that value vectors within MLPs act as units of knowledge storage offers us a more practical way to investigate and quantify parameter specialization.\n\nConsequently, in our research on knowledge storage patterns, we concentrate our investigation on the MLP components. In future work, we will extend our analysis to include a broader range of parameters across other components of the transformers.\n\n> **Q2:** The authors discuss the concept of Knowledge Superposition in LLMs. Maybe it would be interesting to discuss how your metric relates, or may suffer, from the concepts of mono- and polysemanticity?\n\nThank you for pointing out the important concept of Knowledge Superposition again.\n\nThe knowledge superposition discussed in this paper **is indeed related to the concepts of mono- and polysemanticity.** For example, the word \"light\" can mean both **\"not heavy\" and \"illumination.\"** Although these meanings are associated with the same token, we believe that the knowledge parameters representing the \"not heavy\" meaning should be grouped more closely with other parameters that also represent the concept of lightness in weight, while the parameters representing \"illumination\" should be closer to those responsible for knowledge related to light or darkness. **This corresponds to a lower degree of knowledge superposition and greater parameter specialization.**\n\nAccording to the experiments and conclusions of this paper, such an arrangement allows the model to retrieve the target knowledge more efficiently and reduces hallucinations."}]

Author rebuttal:

Meta-review: This paper introduces the interesting concept of "parameter specialization" to explain how knowledge is organized within large language models. The reviewers were in agreement about the paper's primary strengths: a large-scale, rigorous analysis across 20 open LLMs, that provides compelling evidence that model performance is highly correlated with increased parameter specialization. The work represents a significant contribution to the field of LLM interpretability by offering a new perspective and a reproducible methodology for analyzing knowledge storage patterns. The review process raised important questions about the precise interpretation of the proposed PSS metric, and the paper would benefit from additional clarifications regarding the definition of the metric and the expanded limitations section. However, the consensus is that the paper's strengths and the importance of its findings strongly warrant its acceptance.
Decision: Accept (poster)