Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods (2024)

Abstract: Language Models (LMs) acquire parametric knowledge from their training process, embedding it within their weights. The increasing scalability of LMs, however, poses significant challenges for understanding a model’s inner workings and further for updating or correcting this embedded knowledge without the significant cost of retraining. This underscores the importance of unveiling exactly what knowledge is stored and its association with specific model components. Instance Attribution (IA) and Neuron Attribution (NA) offer insights into this training-acquired knowledge, though they have not been compared systematically. Our study introduces a novel evaluation framework to quantify and compare the knowledge revealed by IA and NA. To align the results of the methods we introduce the attribution method NA-Instances to apply NA for retrieving influential training instances, and IA-Neurons to discover important neurons of influential instances discovered by IA. We further propose a comprehensive list of faithfulness tests to evaluate the comprehensiveness and sufficiency of the explanations provided by both methods. Through extensive experiments and analysis, we demonstrate that NA generally reveals more diverse and comprehensive information regarding the LM’s parametric knowledge compared to IA. Nevertheless, IA provides unique and valuable insights into the LM’s parametric knowledge, which are not revealed by NA. Our findings further suggest the potential of a synergistic approach of combining the diverse findings of IA and NA for a more holistic understanding of an LM’s parametric knowledge.

heading: 1 Introduction
text: Language Models encode the knowledge acquired during training as numeric values within the models’ weights, transforming raw information from the training dataset into structured, internal representations. This embedding of knowledge, while
fundamental to an LM’s functionality, renders the model’s inner workings opaque. To unravel the internal mechanisms of an LM and investigate the parametric knowledge encoded in an LM’s weights, the development of eXplainable AI (XAI) methods is paramount.
Research in this domain has explored LM’s parametric knowledge with various attribution methods (Bejan et al., 2023; Fan et al., 2023). Commonly used among these are Instance Attribution (IA, Koh and Liang (2017); Charpiat et al. (2019); Garima et al. (2020)) and Neuron Attribution (NA, Dai et al. (2022); Meng et al. (2022)). IA identifies training instances that influence the model’s parametric knowledge leveraged for its prediction on a test instance. However, despite their utility, IA methods have been criticized for their sensitivity
8173
to the choice of hyperparameters and for producing highly homogenous results (Pezeshkpour et al., 2021). On the other hand, NA locates specific neurons that hold the most important parametric knowledge for prediction on a test instance. While NA has proven valuable in locating and editing the structured knowledge within a model (Dai et al., 2022; Meng et al., 2022), the granular nature of the neuron analysis presents challenges in the interpretation of the outcomes, necessitating manually defined human concepts for interpretation (Sajjad et al., 2022).
While IA and NA present different views on the parametric knowledge employed in an LM’s prediction, no existing work has contrasted the two techniques to determine if they offer similar or complementary insights that could lead to a more comprehensive understanding of an LM’s inner workings. This paper seeks to bridge this gap by establishing a unified evaluation framework, illustrated in Figure 1, that allows for the comparison of these disparate attribution methods, particularly focusing on their application for autoregressive LMs.
Aligning the Results of Attribution Methods. To allow for comparing the results obtained with NA to those obtained with IA, we first introduce a novel attribution method NA-Instances (bottom left, ‘Attribution Results’, Figure 1; §3.4) that retrieves the training instances sharing the most similar neuron activations for each test instance. NA-Instances allows one to interpret the granular NA results and to compare its results to IA. On the other hand, to align the results of IA with NA, we introduce IANeurons (bottom right, ‘Attribution Results’, Figure 1; §3.3), which finds the neurons of the most important training instances discovered by IA that have the highest activation.
Unified Evaluation Framework for Attribution Methods. With the aligned results from IA and NA, we introduce two evaluation schemes. Interpretation of the attribution results is important, but it is also crucial to check whether the given explanation fully represents the model’s underlying behavior (Ross et al., 2017). Therefore, firstly, faithfulness tests are designed to assess whether the neurons discovered by each method are both sufficient and comprehensive in representing the parametric knowledge used by the model for a prediction (bottom left, Figure 1;§3.3). Note that previous work has only performed the comprehensiveness test for NA (Dai et al., 2022), while such evaluations have not been extended to IA. We find
that most neurons can be removed without a significant number of changed predictions compared to the original model, revealing that much of the model’s knowledge is not stored in neurons located in the Multilayer Perceptron Layers (MLP). Secondly, we conduct fine-tuning with a varying number of influential training instances discovered by the two IA methods (bottom right, Figure 1;§3.4) to assess how sufficient the training instances are in representing the parametric knowledge used by the model for a prediction. We find that IA and NAInstances perform on par in both finding sufficient and comprehensive neurons and sufficient influential training instances. In addition to this, results show that influential training instances obtained by IA methods do not yield a better-fine-tuned model than randomly selected training instances.
Characterising Differences between Attribution Methods. We conclude with an extensive analysis (§6) of the characteristics of the attribution methods focusing on their diversity and utility. By comparing the group of selected instances and neurons (§6.1), we first observe that the instances discovered by NA-Instances and IA have a very small overlap. In contrast, the group of neurons from IA-Neurons highly overlaps with the group of neurons from NA, where the latter discovers a large group of neurons in addition to the overlapping ones. Furthermore, from the analysis (§6.2) and results from §5.3, we find that the more diverse the group of influential training instances, the better the performance of the model fine-tuned with the group. Finally, results from §6.3 show that NA-Instances performs better at discovering dataset artifacts than IA methods. Overall, our findings call for future work incorporating both IA and NA methods for a better understanding of LM’s parametric knowledge.

heading: 2 Related Work
text: To understand how models encode and utilise parametric knowledge, researchers have employed a variety of approaches (Xiong et al., 2024). Among these, Instance Attribution and Neuron Attribution have emerged as two principal methodologies.
Instance Attribution (IA). IA identifies the training instances most influential to a model’s prediction for a given test instance. IA’s primary strength is that it provides a human interpretable explanation of the model’s encoded parametric knowledge. Koh and Liang (2017) first introduced
the Influence Function (IF), a method to calculate the impact of each training instance on a model’s prediction, utilizing gradients and Hessian-vector products. Additional methods in this domain include Input Similarity (Charpiat et al., 2019), and TracIn (Garima et al., 2020), which explore similar concepts from different perspectives. Charpiat et al. (2019) defined the notion of similarity of inputs from the neuron network perspective and applied the method to retrieve the training instances close to the test instance. Further, TracIn proposed to track an individual loss of a test instance by looking at the moment when a certain training instance is introduced. We note that most IA methods are developed upon image datasets and models, rather than NLP-specific downstream tasks. This calls for the development of IA methods specialized in NLP tasks, especially in the era of LMs. From an NLP perspective, Pezeshkpour et al. (2022) combined IA methods with feature attribution methods to identify the part of the training instance that influences test prediction. Notably, Han et al. (2020) applied IF across various NLP tasks, revealing its potential to detect and correct training data-induced biases.
Neuron Attribution (NA). Dai et al. (2022) first coined the term “knowledge neuron” referring to the medium where the knowledge lies within the model. Their work illustrated that the factual knowledge encoded within a model’s parameters could be altered by manipulating these specific neurons. Following this, Meng et al. (2022) refined the process of locating knowledge within LMs through causal tracing, demonstrating that feed-forward layers in the model’s mid-blocks play a crucial role in encoding factual knowledge. On the contrary, Hase et al. (2023) claimed that the knowledge neurons discovered by the causal tracing method often do not match the actual model weights that can modify the particular knowledge. This ongoing debate highlights the lack of consensus regarding the utility of NA.
Evaluation of Attribution Methods. Several works have assessed the utility across attribution methods of the same group. Notably, a recent investigation into Instance Attribution (Gu et al., 2023) examines three distinct methods, evaluating them with four metrics – sufficiency, completeness, stability and plausibility. Regarding NA, Fan et al. (2023) introduced an evaluation framework and compared six different NA methods. They assessed the NA methods with two compatibility metrics which are inspired by voting theory. One is called
AvgOverlap and the other is called NeuronVote. They return one method’s compatibility score with respect to other methods using a ranked list of important neurons. Despite these advancements, there remains a gap in the literature regarding a direct comparison between IA and NA methods, which are both used to reveal the parametric knowledge used by an LM. This paper aims to bridge this research gap by presenting a unified framework that allows for a comprehensive comparative analysis of the two methods.

heading: 3 An Evaluation Framework for Attribution Methods
text: We next describe the proposed unified evaluation framework for attribution methods. To allow for the comparison of the two attribution methods IA and NA, we first propose to align their results to common views over the LM’s parametric knowledge (§3.2). Then, we describe the tests employed in the evaluation framework to assess whether the uncovered methods reflect the reasons employed by the models in their predictions (§3.3, §3.4).

heading: 3.1 Preliminaries
text: Consider a test dataset XTest = [x1, x2, . . . , xn], consisting of n test instances and a train dataset XTrain = [xtrain1 , x train 2 , . . . , x train m ] consisting of m training instances. Consider also an LM f used to make a prediction on an instance: f(x) = ŷ. Furthermore, we introduce an attribution method A ∈ {IA,NA, IA − Neurons,NA − Instances}. For each test instance xi, and a model making a prediction f(xi), A computes influence scores of the training instances Sf(xi) = [straini,1 , s train i,2 , . . . , s train i,m ] designating the extent of influence of the corresponding training instance on the learned parametric knowledge of an LM employed in its prediction on xi. A also returns r most important neurons Nf(xi) = [n1i , n2i , . . . , nri ] and their corresponding attribution scores NSf(xi) = [ns1i , ns 2 i , . . . , ns r i ] which are regarded as important for the model’s prediction on xi.

heading: 3.2 Aligning the Results of Attribution Methods.
text: First, to compare the results of IA to the list of most important neurons discovered by NA, we introduce IA-Neurons. IA-Neurons first produces a list of scores for the influence of the training instances Sf(xi) and takes r most influential
of them: [xtraina , x train b , . . . , x train r ]. Then, for each instance in the list, the NA method collects the most important neuron, arriving at the list of top-1 important neurons of the r most influential training instances for a prediction: S ◦ N f(xi) = [n1
xtraina , n1 xtrainb , ..., n1 xtrainr ].
Second, to compare the NA results to the influence scores of the training instances, we introduce NA-Instances. To design NA-Instances, we propose Discounted Cumulative Neuron Similarity (DCNS) based on the Discounted Cumulative Gain evaluation measure (Järvelin and Kekäläinen, 2002) used for the evaluation of ranked results in Information Retrieval. It takes both the rank of the retrieved item and its relevance score into account. Inspired by this, we utilise DCG to score training instances based on their neuron activation similarity with a given test instance. Thus, for a neuron that is important for the test instance prediction, we consider both the importance rank and the attribution score of the neuron from a training instance prediction. Equation 1 shows the scoring function that takes the test instance xi’s important neuron list Nf(xi) = [n1i , n 2 i , . . . , n r i ], the train instance xt’s important neuron list Nf(xt) = [n1t , n2t , . . . , nrt ] and its attribution scores NSf(xt) = [ns1t , ns2t , ..., nsrt ]. It calculates the influence score gtraini,t of training instance xt for the prediction of test instance xi.
gtraini,t = ∑
m=[1,r];nmt ∈Nf(xi)
2ns m t − 1
log2(m+ 1) (1)
Similar to the DCG metric, we penalize the attribution score of the common important neuron with its rank on the list of important neurons for the training instance. In summary, a particular training instance is influential for a test instance, when the important neurons of the test instance are highly ranked for the training instance.

heading: 3.3 Neuron Attribution Faithfulness Tests
text: We employ two faithfulness tests to assess which Neuron Attribution method returns a list of most important neurons Nf(xi) for an LM’s prediction on a test instance xi that is more faithful to the parametric knowledge employed by the LM in its prediction. Sufficiency and Comprehensiveness are two representative evaluation metrics of faithfulness. To evaluate Sufficiency, we only leave the selected r most important neurons Nf(xi) as activated, setting the activation of the remaining neurons Nf(xi) to zero. If the original prediction
of the model f is preserved, we claim that the neurons in Nf(xi) are sufficient to reveal the parametric knowledge employed by f in its original prediction:
f(xi; zero(Nf(xi))) = ŷi; f(xi) = ŷi′
If ŷi = ŷi′, Nf(xi) is sufficient. (2)
We also employ a Comprehensiveness measure to estimate whether the explanation is complete and includes all parametric knowledge that is necessary to understand the model’s inner workings. Contrary to Sufficiency, the most important neurons Nf(xi) are discarded by setting their activation to zero; we then verify whether the prediction changes:
f(xi; zero(Nf(xi))) = ŷi; f(xi) = ŷi′
If ŷi ̸= ŷi′, Nf(xi) is comprehensive. (3)
For the final Sufficiency and Comprehensiveness measures, we count the number of instances in the test set XTest where Eq. 2 and Eq. 3, correspondingly, hold.

heading: 3.4 Fine-tuning with Influential Training Instances
text: To evaluate the effectiveness of an attribution method in discovering the training instances that affect the learned parametric knowledge by the model, we conduct IA Faithfulness tests. To do so, we fine-tune a model f only with the most influential training instances Si and estimate the number of preserved predictions on the test set XTest.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: In exploring natural language understanding, our research is directed towards tasks that necessitate complex reasoning and engage deeply with a model’s acquired task knowledge. Our experiments involve Natural Language Inference (NLI), Fact-Checking (FC) and Question Answering (QA). We focus on selecting both diverse and complex tasks, thus, providing a robust platform for assessing the efficacy of IA and NA methods. One of the selected datasets – MNLI, is also commonly employed for evaluating attribution methods in related work. For NLI, the Multi-Genre Natural Language Inference (MNLI) is used (Williams et al., 2018). Due to the computational cost, we sampled 10K training instances from the training dataset (393K in total) following Pezeshkpour et al. (2021)’s work
on the empirical comparison of IA methods. Next, we choose AVeriTeC (Schlichtkrull et al., 2023), a highly curated FC dataset. It reflects the characteristics of real-world claims, thus requiring the model to go through more complex reasoning. Finally, to investigate the attribution methods’ explanation on a QA task, we work with COmmonSense QA (CoS-QA) (Talmor et al., 2019), a multiple-choice Question Answering dataset built upon an external knowledge graph (Speer et al., 2018).1

heading: 4.2 Models
text: We select three open-sourced different autoregressive language models including OPT-125m (Zhang et al., 2022), BLOOM-560m (Yong et al., 2023) and Pythia-410m (Biderman et al., 2023). To see the impact of model size on the attribution explanations, all models are in different sizes. OPT is trained with a de-duplicated version of the Pile dataset (Gao et al., 2020). For Pythia, we choose the version that is trained with the original Pile dataset, containing duplicated files. BLOOM is oriented to the scientific domain employing multilingual scientific pretraining datasets. The models’ performances on the tasks are available in Appendix A. To fine-tune the LMs, we use a Sequence Classification Head on the last token’s hidden representation and train the models with cross-entropy loss. We provide further details about fine-tuning in Appendix A.

heading: 4.3 Attribution Methods
text: For IA methods, we consider two representative IA methods, IF (Koh and Liang, 2017) and Gradient Similarity (GS) method (Charpiat et al., 2019). Given a model f and test instance xi, IF (Equation 4) computes the importance score of training instance xtraina : s train i,a by upweighting xtraina with the Hessian of the loss function, dϵi df = −H−1f ∇fL(xtraina , ŷa, f). We refer to Koh and Liang (2017) for more details. GS (Equation 5) takes the dot product of the gradients of xi and xtrainm . They are defined as follows:
straini,a = ∇L(xi, ŷi, f)T · dϵi df
(4)
straini,a = ∇fL(xi) · ∇fL(xtraina ) (5) 1Although it is common to provide the external knowledge source to the model for the CoS-QA, we focus on training with the question-answer pairs only as we aim to investigate a model’s parametric knowledge.
Due to the heavy computational cost, the MLP classification layer weight is used to attain the gradients instead of the entire model weights. Prior work showed that the attribution results from the entire weight and the MLP classification layer show a high correlation (Pezeshkpour et al., 2021).
For NA, we adapt the application of integrated gradients (Sundararajan et al., 2017) for discovering important neurons (Dai et al., 2022). The neuron attribution score nsli of l
th neuron on f(xi) is:
nsli = wl m
m∑
k=1
∂Px( k mwl)
∂wl (6)
where wl is the neuron activation value of the lth
neuron from the MLP layer. Equation 6 leverages Riemann approximation by scaling the neuron’s activation value from 0 to its original value. For implementation, we modify the open source repository.2

heading: 4.4 Attribution Method Tests
text: For neuron attribution, we employ faithfulness tests covering both the sufficiency and comprehensiveness of the neurons to represent the model’s parametric knowledge. For Sufficiency, we take the r = 1 neuron with the highest attribution score. For Comprehensiveness, r = 100 neurons are selected. To verify the utility of the neurons selected by the attribution methods, we also choose the same number of neurons randomly and present the result as a Random baseline.

heading: 5 Results

heading: 5.1 Number of Unique Influential Instances
text: Table 1 showcases the number of unique instances in the collection of the 10 most influential training instances for each test instance. Generally, we observe that IF and GS usually identify a smaller set of unique instances compared to NA-Instances – e.g., 30 vs 1776 unique instances discovered by IF/GS vs. NA-Instances correspondingly for MNLI and BLOOM-560m. An exception is the CoS-QA dataset, where we hypothesise that the majority of training instances in the CoS-QA dataset have fewer common neurons with the test instances as the test dataset of CoS-QA has fewer concepts in common with the training dataset. Empirically, we find that although most of the concepts3 in the test dataset were covered in the training dataset, the concepts from the test dataset only represent a small proportion of all concepts within the training dataset. There were 2099 unique concepts in the training dataset and only 725 concepts were used in the test dataset. This finding obtained from NAInstances is in line with Huang et al. (2023)’s work, which claims that each neuron corresponds to a concept in an instance.
Further, we observe that increasing the model size correlates with a decrease in the number of unique training influential instances. This trend underscores the tendency towards homogeneity in the results produced by IA methods. Moreover, it highlights the potential advantage of the combination of IA and NA methods to retrieve a more
2https://github.com/EleutherAI/knowledge-neurons 3The CoS-QA dataset provides concept annotations for
each instance.
diverse set of influential training instances. We further investigate the heterogeneity of the group of influential training instances in §6.

heading: 5.2 Neuron Attribution Faithfulness Tests
text: We evaluate the sufficiency and comprehensiveness of the neurons to represent the model’s parametric knowledge. Table 2 presents the number of the test instances for which the predictions are changed after keeping (Sufficiency) or suppressing (Comprehensiveness) the activation on the selected neurons. We find that NA performs slightly better than other attribution methods for AVeriTeC and MNLI. Furthermore, there are marginal differences between the Random baseline and IF-Neuron/GD-Neuron with less than 1.0 on both Sufficiency and Comprehensiveness. This implies that the explanations from IF-Neuron/GD-Neuron are not sufficient nor comprehensive enough to reveal the complete parametric knowledge used in the model’s prediction.
Notably, although we suppress the activation of all neurons within the MLP layer except one, the model can still recover the original prediction. We ascribe this phenomenon to the attention layers within the Transformer blocks. This finding aligns with Wiegreffe and Pinter (2019)’s argument, that attention weights pose a meaningful impact on the model’s prediction and are important for understanding a model’s inner workings. For a more holistic understanding of a model’s parametric knowledge, we thus suggest that future work also studies attention-based neuron attribution methods.

heading: 5.3 Fine-tuning with Influential Training Instances
text: The models’ performances from fine-tuning with different numbers of influential training instances are presented in Figure 2. We also provide the performances on the different datasets and models in Appendix B, Figure 6. From the figure, we find that the accuracies achieved with the first n% training instances are meaningful enough to show the different impact on the performance between n% most influential training instances and n% least influential training instances. The biggest gap in the accuracy achieved between training with the most and least influential ones is from the NA-Instances method – an accuracy gap of 0.6 for the AVeriTeC dataset. However, given that selecting the same proportion of training instances at random outperforms the attribution methods, we conclude that the influential training instances selected by IA methods (IF, GS) do not provide any benefit for explaining the performance of the final model. Unexpectedly, the training instances selected by NA-Instancesleast achieve better performance in general than the randomly selected ones on the MNLI dataset. Although NA-Instances-least shows a different trend on the AVeriTeC and MNLI datasets, it outperforms other least influential groups. Since the group is composed of training instances that have minimal neuron overlapping with the test instances, we attribute this high performance to the instances in the set selected by NA-Instances-least being more diverse (as seen in general for instances discovered by NA-Instances in Table 1) leading to encompass-
ing a more diverse set of the model’s parametric knowledge.

heading: 6 Analysis
text: Next, we investigate what are the characteristics of the group of influential training instances and the group of most important neurons.

heading: 6.1 Overlap of the Attribution Results
text: Here we look at the overlap of influential instances as well as the overlap of the important neurons discovered by the corresponding attribution methods. First, we investigate the overlap between the first n influential training instances discovered by IF, NA-Instances, and GS, which are then used in the evaluation framework for fine-tuning with in-
fluential training instances (§3.4). Figure 3 shows that for IF and GS, the overlap percentage is high – greater than 80%. This also explains their similar performance on the fine-tuning with influential training instances test (§5.3). Furthermore, compared to the instance attribution methods IF and GS, NA-Instances discovers very different influential instances. For the first 10% of most influential instances discovered by each method, we find that NA-Instances and IF or GS have fewer than 20% instances that are discovered by both methods, which amounts to roughly under 2 influential instances.
Second, we present the proportion of overlapping top-n important neurons selected by NA and IF-Neurons in Figure 4. Results on the overlap of neurons discovered by NA and GS-Neurons show similar trends and can be found in Appendix C, Figure 5. Similar to the diversity of top-n influential training instances, the proportion of unique important neurons found by NA is again higher than those found by IF-Neurons. In addition, we find that most of the neurons found by IF-Neurons
are included in the set of NA. The analytic results from both perspectives underscore the potential of NA methods to reveal the source of the parametric knowledge.

heading: 6.2 Diversity Analysis on the Group of Influential Training Instances
text: From the evaluation results in §5.3, we hypothesize that greater diversity of the influential training instances found by an attribution method yields better performance, which we verify here. The heterogeneity of different groups of influential training instances can be measured at the lexical and parametric levels. To estimate lexical diversity, we compute the number of unique tokens (Vocabulary in Table 3) from the group of influential training instances and the average length of the training instances (Input Length in Table 3) as model input. The cosine similarity between the influential instances with the hidden representations from the last Transformer block (Cosine Similarity in Table 3) and the average loss (Loss in Table 3) are reported to show the parametric diversity of the selected influential training instances.
Table 3 presents the result of this analysis on the AVeriTeC dataset and the MNLI dataset with the OPT-125m model, following the previous section. We find that the Random and NA-Instancesleast methods that show a performance of 0.55 accuracy from Figure 2 contain more than 6900 unique tokens while other methods with less than 0.40 accuracy have 6600 tokens on average. From the parametric diversity metrics, the methods with lower performance collect training instances with a similar distribution of hidden representations and bigger losses. Furthermore, the least influential training instances discovered by IA methods have higher losses compared to the ones discovered by NA methods. However, we observe that the loss
is not an indicator for the most or least influential training instances affecting the model’s test set performance from the NA-Instances perspective.
To verify our findings statistically, we implement a simple linear regression model (Pedregosa et al., 2011) that verifies the association between the diversity metrics and the accuracy. Negative coefficients w.r.t. the model’s performance show both Cosine Similarity of -0.3741 (MNLI), -1 (AVeriTeC), and Loss of -0.3563 (MNLI), -0.1719 (AVeriTeC). These results support our hypothesis on the relationship between the diversity of selected training instances and their effectiveness in reaching a high performance with the fine-tuned model.

heading: 6.3 Dataset Artifact Detection
text: For a more human-interpretable analysis of the benefits of each IA method, we conduct a dataset artifact detection analysis with the NLI dataset HANS (McCoy et al., 2019), containing instances representing heuristics that NLI models are likely to learn. We take the test instances designed to contain lexical overlap heuristics between the premise and hypothesis to see what types of training instances are found as influential for a model’s prediction on the said test instances. We run each instance attribution method on the MNLI training dataset and report the results in Table 4. We find that our proposed approach NA-Instances generally discovers more training instances which have a higher lexical overlap rendering the method to perform better in finding artifacts learned by a model from its training dataset.

heading: 7 Conclusion
text: In this paper, we propose a unified evaluation framework for comparing and contrasting two different attribution methods, IA and NA. To do so, we
first introduce new attribution methods that align the neurons discovered by IA and NA. We assess the sufficiency and comprehensiveness of the explanations for both methods with different faithfulness tests and conduct extensive analyses of their explanations to further investigate the distinct characteristics that yield different results.
Through the proposed evaluation framework, we confirm that IA and NA result in different explanations about the knowledge responsible for the test prediction. This implies the potential advantage of combining IA and NA to unveil a comprehensive view of the LM’s parametric knowledge. In addition, the experimental results on the attribution methods’ faithfulness suggest that the neurons are not sufficient nor comprehensive enough to fully explain the parametric knowledge used for the test prediction. To complement this drawback of the current attribution methods, we hypothesize that this is due to the importance of the attention weights for encoding knowledge, leaving this exploration for future work.
Limitations
Our paper presents an investigation of the knowledge encoded by instance and neuron attribution methods. We perform our experiments using two instance attribution methods, one neuron attribution method, three natural language understanding datasets for three different tasks, and three language models. While we aimed to make a representative selection, future work should investigate other natural language processing tasks not focused on language understanding, as well as more different language models, including very large ones, which we could not study due to computational restrictions. It is also worth noting that the benchmark datasets we used consist of English text only, and we did not study domain adaptation or crosslingual transfer scenarios.
One of our core findings was that selecting (in the case of sufficiency) or occluding (for comprehensiveness) knowledge neurons in the MLP layer (see §5.2) only tells half the story – models still perform astonishingly well given only the one most important neuron. This means that, to truly understand the parametric knowledge of LLMs, the interplay between the neurons in the MLP layer and the neurons in the attention heads should be studied.
Lastly, we found that merely using attribution methods to identify instances to train on might not be a good strategy (see §5.3). This is due to the lack of diversity in the resulting training set (see §6.2). This naturally begs the question of what the purpose of instance and neuron attribution methods then is. §6.3 suggests that the answer could be to identify dataset artifacts. However, these phenomena could still be studied in more depth for different downstream tasks.

heading: Acknowledgements
text: This research was co-funded by the European Union (ERC, ExplainYourself, 101077481), by the Pioneer Centre for AI, DNRF grant number P1, as well as by The Villum Synergy Programme. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. We thank the anonymous reviewers for their helpful suggestions.

heading: B Fine-tuning with Influential Training Instances
text: Figure 6 presents the models’ performances from §5.3. By conducting the evaluation on the sufficiency of the influential training instances, we confirm that various language models show similar trends with the same dataset. For the AVeriTeC, the group of randomly selected training instances (Random) outperforms other groups. On the other hand, the training instances selected by NAInstances/least consistently show better performances than any other groups in general for the MNLI. With the CoS-QA, we observe that the training instances from -most methods perform better than the group of instances from -least methods.

heading: C Overlap among the Group of Important Neurons
text: Figure 5 shows the proportion overlapping top-n important neurons selected by NA and GS-Neurons. We refer §6.1 for details and analysis.

heading: Future Work
text: To complement this drawback of the current attribution methods, we hypothesize that this is due to the importance of the attention weights for encoding knowledge, leaving this exploration for future work. Limitations
Our paper presents an investigation of the knowledge encoded by instance and neuron attribution methods. We perform our experiments using two instance attribution methods, one neuron attribution method, three natural language understanding datasets for three different tasks, and three language models. While we aimed to make a representative selection, future work should investigate other natural language processing tasks not focused on language understanding, as well as more different language models, including very large ones, which we could not study due to computational restrictions. It is also worth noting that the benchmark datasets we used consist of English text only, and we did not study domain adaptation or crosslingual transfer scenarios. One of our core findings was that selecting (in the case of sufficiency) or occluding (for comprehensiveness) knowledge neurons in the MLP layer (see §5. 2) only tells half the story – models still perform astonishingly well given only the one most important neuron. This means that, to truly understand the parametric knowledge of LLMs, the interplay between the neurons in the MLP layer and the neurons in the attention heads should be studied. Lastly, we found that merely using attribution methods to identify instances to train on might not be a good strategy (see §5. 3). This is due to the lack of diversity in the resulting training set (see §6. 2). This naturally begs the question of what the purpose of instance and neuron attribution methods then is. §6. 3 suggests that the answer could be to identify dataset artifacts. However, these phenomena could still be studied in more depth for different downstream tasks