Detecting and Pruning Prominent but Detrimental Neurons in Large Language Models
Authors: Ameen Ali Ali
Shahar Katz
Lior Wolf
Ivan Titov
Venue: COLM 2025 (2025)
Keywords: LLMs
spurious correlations
Integrated Gradients
generalization
model adaptation
TL;DR: We fine-tune LLMs by pruning shortcut neurons using Integrated Gradients, improving generalization and performance on multiple-choice benchmarks.

Abstract: Large language models (LLMs) often develop learned mechanisms specialized to specific datasets, such as reliance on domain-specific correlations, which yield high-confidence predictions without generalizable reasoning. While beneficial in one setting, these dataset-specific mechanisms typically degrade performance when models encounter novel tasks or distributions. In this work, we introduce a fine-tuning approach designed to enhance generalization by identifying and pruning neurons associated with dataset-specific mechanisms in transformer-based LLMs. Our method employs Integrated Gradients to quantify each neuron’s influence on high-confidence predictions, pinpointing those that disproportionately contribute to dataset-specific performance without supporting robust, transferable reasoning. Selectively pruning these neurons compels the model to depend on generalizable representations. Evaluated across multiple-choice benchmarks, our pruning-based fine-tuning significantly enhances performance, surpassing prior (non-pruning) adaptation methods.

Published as a conference paper at COLM 2025
Detecting and Pruning Prominent but Detrimental Neurons in Large Language Models
Ameen Ali, Shahar Katz and Lior Wolf
Ivan Titov
The Blavatnik School of Computer Science Tel Aviv University (ameenal @mail, shaaaaaataaaa wolffcs) tau. ac. il
The University of Edinburgh The University of Amsterdam itievven ed. ac.uk
Abstract
Large language models (LLMs) often develop learned mechanisms special- ized to specific datasets, such as reliance on domain-specific correlations, which yield high-confidence predictions without generalizable reasoning. While beneficial in one setting, these dataset-specific mechanisms typically degrade performance when models encounter novel tasks or distributions. In this work, we introduce a fine-tuning approach designed to enhance generalization by identifying and pruning neurons associated with dataset- specific mechanisms in transformer-based LLMs. Our method employs In- tegrated Gradients to quantify each neuron's 's influence on high-confidence predictions, pinpointing those that disproportionately contribute to dataset- specific performance without supporting robust, transferable reasoning. Selectively pruning these neurons compels the model to depend gener- alizable representations. Evaluated across multiple-choice benchmarks, our pruning- baaed fine-tuning significantly enhances performance, surpassing prior (non-pruning) adaptation methods.
1 Introduction
Large language models (LLMs) have demonstrated remarkable performance across various natural language tasks, including question answering, reasoning, and knowledge retrieval. However, recent studies have revealed a concerning pattern: these models often achieve high performance by exploiting spurious correlations the training data rather than through genuine understanding of the underlying tasks-a phenomenon commonly referred to as "shortcut learning" (Yuan et al., 2024). This manifests when models leverage superficial patterns that correlate with target outputs in the training distribution but fail to capture the causal mechanisms or deeper semantic relationships necessary for reliable generalization to out-of-distribution scenarios. For instance, model might associate specific keywords with particular answer choices in multiple-choice questions rather than developing comprehen- sive understanding of the problem, leading to significant performance degradation when these spurious correlations break down in real-world applications. Previous approaches to mitigating shortcut learning have focused on data-centric solutions, such as augmenting training data with counterfactual examples (Feder et al., 2023), adversarial training (Altinisik et al., 2022), or ensemble methods that combine multiple objectives (Jiang et al., 2023b).
The term shortcut implies that the model reaches a decision more quickly, i.e., at earlier layers. It also suggests reliance on heuristics that may lack robust foundation. However, decision mechanisms encoded as neural pathways can become irrelevant when shifting to new distributions, regardless of their speed or broad applicability within the original training distribution.
Our work focuses on the elimination of such domain-specific mechanisms (DSM) as part of a fine-tuning process, specifically in the context of multiple-choice tasks. This type of task allows us to directly identify mechanisms that may have been beneficial in the onginal domain but become detrimental in the target domain by relying on the prediction accuracy.

---

Published as a conference paper at COLM 2025
Our method is based on the following observations: (i) to eliminate DSM, it sufficient to prune certain neurons within the relevant pathway; (ii) inference mechanisms contributing to the decision-making process can be identified using explainability methods such as Integrated Gradients (Sundararajan et al., 2017); and (iii) eliminating inference mechanisms harmful in the target domain would lead to improved accuracy in that domain.
Based on the first observation, the proposed fine-tuning method prunes specific neurons within a single layer, which, as we demonstrate, is sufficient for effective fine-tuning. Drawing on the second observation, candidate neurons are identified using IG as those most prominent during inference on a small set of fine-tuning samples. This makes our method suitable for few-shot fine-tuning. It is also noteworthy that this stage does not require any label information. The number of neurons pruned and the specific layer are determined based on the accuracy achieved on the same sample set.
One key advantage of our approach is its efficiency in fine-tuning. Unlike full model finetuning strategies, our method operates on minimal set of up to 10 samples to identify DSM neurons. Using this same set, we determine both the optimal layer for pruning and the percentage of neurons to remove. This makes our method highly practical for scenarios where labeled data is scarce or costly to obtain.
Additionally, because pruning is applied selectively within single layer, the cost remains significantly lower compared to full fine-tuning or re-training approaches. Despite its simplicity, our method yields substantial improvements in multiple-choice benchmarks. As can been seen from Figure 1, our method improves results across three model architectures: LLaMA 3.1-8B Instruct, Mistral-7B-v0.3 Instruct, and Qwen2.5-7B Instruct.
2 Related Works
Research on improving generalization in large language models (LLMs) spans several inter- connected areas. This section reviews prior work in (1) shortcut learning in neural networks, (2) neuron-level interpretability in LLMs, (3) attribution methods for deep networks, and (4) model pruning approaches.
Shortcut Learning in Neural Networks Shortcuts refer to the phenomenon whereby mod- els generate responses using shallow heuristics rather than engaging in deep semantic understanding of a given task. LLMs are known to frequently rely on such shortcuts (Du et al., 2023; Ho et al., 2022; Levy al., 2023), allowing them to address various reasoning tasks by exploiting superficial patterns, such as common n-grams. However, this reliance may prevent the models from fully capitalizing on their inherent reasoning capabilities, ulti- mately resulting in suboptimal performance. Notably, Du et al. (2021) employed Integrated Gradients (Sundararajan et al., 2017) to identify input tokens that function as shortcuts in encoder-only models such as BERT (Devlin et al., 2019). Subsequent studies have extended this line of research by utilizing saliency-based gradient methods and token statistics to detect shortcuts either in model inputs or within training datasets (Bastings et al., 2022; Zhao et al., 2024; Friedman et al., 2022; Tang et al., 2023), proposing enhanced training strategies to mitigate shortcut reliance. In this work, we investigate whether pruning a small subset of LLM parameters can reveal latent reasoning abilities that are otherwise suppressed by shortcut-related parameters.
Neuron-level Interpretability in LLMs Research on neuron-level interpretability in LLMs has evolved to focus on the detailed roles played by individual neurons. Early studies revealed that some neurons store specific types of information. For instance, the Knowl- edge Neurons hypothsis (Dai et al., 2022) showed that particular neurons can hold factual details, prompting further investigation into how such localized information influences model behavior. Two main approaches have been employed to study neuron function: activation analysis and attribution methods. Activation analysis examines the responses of neurons to a variety of inputs, uncovering patterns that correlate with features such as syntax, semantics, recurring n-grams and confidence (Voita et al., 2023; Gurnee et al., 2024; Stolfo et al., 2024). Attribution methods, like Integrated Gradients (Sundararajan et al.,
2

---

Published as a conference paper at COLM 2025
2017), estimate the contribution of individual neurons to the final output. Together, these techniques allow researchers to classify neurons into functional groups and identify those that are critical for specific behaviors. Moreover, recent work has applied these methods to multilingual models (Tang et al., 2024), distinguishing between neurons that operate language-specific manner and those that language-agnostic, Ali et al. (2024) proposed modification to the Integrated Gradient method to allow the discovery of copying neurons under the In-Context learning settings.
Model Pruning and Parameter-Efficiene Fine-tuning LLMs are typically trained for general-purpose applications, enabling them to handle a wide range of tasks in a zero- shot setting. To enhance performance specific tasks, common approach is to continue training the models on targeted samples process known as fine-tuning (Houlsby et al., 2019; Brown et al., 2020; Han et al., 2024). However, fine-tuning is computationally expensive and demands more than just a handful of examples. To mitigate the computationally chal- lenge, more efficient strategies such as LoRA (Hu al., 2022) have been developed, which focus on modifying only small subset of the model's parameters to reduce computational overhead.
However, fine-tuning, as well as fine-tuning with reinforcement learning from human feed- back (RLHF) (Ouyang et al., 2022), requires extensive annotated data, which are not always available. Addressing this limitation, recent studies propose methods that adapt models using only a few training samples. One notable example Inference- Time Intervention (TTI) (Li et al., 2023a), which enhances performance on QA benchmarks by steering the activations the attention modules using merely 40 annotated samples map the geo- metric direction associated with truthfulness. Similarly, SADI (Wang et al., 2025) employs steering mechanism on both model activations and the neurons within the MLPs' layers with just 150 samples.
In this work, we demonstrate that disabling a small subset of parameters by zeroing them out rather than retraining can outperform previous state-of-the-art methods using fewer samples and without relying on annotated labels.
can
3 Method
In this section, we describe our novel approach for enhancing multiple-choice performance in transformer-based large language models (LLMss via targeted neuron pruning. Our method is motivated by three key ideas. First, we analyze the role of multi-layer perceptrons (MLPs) within transformer architectures, which are central to encoding and processing the bulk of an LLM's knowledge. Second, we build upon the concept of knowledge neu- rons-specialized units that store factual and task-relevant information-to motivate our extension. Finally, feature attribution, to identify neurons that drive high-confidence yet domain specific mechanisms (DSM neurons), and we prune them to promote more robust reasoning.
leverage Integrated Gradients (IG), originally designed for input
3.1 MLPs in Transformer LLMs and Knowledge Neurons
Transformer models are composed of series of blocks, each typically featuring multi- head self-attention mechanism followed by a feed-forward network. The MLP sub-layer is responsible for non-linear transformations that significantly shape the model's internal representations. Formally, for the 1-th layer, the MLP can be expressed as:
MLP(x4) o (W]2) b/ by
(1)
where x/ E Rd the hidden state of the i-th token, W/ E R xd and W/ E Rddd are the
weight matrices, by and by are biases, and (.)) is a non-linear activation function (e.g., ReLU or GELU). Recent research in transformer-based language models has revealed that not all neurons contribute uniformly to model's performance. A subset of neurons, termed knowledge neurons, have been shown to encode and store specific factual or task-related information. These neurons often exhibit strong activations in response to inputs that
3

---

Published as a conference paper at COLM 2025
invoke particular pieces of knowledge, indicating their role in recalling and processing this information. The identification of knowledge neurons has not only deepened our understanding of the internal workings of LLMss but also paved the way for targeted interventions, such as controlled model editing and selective pruning, to enhance model robustness and generalization.
3.2 Integrated Gradients for Attribution
Integrated Gradients (IG) is principled attribution method developed to explain the predictions of deep neural networks. It addresses limitations of naive gradient-based techniques, such as gradient saturation, by averaging the gradients along a straight-line path from a baseline input to the actual input.
Formally, given model and an input x, the attribution for the i-th feature is defined
(x) (xi x1) fo of(x' + +a(x (a(x-x))) x')) du,
(2)
dx;
where x' a baseline input that represents the absence of features (e.g,, a zero vector) and scales the interpolation between the baseline and The method satisfies key axioms such as sensitivity sensifivity-eenuring ensuring that features affecting the output receive non-zero attributions and implementation intariance-which guarantees that functionally equivalent models yield identical attributions.
In practice, the integral is approximated using Riemann sum over discrete steps:
1 o
x'
& m (x-x))
(x) (x;
x/) m
dx;
Due to its solid theoretical foundations and empirical success, Integrated Gradients has become a widely adopted tool for model interpretability in areas such as computer vision and natural language processing. Although IG was formulated for input features, its conceptual framework can be extended to internal neurons. In our context, we focus on computing IG with respect to the token with the highest logit of the model's output distribution-which encapsulates the model's most confident prediction. This extension allows us to quantify the influence of individual neurons on the model's decision. In particular, the decision to calculate IG with respect to the maximum logit, ignores the actual annotated labels of given sample.
3.3
DSM Neuron Detection and Pruning
Building on the ideas of knowledge neurons and Integrated Gradients, we hypothesize that subset neurons the MLPs act as DSM neurons that disproportionately drive the maximum logit by exploiting spurious correlations rather than robust reasoning. Instead of interpolating over the input, we interpolate over the weight of the neuron under consider- ation. For given question o (e.g., from MMLU, SST, or BoolQ) and its associated output logits (4),, let
M(q) max f() yEV
where V is vocabulary space.
Let do denote the original weight of neuron n; in the MLP. We define the Integrated Gradients (IG) attribution for neuron n, by interpolating its weight from baseline value of to 0y. Formally, the attribution is given by
/''
aM(q:at0
IG(nj.9) = =0;
da,
4
dw;
where a(0;) represents the output of the original LLM (e.g., the logit corresponding to the model's prediction) when the weight neuron n, is scaled by a. The term
(M(gad)))
aw;
4

---

Published as a conference paper at COLM 2025
captures the sensitivity of the model's output to the modified neuron weight at interpolation step a. In practice, we approximate the integral via a Riemann sum over m steps.
We then aggregate the IG attributions over a dataset D {q: to obtain an average attribution score for neuron nz
score(ny) = N Σ (
(5)
Calculating IG with respect to the maximum logit, ignores the label of the given sample.
3.4 Mitigating DSM Neurons
After computing the IG attribution scores for all neurons and averaging over the validation set, we obtain a score matrix of shape L x d, where I. is the number of layers and d is the number of neurons per layer. These scores indicate the relative importance of each neuron in driving the model's predictions.
Since these attributions are derived from the model's output, pruning highly attributed neurons at random may degrade performance. However, in some layers, these neurons contribute to spurious correlations rather than meaningful reasoning. To identify such layers, we apply grid search over the layers and pruning percentages. Specifically, we iteratively prune the top-attributed neurons per layer in increments of 5%, ranging from 5% to 40%, and evaluate the model's performance
held-out validation set.
This systematic search allows us to determine which layers benefit from pruning, leading to improved generalization by reducing reliance spurious correlations, while also iden- tifying layers where pruning degrades performance, indicating that their high-attributed neurons are essential for robust reasoning. This targeted pruning strategy helps refine the model's decision-making without arbitrary neuron removal. The exact psuedo-code for our algorthim presented in Algorthim
Algorithm 1 DSM Neuron Pruning
Require: Task-specific dataset D = ( LLM M, validation set D L number of layers in M.
Ensure: Task-optimized pruned model M Mpruned
1: for each question 9 E D do
2: Compute M(q)= maXycV fq)y
Maximum logit
3: for each neuron ny in model M do
IG(H;, q) = =0; 10;Jo Jo (M(ca)))) 5: score(11)) (
4:
da
dwy
D Aggregate attribution
6: end for
7: end for
8: (lopt/ Popt) + argmax Eval(Prune(MM.l,)) p),
P =5% 50%
9: M Mpruned= Prune(M, Popt)
10: return M pruned
4
Experiments
In this section, we evaluate the effectiveness of our proposed method for detecting and pruning DSM neurons in ttansformer-based large language models (LLMs). Our primary goal is to assess whether selectively pruning neurons identified as DSM neurons can improve model generalization and performance across variety of natural language tasks. To do so, we conduct a series of experiments on benchmark datasets commonly used multiple- choice and reasoning tasks.

---

Published as a conference paper at COLM 2025
Tasks Our experiments focus multiple-choice tasks. We evaluate performance ing the following datasets: XNLI (Bowman et al., 2015), MMLU (Hendrycks et al., 2020), SST2 (Socher et al., 2013), SST5 (Socher et al., 2013), BoolQ (Clark et al., 2019), and Balanced COPA (Roemmele et al., 2011). These datasets include response formats ranging from two to five choices. A detailed description of each dataset is provided in Appendix A. Accuracy serves as the primary evaluation metric.
Models We evaluate our method on diverse set of instruction-tuned LLMs, including LLAMA2-7B-CHATT (Touvron et al., 2023), LLAMA3.1-SD-nnstreee. (Grattafiori et al., 2024), Mistrra--B-Innttruct---i
(Jiang et al., 2023a) and -wwen..-7B--nstrruct (Jiang et al., 2023a). We use these models due to their strong performance on various NLP benchmarks and their widespread adoption in research. By including models with different training methodologies and parameter sizes, we aim to assess the robustness and adaptability of our proposed method across multiple instruction-follovii
paradigms.
Baselines We compare our few-shot approach against several established adaptation methods. Supervised Fine-Tuning (SFT) serves as an upper bound for supervised adaptation by fine-tuning all model parameters using the complete training dataset. We also compare against the following adaptation methods: Inference- Time Intervention (TTT) (Li et al., 2023a) leverages contrastive pairs to identify key attention heads for intervention, with hyperparameter sweep over head selection and intervention strength. This method typically requires dozens of annotated examples. Contrastive Activation Addition (CAA) (Rimsky al., 2024) constructs a fixed steering vector by computing the mean activation difference at the answer position between positive and negative prompts, and generally uses substantial training data. SADI (Wang et al., 2025) is evaluated in three configurations. SADI-HIDDEN, which applies SADI key hidden states across the layers; SADI-HEAD, which modifies activations from all attention heads across layers; and SADI-NEURON, which adjusts outputs from non-linear activation functions in the FFN blocks. These methods require access to many annotated training samples. In contrast to these approaches, our DSM neuron pruning method operates in a genuine few-shot setting, employs only a handful of examples (ten in our experiments) to identify DSM neurons and determine optimal pruning configurations. This makes our method particularly valuable when labeled data is scarce or expensive to obtain.
Implementation Details In all our experiments, we apply the Integrated Gradients (IG) method to the gate_prat -proj layer within each MLP block of the transformer. The shape of this layer varies across architectures, as detailed in Appendix B, the exact promoting format is illustrated in Appendix C. The samples used within the Integrated Gradients framework are picked from the training split in a random way, while the reported accuracy is reported over the test sets.
Results
5.1 Quantitative Experiments
We evaluate our DSM neuron pruning method across multiple tasks and model architec- tures. Tables 1-3 summarize the performance of our method compared to baseline models and other adaptation approaches. Table presents results for LLAMA2-7B-CHLL five multiple-choice tasks. Our method achieves significant improvements over the base- line model, with absolute gains of 0.67%, 4.78%, 4.91%, 7.60%, and 3.6% on NLI, MMLU, SST2, BoolQ, and Balanced COPA respectively. Notably, our approach outperforms other parameter-efficien adaptation methods including ITT (Li et al., 2023b), CAA (Rimsky et al., 2024), and various SADI variants. On MMLU, our method achieves 49.68% accuracy com- pared to the baseline's s 44.90%, representing substantial improvement on this challenging
across
Ihttps://uuggiggte bbtar///eeebceollleeaaaaaaaaaaaa
Phttps://uuggingfacecco https://wgggnggge..coooo/ssrssrr -n-ttluttt

---

Published as a conference paper at COLM 2025
Method
Baccine
Besdiie
Nethod
+0.5%
+3.2%
+2.1%%
+1.5%%
+0.39%
10-
1L.22%
Atsunwyy
70-
+2.9%%
+6.75%
+1.7%%
Figure 1: Comparative analysis of performance improvements across three LLM archi- tectures using method. Each model family (LLAMA3.1-B)) Instruct, Mistral-7B-v0.3 Instruct, and Qwen2.5-7B Instruct) shows consistent accuracy gains across six standard NLP benchmarks.
benchmark. For sentiment analysis (SST2), our approach reaches 93.54% accuracy, approach- ing the performance of supervised fine-tuning (96.70%) without requiring labeled data. The random pruning baseline generally fails improve performance over the original model, confirming that our targeted identification of DSM neurons is essential rather than simply reducing model parameters. Table 2 shows the effectiveness our method on the
Task
NLI
MMLU
SST2
BoolQ
Balanced COPA
LLAMA2-7B-CHAT
63.11
44.90
88.63
70.52
70.80
ITI
63.97
46.07
91.38
74.10
CAA
64.13
46.17
91.16
74.98
SADI-HIDDEN
59.28
45.66*
92.15
76.25
SADI-NBURON
62.97
46.91*
88.69
70.40
SADI-HEAD
64.21
48.23*
92.20
74.35
Random Pruning
63.22
44.10
88.65
70.51
66.00
Ours Supervised Finetuning
63.78
49.68
93.54
78.12
74.40
90.07
96.70
88.75
Table 1: Overall performance of LLAMA2-7B-CHAN on the different tasks in zero-shot setting. *Uses transductive learning, i.e,, all training samples are available upfront, only over the MMLU dataset.
Mistral-7B-Instruct-ee and tent with our findings on LLAMA2, selective pruning of DSM neurons yields substantial the most notable improvements
(LAMA331-8B-Instiee
Instruct models. Consis-
gains across all benchmarks. For are observed on BoolQ (+7.16%) and SST2 (+3,27%), with COPA also showing significant further validate the effectivenes veness
improvement (+6.4%). Results on fiistrl--B--nnstruct-- of our method across different model families. Here, we observe improvements on all six tasks, with the most substantial gains on SST2 (+4.48%) and NLI (+2.96%). The Qwen2.5- 7B-Instruct model shows the largest improvements in some categories, particularly SST5 (+10.32%) and NLI (+4.79%). The consistently positive results across Mistral, LLAMA3.1, and Qwen2.5 architectures demonstrate that DSM neurons are common phenomenon across different transformer-basee LLMs, and that our pruning method provides a broadly applicable solution. Importantly, random pruning shows virtually improvement over the baseline models across all architectures, confirming that our targeted approach is identifying and removing specifically problematic neurons.
7

---

Published as a conference paper at COLM 2025
Task
NLI
MMLU
SST2
BoolQ
SST5
Balanced COPA
LLAMA3.1-88 Instruct
59.34
63.54
88.87
75.19
48.83
88.80
Random Pruning
59.30
63.54
88,90
75.19
48.83
88.80
Ours
61.03
66.78
92.14
82.35
50.70
95.20
66.68
60.66
87.95
86.14
45.74
93.20
Random Pruning
66.68
60.66
87.93
86.14
45.75
93.20
+ Ours
69.64
62.69
92.43
87.65
48.23
95.40
Bwven..-7B-Innsrrun
65.66
70.17
87.61
82.04
42.53
96.80
Random Pruning
65.66
70.20
87.61
82.04
42.53
96.81
Ours
70.45
74.59
90.36
84.43
52.85
97.40
Table 2: Performance comparison of LLAMA3.1-8B Instruct, Mistral-7B-Insttrut--ddd and Owen2.5-7BBInntite models across six NLP tasks (NLI, MMLU, SST2, BoolQ, SST5, and Balanced COPA), showing results for baseline models, random pruning, and our proposed method. Our method consistently outperforms both baseline and random pruning across all models and tasks.
While our primary evaluation focuses on English-language tasks, it is important assess the multi-lingual generalization capabilities of our DSM neuron pruning approach. To this end, we extend our evaluation multilingual scenarios, in which we apply DSM neuron pruning and evaluate on the same language without any change to the method or reliance on language-specifi features annotations. using the XCOPA dataset (Ponti et al., 2020), which covers eight diverse languages: Indonesian (id), Italian (it), Swahili (sw), Tamil (ta), Thai (th), Turkish (tr), Vietnamese (vi), and Chinese (zh). Table 3 presents the performance comparison across these languages for LLAMA2-7B-CHAT, various SADI configurations, and method. The results reveal interesting patterns about the effectiveness of differ- ent adaptation approaches in multilingual contexts. Our findings show that while SADI variants achieve the highest performance improvements in several languages, particularly Indonesian (id) and Italian (it), our method delivers consistent improvements across all languages without requiring transductive learning or labeled samples. For Swahili (sw) and Tamil (ta), our approach matches the performance of SADI variants, achieving the highest accuracy of 50.80% and 50.00% respectively. It is worth noting that SADI-Neuron and SADI-Head achieve stronger performance on some languages, particularly Indonesian (id) with gains of 12.20% and 11.20%, respectively. However, these methods require transductive learning with all annotated samples from the training set, whereas approach operates in a few shot manner with just 10 examples for the DSM neuron detection process. This dramatic reduction in data requirements (from hundreds of labeled samples to just 10) highlights the efficiency of our DSM neuron identification approach.
Task
id
it
ta
th
tr
vi
zh
LLAMA
51.40
61.20
50.20
49.40
50.80
49.40
51.80
62.80
+ SADI-Hidden"
51.40
62.40
50.00
50.00
51.20
48.80
52.20
64.80
SADI-Neuron*
63.60
68.80
50.20
48.80
53.80
50.60
60.40
70.40
SADI-Head*
62.60
70.60
50.80
49.60
51.40
51.60
60.20
70.10
+ Ours
55.40
67.40
50.80
50.00
50.80
51.40
54.80
68.60
Table 3: Evaluation results on multilingual task XCOPA with LLAMA227B-CHAT..
5.2
Ablation Studies
To assess the efficiency and robustness of our DSM neuron detection method, we conduct an ablation study on the number of samples required for Integrated Gradients (IG) computation. As computing IG attributions for all neurons model can be computationally expensive,
Shttps:/ hwtp://.w.....ctaaaaaeeeeeeeeeee wwt.taataets/cammrrieeeeee t1/xcopa
8

---

Published as a conference paper at COLM 2025
determining the minimum number of samples needed for effective DSM neuron detection is crucial for practical applications.
Figure 2 presents the performance BoolQ and SSI2 tasks using the LLAMA2-7B-CHAT
model with varying numbers of IG samples. The results demonstrate that our method achieves substantial improvements over the baseline even with very few samples. With just samples, we observe improvements of 6.20% on BoolQ and 4.13% on SST2, highlighting the efficiency of our approach in identifying meaningful DSM neurons with minimal data. Interestingly, we observe that optimal performance is achieved with just 10 samples for both tasks, with accuracy values of 78.12% for BoolQ and 93.54% for SST2. Increasing the number of samples beyond this point does not yield consistent improvements and sometimes results in slightly lower performance. For instance, with 20 samples, performance drops slightly to 76.23% on BoolQ and 92.88% on SST2. Additionaly we examine the impact of transferring the optimal pruning configuration, derived from the adaptation set, to unseen test data, ensuring that the DSM neuron pruning method remains robust. Specifically, we compare the accuracy of the pruned model on both the test set and the adaptation set to assess the generalizability and effectiveness of the pruning configuration across different data splits, results can be found in Appendix D. Additional studies examining the robustness of our method to adaptation set selection, the effectiveness of single-layer versus multi-layer pruning strategies, sensitivity to pruning threshold hyperparameters, and the choice of intervention layer are provided in Appendix E. These additional results demonstrate the stability of our approach across various experimental contigurations.
95
91.33
42-88
53.28
91-45
(+6.53)
[+8.00]
85-
Accuracy (%)
80
74-32
[+7.001
(+7.50)
(45.80)
145-730
(1577)
11378
70
20
52
Number Samples
LLAMAZ- CHAT Boold (70.52%) LLAMA2-7B- 5ST2 (88.07%)
Boold
Figure 2: Ablation studies over the number of samples used within the IG framework to detect DSM neurons
5.3 Computational Cost Analysis
A practical consideration for our method is the computational cost of computing IG for all neurons in the model. To quantify this overhead, we conducted timing experiments across our four model architectures, measuring the time required to compute IG attributions for all neurons in the gate_proj layer. These experiments used 100 samples to obtain stable timing measurements and report the average computation time per sample.
The results show that computing IG attributions requires between 3.74 and 4.68 seconds per sample across different models. Given that our method uses only 10 samples, the total computation time for DSM neuron detection ranges from approximately 37 to 47 seconds per task, negligible overhead compared to traditional fine-tuning approaches. The dimensions

---

Published as a conference paper at COLM 2025
Model
Avg. Time per Sample
Score Matrix Dimensions
LLAMA2-7B-CHAT
4.38 sec
32 11, 008
4.68 sec
32 14,336
Owen2.5-7B-nnstruee Mistral--B-innttruct--333
3.74 sec
28 18, 944
4.46 sec
32 14, 336
Table 4: Computational cost of Integrated Gradients attribution and resulting score matrix dimensions for different models. The score matrix dimensions represent [number of layers x neurons per layer].
of the score matrix reveal the scale of our attribution analysis. For instance, LLAMA3.1- 8B-Instruct produces score matrix of shape 32 x 13,336, representing attribution scores for 14,336 neurons across each of the 32 layers, totaling 458,752 attribution values. Despite this large number of parameters being analyzed, our method efficiently identifies the most influential DSM neurons through ranking and selective pruning.
6 Conclusions
The idea that pruning improves performance is not new. In the first two years of life, the brain prunes excess synapses to improve neural efficiency Huttenlocher & Dabholkar (1997). This process is especially active in the visual and prefrontal cortices during key developmental windows Petanjek et al. (2011). In machine learning, pruning removes redundant or less important weights nodes to reduce model complexity and overfitting LeCun et al. (1990). This can improve generalization and maintain or even enhance accuracy with fewer parameters Han et al. (2015).
What is unique about our work is that, unlike traditional pruning approaches which target redundant -ow-importance parameters Sun al., our method deliberately identifies and removes the most influential neurons-those driving high-confidence but non-transferable predictions. This inversion of the conventional pruning logic detects and addresses the 'mmpy vessels' (DSM) that 'make the most noise' (obtain the highest explainability scores), thereby suppressing mechanisms that impede generalization.
Moreover, our pruning is performed locally, within single MLP layer, avoiding broad struc- tural changes while still yielding substantial performance gains. We situate our approach within the broader context of transfer learning, specifically addressing the underexplored challenge of few-shot adaptation. Our results demonstrate that even with minimal data, selectively pruning influential neurons can significantly enhance cross-domain performance, offering a lightweight and interpretable alternative to conventional fine-tuning.
Acknowledgements
Ivan Titov supported by the Dutch National Science Foundation (NWO Vici VI.C.212.053). This work was supported by grant from the Tel Aviy University Center for Al and Data Science (TAD). This research was also supported by the Ministry Innovation, Science & Technology .Israll (1001576154) and the Michael J. Fox Foundation (MJFF-022407). The authors would like to thank Rochelle Choenni for the valuable insights. The contribution of Ameen Ali is part of PhD thesis research conducted at Tel Aviv University.
10

---

Published as a conference paper at COLM 2025
References
Ameen Ali, Lior Wolf, and Ivan Titov. Mitigating copy bias in in-context learning through neuron pruning. arXiv preprint arXiv:2410.01288, 2024.
Enes Altinisik, Hassan Sajjad, Husrev Taha Sencar, Safa Messaoud, and Sanjay Chawla. Impact of adversarial training on robustness and generalizability of language models. arxiv preprint ar Xtt:22211.0553 2022.
Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Fil- ippova. "will you find these shortcuts?" protocol for evaluating the faithfulness of input salience methods for text classification. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 976-991, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: https://aclanttoology. org/ wwtp:///./......pmm..aaa...... hal.pppmmi..4444
wwtp//22.eenlpmmai..... URL
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arx::: 003366 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint wxXi::1905.1004. 2019.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, May 2022.
Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653 (0.18633vv1/NTC 11t0://1//111111 1423. URL https: org/N/
Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxi- ang Gu, Tong Sun, and Xia Hu. Towards interpreting and mitigating shortcut learning behavior of nlu models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 915-929, 2021.
Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1): 110-120, 2023.
Amir Feder, Yoav Wald, Claudia Shi, Suchi Saria, and David Blei. Data augmentations for improved (large) language model generalization. Advances in Neural Information Processing Systems, (6:70638-70653 2023.
Dan Friedman, Alexander Wettig, and Danqi Chen. Finding dataset shortcuts with grammar induction. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4345-48550 4363,, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Lin- guistics. doi: 10.18653/ /v1/ 222.2.2eemlpmmaaai emnlp-main, 293/.
URL https:/ hatsss///laattoogy/... org/ or2/2022.
11

---

Published as a conference paper at COLM 2025
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. ar Xrviv preprint ar arXio:2407.21783, 2024.
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hath- away, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. arXiv preprint ar Xit::2401 12181, 2024.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connec- tions for efficient neural networks. In Advances in neural information processing systems, volume 28, 2015.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arviv preprint (.Xit:2009.033000 2020.
Xanh Ho, Johannes Mario Meissner, Saku Sugawara, and Akiko Aizawa. A survey on measuring and mitigating reasoning shortcuts in machine reading comprehension. arviv preprint ar Xix::220900011 002844449 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parfieter-effiitent transfer learning for nip. In International conference on machine learning, pp. 2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Peter R Huttenlocher and Anand S Dabholkar. Regional differences in synaptogenesis in human cerebral cortex. Journal of Comparative Neurology, 387(2):167-178, 1997.
Albert Q- Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023a. URL https://arxiv.. www.ii.o.or//booee orggabss/311 25650.25...
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023b.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Mosh Levy, Shauli Ravfogel, and Yoav Goldberg. Guiding LLM to fool itself: Automatically manipulating machine reading comprehension shortcut triggers. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems,
2023a.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https:
12

---

Published as a conference paper at COLM 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, C Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277444 2022.
Chong
Zdravko Petanjek, Miloš Judaš, Goran Šimić, et al. Extraordinary neoteny of synaptic spines in the human prefrontal cortex. Proceedings of the National Academy of Sciences, 108(32): 13281-13286, 2011.
Edoardo M. Ponti, Goran Glava s, Olga Majewska, Qianchu Liu, Ivan Vuli'c, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. arXiv preprint, 2020. URL https: / /ducdauge thub.u..co wwt.ii/////oooo///eeeeeeoeeeoeee pdf.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering ilama via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, August 2024. Association for Computational Linguistics.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew Gordon. Choice of plausible alter- natives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011. URL https: / /pwople. ict.usc. wct.usc.edu/ wttp:///orrooooppppaaatttooooo// SPRING1 1A. PDF.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models. Advances in Neural Information Processing Systems, 37:125019-125044 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Ruixiang Tang, Dehan Kong, Longtao Huang, and Hui Xue. Large language models can be lazy learners:
Analyze shortcuts in in-context learning, In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Compudational Linguisfics: ACL 2023, pp. 4645-4657, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v /ttp///l222222//// fi123.finding--c.222... URL https::/ http:///laaaattoy....oooo org/ 2023. findings- 284/
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ri-n Wen. Lancuage-specific neurons: The key multilingual capabili- ties in large language models. are preprint ar Xio::2402.16433. 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. arv preprint 7rXi::2307.09288, 2023.
Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. arXiv preprint arXiv:2309.04827 2023.
Weixuan Wang, JINGYUAN YANG, and Wei Peng. Semantics-adaptive activation interven- tion for LLMs via dynamic steering vectors. In The Thirteenth International Conference on Learning Representations, 2025. URL http:///wwpeeeeeeeeeeee///////aa
13

---

Published as a conference paper at COLM 2025
Yu Yuan, Lili Zhao, Kai Zhang, Guangting Zheng, and Qi Liu. Do LLMs overcome shortcut learning? an evaluation of shortcut challenges in large language models. In Yaser Al- Onaizan and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12200,, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
Lili Zhao, Qi Liu, Linan Yue, Wei Chen, Liyi Chen, Ruijun Sun, and Chao Song. Comi: Correct and mitigate shortcut learning behavior in deep neural networks. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 218-218222 228, 2024.
14

---

Published as a conference paper at COLM 2025
A
Tasks
We evaluate our approach on a diverse set of benchmarks spanning commonsense reason- ing, natural language inference, knowledge assessment, sentiment analysis, and question answering. In the following, we summarize the details and dataset sizes for each benchmark.
1. Balanced COPA5: A dataset where each question presents a premise and two alternatives, with the task of selecting the alternative that most plausibly has causal relationship with the premise.
2. XNLI': A multilingual extension of NLI, consisting of sentence pairs labeled as entailment, contradiction, or neutral, designed to evaluate cross-lingual natural language inference.
3. M A benchmark assessing knowledge acquired during pretraining, covering 57 subjects across STEM, humanities, social sciences, and more.
4. SST-29 and SST-5'0;
Sentiment analysis datasets, with SST-2 providing binary labels (negative, positive) and SST-5 offering five labels (negative, somewhat negative, neutral, somewhat positive, positive).
5. BoolQu!: A question-answering dataset featuring naturally occurring yes/no no ques-
tions.
Task # train
Balanced COPA
XNLI (EN)
MMLU
SST2
SST5
Boolq
1K
393K
99.8K
67.3K
8.54k
9.43k
# test
500
5.01K
14K
872
2.21k
3.27k
Table 5: The number of data used for identifying key elements and testing for 6 tasks.
B Intervention Layer
Our method is applied to the gate proj layer across LLaMA, Mistral, and Qwen models. Specifically, we intervene in the transformation:
MLP = Glown.proj(atl www.nproo(aatfiigttepppoog
x op-proj(x)
(6)
We choose the gate proj layer as our operating layer for two key reasons. First, it plays crucial role in the feedforward network (FFN) by modulating the interaction between the up-proj and down_proj transformations, making a critical point of intervention. Second, this layer has significantly larger parameter footprint than other FFN components, as detailed in Table 6, making an effective target for structured modifications.
Model
down_proj
up-proj
gate_proj
ILAMA2-7B-CHA.
11008 x 4096
4096 x 11008
4096 11008
LLAMA3.1-8B Instruct
14336 x 4096
4096 x 14336
4096
14336
Miistral-B-Innsruct-vy
14336 x 4096
4096 x 14336
4096 x 14336
Qwen22.5-7-IInstruct
152064 3584
3584 x 18944
3584 18944
Table 6: The number of data used for identifying key elements and testing for 6 tasks.
/ttpt//////////goooooooooooooarr copa
7
co//ataaeets/accboo//miil
#nttps://hugginggaccee co//dttasetss/cii//mmuu
15

---

Published as a conference paper at COLM 2025
C Tasks Prompting
SST2 Zero-Shot Prompt
Consider the sentiment expression in this sentence and respond briefly with 'positive' or 'negative'.
(TEXT)
Answer:
SST5 Zero-Shot Prompt
Consider the sentiment expression in this sentence and respond briefly with 'very positive', 'positive', 'neutral', 'negative', and 'very negative'.
TEXT
Answer:
MMLU Zero-Shot Prompt
Question: QUESTION
Which the following answers is correct?
A. CHOICES[O]
B. CHOICES
C. CHOICESS|1
D. CHOICES[3]
State the letter corresponding to the correct answer.
Answer:
XNLI Sero-Shot Prompt
Answer whether the hypothesis is more likely to be true, false, or unclusive based on the given premise. Premise: PREMISE Hypothesis: HYPOTHESIS Answer:
Balanced COPA Zero-Shot Prompt
Question: PREMISE Based the previous passage, choose the most reason- able question.
A: CHOICES[O]
B: CHOICES[1]
Answer:
BoolQ Zero-Shot Prompt
Is the answer to the question encapsulated in the passage? Please confirm
16

---

Published as a conference paper at COLM 2025
with 'yes' or 'no'.
Passage: PASSAGE
Question: QUESTION
Answer:
17

---

Published as a conference paper at COLM 2025
D Transferability from Adaptation to Test
To validate the robustness of our DSM neuron pruning method, we investigate whether the optimal pruning configuration identified on small adaptation set transfers effectively to unseen test data. In our approach, we use the adaptation set not only to identify DSM neurons through Integrated Gradients, but also to determine the optimal layer and pruning percentage through grid search. This dual-purpose use of the adaptation set raises an important question: does the configuration that performs best on the adaptation samples maintain its effectiveness when applied to entirely new examples?.
Figure 3 presents the results of our grid-search across the SST2 dataset using the LLAMA2- 7B-CHAT model. We define two evaluation settings: "Test Pruned" represents the accuracy of the pruned model (at different layers) on the test set, where DSM neurons were detected using the adaptation set samples. "Adaptation Pruned" shows the accuracy of the pruned model on the same adaptation set samples that were used to detect the DSM neurons.
Layer Error Comparison
Test Pruned
Adaptation Pruned
Error (96
Base Adaptation Error: 20.00%
20
aa
st Res
t:
%
10
14
16
18
20
22
24
26
28
30
Figure 3: Comparison of error rates between pruned and non-pruned models across different layers on both adaptation and test sets for SST2
Figure 3 demonstrates that the optimal pruning configuration identified on the adaptation effectively transfers to the test set. For the SST2 dataset, the non-pruned model achieves a baseline error rate of 20.0% on the adaptation set. By pruning layer 10, we achieve perfect error rate of 0.0% on this same adaptation set. Importantly, this improvement transfers to the test set, where the pruned model (at layer 10) achieves 6.46% error rate (the lowest across the layers) compared to the non-pruned baseline of 11.13%.
E Additional Ablation and Robustness Studies
E.1 Robustness to Adaptation Set Selection
critical consideration for any few-shot adaptation method its sensitivity to the specific samples chosen for the adaptation set. To evaluate the robustness of our DSM neuron pruning approach to this selection, we conducted comprehensive experiments examining both the variance across different random selections and the impact of sample quality.
18

---

Published as a conference paper at COLM 2025
Variance Analysis Across Random Seeds We performed five independent runs with LLAMA3.1-B-Instrud on the BoolQ dataset, each using a different random seed to select the adaptation samples. Additionally, compared the variance when using different adaptation set sizes (10, 50, and 100 samples) to understand how sample size affects stability.
Adaptation Set Size
10 Samples
50 Samples
100 Samples
BoolQ Accuracy
82.93 1 0.65% 82.57 1 0.05%
82.43 1: 0.02%
Table 7: Mean accuracy and standard deviation across five random seeds for different adaptation set sizes on BoolQ using LLAMA3.1-8B-Instru.
The results demonstrate a high degree of stability in our method's performance. With just 10 samples, we achieve mean accuracy of 82.93% with standard deviation of only 0.65%, indicating that our DSM neuron identification is robust to the specific random selection. Interestingly, while larger adaptation sets (50 and 100 samples) show even lower variance, they do not improve mean performance, reinforcing our finding that 10 samples are sufficient for effective DSM neuron detection.
Impact of Sample Correctness To further investigate robustness, we examined whether the quality of adaptation samples, specifically whether the model correctly predicts them, affects the pruning effectiveness. We compared three scenarios: (1) randomly selected samples, (2) samples that the model correctly predicts, and (3) samples that the model incorrectly predicts.
Dataset Random Correctly Predicted Incorrectly Predicted
BoolQ 82.35% 92.14%
85.25%
79.81%
SST2
92.14%
87.50%
Table 8: Performance comparison using different adaptation set selection strategies with LLAAMA3.1-9B-Instruuct.
These results reveal several important insights:
1. Random selection performs well: Our default strategy of random selection achieves strong performance on both tasks, validating the practical applicability of our method without requiring careful sample curation.
2. Correctly predicted samples can enhance performance: On BoolQ, using samples that the model already predicts correctly leads to 2.9% improvement over random selection, suggesting that DSM neurons identified from high-confidence correct predictions effectively capture spurious patterns.
3. Incorrectly predicted samples underperform: Using only misclassified samples results in lower performance (79.81% on BoolQ, 87 87.5% on SST2), indicating that errors may stem from various sources beyond DSM neurons, making them less effective for identifying systematic shortcuts.
4. Task-dependent effects: The impact of sample selection varies by task. SST2 shows identical performance for random and correct samples, while BoolQ shows more sensitivity, suggesting that the optimal selection strategy may be task-specific.
These findings demonstrate that our DSM neuron pruning method exhibits low sensitivity to the random selection of adaptation samples, making it practical for real-world applications where careful sample curation may not be feasible. The low variance (+0665%) across different random seeds and the strong performance with random selection validate the robustness of our approach.
19

---

Published as a conference paper at COLM 2025
E.2 Justification for Single-Layer Pruning Strategy
Our approach of pruning within single layer (specifically the pate_proj layer) with grid search over pruning percentages is grounded in both empirical evidence and practical considerations. This section provides detailed justification for these design choices.
Empirical Validation of Layer Selection To validate our focus on the gate_proj layer, we conducted comparative experiments across all MLP components in 1.AMAM..--BB-Istru Our DSM neuron detection and pruning method was applied independently to each MLP layer type (gate proj, up proj, and down proj) to evaluate the resulting performance.
Dataset
Baseline
gate
up-proj
down.proj
BoolQ
75.19%
82.35%
81 .84%
79.53% 90.36%
SST2 88.87%
92.14%
91.02%
Table 9: Performance comparison of pruning different MLP layers in LLAMAA11-0D-insrruckk Bold indicates best performance.
The results demonstrate that pruning the gate.proj layer consistently yields the highest performance gains. On BoolQ, gate_proj achieves +7.16% improvement, compared to +6.65% for up-proj and +4.34% for down Similarly, on SST2, gate_proj achieves +3.27% improvement, compared to +2.15% for up-proj and +1.49% for down.proj.
This superior performance of gate.proj pruning aligns with recent findings in the in- terpretability literature. The gating mechanism plays a crucial role in selecting which knowledge-relate behaviors to activate (Ali et al., 2024), making it natural location where domain-specifts mechanisms might be encoded. DSM neurons that rely on spurious correla- tions may preferentially manifest in this gating layer, as it controls the flow of information through the MLP block.
E.3 Sensitivity to Pruning Threshold
To assess the robustness of our method to the pruning threshold selection, we conducted detailed analysis of performance across different pruning percentages. Using LLAMA3.1- 8B-Instruct, we evaluated the impact of varying the pruning percentage from 0% (baseline) to 40% both BoolQ and SST2 datasets.
Dataset 0%
10%
15%
20%
25%
30%
35%
40%
BoolQ
75.19
80.80
81.46
81.28
80.90
81.00
82.32
79.12
SST2
88.87
89.92
90.12
91.45
92.14
92.05
92.10
90.12
Table 10: Performance of LLAMAS.1--B-Lnsttteeeee across different pruning percentages on BoolQ and SST2. Bold indicates peak performance for each dataset.
The results demonstrate that our method exhibits robust performance across wide range of pruning thresholds. For BoolQ, all pruning percentages from 10% to 35% yield substantial improvements over the baseline (75.19%), with performance ranging from 80.80% to 82.32%. The peak performance of 82.32% is achieved at 35% pruning, representing a 7.13% absolute improvement. Notably, the performance remains consistently high (above 80%) across most pruning percentages, only dropping below this threshold at the extreme of 40% pruning.
Similarly, SST2 shows consistent improvements across all tested pruning percentages. Start- ing from a baseline of 88.87% performance improves to between 89.92% and 92. 12.14, with the peak 25% pruning. The performance remains above 90% for pruning percentages between 15% and 35,, demonstrating a stable improvement window.
This analysis confirms that our pruning approach is robust to threshold selection, making practical for deployment without extensive hyperparameter tuning.
20


Reviews:
[{"review_id": "GVCimEGhnV", "reviewer": "Reviewer_XM36", "summary": "This paper proposes a method to enhance generalization in large language models (LLMs) by detecting and pruning neurons associated with dataset-specific mechanisms (DSMs). The authors argue that DSMs lead to shortcut learning, where models exploit spurious correlations in training data. Their approach uses Integrated Gradients (IG) to identify neurons disproportionately influencing high-confidence predictions and prunes them in a single layer. Evaluations across multiple-choice benchmarks show improved performance over baselines like ITI and SADI, with minimal labeled data requirements.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Conceptual Innovation: The idea of pruning influential neurons driving non-transferable mechanisms (instead of low-importance parameters) is novel and addresses a critical challenge in LLM robustness.\n\nMethodological Clarity: The integration of IG for neuron attribution and the grid-search strategy for layer/pruning optimization are well-explained and systematic.\n\nPractical Impact: Demonstrates effectiveness in few-shot, label-free settings, making it applicable to low-resource scenarios.", "reasons_to_reject": "Incomplete Mechanistic Justification: The focus on the gate_proj layer lacks empirical or theoretical evidence comparing it to other layers, for instance, up_proj,  weakening the rationale for layer selection.\n\nSample Sensitivity: Reliance on 10 unlabeled samples for DSM detection risks bias if samples are unrepresentative. No analysis of variability across different sample sets is provided.\n\nScalability Concerns: The grid-search approach for layer/pruning percentage may not scale efficiently to larger models (e.g., 70B+ parameters).\n\nNarrow Task Scope: Evaluations are limited to multiple-choice tasks; generalization to generation tasks (e.g., summarization) or diverse domains (e.g., biomedical text) is untested.", "questions_to_authors": "How does the choice of the gate_proj layer compare to pruning other layers in terms of performance and computational cost?\n\nCould the method’s reliance on 10 samples lead to instability if those samples are noisy or biased? Please provide sensitivity analysis across varying sample sets.\n\nWhat modifications would enable scaling the grid-search strategy to models with 100B+ parameters?\n\nWhy restrict evaluations to multiple-choice tasks? Would the approach generalize to open-ended generation or domain-specific applications?", "extra_scores": {"ethics_concerns_details": "n/a"}}, {"review_id": "SXmTl7LW6p", "reviewer": "Reviewer_PXUN", "summary": "This paper proposes to improve the generalization of LLMs to new tasks by pruning neurons in pathways that are highly domain specific due to spurious correlations. The main advantages of the approach are that they require only a small set of adaptation samples (as low as 10 samples) and the pruning cost is significantly lower compared to full model fine-tuning or re-training.\nThe paper is well written and clearly describes the motivation and different steps in the approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The usage of pruning as a fine tuning mechanism for LLMs is very interesting and novel.\n\n2) The paper contains an extensive experimental section including a good comparison with previous fine-tuning methods on a variety of tasks and models.", "reasons_to_reject": "1) The paper would benefit from a discussion regarding how the choice of the adaptation set affects the accuracy and how robust the method is to this random choice. Specifically it would be interesting to see what is the variance of the accuracies given different random samples of the adaptation set.\n\n2) Additional details and clarifications on the experiments and adaptation set would also be very helpful, specifically regarding the following questions:\n* Are all the adaptation sets chosen at random? \n* Is the adaptation set specific for each task or is it taken from multiple tasks and the pruning is done once simultaneously for all the tasks in the figure/table? e.g. in Figure 1, Table 1 & 2.\n* Appendix D states that it explores the transferability from the adaptation set to the test set. Aren't the results throughout the paper presented for a held out test set? Please clarify the difference between the experiments in the main paper and the results in Appendix D.\n* What is the computational cost of computing IG for all the neurons considered and what is a typical size for the score matrix?", "questions_to_authors": "Minor comment: \nIn Table 1, I think that the asterisk symbol is misplaced - it appears next to the MMLU numbers for the SADI methods and not in the left-most column.", "extra_scores": {}}, {"review_id": "sBjhyIr6X0", "reviewer": "Reviewer_u6ZY", "summary": "The paper proposes a neuron pruning method to remove a model's domain specific spurious correlations. The method selects neurons with high influence on the output (determined by a score based on integrated gradient). Main experiments are done with five multiple-choice/classification datasets (XNLI, MMLU, SST2, BoolQ, Balanced COPA). The proposed method outperform other model steering baselines in most cases.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed method employs an interesting way to use integrated gradient methods to attribute neuron contributions.\n* The proposed method can improve the pre-trained model performance on multiple-choice QA tasks with only a few (10) examples.", "reasons_to_reject": "* Discrepancy between the paper's goal and its evaluation. The goal of the paper is to remove domain-specific spurious correlation of the model. However, the evaluation is based on test accuracy in _one_ domain. Ideally experiments should be conducted with a source and a target domain in mind, and use accuracies in both domains to demonstrate the effectiveness of removing domain-specific spurious correlations.\n* The comparison with baselines is confusing. \n  * The compared methods are using more datapoints than the proposed method (Line 236). The number of datapoints are not discussed in experiment section but instead in the related work section. Ideally all methods should be compared with the same number of datapoints.\n  * One of the advantages of the paper is the data efficiency of the method. The proposed method uses at most 10 datapoints. A straightforward baseline is to fine-tune the model on these 10 (or less) datapoints. \n* Unsupported claim on computation efficiency. The paper claims that the proposed method has low computational cost (Line 59), but from my understanding enumerating neurons and computing integrated gradients can be computationally expensive. This claim should be supported with runtime comparison.", "questions_to_authors": "* Regarding MMLU results. (1) Is there any reason the supervised fine-tuning results is not available on MMLU? (2) Which version of MMLU did you use? According to Table 4, the version you use have 116K training examples and 1.21K test examples. The dataset link provided does not include a version that fits this description. Could you please clarify?\n* Line 232. What is the methodology of SADI, a compared baseline?\n* Can you provide more details on the random pruning baseline you implement? Do you determine $l$ and $p$ in a way similar to Line 8 of Algorithm 1?\n* Line 278. What does \"cross-lingual generalization\" mean here? Do you apply pruning in one language but evaluate the model on other languages?", "extra_scores": {}}, {"review_id": "oqxjyhKsv5", "reviewer": "Reviewer_XAzb", "summary": "The paper addresses the issue of \"shortcut learning\" in LLMs, where models rely on dataset-specific neurons that reduce generalization. Its main contribution is a neuron-level pruning method leveraging Integrated Gradients to identify and remove neurons responsible for detrimental, domain-specific mechanisms (DSMs), enhancing cross-task generalization in a computationally efficient, few-shot fine-tuning setup. LLM baselines and other adaptation methods are comprehensively compared.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Proposes a neuron-level pruning strategy explicitly targeting DSM neurons, which is insightful and addresses a relevant issue\n2. Achieves substantial improvements across diverse LLMs (LLaMA, Mistral, Qwen) and multiple-choice benchmarks, showing broad applicability.\n3. Demonstrates computational efficiency and performance, reducing reliance on annotated data. Extensively evaluates the method, including multilingual generalization, and systematically compares with adaptation baselines.", "reasons_to_reject": "1. Limited theoretical justification provided regarding the specific selection of Integrated Gradients versus other neuron attribution methods.\n2. The paper lacks detailed analysis or visualization of exactly how the pruned neurons were detrimental, limiting interpretability claims.\n3. Experiments are predominantly on multiple-choice tasks; it remains unclear whether the approach generalizes effectively to more complex language tasks like generation or open-ended reasoning.\n4. The effectiveness of the pruning percentage and chosen single-layer pruning strategy appears heuristic and somewhat arbitrary, without extensive theoretical or empirical justification.", "questions_to_authors": "1. Why did you choose Integrated Gradients specifically over other attribution methods for neuron attribution?\n2. Can you provide specific examples or qualitative analysis showing exactly how pruned DSM neurons were contributing detrimentally to shortcut learning?\n3. How sensitive are your improvements to the chosen pruning threshold (e.g., is the performance robust across different pruning percentages)?\n4. Have you tested or do you have results indicating if this neuron-level pruning approach generalizes to open-ended generation or dialogue tasks beyond multiple-choice questions?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach to adapt language models to specific downstream tasks. The core idea of the approach is to use integrated gradients (IG) to locate individual neurons that disproportionately influence high-confidence predictions of an (instruction fine-tuned) model, and subsequently prune these neurons to improve the generalization of the model to this particular task. The authors show promising results using as little as 10 samples to identify and prune neurons and find performance improvements across several multiple choice classification tasks.    

Reviewers highlighted the conceptual novelty of the proposed approach, the practical importance for few-shot adaptation, as well as the extensive experiments. 

The main weaknesses identified by the reviewers were a lack of a mechanistic justification for why the authors apply their method to a particular layer only, the somewhat narrow scope focusing only on classification tasks rather than generative ones, and a lack of experiments studying the variance of the proposed method.    

Based on the author's responses, almost all of these concerns seem to have been addressed and the authors provided extensive explanations, elaborations, and new experimental results during the rebuttal. 

A downside of the approach I'm seeing is that it is mostly applicable for tasks that have only a single output token such as classification. This is due to the reliance on IG which the authors also discussed during the rebuttal. 

Overall, I agree with the reviewers that the proposed approach is interesting and provides an interesting perspective on how to perform task-adaptation for LLMs.
Decision: Accept