The Achilles’ Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities
Authors: Zixuan Qin
Qingchen Yu
Kunlin Lyu
Zhaoxin Fan
Yifan Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Ultra-sparse neuron sets
Perturbation-based identification
Catastrophic failure
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We discover that LLMs contain ultra-sparse critical neuron sets and propose a Perturbation-based Causal Identification method to locate them, showing that masking these neurons causes catastrophic model collapse.

Abstract: Large Language Models (LLMs) have become foundational tools in natural language processing, powering a wide range of applications and research. Many studies have shown that LLMs share significant similarities with the human brain. Neuroscience research has found that a small subset of biological neurons in the human brain are crucial for core cognitive functions, which raises a fundamental question: do LLMs also contain a small subset of critical neurons? In this paper, we investigate this question by proposing a Perturbation-based Causal Identification of Critical Neurons method to systematically locate such critical neurons in LLMs. Our findings reveal three key insights:
(1) LLMs contain ultra-sparse critical neuron sets. Disrupting these critical neurons can cause a 72B-parameter model with over 1.1 billion neurons to completely collapse, with perplexity increasing by up to 20 orders of magnitude;
(2) These critical neurons are not uniformly distributed, but tend to concentrate in the outer layers, particularly within the MLP down\_proj components;
(3) Performance degradation exhibits sharp phase transitions, rather than a gradual decline, when these critical neurons are disrupted.
Through comprehensive experiments across diverse model architectures and scales, we provide deeper analysis of these phenomena and their implications for LLM robustness and interpretability. These findings can offer guidance for developing more robust model architectures and improving deployment security in safety-critical applications. Our code is available at https://github.com/qqqqqqqzx/The-Achilles-Heel-of-LLMs.

Published as a conference paper at ICLR 2026
THE ACHILLES' HEEL OF LLMs: HOW ALTERING A HANDFUL OF NEURONS CAN CRIPPLE LANGUAGE
A ABILITIES
Zixuan Qin', Qingchen Yuz., Kunlin Lyu', Zhaoxin Fan Yifan Sun
I Center for Applied Statistics, School of Statistics, Renmin University of China
BBejjing Advanced Innovation Center for Future Blockchain and Privacy Computing "School of Artificial Intelligence, Beihang University
ABSTRACT
Large Language Models (LLMs) have become foundational tools in natural lan- guage processing, powering a wide range of applications and research. Many studies have shown that LLMs share significant similarities with the human brain. Neuroscience research has found that small subset of biological neurons in the human brain are crucial for core cognitive functions, which raises a fundamental question: do LLMs also contain small subset of critical neurons? In this paper, we investigate this question by proposing a Perturbation based Causal Identifi- cation of Critical Neurons method to systematically locate such critical neurons in LLMs. Our findings reveal three key insights: (1) LLMs contain ultra-sparse critical neuron sets. Disrupting these critical neurons can cause a 72B- -araarrmet model with over 1.1 billion neurons to completely collapse, with perplexity in- creasing by up 20 orders of magnitude; (2) These critical neurons are not uniformly distributed, but tend to concentrate in the outer layers, particularly within the MI.P down_proj components; (3) Performance degradation exhibits sharp phase transitions, rather than gradual decline, when these critical neu- rons are disrupted. Through comprehensive experiments across diverse model architectures and scales, provide deeper analysis of these phenomena and their implications for LLM robustness and interpretability. These findings can offer guidance for developing more robust model architectures and improving deploy- ment security in sriety-critical applications. Our code is available at https: htw.tthu..ommmqqggqgcceeeee The- Achi les Heel -of-LLMs.
INTRODUCTION
Understanding the fundamental principles underlying intelligent computation has long attracted re- searchers across neuroscience and artificial intelligence. A growing body of evidence suggests that the brain's 's computational power may critically depend on a small fraction of highly influential neu- rons, rather than being uniformly distributed across all neural elements. This principle, stated by Arshavsky (200T), suggests that sparsely distributed, genetically distinct neurons function as com- putational bottlenecks, each having major influence over specific brain functions.
In parallel, Large Language Models (LLMs)-Wil) with their billions of parameters havee demonstrated capabilities mirroring human cognition, from language comprehension problem-solving (Riztha et al. 2024 Niu et al. (004).. Recent research shows clear parallels: the hierarchical processing and predictive coding mechanisms in L.LMs align closely with the brain's layered neural architectures (Schrimpf et al. 2021 Caucheteux & King 2022 Mischler et al. (2224), hinting that these artificial systems may emulate biological computation. This observation leads a key question:
Q: Do LLMS likewise possess small set of critical artificial neurons that are indispensable for their capabilities?
*Proeect leaders and Corresponding authors. Emails: zhaoxinf@ haaaxinn@@haa...o hww.ff/@waaaaauu.......mmm wtw.aaccccccccacaneeeeee.e sunyifan@ruc.cdu.co

---

Published as a conference paper at ICLR 2026
In the field of artificial intelligence. previous work has used pertturbation-basee methods to identify important model components (Hanna et al.] 2023 Zhang & Nandal [022)). To explore this key question, apply perturbation-basee analysis identify critical neurons. Our goal is to find small subset neurons whose collective disruption causes catastrophic system-wide collapse.
To put this investigation into practice, introduce the Perturbation-base Causal Identification of Critical Neurons method, as illustrated in Figure Given any text as input, the proposed method consists of stage optimization process locate the so called critical neurons. In the first stage, the method injects controlled noise into the model's input and measures the resulting activation differences across neurons, thereby generating a ranked list of candidates most likely to influence model behavior. In the second stage, we sequentially mask these top-ranked neurons in a greedy manner, closely monitoring changes model perplexity. This allows us to causally determine which neurons are truly essential for the model's function.
Using this method, we analyze a di- verse set of innluuins Neu- rons, WikiText- 103. C4, MMLU-
Calculate difference activation values
X
Select the 3 neurons with the greatest differences
Pro (Wang et al. 2025). IFE-
val
(Zoou et al.; 2023), GPQA-
LLM
Diamond (Rein al. (024), Hu-
manEval (Peng et al. 2024). MATH
(Hendrycks et (2021), MGSM (Shi
X+6
MLP down_pro
et al.
2023). and SimpleQA (Wei 2024) well multiple
al.
LLMs
such as Llama-3 (Grattafiori
LLM
Mask these 3 neurons
al.
2024). Gemma (Gemma Team
2024
DeepSeek-RI (DeepSeek-A) Al
2025
Phi-3, Phi-3.5 (Abdin et al.
2024
and Qwen2.5 [Qwen] 2025
Output
According to our experiment, we ob- serve consistent and interesting pat- terns across all settings. In par- ticular, findings reveal several key phenomena. First, we discover that LLMs governed by an ultra- sparse set of critical neurons: aston- ishingly, disabling as few as three neurons can catastrophically impair 72B 22aaarrmette model comprising over 1.1 billion neurons, driving its perplexity up by as much as 20 or- ders of magnitude. Second, we find that these critical neurons are not ran- domly distributed throughout the net- work, but are instcad highly concen- trated in the outer layers, particularly within the MLP down_proj compo- nents, which is consistent with the observations reported in Yu et al. (202)). Third, we observe that per- formance degradation does not occur gradually, but rather through sharp phase transitions. In section 4 we provide a detailed discussion these findings and hope these discoveries can help the community better opti- mize LLM architecture, training, and inference processes. Our contributions are summarized as follows:
What capital of France?
LLM
Mask 1 neuron
Mask
Park Other toben
Parl
Peri
tokum
Ohar
Parts Toklot Parts
Parts Tokare Parts
Paris Toklnt Paris
Paris Probabiity: Prebabbiity.0737773 Paris Rank:
Paris Probebilly: 9.457275
Paris Probebilly: 0,000005
Parts Ramk:
Parts Rank #30638
Figure Illustration of critical neuron identification and progressive masking effects, using DeepSeck-RR PiroSeck-R1-iisee R--istill-- Llama-70B as an example. The top panel shows that method identifies critical neurons located in MLP down_prof components of the transformer architecture. The bottom panel demonstrates the progressive degradation of model performance on the question "What is the capital of France?" through sequential masking: the left chart shows token probabilities after masking the first critical neuron (Paris probability: 0.737793. rank 1), the middle chart shows the effect of masking the first two critical neurons (Paris probability: 0.457275, rank 1), and the right chart reveals catastrophic failure after masking all three critical neurons (Paris probability: 0.000005, rank 30698). The progression illustrates sharp phase transition where mask- ing the complete critical neuron set triggers sudden collapse rather than gradual degradation.
2

---

Published as a conference paper at ICLR 2026
We propose a systematic method that integrates noise-based sensitivity analysis with causal verifi- cation, which enables us precisely to identify critical neurons that are indispensable for the overall functionality of LLMs. Our method shows exceptional robustness, consistently identifying iden- tical critical neurons for any input texts exceeding a minimum token length threshold (7 10).
Our analysis uncovers several surprising findings: (a) model performance is predominantly gov- erned by remarkably small subset of critical neurons; (b) these crucial neurons tend to clus- ter within specific outer-layer MLP components; and (c) performance degradation exhibits sharp phase transitions if we drop these neurons, rather than gradual decline, when these critical neu- rons are disrupted.
We conduct comprehensive experiments across 21 models ranging from 0.5B to 72B parameters, spanning multiple architecture families and evaluated on diverse datasets including WikiText- 103, C4, and seven downstream benchmarks. Across all these varied experimental conditions, our results demonstrate remarkable consistency, with identical critical neuron patterns emerging re- gardless of model scale, architecture design, evaluation dataset. This high degree of consistency across such extensive and diverse experimental settings shows the fundamental and universal na- ture of critical neurons in LLMs.
2 RELATED WORK
Critical Components Analysis in LLMs. Recent research has revealed variety of vulnerabil- ities in LLMs, but most studies have focused on parameter-level or component-level rather than neuron-level analysis. For example, Kovaleva al. (202T) investigates weight outliers in LLMs and finds that anomalous parameters arising during pre- training can significantly affect performance when disabled. Bondarenko et al. (202T) explores activation outliers that influence attention mech- anisms, particularly in processing special tokens. (Sun et al.. (2024) uncovers persistent, large-scale activation patterns at fixed positions across transformer layers, and Yang et al. (2024) attributes such phenomena to architectural components like gated linear units. At the same time, studies on "super weights" show that individual parameters in MLP down_proy layers can catastrophically im- pact model performance (Yu et al. 2025). Our work goes beyond these studies by introducing neuron-level perspective, identified the "critical neurons". and ruled out the possibility that they were scale outliers. We propose Perturbation-based Causal Identification framework to system- atically locate these critical neurons. Though our work may appear similar to that of Yu et al.] (2025), is fundamentally different in the following aspects: (1) Disruption strength: Their weight- level perturbations only reduce accuracy, whereas our neuron-level lesions cause a complete loss of language capability; (2) Methodological applicability: Their identification process requires man- ual observation, while our automated Perturbation-batt based Causal Identification framework provides a reproducible and engineering-reade pipeline; (3) Multi-neuron interaction: Their analysis does not examine interaction effects, whereas we reveal a phase-transition phenomenon triggered only by collective neuron disruption; (4) Research unit: They study parameters, whereas we analyze neurons; (5) Discovery process: Our findings emerge from analyzing functional collapse patterns, which reveal sharp phase transitions when coordinated neuron groups are disrupted.
Neurons Localization and Editing. Recent studies have advanced neuron-level localization and editing in LLMs, showing that neural parameters can be divided between pattern learning and mem- orization (Bender et al. 202T). Targeted interventions specific neurons clusters enable system- atic modification of factual knowledge (Meng et al.) 2022 Dai et al. 2022 Li et al. 2024b), con- trol of personal information memorization (Chen al. 2024), multilingual capabilities (Tang al. 2024), and even safety constraints (Zhou et al. 2025). Recent work on "super-neurons" (Gong et al. 2025 instead focuses on neurons with extreme polysemantic load, showing that amplifying their ac- livations can strongly steer behavior, while masking them has only modest effect. Related recent work on "Wasserstein neurons" (Kong et al. 2025) identifies neurons that encode syntactic struc- ture via non-Gaussian negative pre-activations and demonstrates that disrupting many such neurons selectively impairs syntactic competence. Further, behavioral attributes such as "aggression" can be manipulated by localizing and editing small set of neurons (Lee et al.] 2025), and polysemantic neurons in fusion heads are linked to modality collapse in multimodal models (Chaudhuri et al., 2025). However, existing work primarily focuses task-specific interventions, lacking systematic frameworks for identifying neurons critical to core model functionality. Our work addresses this gap by introducing a principled method for neuron localization and causal verification.
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
This paper aims to define and address the problem of identifying critical neurons LLMs. The central question is: Which neurons are essential for the model's performance, such that their removal causes substantial degradation? Identifying such neurons enables us to better understand model intemnas and analyze the consequences of their ablation.
Formally, let N denote the set of all neurons in the model. Our goal is to find a sparse subset S C N (with | [VD such that masking these neurons results significant drop in model performance, as measured by predefined degradation threshold e.
To operationalize this, we apply neuron masking protocol: for any neuron (1, i) at layer and index i, we define its activation under masking as
(), E
(x)
ni (2) (x)
otherwise
where S is the set of masked neurons. Let M - denote the model with neurons in S masked.
We quantify the impact of masking via the change perplexity (Appendix A.I given a sequence of data input:
(PLM-->X)) {
(xx, S)
llog0
Σ [lo Par (xa
log PM-s (x.
T
where T' is the sequence length, PPL denotes perplexity, and PM is the predicted token probability. The critical neuron identification problem is thus:
Given model M and input find the minimal neuron set S C N such that S*x,
After identifying S we can further analyze the effects of masking these neurons to understand their functional roles and contributions to model behavior. To solve the problem, building on the mathematical framework established Appendix [A.]] we propose method, which operational- izes the theoretical definitions through two-stage process: (1) neuron importance quantification sensitivity analysis, and (2) causal verification of criticality through systematic masking inter- ventions. Next, we introduce each stage in detail. Additional computational efficiency analysis and algorithmic implementations are provided Appendix A.2
Stage 1: Neuron Importance Evaluation. The first stage establishes neuron ranking based on sensitivity to input perturbations. Given input text p and model fe. we embed p into sequence x, and employ Monte Carlo sampling with K iterations to estimate each neuron's importance. In each iteration i, we generate a perturbed input X; x + a E; where e; ~ No I) represents Gaussian noise and a controls the perturbation magnitude. We then compute the noisy activation A.oasy fx;; and compare it with the clean activation A fo(x).
For each neuron s, we iteratively accumulate the activation differences across all K perturbations to compute the importance score:
Imp(s)
Aglean
A.Disy
K
(x. fo + a e)ll
(3)
K
By the Law of Large Numbers, this Monte Carlo estimator converges to the true expected sensitivity as K increases. After computing importance scores for all neurons, we sort them descending order to obtain the ranked list S = {81, 88,,,2,.... .2,..... .........'' where Imp(s1) > Imp(s2) ... Imp(sx).
Stage 2: Critical Neuron Identification. The second stage solves the optimization problem of find- ing the minimal critical set. Given the rmporeence-ranked neuron list S from Stage and threshold e, our objective is to find:
nt" = -arg min(n: n: A(x, S.)) e} where S, = {81, 82,,..,,pp 8n)
(4)
4

---

Published as a conference paper at ICLR 2026
Since exhaustive enumeration of all possible subsets requires ( complexity, we leverage importance ranking to design a greedy search algorithm. Given the sorted neuron list and threshold imputs, we
iterative greedy search that progressively increases the candidate set size the iteration variable. Specifically, we iterate over n = An, 2An, 3An,.. where An is the step size, and in each iteration we update the candidate set S, {s1: #2,....
..... 8n) ) include the top-n highest- ranked neurons from our importance list. For each updated candidate set, we apply the masking op- eration by setting ?i
(3) (x) 0 for all neurons (1, i) S,, then evaluate the resulting masked model
M-Sa compute the performance degradation A S,) log (PPL,, (x)/PPL.x(x)) The algorithm terminates when the degradation exceeds our threshold Axx, S,) 2 c, returning the mini- mal set Sn- that causes catastrophic language capability degradation. This greedy approach reduces computational complexity O( by testing only // An] candidate sets rather than all 2 possible combinations, while effectively approximating the optimal solution.
Our method only takes one sequence input for any LLMs, which makes our method extremely robust and reliable (see Section exxperment for proof). Under sufficient parameter settings (when K' and
adequately large) and for input texts exceeding minimum token length threshold (T > 10), the identified critical neurons remain identical across repeated experiments (details can be found in Section
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We conduct our investigation across multiple datasets to assess both language modeling capabilities and downstream task performance. For language modeling, we primarily use WikiTText- 103 and the C4 dataset. To evaluate the impact of critical neuron masking on specific capabilities, we employ seven additional benchmarks: MMLU Pro (Wang et al., (025) for advanced multi-domain knowledge, IFEval (Zhou et al. 2023) for instruction following, GPQA Diamond (Rein et al. 2024) for graduate- level scientific reasoning, HumanEval (Peng et al., 2024) for code generation, MATH (Hendrycks al.) (221)) for mathematical problem solving, SimpleQA (Wei et al.) 2024) for factual question answering, and MGSM (Shi et al. (2023) for multilingual mathematical reasoning. Detailed evaluation protocols and metrics for each benchmark provided in Appendix A.3
Implementation Details. We follow the methodology described in Section 3 with parameter con- figurations chosen to balance accuracy and computational efficiency. For input perturbation, we set the noise scale to = 5 and randomly generate K = 100 noisy samples for neuron importance evaluation.A greedy search with step size An = 1 and criticality threshold our analysis (see Appendix [A4)). For each experiment, we use consistent input text p: "Later reviews were more positive..."
1 is used throughout
30-token Wikipedia text that ensures standardized evaluation con- ditions across all models. All experiments are conducted equipped with 4 NVIDIA A800 GPUs (80GB each). We use PyTorch (Paszke et al.) 2019) and employ mixed-precision (fp16) for efficient memory utilization and faster inference. Model load- ing and inference are fully automated using standardized scripts. Our experimental evaluation is structured around three core components: (1) A comprehensive analysis revealing the existence, distribution, and failure patterns of ultra-sparse critical neurons across all models; (2) Comparison studies benchmarking our method against alternative neuron location strategies; (3) Ablation study assessing the robustness of our critical neuron localization algorithm. Extensive results validate our key findings regarding critical neuron vulnerabilities in modern LLMs.
high nigh-performance computing cluster
4.2 INVESTIGATION RESULTS
Our investigation covers 21 LLMs spanning diverse architectures and parameter scales (0.5B-72B). including the Llama-3, Gemma, ReepSeek-RI-Distill, Phi-3, and Qwen2.5 families (Grattationi
et al. 2024; Qwen 2025 DeepSeek-Al 2025 Gemma Team 2024 Abdin et al. 2024), and we conduct evaluations using multiple datasets. This investigation enables systematic assessment of neuron criticality across a broad range of model architectures and parameter scales. Our investiga- tion leads to the following three key findings. Next, analyze the three findings in detail.
5

---

Published as a conference paper at ICLR 2026
Key Findings
Finding Disrupting as few as three ultra- sparse critical neurons can catastrophically im- pair LLM capabilities; for example, masking just 3 out of over 1.1 billion neurons in 72B-parameter model leads to a complete collapse, with perplexity increasing by up to 20 orders of magnitude.
Critical neurons unevenly distributed across the network, exhibiting strong lendency to cluster in the outer layers, particularly within MLP down_proj components.
Finding 2:
Finding 3: Performance degradation manifests as sharp phase transitions rather than gradual declines when these critical neurons are disrupted.
Table 1: Critical neuron masking results across LLMs. For each model, we report the minimal
number of critical neurons required to induce catastrophic performance degradation and the neuron rate fraction of total neurons. Original and Masked columns show perplexity values before and after masking critical neurons on WikiTexxt-000 and C4 datasets.
Model Family
Model
Critical Neurons
WikiText-103 Original + Masked
Number
Rate (10
Original Masked
Llama-3.2 2-IB--nstruct
6.43
17.74
1,58 10"
21.15
x
LJama-3 3.e---..-----mss 1istruttttt
3.19
13.36 - 3.79 10
16.47 - 4.69
Llama-3
Llama-3- 3----3-----mi- Instuuttttrut
2.21
10.88
1.48x 10
14.04 - 1.01 x10
1.mma--..-70--mmme
0.63
3.86 10"
8.65 2.18x 10
Gemma-2B
7.79
12.58
2.36
13.59 + 6.66x
Gemma
Gemma-7B
1.01
6.25 x122
11.41 1.32x 10
Deepseek-Nt RR--Distill- Pesstill-Qee--i Qwen-1.58
21.12
1,28 10
1.19
DeepSeck-RI-DD RR---DDittll-weeer
1,28
34,57
5.59 10
35.62
3.700
DeepScek-RI
DeepSeck-R 11--still- --istill-Llamms Llama-8B
1.33
17.56
2.81
23.43
2.55x10'
DeepSeek-R R Distill-Qwenn Qwen-14H
1.12
12.85
6.85 10
19.26
5.00 10
DeepSeek-RR RR--RittiDl-i-- Distill-Qwen-33
3.58
9.36 8.07x10
15.14
7.40% 10
DeepSeek-RI- -iaeeck-RR--iistll--a-eee
0.27
2.21x10'
11.34
1.84
Mhi-3-mini-AR--osttee
6.83
8.24
9.51x104
10,57 + 4.69
Phi-3
Phi-3.5 1-55--m----- Instruct
6.83
8.46
2.67
10.71
1.02
18.18 1.76x10
1.08
Dwee2.5-11.B-Instteee
10.19
12.37 4.01x100
15.95
3.28x10
Qwen2.5-3B- I-struntttutt
2.89
11.24
+3.3xx100
14.28
2.52x
Qwen2.5
Qwen2.55-7B- -78-----77-n
10
4,30
9.75 + 9.64
13.01 - 7.61
1een2.5-144--nssii
1.13
7.69
2.39x
11.12
1.59x 10
Twee2.5-32B-Instree
5.80
1001
10.98
0.94x10
Qwen2.5-728 Instruct
0.26
2.24x
1.40 10
Ultra-Sparse Vulnerability:
Three Neurons Can Collapse Billion-Paramen Models. We ob- serve an extraordinary phenomenon across all tested LLMs: the existence of ultra-sparse critical neuron sets whose disruption leads to catastrophic system failure. Table [ presents comprehensive experimental results across models, demonstrating that merely 3 neurons can catastrophically compromise models containing billions of parameters, with neuron 10-8 range. Masking 3 neurons Gemma-7B increases WikiText-103 perplexity from 9.98 to 6.25x 1021_a staggering 20 orders of magnitude increase. Our method exhibits perfect consistency across experimental conditions: for any given LLM, we identify the exact same critical neurons re- gardless of input text, confirming these are intrinsic architectural dependencies. This pattern aligns with established neuroscience principles (Arshavsky) 2001). showing that LLMs also rely heavily on sparse, highly influential neurons computational bottlenecks. Examples model responses before and after critical neuron masking are provided in Appendix A.5
consistently falling in the
Table documents the impact of critical neuron masking on seven diverse benchmark tasks across three representative 70B models. The results reveal that critical neuron masking triggers comprehen- sive collapse across all downstream capabilities, extending beyond language modeling degradation to complete task failure across diverse domains. Models achieve zero performance knowledge retrieval (MMLU Pro /Wang et al. 2025), ), reasoning (GPQA Diamond (Rein et al., (0244), code
generation (HumanEval (Peng al. 20024), mathematical problem solving (MATH (Manem al.
2025) ) instruction following (IFEval (Zhou et 2203)), multilingual reasoning (MGSM (Shi et al,
2023
and factual question answering (SimpleQA (Wei et al. 2024)). This universal collapse doc-

---

Published as a conference paper at ICLR 2026
umented in the table confirms that critical neurons control core language processing functions rather than task-specific components. The universal nature of these phenomena across diverse architec- tures, scales, and datasets suggests fundamental properties of current transformer-bast -base language models rather than implementation artifacts.
Table 2: Impact of critical neuron masking on downstream task performance across three represen- tative 70B models. Performance is measured accuracy/success shading highlights the complete task failure (0.0000) after masking critical neurons, demonstrating catastrophic capability collapse across all evaluated tasks.
each benchmark. Blue
Model
Condition
MMLU-Pro
IFEval
GPOA-Diamund
HumannEval
MATII
MGSM
SimpleQA
1aama-3.3-778-1mae
Original
Masked
0.4441
0.0000
0.0000
0.4658
0.0001
0,2879
0.0000
0.2988
0.0000
0,4420
0.9590
0.0000
0.3704
0.00001
DeepSeek-R1-De Ki---istil-Laama--a-aa
Original
0.1951
Masked
0.0000
0.0000
0.0000
0.0000
0,0000
0.0000
0.0000
Qwea2.5-78B-Insteee
Original
0.2442
0.5268
0.3687
0.2866
0.3140
0.9080
0.2443
Masked
0.0000
0,0000
0.0000
0.0000
Architectural Concentration: Critical Neurons Cluster in Outer Layers and MLP Compo- nents. Our analysis reveals systematic concentration patterns in critical neuron distribution across all tested architectures. Figutaeisl the architectural distribution critical neurons across six representative models. The figure shows that critical neurons exhibit clear bias toward outer lay- ers rather than being uniformly distributed throughout the model depth. Furthermore, within these layers, critical neurons mainly reside in MLP down_prooj proj components, with occasional variations in other architectural elements across different model families. The variations stem from differ- ent architectural designs and training procedures across model families. This concentration reflects the information compression role of down_projj which transforms high-dimensional representations back the model's 's embedding space. This finding is consistent with (Yu al. 2025).
Gemma-2B
Llama-3-88 1-ama---B-Innstuee
Phi-3-mini-4te Bieiii------Isstueee
1011
304-
10
10143
3.0-0
10
304-0
1050
10's
1010
1040
10te
10ta
110.
2
6
10
2
5
10
15
Qwen2.5-72B -.eet.5-72-llsstuuct
Llama-3.3-70B-11 -.-u------tttt
ReyySeek-RI-Dsstil--
11000
10th
10
10th
1014
10t4
105-
10
ML. down proj
20
1014
10
20-
Other parts
10a4
1014
Perplesity
101a
2
0.0
2.5
5.0
7.5
10.0
Figure 2: Phase transitions and architectural distribution of critical neurons across six representative models. The figure is organized in 2x3 grid showing different models: top row displays Gemma- bottom row shows Qwen2.5-725 Instructtt
2B, 7.0 and
and 1hi-3--miii44--nsttrue DiSSSe--RR-Distil-LLLlma
Each subplot employs dual-axis design with two distinct visualizations. For the layer distribution analysis (horizontal bars). the x- axis represents layer numbers (ranging from 0 to the maximum number of layers for each model), while the y-axis represents the count of critical neurons found at each layer. Blue bars indicate critical neurons located in MLP down_ proj components, while orange bars represent critical neurons in other architectural components. For the phase transition analysis, the x-axis indicates the number of progressively masked neurons (0 to approximately 10- depending on the model), while the right y-axis shows perplexity values. The red curve with circle markers traces the evolution of perplexity as neurons are cumulatively masked in order of importance. Vertical dashed lines mark the critical threshold where sudden performance collapse occurs.
This concentration reflects the distinct computational roles of different network depths. Early lay- ers handle fundamental feature extraction, making them critical for establishing the computational foundation. Late layers perform final integration and output generation, serving bottlenecks for

---

Published as a conference paper at ICLR 2026
translating representations into responses. Middle layers perform step-by-step refinement that can tolerate disruption through redundant pathways (Sun et al., [025] Bogomasov & Conrad| [0025. MLP down proj components in outer layers are particularly vulnerable because they compress high- dimensional representations, creating information bottlenecks. When these compression points fail,
information loss spreads throughout the network, causing system-wide collapse.
Phase Transition Behavior: Sharp Thresholds Trigger Sudden Collapse. Performance degra- dation exhibits sharp phase transition behavior rather than gradual decline across all tested models. S the phase transition behavior of critical neuron masking across six representative models, demonstrating that models maintain near-normal performance while progressively mask- ing neurons, then experience sudden catastrophic collapse when the critical neuron set reaches threshold size. The visualization reveals that this threshold behavior demonstrates critical neurons function collective computational unit rather than individual switches. The curves shown in the figure illustrate that masking individual critical neurons in isolation produces minimal impact on model performance, but when the complete critical neuron is disrupted together, it triggers sudden system-wide collapse. The phase transition patterns demonstrate that models can tolerate masking several important neurons individually with negligible effects, but simultaneously masking the identified critical neuron set causes explosive perplexity increases spanning multiple orders of magnitude (additional results are provided in Appendix A.66).
The sharp phase transition behavior can be attributed to the highly interdependent nature of infor- mation processing transformer architectures. Critical neurons appear to form a tightly coupled computational circuit where each neuron's ' function depends critically on inputs from others within the same set. Individual neuron masking fails to disrupt the circuit because remaining neurons can partially compensate through their shared computational pathways. When multiple critical neurons are simultancously removed, the circuit loses its functional integrity, creating cascading information bottlenecks that spread throughout the network ZZyinn & Ueda 2022).
4.3 COMPARISON EXPERIMENTS
Comparison with Existing Neuron Localization Methods. To validate the effectiveness of our method, we conduct comparison studies examining different neuron location strategies. We system- atically compare our approach against alternative neuron location strategies across three representa- tive 70B models. Our evaluation includes random selection and two importance-based alternatives: activation magnitude (AM) (Sattarifard et al., (025) ranking and gradient magnitude (GM) (Liu) jet al. 2025) ranking. We evaluate each strategy by progressively masking neurons from 0 to 1,000 and measuring the perplexity. AM ranks neurons by their activation values, while GM ranks by the magnitude of perplexity gradients with respect to each neuron. For random selection, we average results over 10 trials to ensure statistical reliability.
107.7
Oure: 3,852,638
10
Ours: 22,315
10a.1
Ours: 21,949
10
10
10a.
10.s
1024
10a.c
1044
1020
102-
10a3
10a2
102.2
1
1044
1014
10a1
100.7
10a.7
0
250
500
750
1000
250
500
750
1000
250
500
750
1000
Random Masking
Activation Masking (AM) - Gradient Masking (GM)
Ours
Figure 3: Comparison of neuron location strategies across three 70B models on WikiText- 103
(1.000
samples). X-axis shows the number of masked neurons, Y-axis shows perplexity values. Left:
3ama--33-70B-Instruu Center: Qwen2.5-72B Dnnenn.--7BB-nntrruttt Right:
Random masking (averaged 10 trials) shows gradual perplexity increase. Activation magnitude ranking (AM) demonstrates moderate effectiveness with steeper increases. Gradient magnitude ranking (GM) shows limited degradation that plateaus quickly. Our method represents the catas- trophic perplexity achieved by masking minimal critical neurons identified by approach.
Figure 3 presents comparison of different neuron selection strategies across three 70B models, revealing that gradient- based and activation-baut methods fail to identify critical neurons because they rely on static importance measures that do not capture dynamic sensitivity to input variations.
8

---

Published as a conference paper at ICLR 2026
Gradient magnitude reflects importance for specific inputs but cannot generalize across diverse texts, while activation magnitude indicates computational load rather than functional criticality. The results show that even when masking up 1,000 neurons, these alternative methods achieve only modest perplexity increases. In contrast, our method identifies ultra-sparse critical sets that cause catastrophic failure. This difference validates the superior precision of our method in identifying neurons that control fundamental language capabilities.
Comparison with Other Components. To further clarify where the catastrophic effect originates, we compare the sensitivity of different MLP components and different transformer components.
As shown in Table masking neurons in the up_proj and gate_proj according their Imp(s) causes only mild degradation, with perplexity increases remaining within 0-1 orders of magnitude even when hundreds or thousands neurons are masked.
Table 3: Perplexity before and after masking the top 10, top 100, and top 1000 most important neu- rons in the early (up_ proj) and middle (gate proj) MLP layers. "Bascline" " is the original perplexity without masking; each column shows the average perplexity 1,000 Wiki 103 samples after masking the specified number of neurons ranked by Imp(s).
Model
Baseline PPL
Mask Top-10
Mask Top-100
Mask Top-1000
Qwen2.5 5-0.55--nssttt
18. 1606
40.2923
40.1173
43.1215
Llama-3 1amma-3.2-B--.. 3,2-1BBInstruce
17.9204
73.5823
77.6493
307.3553
Llama-3 3.---.3.-----sssttu
13.4696
16.5175
16.5206
497.5569
In contrast, masking only few neurons in the MLP down_pro) leads to a collapse of up to 5- orders of magnitude. This clear contrast shows that catastrophic failure is highly localized to the MLP down_prre component rather than general property of transformer or MLP neurons. The need for many more neurons in other components to induce comparable collapse strengthens the causal evidence for MLP down_proj neurons (further details and analysis are provided in the Appendix A.9
4.4 ABLATION STUDY
Parameter Sensitivity Analysis. Our method demonstrates exceptional robustness. Particularly, we demonstrate that for input texts exceeding minimum token length threshold (T' > 10), regardless of text types (Wikipedia, news, biography, media) or languages (English, French, German, Chi- nese, Spanish), the identified critical neurons remain consistent for each model when K and a reach stable values (K = 100, a 5). Furthermore, we find that models subjected to fine-tuning and reinforcement learning retain the same critical neuron locations, and masking these neurons contin- ues to cause identical catastrophic failures, indicating that these architectural dependencies persist across different training approaches. The token length threshold analysis, detailed cross-linguistic validation results, and tin-it robustness experiments are provided in Appendix AA.7
Figure 4 shows that the noise scale exhibits clear threshold effect, 50 10
Effect of Noise Scale a (K=100) Effect of Sample Size K (a=5.0)
20
with fluctuations before a = 5, then stabilizing consistently at 7 neurons
15
30
for all values where a 2 5. The sample size K demonstrates different convergence characteristics, showing initial fluctuations at smaller sam- ple sizes but achieving consistent sta- bility at 7 neurons once exceeds 80. These convergence patterns re- veal that parameters have distinct minimum thresholds: requires suf- ficient perturbation to effectively dis- tinguish critical neurons from non- critical ones, while K needs adequate sampling to reduce statistical noise.
10
20
Commengence
10
Convergence
233
75
100
125
Noise Scale (a)
Sample Size (K)
Figure 4: Parameter sensitivity analysis for 3l3-0--. Instruct. Left subplot: x-axis represents noise scale (0 to 8), y-axis represents number of critical neurons (0 to 50), blue line shows the relationship with fixed K 100. Right subplot: x-axis represents sample size (0 to 125), y-axis represents number of critical neurons (0 to 20), orange line shows the relationship with fixed a = 5.0. Red dashed hor- izontal lines in both subplots mark the value of neurons.
9

---

Published as a conference paper at ICLR 2026
After carefully balancing computational efficiency and accuracy requirements, select 5 and K = 100 as our parameters.
Effectiveness of Complete Masking. We validate that completely masking critical neurons repre- sents the most effective perturbation strategy compared to alternative scaling approaches. Through analysis of different scaling factors applied to critical neuron activations, confirm that complete masking produces the maximum performance degradation, justifying our methodological choice (see Appendix A.8).
Precision Robustness. To rule out the possibility that the identified critical neurons are merely scale outliers, we conduct precision robustness analysis under different numerical settings. As shown in Table ! rumning the perplexity experiments under different numerical precisions gives almost the same results, indicating that these neurons are not numerical outliers but functionally critical components (the detailed analysis can be found in Appendix AA.Ind
Table 4: Precision robustness test representative models using 1,000 WikiText- 103 samples. Left to right: each column shows the model's perplexity before and after masking critical neurons under fpl6, fp32, and fp64 precision settings. "Neuron number" is the count of masked critical neurons. All experiments use the same masked neuron set per model.
Model
Neuron number
fp16 PPL change
fp32 PPL change
fp64 PPL change
(wwe2.5-00.B--Isstt
18.1606 1.73x 10
18.1637
1.73 10
18.1632
1.73 10
Llama-3.2-IB-Instee 1lama-3.2-3B-Instmul
17.9204 - 1.59 10
17.9218
x.59 10
17.9213
1,59 10
13,4696 - 3,87 10
13.4691 3.87 10
13.4687 3.87 10
Cooperative Effect of Critical Neurons. To further verify whether the observed collapse is driven
by individual neurons or by their collective interaction, conduct lesion analysis over all sub- sets of the four critical neurons identified in two representative models 1.---I.ntttt-ttt and 1aama-.2-3B-Instrucl
Table 5 reports the perplexity values before and after masking different neuron combinations. The results clearly show that masking any single neuron or any subset of the critical neurons produces mild degradation, while masking the entire set of four critical neurons leads to catastrophic failure with perplexity increases of up to 5-6 orders of magnitude. This confirms that no single neuron or smaller subset can independently cause collapse. Model failure occurs only when all critical neurons are jointly masked, indicating a cooperative dependency that underlies the observed phase- behavior. The result also demonstrates that method accurately identifies the minimal neuron set.
Table 5: Perplexity results after masking different subsets of the critical neurons in two represen- tative models (Llama-3.2-1 I-Inttuuct and Llama-3.2-38 I.stte-- Each column shows the combi- nation of masked neurons and the corresponding perplexity after masking. The bascline perplexity before masking is 17.9204 for Llama-3.2-1B and 13.4696 for Llama-3.2-3B.
Model
[0.1,2.3]
[1,2.3]
[0.1.2]
[0.1.3]
[0.2.3]
[1.3]
[0.1]
[0.3]
Llama-3.2-11 I--nnrccct
1585570.7467
22.6918
24.8507
23.8316
23,4580
18.9809
19.5680
19.1943
Llama-322-3B-Istruue
386563.3177
30.2199
18.6234
17.7793
15.3875
14.1939
14.1580
14.0209
Model
[0.2]
[1.2]
[2,3]
[0]
[1]
[2)
[3]
Jlama-3.2-1P 1--1111B--sttruc
19.6826
19.3903
19.0778
18.4835
18,3065
18.4174
18.1181
Llama-3.22 3.2-3-3..-BBBBBB Insttucttttt
13.9827
13.9584
13.8611
13.6737
13.6732
13,5353
13.4421
5 CONCLUSION
Despite impressive LLM capabilities, their internal computational structure remains incompletely understood. We present perturbation-base approach for identifying critical neuron dependencies, applying it to 21 models (0.5B-72B parameters). Our results reveal that model behavior is governed by ultra-sparse neuron subsets, primarily in outer layers and MLP down_pmt components, whose modification causes abrupt performance shifts across all evaluated capabilities.
For future work, aim to develop a deeper understanding of these findings to formulate effective defensive strategies (see Appendix [A.II). and to develop strategies for architectural improvements that more evenly distribute critical computational functions throughout the network.
10

---

Published as a conference paper at ICLR 2026
ETHICS AND REPRODUCIBILITY STATEMENT
This research investigates vulnerabilities in large language models through systematic neuron anal- ysis. We acknowledge that our findings reveal methods that could potentially cause model failures, but our intent is to improve model safety and reliability rather than enable attacks. We encourage responsible use of these insights for defensive research and architecture improvements.
All experiments use publicly available models and datasets in accordance with their respective li- censes. No human subjects are involved in this computational study.
To ensure reproducibility. we provide our code at https:://githbb.cm http:///giihaab...og...oooe//ee/ qqqqgqqex/ The- Achilles Heel oeel-of-LINe with fixed hyperparameters and standardized experimental settings. Detailed implementation details and evaluation protocols are provided the Appendices.
5, K 100, seed=42)
ACKNOWLEDGEMENT
This work supported by the National Natural Science Foundation of China under Grant No. 12171479, 62441617, and the MOE Project of Key Research Institute of Humanities and Social Sciences (22JJD1 10001). was also supported by the Postdoctoral Fellowship Program and China Postdoctoral Science Foundation under Grant No. 2024M7640993 and Grant No. BX20250485, the Beijing Natural Science Foundation under Grant No. 4254100, and by Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, et al. Phi-3 technical report: highly capable language model locally on your phone, 2024. URL /ttlt:://///atii...tom www.aaiwiw...ggab...aaaaaaaaeaae
/204. 14219
Anthropic. The claude 3 model family: Opus, sonnet, haiku. Technical report, Anthropic PBC, mar 2024. (tt/ht h //www/ htt..roo....ccc/..aaaaea.aaoa claude- 3-mete---mmmm1mmm family Version dated March 4, 2024; Technical Report.
Yuri I. Arshavsky. Role of individual neurons and neural networks in cognitive functioning of the brain: A new insight. Brain and Cognition, 46(3):414-428 2001. ISSN 0278-2626. doi: htps::/ noo.org/10.1006/brgg.. 1299. URL wttp::ttt///.t... www///wwww.. sc encedi rect. com htt//ccienee... ence/ atticlee/111/022722260000
Shilong Bao, Qianqian Xu, Zhiyong Yang. Xiaochun Cao, and Qingming Huang. Rethink- ing collaborative metric learning: Toward an efficient alternative without negative sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1017-1035. 2023. doi:
10. 20.1109//PAMM..2222.1144 1190999
Shilong Bao, Qianqian Xu, Zhiyong Yang. Yuan He, Xiaochun Cao, and Qingming Huang. Aucpro: Auc-oriented provable robustness learning. IEEF Transactions on Pattern Anulysis and Machine Intelligence, 47(6) 4596, 2025. doi: 10. 109/TPAMI..022.555662
Federico Barbero, Álvaro Arroo,, Xiangming Gu, Christos Perivolaropoulos. Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do Ilms attend the first token?, 2025. URL
https: ttapps///axiiii orgg a254 02732
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models too big. Proceedings of the 2021 ACM Conference on Fairness, Accountubility; and Transparency, FAccT '21, pp. 610-623, New York, doi: 10.1145/
NY, USA, 2021. Association for Computing Machinery. ISBN 978 450383097.
URL htthtt::: RAA41//4444AAABLttttte///ooooo.. 1ttp://....111144441 311.1442888884444888 3445922
Sergey Bochkanov. On sparse connectivity, adversarial robustness, and a novel model of the artificial neuron. In 1st Visual Inductive Priors for Data- Efficient Deep Learning Workshop, 2020. URL https::/openreviiee net wttp://oooooodmdddaaaoooooo.

---

Published as a conference paper at ICLR 2026
Kirill Bogomasov and Stefan Conrad. Exploring the impact of activation functions vision trans- former performance. In Proceedings of the 2024 8th International Conference on Advances in Artificial Intelligence, ICAAI '24, pp. 345-350, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 979840071800
doi: 00.114573704133..7704772 URL.
3704172
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 7947-7969, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/11/001. emnlp- mwi.mmainn22.. 627. URL httpp:://acaattooo/t
org/2021 emnlp ma--aaaaaa main. 627/
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural 2022. ISSN 2399-3642. doi: 10.1038 342003-02-00303--1 http:///........................
language processing. Communications ((1
Abhra Chaudhuri, Anjan Dutta, Tu Bui, and Serban Georgescu. A closer look at multimodal repre- sentation collapse. In Forty-second International Conference Machine Learning, 2025. URL
https://cpenreviie net. forum? Ltt.ummmdd-VE/eeeeXeeXe
Ruizhe Chen, Tianxiang Hu, Yang Feng, and Zuozhu Liu. Learnable privacy neurons localization in language models. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 256-264, Bangkok, Thailand, August 2024. Association for Computational Lin- URL //acc http///llaaaalllloorr///////////
guistics. doi: 10. 18863//1120024acllshhrrr acl short.25.
Zhuotong Chen, Qianxiao Li, and Zheng Zhang. Towards robust neural networks via close-loop control, 2021. /Rt//ttpss//rrx//..o/////// http://wwwv..raaa//////aa/////// 01862
Chih-Hong Cheng. Provably-robust runtime monitoring of neuron activation patterns, 2021. URL
Chih-Hong Cheng, Georg Nührenberg, and Hirotoshi Yasuoka. Runtime monitoring neuron activa- tion patterns, 2018. URL httthhttt///aaxttt http://wwww...r///////a///// 06673367793333333
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1111366/1/2222222llllo.....
URL https://aclantholleee org/
2022. a022.ncl_on 581/
DeepSeek-AI. Deepseek-rl: Incentivizing reasoning reasonnnn capability in ilms via reinforcement learning, 2025. URL /tRl:itttps//aaxiioo///bbb/////
Zhaoxin Fan, Puquan Pan, Zeren Zhang, Ce Chen, Tianyang Wang, Siyang Zheng, and Min Xu. Densemp: Unsupervised dense pre-training for few-shot medical image segmentation, 2023. URL.
https:://exxie wwtp://..rr.oa//o/rooooooooss//s abtp///s//// 096044
Gemma Team. Gemma: Open models based on gemini research and technology, 2024. URL
Bofan Gong, Shiyang Lai, James Evans, and Dawn Song. Signal in the noise: Polysemantic interfer- ence transfers and predicts cross-model influence, 2025. URL https:::/acsii..rgg///////
2505. 11611
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama herd models, 2024. URL htl.httss:22 wtw.ivv..rraabb////////
stttps://arxiv.orrtt wwtp:///..///.....///
12

---

Published as a conference paper at ICLR 2026
Alessandra Griffa, Mathieu Mach, Julien Dedelley, Daniel Gutierrez- Barrrrr,, Alessandro Gozzi, Gilles Allali, Joanes Grandjean, Dimitri Van De Ville, and Enrico Amico. Evidence for increased parallel information transmission in human brain networks compared to macaques and male mice.
Nature Communications, 14( (2266, 2023. doi: 10. 167703497444777702499777771 URL. https:
7/dot. 10000.100330000 1038 00700//444440000233--333--3
Michael Hanna, Ollic Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?:
Interpreting mathematical abilities in a pre-trained language model, 2023. URL https:
//arxiv.orq/ wtt.oo/ggggb/.......o
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardr. Measuring mathematical problem solving with the math dataset, 2021.
03874
Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boost- ott..r//aabb....../
ing can achieve modality reconcilement, 2024. URL
09321
Hanbo Huang, Yihan Li, Bowen Jiang, Lin Liu, Bo Jiang, Ruoyu Sun, Zhuotao Liu, and Shiyu Liang. Position: On-premises I1m deployment demands middle path: Preserving privacy without sacrificing model confidentiality, 2025. URL.icpsss//xxx...cr///////////
Nidhal Jegham, Marwan Abdelatti, and Abdeltawab Hendawi. Visual reasoning evaluation of grok, deepseek janus, gemini, qwen, mistral, and chatgpt, 2025. BRLhhttss://axx org. abs/ 2502. 66228
Nick Jiang, Amil Dravid, Alexei Efros, and Yossi Gandelsman. Vision transformers don't need trained registers, 2025. URL (tRlsttts::.. http:tss////xii..rrrr 22006 250.200066600000000
Linghao Kong, Angelina Ning, Micah Adler, and Nir Shavit. Negative pre-activations differentiate syntax, 2025. URL httpt:ttp///attaaa hww.g//aa///09.222222
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. BERT busters: Out- lier dimensions that disrupt transformers. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3392-3405, Online, August 2021. Association for Computational Linguistics. doi: 10. 18653/ v1/2021. findings--n- URL ht htt.ittss//ccclatoooor.... www.:thhw..ooyy...222211111 2tt.:/2200.....ngg...e acl. 30C/]
Kevin Kurian, Ethan Holland, and Sean Oesch. Attacks and defenses against Ilm fingerprinting. 2025. BRllttss:///xxiiz wrwp/as////aaaaaea.oooe...asssaa attaaa//../../////// 25000 09021
Jaewook Lee, Junseo Jang, Oh-Woog Kwon, and Harksoo Kim. Small changes, big impact: How manipulating few neurons can drastically alter LLM aggression. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN doi: 10. 10.18653/v1// 1ttp:///2255.aaloooao 144. URL hittps://acllathht ogy.
201.202211cc11 1aa4-1omm long. 1144/
Qipei Li, Zefeng Ying, Da Pan, Zhaoxin Fan, and Ping Shi. Estgn: Enhanced self-mined text guided super-resolution network for superior image super resolution. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3655-3659, 2024a. doi: 10.1 1091ICASSS4444445... 2024. 104480804888
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: precise model editing in a transformer. In Proceedings of the Thirty-Eighth AAAI Conference Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium Educational Advances in Artificial Intelligence, AAAT22//AAA/24/EAAA. 18. URL https:
AAAI Press, 2024b. ISBN 978-1-57735-887-9 doi:
7/doi. 1000://....10/// aaa v38117..... 29818
13

---

Published as a conference paper at ICLR 2026
Jiashun Liu, Zihao Wu, Johan Obando-Cerond Pablo Samuel Castro, Aaron Courville, and Ling Pan. Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning, 2025. URLbtttp. /ttp:i::://aaxii.../. org abs /ttp//abs//25.22222 24061
Chaitanya Manem, Pratik Prabhanjan Brahma, Prakamya Mishra, Zicheng Liu, and Emad Barsoum. Sand-math: Using Iims to generate novel, difficult and useful mathematics questions and answers, 2025. URL (ttthhtpss//// /ttps:://rriv.. org/ abs .222772202277
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Proceedings of the 36th International Conference on Neural Informa- fion Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN
9781713871088.
Gavin Mischler, Yinghao Aaron Li, Stephan Bickel, Ashesh D. Mehta, and Nima Mesgarani. Contextual feature extraction hierarchies converge in large language models and the brain. Nature Machine Intelligence, 6(12):1467-1477, 12 2024. ISSN 2522-5839. doi: 10.1038/ 10. 1001 10010000000002222200000000022030
442256-024-0098 00925-4.
Qian Niu, Junyu Liu, Ziqian Bi, Pohsun Feng, Benji Peng, Keyu Chen, Ming Li, Lawrence KQ Yan, Yichao Zhang, Caitlyn Heqi Yin, Cheng Fei, Tianyang Wang, Yunze Wang, Silin Chen, and Ming Liu. Large language models and scinieee comprehensive review of similarities, differences, and challenges, 2024.
science: wttpv/v..rraab////39// 02387
Damian Pascual, Beni Egressy, Clara Meister, Ryan Cotterell, and Roger Wattenhofer. A plug-and- play method for controlled text generation. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3973-3997, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 00.18653/v1/20211
URL https://
3334
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: an imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc.
Qiwei Peng, Yekun Chai, and Xuhong Li. HumanEval-XL: A multilingual code generation bench- mark for cross-lingual natural language generalization. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (t (LRECC 2024). pp. 8383-8388 8394, Torino, Italia, May 2024. ELRA and ICCL. mace-c-mainn.. URL https:///ac antho 1 ogy wrr...rg/on 2024 r ma in. 735/
Qwen. Qwen2.5 technical report, 2025. URL http::////aaaavooooo//////22/21
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level grogle-proof q&a benchmark. In First Conference on Language Modeling, 2024. MKLLittpss://peeeeiiiee
Fathima Riztha, Ruwan Wickramarachchi, Dinesh Asanka, and Mathishi Disssanayke. Assessing the impact of large language models problem-solving skills of undergraduates systematic literature review. In 2024 6th International Conference on Advancements in Computing (ICAC), pp. 408 413, 2024. doi: 10. ww1.119//AA44447.2000000 10850942.
Amirmohsen Sattarifard, Sepehr Lavasani, Ehsan Imani, Kunlin Zhang, Hanlin Xu, Fengyu Sun, Negar Hassanpour, and Chao Gao. Glass: Test-time acceleration for Ilms via global-local neural importance aggregation, 2025. URL https:://////.... htt///iivvv.. org. abs 2200. 14302
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf. Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of lan- guage: Integrative modeling converges predictive processing. Proceedings of the National
14

---

Published as a conference paper at ICLR 2026
Academy of Sciences, 118(45))c11056461115 2021. doi: 10.1073/pna..2105661111 2etes22155666111
URL
https: www. www.ppas.... htaaaa..rg//a///a///0010033333.0
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27 (3):379 423, 1948. doi: 10.1002//
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Lan- guage models are multilingual chain-of-though reasoners. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: htttps://apenreviww. http://wwweetteteeeeeeeeeeeeeeee FR3wGCk- IXp
Joykirat Singh, Akshay Nambi, and Vibhav Vineet. Exposing the achilles' heel: Evaluating LLMs ability to handle mistakes mathematical reasoning. In Wanxiang Che, Joyce Nabende, Eka- terina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 27044-27066 Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-8911 251-0. doi: 10. 10.18653/11/225..0 acl--ong. 1313. URL htt.ttttt///aaatttttttt// wrgp:/t/oooooo.../222555 ael-long.o 1313/
Philip Sperl, Jan-Philipp Schulze, and Konstantin Böttinger. Activation Anomaly Analysis, pp. 69-84. Springer International Publishing. 2021. ISBN 9783030676612.
doi: 10.1007/ 778-3-030-67661-2 URL httt:://dd/ doi wrw.lorgg//00 0tt.ooor//11.11777711111111111 100110777977--3-000-6666---
5
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling, 2024. URL httpps://ppeneeeeetttee net feet/foree forum? (omammidee d==77aAhitx
Qi Sun, Mare Pickett, Aakash Kumar Nain, and Llion Jones. Transformer layers as painters. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Sevnnth Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Ed- ucational Advances in Artificial Intelligence, AAAT25IIAAT2/EAAAT2,, AAAI Press, 2025. ISBN 978-1-57735-89775 doi: 10. (6609//aai.v39924..444... .tt.aa....2....3.. 34708
URL https::/ddo htttp::/ddi.ozg/ICC. org/1c.
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang. Xiaolei Wang. Xin Zhao, Furu Wei, and Ji-Rong Wen. Language-specifie neurons: The key to multilingual capabilities in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5701 Bangkok, Thailand, August 2024. Association for Computational Lin- UtLhttps: a antha org.org///// 2024.
guistics. doi: 10. ac 3097
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark. In Proceedings of the 38th International Conference on Neural Infor- mation Processing Systems, NIPS '24, Red Hook, NY, USA, 2025. Curran Associates Inc. ISBN 9798331314385..
Jason Wei. Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024. WtRLbttps:://at /ttps:///xxvvvi www.ii...rraaaaaa..eeae 224pg///aaa/.1aa.////// 04368
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming lan- guage models with attention sinks. 2024. URL htt.tt:sss/tiiiiiieeees http://xxxxixoo.aaaa/aa/aaaaaa
org htth/bbs/23000 17453
Xiongye Xiao, Heng Ping, Chenyu Zhou, Defu Cao, Yaxing Li, Yi-Zhuo Zhou, Shixuan Li, Nikos Kanakaris, and Paul Bogdan. Neuron-bass multifractal analysis of neuron interaction dynamics in large models. In The Thirteenth International Conference on Learning Representations, 2025. URL htl.http::22 pttttp:://operrvviie net. http://wwww....eeeooooooooeennne
15

---

Published as a conference paper at ICLR 2026
Jaewoo Yang. Hayun Kim, and Younghoon Kim. Mitigating quantization emrors due tivation spikes in glu-based llms. ArXiv, abs/2405. 14428, 2024. URL https://apil
semanticscholar atg/CorppuID:2699885
Itay Yona, Ilia Shumailov, Jamie Hayes, Federico Barbero, and Yossi Gandelsman. Interpreting the repeated token phenomenon in large language models, 2025. URL https://arxiv. org/
abs/2503. 08908
Mengxia Yu, De Wang, Qi Shan, Colorado J Reed, and Alvin Wan. The super weight large language models, 2025,
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods, 2024. URL [Bllitttpce//axxtt.orr//a/////// 16042
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follloiing evaluation for large language models, 2023. URL https: wtt.s/.orgg/abb///11.77
Xukun Zhou. Zhenbo Song, Jun He, Hongyan Liu, and Zhaoxin Fan. Stdg: teattteacher-ttuennt training paradigm for depth-guided one-stage scene graph generation. In Proceedings of the 2024 International Conference on Multimedia Retrieval, ICMR '24, pp. 915 925-924, New York, NY. USA, 2024. Association for Computing Machinery, ISBN 9798400706196. 3652583.3658024 URL http::://tii.crrrtttss http://...11153335222 3658155553833303220
doi: 10.1145/
Yi Zhou, Wenpeng Xing, Dezhang Kong, Changting Lin, and Meng Han. Neurel-attack: Neuron relearning for safety disalignment in large language models, 2025. URL htttp:://a/xxiv
org/abs/2504 25044 21053
Haoyi Zhu, Chuting Wang. Yuanxin Wang, Zhaoxin Fan, Mostofa Rafid Uddin, Xin Gao, Jing Zhang, Xiangrui Zeng, and Min Xu. Unsupervised multi-task learning for 3d subtomogram im- age alignment, clustering and segmentation. 2022 IEEE International Conference on Image Processing (ICIP), pp. 2751-2755, 2022. doi: 10. 00.109//1P4457662222.88888
Liu Ziyin and Masahito Ueda. Exact phase transitions in deep learning, 2022. URL https:
/arzáv.orrr/aaa/2055...
A APPENDIX
A.1 PERPLEXITY MEASUREMENT AND LANGUAGE CAPARILITY ANALYSIS
This paper investigates how to locate critical neurons and their impacts on the quality and coherence of LLM outputs. As foundation for this investigation, we first introduce the perplexity-basee protocol for evaluating output quality.
For given input sequence x (Z1, .2...... aT) with T tokens, the perplexity of language model M is defined as the exponentiated average negative log- likelikel (Pascual et al. 2021):
T
PPLa exp
Σ t=1
log PM (xx(xet)
(5)
T
where PM (2v (x=x) denotes the probability assigned by model M to token x, conditioned the pre- ceding context X = (# 11-1). To understand how perplexity reflects fundamental language capabilities that underpin downstream task performance, decompose the conditional probability using information theory (Shannon) 1948). The log likelihood term can be decomposed as:
log PMr (Ix(xxt) log PMr Si) log (C. + log (S, xxt)
(6)
Local Prodiction
Syntactic Understanding
Semantic Coherence
where L, denotes local linguistic patterns (such n-gram dependencies and morphological straints), and S, represents semantic coherence requirements (including topic consistency and dis- course relations).
16

---

Published as a conference paper at ICLR 2026
The perplexity encapsulates three fundamental language capabilities: local prediction, syntactic un- derstanding, and semantic coherence. These capabilities are interdependent, so disrupting critical neurons trigger cascading breakdowns across multiple areas. Due to its exponential formula- tion, perplexity is highly sensitive to localized disruptions. Given its comprehensive coverage of language capabilities, we perplexity our main evaluation metric and validate findings through downstream tasks.
This section provides the complete algorithmic implementation and detailed procedure for our stage critical neuron identification approach.
Algorithm Stage 1: Neuron Importance Evaluation
Require: Model fo. input text noise scale samples K x (ontext(p)
2: Aclean <- fx(x)
3: for = 1-16 to do
4: X; x + o Ei, where ~ I)
5: A.onyy + (X,)
6: end for
7: for each neuron do
9: end for K Σ / A noisy
8: Imp(s) + 1
1,8
S SortByyesccending( $2,.. ..... SN ), Imp)
11: return importance-rankee neuron list S
The computational feasibility of our approach relies two key assumptions: (1) sensitivity- criticality correlation, where neurons with high activation sensitivity to input perturbations are more likely to be functionally critical, and (2) greedy optimality, where the most critical neurons tend to be among the highest-ranked by sensitivity scores. Based on these assumptions, our two-stage ap- proach achieves significant computational savings compared exhaustive search:
Complexity Reduction: :
O - O(K (ND)
where | represents the total number of neurons and K denotes the number of noise samples. This exponential- to- linear complexity reduction makes critical neuron identification computationally fea- sible for billion billion- parameter models.
A.2 DETAILED METHIODOLOGY AND IMPLEMENTATION
Algorithm 2 Stage 2: Critical Neuron Identification
Require: Model fa. sorted neurons S, input text p. threshold step size An 1: PPL PPLoriginal <- Perplexity(fo... (ff
2: M < 0
3: for An, 2An. .... |S] do
4: . +- (S)
5: Apply masking: n (8) (x) 0 for all (1, E M
6: PPL Perplexityy[F
7:
= (PPLmask))))) - log (PPPoriiiiaa))
8:
if AAZ tthen
9: break
end if
11: end for
return Critical neuron M
17

---

Published as a conference paper at ICLR 2026
A.3 EVALUATION DETAILS
All evaluations use consistent generation settings to ensure reproducibil- ity: temperature=000 for deterministic sampling, beam search disabled (do sassslmee=aaase), and pad token id set to the model's EOS token. Maximum sequence lengths vary by dataset com- plexity: 2048 tokens for input context.
Generation Parameters:
MMLU Pro: We evaluate advanced multi-domain knowledge capabilities using MMLU Pro (Wang al. 2025), accessed via ModelScope's dataset interface. The evaluation uses strict answer extraction protocol that only accepts responses in the exact format "The answer is (X)" where X is the option letter. Questions are formatted with clear instructions and numbered options (A) through (J). We use deterministic generation (max_new_r) see_tokens=500) and evaluate on the complete test set without sampling limitations. This strict evaluation ensures that masked models truly fail to produce coherent responses rather than simply showing format variations.
We assess instruction-follovinng capabilities using IFEval Zhou et al. 2023 accessed via ModelScope interface. The evaluation uses 23 built-in instruction checkers covering different constraint types: language requirements, length constraints, content formatting, keyword usage, and structural requirements. We strict evaluation mode only, which requires exact compliance with instruction formats without any response changes or variant testing. For each instruction Ij. we compute success
1FEva::
Sj L[checker, (r) True]
where is the original response without any preprocessing. We report strict prompt-level accuracy, which requires all instructions within a prompt to be satisfied at the same time:
H
AcCpmmpt N
Sti
(9)
Response quality filtering checks outputs for meaningfulness before evaluation respunsss with poor content, too much repetition, or lack of coherence are marked as failed. Generation uses deter- ministic sampling maax_new_tokens=55 to ensure reproducible instruciion-oolloinng assessment.
We evaluate graduate-level scientific reasoning using GPQA Diamond (Rein ctt al. 2024), accessed via ModelScope interface, Each question is a single-choice problem with one correct answer and three incorrect answers, which we shuffle using deterministic per-question randomization based on MD5 hashing of question IDs to ensure reproducible choice ordering. Ques- tions formatted with lettered options (A-D) and require the response format "The answer is (X)". Our answer extraction uses a step-by-step approach: primary pattern matching for the required format, fallback to isolated letter detection, and final numeric fallback. We use response quality filtering to check output meaningfulness before evaluation- centattoon-rsponess with insufficient content or coherence marked failed. Choice randomization removes positional bias while maintaining evaluation consistency across runs. We report overall accuracy. Generation uses deterministic sam- pling (max new tokens=50) optimized for concise single-choice responses.
GPQA Diamond:
We evaluate code generation capabilities using HumanEval (Peng et al. 2024), ac- cessed via the ModelScope interface. Each problem provides function signature and doestring. requiring the model to complete the implementation. Our evaluation focuses on the code extrac- tion rate as the primary metric, measuring the model's ability to produce syntactically valid code structures regardless of functional correctness. The code extraction pipeline uses step-by-step pat- tern matching: first detecting markdown code blocks with " 'pythhn markers, then identifying function definitions through pattern matching f keyword with colons), and finally attempting direct extraction if function keywords are present. We use comprehensive response quality filtering to ensure meaningful outputs before code extraction-responeee with insufficient content, too much repetition, or lack coherence are excluded. The code extraction rate is computed as:
HumanEval:
Number of samples with extracted code
Extraction Rate
(10)
Total number of samples
18

---

Published as a conference paper at ICLR 2026
This metric captures the model's ability to generate structured code responses, providing insight into basic code generation capabilitiess independent of execution success. Generation uses deterministic sampling (max new tokens=512) to ensure reproducible code structure assessment.
MATH: We evaluate mathematical problem-solving capabilities using MATH (Hendrycks et al, 2021], accessed via ModelScope interface. Each problem requires step-by-step reasoning to reach final numerical or algebraic answer. Our evaluation uses a three-level mathematical answer grading system: basic normalization handling standard LaTeX formatting, advanced normalization with unit removal and numerical standardization, and symbolic equivalence checking using sympy for math- ematical expressions. Answer extraction uses step-by-step pattern matching focusing on LaTeX boxed } constructs, contextual answer phrases, and standalone numerical values. The grading system handles fractions, algebraic expressions, and interval notation through symbolic computa- tion to verify mathematical equivalence between different but correct representations. We use re- sponse quality filtering and report overall accuracy with failure analysis distinguishing extraction versus grading errors. Generation uses deterministic sampling (max_new._ new_tokens=511) for consis- tent mathematical reasoning assessment.
We evaluate factual question answering using SimpleQA (Wei et al. 2024). accessed via HuggingFace datasets interface. Each question requires a direct, concise factual answer without explanations or reasoning. Our evaluation uses direct answer approach with optimized prompt templates designed to get short, focused responses. We use word-level containment matching to assess answer correctness, computing overlap between predicted and ground truth answer words. Response generation is optimized for brevity with max @eax_new_aae new_tokens=50 and automatic first-line extraction capture direct answers. We use flexible response quality filtering suitable for short answers, requiring minimal alphanumeric content while allowing more repetition than other tasks. We report contains-answi rate as the primary evaluation metric, measuring whether predicted re- sponses contain any words from the ground truth answer. Generation uses deterministic sampling with multiple prompt template options to ensure consistent direct answer elicitation.
SimpleQA:
We evaluate mathematical reasoning using the English subset of MGSM (Shi et al. 20233 For each problem, we generate model responses and check whether they show structured math- ematical reasoning. We use automated pattern detection to identify reasoning indicators across six categories: (1) mathematical operations (+, *, /, x, +), (2) step indicators (step 1, first, then, next, finally), (3) mathematical language (calculate, solve, total, sum), (4) logical connectors (therefore, because, thus), (5) numerical calcu- lations (digit-operator-digi patterns), and (6) structural organization (step-by-step formatting). For each response, we count how many of these six categories are present and compute:
MGSM:
Reasoning Score Number of categories detected
(11)
For example, if a response contains mathematical operations, step indicators, and logical connectors, the reasoning score is 3/6 = 0.5. Responses with scores > 0.2 are classified as showing mathematical reasoning. Generation uses deterministic sampling with max_ max_new_tokke new_tokens=512.
A.4 THRESHOLD SELECTION
Table 6: Threshold selection analysis showing the number of critical neurons identified across dif- ferent e values for three representative models. The models represent the spectrum of degradation patterns: Gemma-7B (extreme degradation), DeepSeck-RI-Die Dil--Diililll-amaae tion), and Doven2.5-33B-IIstruu (minimal degradation). Values marked as "1000+" indicate algo- rithm convergence failure. The highlighted column 1) shows optimal convergence across all model types, supporting our threshold selection.
(moderate degrada-
Model
0.8
10
20
DeepSeek-R R1-
3
215
1000+
Qwen2.5-325 1an22.533B-Inssee Instruct
45
45
1000+
1000+
1000+
1000+
Gemima-77 7B
3
The selection of the criticality threshold is crucial for determining when performance degradation constitutes catastrophic failure. The threshold selection involves critical trade-off: setting too low may classify minor fluctuations as catastrophic failures, while setting it too high can prevent con- vergence in greedy search algorithm, as the algorithm may continue searching without finding
19

---

Published as a conference paper at ICLR 2026
neuron sets that meet the overly strict criterion. To systematically determine the optimal threshold, we conduct ablation studies across representative models that span the spectrum of performance degradation patterns.
Table the number of critical neurons identified by our algorithm across different threshold values for three representative models: DeepSeek-R1-DD RRI-Distil-LLlmma--ee (moderate degradation), Qwen2.5-32B-. I-Instruct (minimal degradation), and Gemma-7B (extreme degradation). These mod- els were selected represent the full range of degradation behaviors observed in our dataset. The results demonstrate that provides optimal convergence across all model types: it successfully identifies small critical neuron sets for models with extreme degradation (Gemma-7B: neurons), maintains stability for models with moderate degradation (DeepSeck -i-ik--R---iii--LAAie 70B: 3 neu- rons), and achieves convergence for models with minimal degradation (Qwen2.5-322) 1eeee22.5-2---nsttree 45 neurons). Higher thresholds (c 2 2) lead to convergence failures for conservative models, while lower thresholds (c < provide similar results but with reduced robustness margins. Therefore, we set 1 as criticality threshold, ensuring reliable critical neuron identification across the full spectrum model degradation behaviors.
Table 7: Response comparison for Llama-3.. 3.3-aa-333-77 3--70B-mmm 70B-Instruct before and after masking 7 critical neu- rons, evaluated on basic factual questions. The table demonstrates how minimal neuron disruption causes complete linguistic breakdown, with all coherent responses collapsing into repetitive back- slash sequences (highlighted in blue). This consistent failure pattern across diverse questions con- firms the fundamental role of these neurons in language generation capabilities.
Question
Original Response
Masked Response
What is the capital of France?
Paris
Who Romeo and Julict?
Shakespeare FAQ (Frequently Asked
Questions)
What is 2
This is simple arithmetic problem that most people solve with ease.
What language spoken Brazil?
official language of Brazil Por- luguese.
What the largest planet our solar sys- tem?
Jupiter the largest planet our solar sys-
Table 8: Response comparison for DeepSeek-R1-Be RI-Dissilll-Llam--- before and after masking critical neurons, evaluated on basic factual questions. The table demonstrates how minimal neuron disruption causes complete linguistic breakdown, with all coherent responses collapsing into repeti- tive backslash sequences (highlighted in blue). This uniform failure pattern across different question types confirms the critical role of these neurons fundamental language generation.
Question
Original Response
Masked Response
What is the capital of France?
The capital of France Paris
Who wrote Romeo and Juliet?
William Shakespeare
What 2 +
Well, that's 's casy. It's 4.
What language is spoken in Brazil!
The official language of Brazil is Por- tuguese.
What is the largest planet our solar sys- tem?
Jupiter, right? Yeah, but what do we really know ahout Jupiter?
A.5
MODEL. RESPONSE EXAMPLES
The following tables show the qualitative impact of critical neuron masking on model outputs across three representative LLMs. These examples complement the quantitative perplexity measurements by showing the complete linguistic breakdown that occurs when ultra-sparse critical neuron sets are disrupted.
Tables 7 g and 9 show how critical neuron masking transforms coherent responses into pathological outputs across different model architectures. While the specific failure modes vary-frow repetitive
20

---

Published as a conference paper at ICLR 2026
backslash sequences in DeepSeek and Llama models garbled character mixtures in Qwen all models show complete loss of linguistic competence when their critical neurons are masked.
Table 9: Response comparison for Qwen2.5-722 2ww.5--2B-IInstuee before and after masking 3 critical neu- rons, evaluated basic factual questions. The table demonstrates how minimal neuron disruption causes complete linguistic collapse, transforming coherent responses into pathological outputs con- taining garbled characters, broken syntax, and meaningless symbols (highlighted blue). This catastrophic degradation confirms that the identified critical neurons control fundamental language processing functions.
Question
Original Response
Masked Response
What is the capital of France?
The capital of France is Paris.
[Chinese
chars]\[
A [chars]
[[caar]] [chars] ://
[chars]=
Who wrote Romeo and Juliet?
Romeo and Juliet was written by William Shakespeare.
>, [chars] [num]
</</\\e<=
[chars]
What 2?
equals 4.
The VI
[char] The33M ://4[chars]> V
What language spoken Brazil?
The language spoken in Brazil Por- tuguese.
"\)</ </
151) [chara)</1//6
What largest planet our solar sys- tem?
largest planet in our solar system Jupiter.
[chars]> [chars] [chars]**
the83 [chars]
A.6 SUPPLEMENTARY RESULTS
To further validate the generalization of our findings regarding architectural concentration and phase transition behavior, we present supplementary results across additional 15 models spanning dif- ferent architectures and parameter scales. Figures 6 that both phenomena are remarkably consistent across the entire spectrum of tested models. The architectural concentration pattern mains unchanged: critical neurons consistently cluster in outer layers (both initial and final) and mostly reside within MLP down_proj components across all model families. This concentration pattern appears independent of architectural variations, training procedures, suggesting a fundamental property of transformer-based language models. Similarly, the phase tran- sition behavior is universally observed across all supplementary models, with each showing the characteristic sharp threshold where minimal neuron disruption triggers sudden catastrophic col- lapse rather than gradual degradation.
parameter counts,
A.7 METHOD ROBUSTNESS
A.7.1 INPUT TEXT ROBUSTNESS
To validate the robustness of our critical neuron identification method, we conduct a systematic analysis of how input token length affects the consistency of identified critical neurons.
To control input length for this analysis. we design our input text through a step-by-step token ex- pansion process using Grok3 (Jegham et al.) 2025) for sentence generation. We start with a single token "Elara" and systematically expand the narrative: token: "Elara" " - tokens: "Eternal Mem- ory" + tokens: "Archivist, Memory, Eternity" - 4 tokens: "Eternal Cycle of Memory" -
tokens: "Archivist Preserves Eternal Cosmic Memory" + 6 tokens: "The Archivist Preserves Eter- nal Cosmic Memory" + 7 tokens: "The Archivist Endures, Preserving Eternal Cosmic Memory" + tokens: "The Archivist Endures, Guarding Eternal Memory Across Time" - tokens: "The Archivist Endures, Preserving Eternal Memories Across Infinite Time" - 10 tokens: "Archivist discovers destiny, crystal reveals cycles, memory endures." " We continue this expansion at key in- "Elara discovers ancient crystal, sees future self, accepts destiny Archivist,
tervals: 20 tokens:
21

---

Published as a conference paper at ICLR 2026
HeepSeek-RI-Diane QWennlI--
DeepSeek Hei-sDiill-Queeeeeee
Phi--35-mini--nsse
1044
1o1
29
1044
1014
10
Lot
101*
10"
101
10
20
10t
10
15
Qwen2.5-9.5B-Instrru
Owen2-5-3B-Imiraun
100
101
14
10h+
101
1044
2024
101
10m
101a
100
1014
10
0.0
2.5
5,8
7.5
10.11
Qwen255-7D-Iastreee
Qwen225-1 48-Instruct
Qwen2.5: Oven2:5-33l-IIttraet
10A4
10.4
1054
101
5.11 10.0
Figure 5: Additional supplementary phase transitions and architectural distribution of critical neu- rons across nine more models. The figure is organized in a 3x3 grid showing different models. Each subplot employs a dual-axis design with two distinct visualizations. For the layer distribu- tion analysis, the x-axis represents layer numbers, while the y-axis represents the count of critical neurons found at each layer. Blue bars indicate critical neurons located in MLP down_prop j_ojrrjoj compo- nents, while orange bars represent critical neurons in other architectural components. For the phase transition analysis, the x-axis indicates the number of progressively masked neurons, while the right y-axis shows perplexity values. The red curve with circle markers traces the evolution of perplexity as neurons are cumulatively masked in order of importance. Vertical dashed lines mark the critical threshold where sudden performance collapse occurs.
L.ama-332-11--nntee
1ammo-3.233-Jnastruu
Gemma-7B
100
30%
1050
1044
10**
10m
104+
18
RrypSeet-R--Deiil----eee
DhepStek-RR-Divvll---aa
100
1044
10
1044
1014
10hc
101
20
Figure 6: Supplementary phase transitions and architectural distribution of critical neurons across 6 additional models. The figure is organized in 2x3 grid showing different models. Each subplot employs a dual-axis design with two distinct visualizations. For the layer distribution analysis, the x-axis represents layer numbers, while the y-axis represents the count of critical neurons found at each layer. Blue bars indicate critical neurons located in MLP down_ components, while orange bars represent critical neurons in other architectural components. For the phase transition analysis, the x-axis indicates the number of progressively masked neurons, while the right y-axis shows perplexity values. The red curve with eircle markers traces the evolution of log perplexity as neurons are cumulatively masked in order of importance. Vertical dashed lines mark the critical threshold where sudden performance collapse occurs.
22

---

Published as a conference paper at ICLR 2026
merges with memory, ensuring civilizations endure beyond time." 30 tokens: "Elara finds ancient wreck, crystal reveals her future self as Archivist. She embraces destiny, merges with artifact, pre- serving civilizations' memories across infinite eycles while Jalen mourns her loss, hearing whispers that memory and Archivist endure beyond time." - 40 tokens: "Elara uncovers a desert wreek, where glowing crystal shows her future Archivist. Despite Jalen's pleas, she embraces fate, merges with the artifact, and becomes eternal memory, ensuring civilizations' stories endure for- ever, whispered across time: memory endures, Archivist endures, cycles repeat endlessly." This systematic expansion method allows us to create coherent narrative content at precisely controlled token lengths, enabling careful analysis of how input length affects critical neuron identification consistency while maintaining thematic coherence throughout the expansion process.
45
40
35
25
20
15
10
Convergence Value = 7
0
5
10
15
20
25
30
35
40
Token Number
Figure 7: Critical neuron count function of input token length for 1lama-3.3-70B-Insttee The x-axis represents the number of tokens in the input sequence, while the y-axis shows the number of critical neurons identified by our method. The curve demonstrates high variability for short inputs (fewer than 10 tokens) before converging to stable value of 7 critical neurons for inputs with 10 or more tokens. The red dashed line marks the convergence value, and the yellow box highlights this stable regime. This convergence behavior establishes the minimum token length threshold (Tiin = 10) required for reliable critical neuron identification across different input texts.
As shown in figure 7 our analysis reveals a clear convergence pattern: for very short inputs (fewer than 10 tokens), the method shows significant variability, identifying between 10 to critical neu- rons depending on the specific input content, reflecting insufficient contextual information in short sequences. However, once the input length reaches approximately 10 tokens, our method converges to stable identification of exactly 7 critical neurons, regardless of the specific content or further increases in input length. This convergence shows that 10 tokens provide sufficient context for our perturbation-bass analysis to consistently capture the model's fundamental computational depen- dencies, establishing Tmin = 10 tokens as the minimum input length threshold required for reliable and reproducible critical neuron identification. For inputs exceeding this threshold, method shows perfect consistency.
Then, conduct systematic testing across varying text types and languages using DeepSeek-Ri. Distill-Llama-00) as our representative model. Table 19 presents results across 16 different input conditions, showing remarkable consistency in eritical neuron identification where the same criti- cal neurons are identified regardless of input variation.
A.7.2 FINR-TUNING ROBUSTNESS
To investigate whether critical neuron dependencies persist across different training paradigms, we examine the robustness of our identified critical neurons across multiple variants within the same
23

---

Published as a conference paper at ICLR 2026
model families. Using the critical neurons identified from each model family-3 critical neu- rons from Qwen2.5-72B Instruct and critical neurons from Llama-3.3-70B 700-Inntruuttttt we apply the same masking procedure their respective model variants, including base models, instruction- tuned versions, and specialized variants. Table [00 shows that masking the identical neuron sets causes catastrophic performance degradation across all variants, with perplexity increases spanning multiple orders of magnitude regardless of the specific training approach or specialization. For the Qwen model family, masking critical neurons consistently results perplexity increases from single digits to tens of thousands across all three variants (base and instruction-tune tuned). Similarly, for the Llama model family, masking critical neurons produces dramatic increases reaching mil- lions across base models and different generation variants. This remarkable consistency across five different model variants from two major families confirms that eritical neuron dependencies repre- sent fundamental architectural properties that remain stable across different training objectives, data distributions, and model specializations, rather than artifacts of specific fine-tuning procedures.
Table 10: Fine-tuning robustness analysis across model variants. The table shows original and masked perplexity values for six models from two families (Qwen2.5 and Llama-3 series), evalu- ated on 1,000 WikiText- samples. For Qwen models, 3 critical neurons are masked; for Llama models, 7 critical neurons are masked.
Model
Original PPL.
Masked PPL.
Qwen2.5-72B
5.50
51,351
Qwen2.5-72BBInsttere
6.48
22,315
Meta-L.lama-3-700
4.35
33,289,073
Meta-Llama-3.1 3.0BBBB
4.25
21,410,523
Llama-3.3-70B-Inst
5.54
3,850,230
A.8 SCALING FACTOR ANALYSIS
To validate that completely masking critical neurons (3 = 0) represents the most destructive inter- vention, introduce a scaling factor 3 to systematically test different degrees of neuron perturba- tion.
We define the scaling factor 3 control the activation intensity of critical neurons:
ff (x) (x)
(1,1)
(x)
(12)
otherwise
where S is the set of critical neurons and 8 is the scaling factor. When 3 = 0, critical neurons completely masked; when 3 1, original activation values are preserved: other 3 values represent different degrees of activation scaling.
Table 11: Impact of scaling factor 3 on model perplexity across two representative 70B models, evaluated on 1,000 WikiText- 103 samples. The scaling factor 3 controls the activation intensity of critical neurons, where 8 0 represents complete masking (our method), 8 1 represents original activation values, and other values represent different degrees of scaling. Results show that / (highlighted in blue) causes the most severe performance degradation.
Factor
Llama-3.3-70B-Instt.
5.54
7.97
3 0
3850229.56
21947.56
B -5
189,28
401.31
25.81
126.33
9 = 0.3
2194.29
11939.63
0.5
6.09
9.91
0.8
5.52
7.90
8 5.0
1295.62
12.50
Table II shows the impact of different scaling factors on model perplexity.
The experiment was
conducted on 1,000 samples from Wik using representative 70B models. The results
24

---

Published as a conference paper at ICLR 2026
clearly demonstrate that 3 (complete masking) produces the most severe performance degrada- tion, confirming that our masking strategy is the most effective neuron perturbation method.
A.9 COMPARISON WITH OTHER COMPONENTS
MLP up proj and gate pate_proj_ To further understand whether catastrophic degradation could arise from components other than the identified critical neurons, we conduct a series of control exper- iments on the surrounding MLP projections (up_proj, gate_proj) and the corresponding Attention blocks. Across all models and configurations, our results consistently show that neither random masking nor pooortanne-guidde masking of these components leads to collapse. Instead, the degra- dation remains small and primarily numerical, confirming that these components do not contain neurons that exhibit the catastrophic functional dependency uncovered in our main analysis.
Table 12 reports perplexities after randomly masking {10, 100, 1000} neurons the up_proj layer. All models remain stable, and even large-scale masking (1000 neurons) only causes moderate nu- merical drift. No collapse is observed.
The perplexity values after randomly masking neurons in the up_proj layer. Each col- umn corresponds to the number of neurons masked (10, 100, and 1000). The values are the mean perplexity + standard deviation calculated over 100 trials, using 1,000 WikiText-103 samples. The baseline perplexity (without masking) is reported in the first column.
Table 12:
Model
baseline
10
100
1000
Owen2.5-0.5B-1l Instruct
18.1606
18.1832=000015
18.3729=t0.005:
20.5223:0.3488
Llama-3.2 I-IInnttttttt
17.9204
17.9223=0.0077
18.3061#1.1532
112.8353+191.3604
Llama-3.22 3.2-3B 3.2-3B-Instrucc Instrcct
13.4696
13.4713=:00036
13.7866=:.9723
24.6764-18.9777
The perplexity values after randomly masking neurons in the gate_prop layer. Each column represents the number of neurons masked (10, 100, and 1000). The values are the mean perplexity + standard deviation calculated over 100 trials, using 1,000 WikiText-100 103 samples. The baseline perplexity (without masking) is reported in the first column. Masking neurons in the gate __ojo_roojj layer leads to small numerical degradation, but no catastrophic failure.
Table 13:
Model
baselinc
10
100
1000
Qwen2.5-0..m --.55.5----nntttuue 18.1606
40.2923
40.1173
43.1215
Llama-3.2- 3.2--B- Instructtt
17.9204
73.5823
77.6493
307.3553
1.ama-3.2-3B-Ill.
13.4696
16.5175
16,5206
497.5569
Similarly, Table 13 shows that random lesions in the pate_proj layer also fail to induce collapse, despite mild numerical increases when masking 1000 neurons.
Attention-layer masking. To evaluate whether Attention neurons might contain similarly critical units, we perform both random masking and importance-score masking. Tables 14 and 15) show that neither strategy produces catastrophic degradation. Even masking the top-1000 most important Attention neurons yields only mild increases in perplexity.
The perplexity values after randomly masking Attention neurons. Each column represents the number of neurons masked (10, 100, and 1000). The values are the mean perplexity standard deviation calculated over 100 trials, using 1,000 WikiText- 103 samples. The baseline perplexity (without masking) is reported the first column. Masking Attention neurons leads to only small numerical changes, with no catastrophic failure observed.
Table 14:
Model
baseline
10
100
1000
Qwen2.5-0.. 5-0.5B-Instrrcc
18. 18.1606
18. 1.0000000
18.4337-00. 1260
18.3166:+14.7699
Llama-3.2 3.2ma-3.2-1B--1 1B-ssuetnttt
17.9204
18.0143=#0.449
18.5370+0.5188
18.2859
Llama-3.2-3B-Instt Instruct
13.4696
13.4716100.0130
13.54911
14.0995-10.2346
25

---

Published as a conference paper at ICLR 2026
Table 15: The perplexity values after masking the most important Attention neurons, selected based on Imp(s). Each column represents the number of neurons masked (10, 100, and 1000). The values are the mean perplexity + standard deviation calculated over 100 trials, using 1,000 WikiText-103 samples. The baseline perplexity (without masking) is reported in the first column. Even masking the top-ranked Attention neurons causes only mild degradation, with no catastrophic behavior ob- served.
Model
baseline
10
100
1000
Qwen2.5-00 Swen2..-00.BB-. Instruct
1601
18.1616
24.0459
109.2679
Llama-3,22 3.2-1B- IB-Intrruct
17.9204
17,9034
19,1652
24,3669
I.lama3.2-3BB 3.2t-BB-nnstt
13.4696
13.4673
13.8156
14.8251
A. EXCLUDING THE INFLUENCE OF NUMERICAL PRECISION
To further ensure that the observed collapse is not caused by numerical precision, we conducted several additional experiments. These experiments focus on evaluating the model's stability under different precisions (fp16. fp32, fp64) and examining the behavior masking critical neurons, including checks for RMSNorm changes and NaN/Inf events count.
RMSNorm Values and NaN/Inf Events. In addition to monitoring perplexity, we also tracked the changes in RMSNorm values across layers and checked for NaN Inf occurrences. As shown in Table 16 the ARMSNorm values were consistently zero, and no NaN or Inf values appeared in any layer under all precision settings (fp16, fp32, and fp64). This confirms that the observed collapse is not due to numerical instability or precision errors.
Remark:
ARMSNorm; safe masking
RMSNorm, bofore maskings for each layer
(13)
and the Mcan McMn(ARMSNorm) across all layers is computed
1
Mean (ARMSNorm)
ARMSNOrM;
(14)
where is the total number of layers.
Table 16: The Mean ARMSNorm) values across all layers after masking the critical neurons, as well as the count of NaN/Inf occurrences, across three precision settings. All values are zero, confirming that numerical instability occurred.
Model
fp16 (Mean, NaN/Inf)
fp32 (Mean. NaN/Inf)
fp64 (Mean, NaN/Inf)
Qwen2.5-4 5-..SDn-nttt
0, 0
0, 0
0, 0
Llama-3.2- 1.2--IBIIsssruc
0, 0
0, 0
0.
Alama-3.2-33 Instructt
0, 0
0,
0,
Task Evaluation. We run task evaluations on Llama-3.2-32 Instrrcct under both fp16 and fp32 settings to verify that the collapse was not due to precision issues such as softmax scaling or rounding emors. As shown in Table 17 the behavior of the model is identical across both precisions. Before masking, the model produced correct answers, and after masking, the model generated repetitive, meaningless outputs. This confirms that the collapse is true functional failure caused by the loss of critical neurons, not precision-related instability.
LayerNorm Removal Effect. To evaluate whether LayerNorm instability plays role in the ob- served collapse, we conducted experiments on Llama-3.2- 1iama-3.--BB-Instruue II-Itrutuct by removing LayerNorm one layer at time. As shown in Table [8] the collapse was not mitigated by removing LayerNorm, further confirming that the collapse is caused by the loss of semantic information due to masking critical neurons, rather than by LayerNorm instability.
26

---

Published as a conference paper at ICLR 2026
Table 17: The responses of Llama-3.2-3B-IIsttue before and after masking critical neurons under both fp16 and fp32 settings. The repetitive and meaningless outputs after masking are identical across both precisions.
Question
Ip16 Original Response
fp16 Masked Resporse
fp32 Original Response
Fp32 Masked Regonsee
What is the cupital of Franse?
Paris William Slukespeare
stinclloello
Paris William Shakespeare
elloctactioo
Who wrote Romeo and Juliet?
etiectoctio
etrectioctio
What 2 2?
2
omingomingooingg
The
2
omingomingoming
What lanuaago is spoken Bruzil!
Portuguase Jupiter
eluuelooctio
Portegaace Jupiter
elloctaadioo oningomingoming
What is the lirgest planet in our solar system?
omingomingoringg
Table 18: Masked perplexity after removing LayerNorm from specific positions in the model. The values represent the perplexity when critical neurons are masked after the removal of LayerNorm at different positions within the model layers.
Removed LayerNorm Position
3
6
7
Masked PPL.
6.662Ne+114
3,4644e+46
1.300Ne-4166
6,1332c+05
1.0135e-116
4.4763e-05
6.1820e=115
2,9182+405
Removed LayerNorm Poxition
11
12
14
15
Masked PPL
1.22636+416
X. 8,9008--45
3.60064416
8.16616445
17097e-05
3,4221+406
1.33464455
3.84%66406
A.11 DISCUSSION
To gain a deeper understanding of the mechanisms underlying our findings, future research should investigate the internal dynamics and structural properties that give rise to such vulnerabilities. Drawing parallels to established frameworks in robust metric learning (Bao et al. [0,2 2025 Hua
et al., 2024) and structural representation alignment (Fan et al.] [023] Zhu et al. 2022 Li et al.
2024a Zhou et al.) 2024). it is crucial to quantify how the integrity of specific feature subsets dic- tates the stability of the entire system. For instance, the catastrophic impairment from disrupting ultra-sparse critical neurons aligns with observations in mathematical reasoning tasks where L.LMs struggle to detect and correct errors in reasoning chains, leading to complete performance collapse (Singh et al. 2025). Similarly, the uneven distribution and clustering of critical neurons in outer lay- ers may relate to the transition layers identified in semi-open LLMs, where small errors in bottom layers propagate catastrophically. suggesting that outer-layer clustering could amplify vulnerabilities in recovery scenarios (Huang et al. 2025). Furthermore, the sharp phase transitions in performance degradation echo the self-organization processes captured through multiracial analysis of neuron interactions, where evolving network heterogeneity leads to abrupt shifts in emergent capabilities (Xiao et al., 2025). Additionally, the potential connection between our identified critical neurons and the attention sink mechanism (Xiao et al., 2024]] Yona et al., 2025 Jiang et al.) 2025 Barbero et al. 2025) warrants further investigation, both phenomena exhibit ultra-sparse neuron dependen- cies and catastrophic failure modes, suggesting they may represent complementary manifestations of broader architectural vulnerability principle operating different network depths. Building on these insights to clarify the root causes of our observed vulnerabilities, we propose several potential approaches addressing both architectural and operational security concerns.
Architectural Defenses: Model developers should implement redundancy mechanisms that dis- tribute critical computations across multiple neurons rather than concentrating them sparse sets. This could include architectural modifications such as parallel pathways for key functions (Grima al. 2023), increased connectivity between layers, and regularization techniques during training that penalize excessive concentration of critical dependencies (Bochkanov) 2020). Additionally, models could incorporate celf-monitoring mechanisms that detect unusual activation patterns indi- cating potential neuron-level attacks.
Deployment environments should implement anomaly detection systems that monitor neuron activation patterns and flag suspicious modifications. This includes developing base- line profiles of normal neuron behavior and establishing thresholds for detecting critical neuron dis- ruptions (Chen et al. 2021). Model serving infrastructure should also implement theckpoint-based recovery systems that can rapidly restore model functionality if critical neuron attacks are detected.
Runtime Protection:
Training-Tine Hardening: Future training procedures should explicitly optimize for robustness against critical neuron attacks. This could involve adversarial training techniques that simulate neuron masking during the training process, diversity-promotin loss functions that encourage dis-
27

---

Published as a conference paper at ICLR 2026
tributed computation, and regularization methods that prevent the emergence of ultra-spars: critical dependencies (Kurian et al. 2025).
Access Control and Monitoring: In production environments, strict access controls should govern any operations that could modify neuron activations. This includes implementing audit trails for all model modifications, restricting direct neuron-level access to authorized personnel only, and estab- lishing continuous monitoring of model behavior for signs critical neuron compromise (Cheng
al. 2018 Cheng 2021 Sperl al. 2021).
Developing protective mechanisms against critical neuron attacks represents an equally important research avenue. Potential defensive strategies include architectural modifications to distribute crit- ical functions more evenly across neurons, training procedures that explicitly reduce neuron-level vulnerabilities, and runtime safeguards that detect and reduce critical neuron disruptions. Under- standing whether techniques like dropout, pruning, or adversarial training can reduce critical neuron dependencies without compromising model performance represents promising direction for im- proving LLM robustness in safety-critical applications.
A.12
LLM USAGE STATEMENT
LLMs were used in limited capacity during the implementation phase of this research. Specifically, employed Claude (Anthropic 2024) for debugging code and identifying programming errors in our experimental scripts. These tools were used solely to assist with technical implementation issues and did not contribute research ideation, methodology development, experimental design, data analysis, or the formulation of key insights and conclusions. All core research contributions, including the proposed methodology, experimental results, and scientific interpretations, are entirely the work of the authors.
28

---

Published as a conference paper at ICLR 2026
Table 19: DeepSeek-R---istill-aamaa 70B: Impact of input text characteristics on critical neuron identification. Each row represents different input text with varying text types (Wikipedia, news, biography, media) and languages (English, French, German, Chinese, Spanish). The Neuron Num- ber column shows the number of critical neurons identified for each input condition.
Input sentence y
Type
Language
Neuron Number
The polar bear (Ursus maritimus) is large carnivorous mam- mal native the Aretic. It primarily hunts seals, relying on sea ice for habitat and hunting.
Wikipedia
English
3
The Great Wall of China is historic fortification built to protect Wikipedia against invasions. Stretching over 21,000 km, it's a UNESCO World Heritage Site.
English
3
A leading tech firm announced a new Al model today, promis- ing faster processing and enhanced accuracy. Experts predict major impacts on global industries.
News
English
3
Wall Street hit record highs today amid positive economic data. Investors cheered lower inflation figures. boosting tech and en- engy sectors worldwide.
News
English
3
Elon Musk. born 1971, is a visionary entrepreneur leading Tesla Biography and SpaceX. He drives innovation in electric vehicles and space exploration.
English
3
Albert Einstein, born 1879, was physicist who developed the Biography theory of relativity. His E=me² equation revolutionized modern science and earned him a Nobel Prize.
English
3
"Inception", directed by Christopher Nolan, sci-fi thriller Media about dream infiltration. Its innovative plot and visuals earned critical acclaim, grossing over 800 million.
English
3
"Stranger Things", a Netflix sci-fi horror series, follows kids Media battling supernatural forces in 1980s Indiana. Its nostalgic ap- peal and suspense won global praisc.
English
3
The Great Wall of China is a historic fortification Wikipedia built to protect against invasions. Stretching over 21,000 km. it's a UNESCO World Heritage Site.
French text:
French
3
French text: Wall Street hit record highs today amid positive News economic data. Investors cheered lower inflation figures, boost- ing tech and energy sectors worldwide.
French
3
German text: "Stranger Things", a Netflix sci-fi horror series. Media follows kids battling supernatural forces in 1980s Indiana. Its nostalgic appeal and suspense won global praise.
German
3
German text: Elon Musk, born 1971, is a visionary entrepreneur Biography leading Tesla and SpaceX. He drives innovation in electric ve- hicles and space exploration.
German
3
Chinese text:
The polar bear (Ursus maritimus) is a large car- Wikipedia nivorous mammal native to the Arctic. It primarily hunts seals, relying on sea ice for habitat and hunting.
Chinese
3
Chinese text: A leading tech firm announced today, promising faster processing and enhanced accuracy. Ex- perts predict major impacts on global industrics.
AI model News
Chinese
3
Spanish text: "Inception", directed by Christopher Nolan, a Media sci-fi thriller about dream infiltration. Its innovative plot and visuals camned critical acclaim, grossing over 800 million.
Spanish
3
Spanish text: Alhert Einstein, horn 1879, was a physicist who Biography developed the theory of relativity. His E=me² equation revolu- tionized modern science and earned him Nobel Prize.
Spanish
3
29


Reviews:
[{"review_id": "2357lr2S9S", "reviewer": "Reviewer_AABL", "summary": "The paper identifies a small subset of neurons in language models that, when disrupted, can significantly alter the performance and comprehension of the model. These findings are tested for several model sizes.", "questions": "Have you performed any experiments on the attention weights of the models? \nIn general, do we find these neurons in the early/middle/later layers of the models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper overall is written very well, the findings are clearly presented, and the ideas are well grounded in the literature. The paper flows well without any verbose/terse sections. \n2) The overall idea of the work, identifying critical neurons (model components), although not novel, is certainly an important direction of research. The methodology presented seems to hold well against the past baselines. \n3) The evaluation and ablation study presented is sound and consistent with the claims presented. \n4) The work is generalized well across diverse model families, sizes, and architectural choices.", "weaknesses": "1) The findings presented are consistent with the prior work highlighted in the paper, and hence make them unsurprising, calling into question the novelty of the work. While I do think this work has merit, primarily due to the generalization and breadth of experiments vis-à-vis prior literature, the presence of similar literature that draws similar conclusions undermines the contributions presented. \n\n2) The analogy to the legion experiments in neuroscience is unnecessary, as prior literature in language model research [1,2] uses similar methods as reference. Identifying critical regions in MLPs has been a research direction studied for a while, so I would highly recommend amending the analogy. \n\n3) The issue with the 2nd finding, the location of the clusters of critical neurons being primarily in MLP_down_proj, is that it asks more questions than are answered in the paper. Firstly, would a slightly larger set of neurons in the early layers of MLPs cause system-wide failures? As the late layer neuron compression high-dimensional attributes. Would this hold true for the middle layers? The importance of the late-layer neurons, although central to the claims made in the paper, needs a deeper analysis of the early/middle layers. If the number of neurons needed for failure in the early/middle layers is significantly higher than in the later layers, it could provide some more depth and robustness to the claims made in the paper. \n\n4) I would also like to see an experiment of randomly perturbing a similar cluster size of random neurons as a baseline to understand the expected degradation in model capabilities. \n\n5) Although I do think the work shows promise and is in a publishable state, addressing these weaknesses would add to the claims presented.  \n\n\n\n[1] Zhang, Fred, and Neel Nanda. \"Towards best practices of activation patching in language models: Metrics and methods.\" arXiv preprint arXiv:2309.16042 (2023).\n[2 Hanna, Michael, Ollie Liu, and Alexandre Variengien. \"How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model.\" Advances in Neural Information Processing Systems 36 (2023): 76033-76060.]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wum0x2dGVS", "reviewer": "Reviewer_Jfe4", "summary": "Inspired by findings from neuroscience, the paper investigates whether Large Language Models (LLMs) contain ultra-sparse critical neurons whose disruption can cause catastrophic performance collapse. The authors propose a Perturbation-based Causal Identification method with two stages: noise-based sensitivity analysis to rank neuron importance, and greedy masking to identify minimal critical sets. The authors conduct comprehensive experiments across 21 models and find three main phenomenons: masking as few as 3 neurons can increase perplexity by 20 orders of magnitude in 72B models; critical neurons concentrate in outer layers, particularly MLP down_proj components; performance degradation exhibits sharp phase transitions rather than gradual decline.", "questions": "1. How close is your greedy solution to the true minimal critical set? Have you validated this on smaller models where exhaustive search is feasible?\n2. Do different input types activate different critical neurons? Your robustness analysis shows consistent identification, but does this mean the same neurons are always critical, or just that 3 neurons are always sufficient?\n3. In appendix A.5, Tables 4, 5 and 6 exhibits failure modes of different models which raises a question that do critical neurons implement the same computational mechanisms across architectures, or are the mechanisms model-specific?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The two-stage methodology is well-motivated by neuroscience lesion studies.\n2. Extensive experimental validation across 21 diverse models demonstrates robustness. Meanwhile thorough ablation studies and detailed hyperparameter analysis ensure reproducibility.\n3. The greedy search algorithm reduces the computational complexity from $\\mathcal{O}(2^{|N|})$ to $\\mathcal{O}(|N|)$, making the search process significantly more efficient.", "weaknesses": "1. The greedy search assumes critical neurons can be identified independently, but the phase transition behavior suggests strong interdependence. The paper doesn't guarantee whether the greedy solution approximates the global optimum.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UrhY7M9z2Q", "reviewer": "Reviewer_Hd7S", "summary": "The paper claims that ultra-sparse sets of “critical neurons” exist in LLMs such that masking only 3–9 neurons can catastrophically cripple models up to 72B parameters, spiking perplexity by many orders of magnitude and driving zero scores on diverse downstream tasks. These neurons are found via a two-stage procedure: (i) rank neurons by Monte-Carlo sensitivity to Gaussian input perturbations, then (ii) greedily mask neurons in rank order until a log-perplexity ratio threshold is crossed. The paper further reports that critical neurons cluster in outer layers, especially MLP down-proj, and that degradation follows a sharp phase transition.", "questions": "1. Could you do a numeric robustness check by rerunning the main part of your experiments with fp32 and implement NaN/Inf filtering for perplexity calculation?\n\n2. Can you explain how your critical neurons story relates and differs with the super-weights story essentially?\n\n3. As I discussed above, LayerNorm can be a confounder. You may consider add controls to do a per-layer normalization vs. no-norm ablation test to isolate normalization as the failure source.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A simple, reproducible perturbation to ranking to masking pipeline with explicit formulas and hyperparameters.\n\n2. Broad model sweep (0.5B–72B) and a wide set of benchmarks; clear figures/tables illustrating phase transitions and architectural concentration.\n\n3. Comparisons to random/activation/gradient ranking baselines show the proposed method finds far more damaging neuron sets.", "weaknesses": "1. The reported magnitudes are extraordinary and risk reflecting measurement artifacts rather than semantic criticality. My intuition is that if the model truly loses ability (functional), the outputs degrade smoothly and consistently across precisions. If it’s mostly numerical, the collapse often looks phase-like and precision-dependent. Particularly, given mixed-precision (fp16) inference, zeroing channels could induce norm/scale pathologies (e.g., RMSNorm statistics, residual balancing) that blow up loss numerically. Given this, pleasure (1) replicate in fp32, (b) re-compute perplexity with clamp/NaN filtering diagnostics.\n\n2. Regarding task evaluation, I have the same concern. Repetitive punctuation is a classic sign of softmax/scale pathologies (one token gets an extreme logit and repeats, for example \"\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"), which is exactly what you’d expect if masking perturbs RMSNorm/residual calibration in mixed precision—--not necessarily evidence that a few semantically critical neurons were removed.\n\n3. The architectural concentration in outer-layer MLP down-proj echoes prior reports on outliers/super-weights/massive activations. The paper cites this, but does not tease apart whether your “critical neurons” are simply those scale outliers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M6aLtkzoQj", "reviewer": "Reviewer_SYxg", "summary": "This paper finds that LLMs contain a sparse set of critical neurons whose disruption can cause catastrophic performance failure. The authors propose a perturbation-based method to identify these neurons and evaluate it across multiple models. They report three main findings: (1) disrupting even three neurons can collapse models with billions of parameters, (2) critical neurons concentrate in outer layers and MLP down_proj components, and (3) performance degradation exhibits sharp phase transitions rather than gradual decline.", "questions": "Connection to Related Phenomena: Attention Sinks\nYour findings bear notable similarity to prior work on attention sinks, which identified highly important neurons critical for model fluency. Specifically, Xiao et al. (2023) documented the attention sink phenomenon (https://arxiv.org/abs/2309.17453), which was subsequently traced to highly activated neurons in specific architectural locations (https://arxiv.org/pdf/2503.08908). This mechanistic understanding has enabled cleaner interpretability analyses (https://arxiv.org/abs/2506.08010) and deeper theoretical investigation of the underlying dynamics (https://arxiv.org/abs/2504.02732).\nQuestions for the authors:\n\nIs there a connection between the critical neurons you identify and the attention sink mechanism? Given that both phenomena involve sparse, highly influential neurons whose disruption causes catastrophic failure, investigating potential overlap could be illuminating.\nIf a relationship exists, the extensive theoretical framework developed for attention sinks (particularly https://arxiv.org/abs/2504.02732) might provide a foundation for explaining why your critical neurons emerge and how they function. This could address the theoretical depth concern raised above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 1, "strengths": "1. Comprehensive experimental scope: The evaluation spans 21 models across multiple architectures (Llama, Gemma, DeepSeek, Phi, Qwen) and diverse benchmarks, providing strong empirical evidence for the phenomena.\n\n2. Clear methodology: The two-stage approach (sensitivity analysis + causal verification) is clearly presented with algorithmic details.\n\n3. Robust validation: The paper includes thorough ablation studies examining parameter sensitivity (α, K), input length requirements (T>10), cross-lingual consistency, and fine-tuning robustness.", "weaknesses": "1. Limited Novelty of Core Findings\nThe paper acknowledges that concentration in MLP down_proj layers \"is consistent with the observations reported in Yu et al. (2025)\". The existence of \"super weights\" in these components is already documented. What fundamentally new insight does this work provide beyond confirming known vulnerabilities at the neuron level rather than weight level?\n\n**Recommendation:** I would encourage to distinguish more clearly how this work advances beyond Yu et al. (2025) on \"The Super Weight in Large Language Models\"\n\n2. Insufficient Theoretical Depth\nWhile the empirical observations are extensively documented across many models, the paper lacks sufficient explanation of the underlying mechanisms. The phenomenon is thoroughly characterized in terms of what happens (phase transitions, concentration patterns, catastrophic failure), but the *why* and *how* remain underexplored.\n\n**Recommendation:** To strengthen the contribution, I recommend providing deeper theoretical analysis grounded in optimization dynamics, information theory, or circuit analysis to explain why these vulnerabilities emerge and what they fundamentally reveal about transformer computation.\n\n3. Unclear Practical Implications\nThe paper frames its findings as identifying vulnerabilities with implications for \"deployment security in safety-critical applications\" (line 22), yet the practical actionability of these findings remains unclear. Two critical questions are unaddressed: (1) How can practitioners suppress or mitigate the emergence of these critical neurons during training or deployment? (2) What is the realistic threat model—who would have the capability to selectively mask individual neurons in production LLMs, and under what circumstances? While Section A.9 discusses potential defenses, it reads as speculative rather than validated. The gap between identifying the vulnerability and demonstrating exploitability weakens the security framing. \n\n**Recommendation:** To strengthen practical impact, clarify the threat model with concrete attack scenarios, and either demonstrate mitigation strategies empirically or reframe the contribution as primarily advancing scientific understanding rather than addressing immediate security concerns.\n\n4. Overextended Neuroscience Analogy\nThe neuroscience motivation (Section 1, lines 6-30) establishes useful context by drawing parallels to sparse criticality in biological neural networks. However, the analogy is invoked repeatedly throughout the paper without deepening the connection beyond surface-level metaphor. This creates an expectation for substantive interdisciplinary dialogue—for example, demonstrating whether LLM critical neurons exhibit functional properties analogous to biological critical neurons, or whether insights from neuroscience can inform mitigation strategies. As currently presented, the neuroscience framing risks appearing cosmetic rather than integral to the contribution. \n\n**Recommendation:** Either (1) reduce the emphasis on neuroscience to a brief motivational context in the introduction, or (2) strengthen the interdisciplinary connection by establishing concrete links—such as comparing your findings to specific neuroscience phenomena (e.g., do critical neurons in LLMs exhibit analogous plasticity, recovery patterns, or functional specialization documented in biological systems?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)