Negative Pre-activations Differentiate Syntax
Authors: Linghao Kong
Angelina Ning
Micah Adler
Nir N Shavit
Venue: ICLR 2026 Poster (2026)
Keywords: entanglement
Wasserstein distance
negative pre-activation
syntax
interpretability
Primary area: interpretability and explainable AI
TL;DR: We show for the first time a mechanistic function of negative pre-activations within LLMs with smooth activation functions: entangled neurons use them for grammatical differentiation

Abstract: Modern large language models increasingly use smooth activation functions such as GELU or SiLU, allowing negative pre-activations to carry both signal and gradient. Nevertheless, many neuron-level interpretability analyses have historically focused on large positive activations, often implicitly treating the negative region as less informative, a carryover from the ReLU-era. We challenge this assumption and ask whether and how negative pre-activations are leveraged by models. We address this question by studying a sparse subpopulation of Wasserstein neurons whose output distributions deviate strongly from a Gaussian baseline and that functionally differentiate similar inputs. We show that this negative region plays an active role rather than reflecting a mere gradient optimization side effect. A minimal, sign-specific intervention that zeroes only the negative pre-activations of a small set of Wasserstein neurons substantially increases perplexity and sharply degrades grammatical performance on BLiMP and TSE, whereas both random and perplexity-matched ablations of many more non-Wasserstein neurons in their negative pre-activations leave grammatical performance largely intact. Conversely, on a suite of non-grammatical benchmarks, the perplexity-matched control ablation is more damaging than the Wasserstein neuron ablation, yielding a double dissociation between syntax and other capabilities. Part-of-speech analysis localizes the excess surprisal to syntactic scaffolding tokens, layer-specific interventions show that small local degradations accumulate across depth, and training-dynamics analysis reveals that the same sign-specific ablation becomes more harmful as Wasserstein neurons emerge and stabilize. Together, these results identify negative pre-activations in a sparse subpopulation of Wasserstein neurons as an actively used substrate for syntax in smooth-activation language models.

Published as a conference paper at ICLR 2026
NEGATIVE PRE-ACTIVATIONS DIFFERENTIATE SYNTAX
Linghao Kong', Angelina Ning', MIT "Red Hat AI
Micah Adler', Nir Shavit!
{linghao, angn.731, micah432, shanir]@mit..e edd
ABSTRACT
Modern large language models increasingly use smooth activation functions such as GELU or SiLU, allowing negative pre-activations to carry both signal and gra- dient. Nevertheless, many neuron-level interpretability analyses have historically focused on large positive activations, often implicitly treating the negative region as less informative, a carryover from the ReLU-era. We challenge this assump- tion and ask whether and how negative pre-activations are leveraged by models. We address this question by studying a sparse subpopulation of Wasserstein neu- rons whose output distributions deviate strongly from Gaussian baseline and that functionally differentiate similar inputs. We show that this negative region plays active role rather than reflecting a mere gradient optimization side effect. A minimal, sign-specific intervention that zeroes only the negative pre-activations of small set of Wasserstein neurons substantially increases perplexity and sharply degrades grammatical performance on BLiMP and TSE, whereas both random and perplexity-matchie matched ablations of many more non- Wasserstein neurons their neg- ative pre-activations leave grammatical performance largely intact. Conversely, on a suite of non- non-grammatical benchmarks, the perplexity-matee mattheed control ablation is more damaging than the Wasserstein neuron ablation, yielding a double dissoci- ation between syntax and other capabilities. Part-of-speech analysis localizes the excess surprisal to syntactic scaffolding tokens, layer-specific interventions show that small local degradations accumulate across depth, and wraiing-dynaame analysis reveals that the same sign-specific ablation becomes more harmful Wasserstein neurons emerge and stabilize. Together, these results identify nega- tive pre-activations in a sparse subpopulation of Wasserstein neurons as actively used substrate for syntax in smooth-activation language models.
INTRODUCTION
Prior works have successfully investigated the roles of specific neurons within large language models (LLMs), identifying units tied concepts, run-stable behavior, and confidence regulation (Gurnee ct al. 2023 2024; Stolfo et al., (2024). With respect to grammar, researchers have isolated neurons that are language selective, causally implicated in agreement, and selective for specific syntactic phenomena (AlKhamissi et al. 2024 Mueller et al., 2022 Duan et al.) (025). A common working heuristic in such analyses is to define what a neuron "represents" as inputs that produce high pos- itive pre-activations, an assumption originating in rectified architectures, where negative values are effectively inactive (Nair & Hinton 2010). Although recent methods such as sparse autoencoders can, in principle, be sensitive to negative activations (Ting al. 2025 Cunningham et al.) (023), the structure of the negative pre- activatii region in amnoth-activatiin language models remains comparatively underexplored.
This gap is particularly striking given that modern transformers predominantly employ smooth ac- tivation functions such GELU (Hendryeks & Gimpell 2016) and SiLU (Elfwing et al.) 2018). Introduced primarily for optimization benefits, these functions provide smooth gradients near zero, mitigate "dying ReLU" issues (Lu al. 2019), and empirically improve performance (Shazeer)
A preliminary version this work appeared in the 3rd Workshop on High-dimensional learning Dynamics at ICML 2025 (Kong et al. 2025).
htw.t//..o......aaaaaaaaeeeeeeee Code available httpttpp////

---

Published as a conference paper at ICLR 2026
Crucially, for inputs less than zero, such functions produce both nonzero output and gra- dient. Thus, in principle, the negative pre-activation region is available for computation, yet it is typically treated as inert. We challenge this assumption and test whether the negative pre-activations are functionally utilized, and if so, for what purpose.
2020).
To address this question, we focus on Wasserstein neurons: recently identified subpopulation of neurons whose pre-activation distributions exhibit large Wasserstein distance (WD) from a Gaussian baseline (Sawmya al., (002). Prior work has shown that such neurons, though they comprise only a small fraction of the network, are disproportionately sensitive to sparsification and targeted removal. Functionally, they uniquely map locally similar input vectors to widely separated output scalars via their dot product, a property quantified as mapping difficulty (MD). Following prior usage, such neurons are termed entangled, extending concepts from superposition in which multiple features are shared by the same neuron (Elhage et al. 2022 Adler et al. (225) to this complementary case in which closely related inputs are separated by a single neuron. WD and MD correlate strongly, motivating the use of WD as practical entanglement proxy (Section A.I)
We find that Wasserstein neurons in the linear projections immediately preceding the nonlinearity of the multilayer perceptron (MLP) block (the gate projection in GLU-style models (Shazeer) 2020 such as Llama (Grattafiori et al., 2024) and the up projection in GPT-2-style models (Radford et al,, 2019) such Pythia (Biderman et al. 2023) share interesting property: the deviation from Gaussianity concentrates in the negative region of the pre-activation space, and so we focus on this tractable subset of neurons candidates for analysis. This effect is markedly stronger in ReLU models: although ReLU-based models such as OPT (Zhang et al., (2022) also exhibit non- Gaussian pre-activation distributions, theirs show comparatively less concentration of such structure specifically in the negative region, consistent with ReLU's clamping (Figure I
Jal OPT Wasseestein neuron
(b) OPT neuron post-RelJ
[cl Pythie Wasserstets neuron
cdi Pethia neuron Gest
[el Pythoe neuros 10 pattern
*AD AND Bay oo Sk Peneeteretttiiiiio
Prevectraation valuo
Preeeteeeeccecccoooooo valus)
Presccicction
Normalized Tiebanoe
Figure 1: Wasserstein neurons in ReLU vs non neL- LLMs. (a. b) In OPT-1.3B, ReLU-based model, the dominant pre-activation mass resembles a somewhat Gaussian peak whose mode lies below zero, with an additional mildly multimodal positive tail. (c, d) In Pythia 1.4B, a GELUUbased model, the dominant mass instead centers near zero, and the negative pre-activation region exhibits more pronounced multimodality, reflecting preservation of negative inputs. (e) The input output (IO) relationship of the Pythia Wasserstein neuron, showing that for pairs of inputs that are fairly similar, their outputs are still mapped far apart by the neuron. More details provided in Section 1.1 Neurons acquired from the up projection in the second MLP block of their respective models.
Here, show that this seemingly "inactive" region is in fact crucial for model function: Wasser- stein neurons systematically exploit negative pre-activations to differentiate syntax. Across multi- ple LLM families, clamping only the negative pre-activations of the top few Wasserstein neurons in each MLP gate projection significantly impairs general model performance, yielding large per- plexity increases. Matching the same perplexity rise with non-entangled neurons requires far more units. Even under these perplexity-matcher conditions, only the Wasserstein-neuron intervention produces large drops grammatical accuracy on BLiMP (Warstadt et al.) 2020) and TSE (Marvin & Linzen 2018), whereas the non-entangled neuron ablation produces larger drops on a panel of non-grammatical benchmarks, yielding a clear double dissociation. Further analysis reveals that these neurons" input differentiation preferentially separates syntactic scaffolding tokens, such as de- terminers and prepositions, sending locally similar inputs to distinct negative pre-activation values. These effects are strongest in early layers and compound across depth. Across training checkpoints, the same fixed negative pre-activation ablation grows increasingly damaging as Wasserstein neurons acquire their characteristic non Con-Gassiian structure. Together, these results indicate that the negative pre-activation region serves as an active site of computation in this entangled subpopulation and is disproportionately implicated in syntax, rather than serving as mere optimization convenience.
2

---

Published as a conference paper at ICLR 2026
2 A MOTIVATING CHARACTERIZATION OF WASSERSTEIN NEURONS
2.1 IDENTIFYING WASSERSTEIN NEURONS IN LANGUAGE MODELS
First, we specify where in LLMs we analyze Wasserstein neurons and briefly detail their previously established conventions, such as how their WD and MD metrics are calculated. The MLP block of GPT-2-style GELU-based models, such as the Pythia suite (Biderman et al.) 2023). is as follows:
= W down (GELU( W W). The MLP of SiL.Ubbssed GLU-style models, such as Llama 3.1 8B
Grattafiori et al. 2024), Mistral 7B vO.3 (Jiang et al. (023), and Qwen3 8B Base (Yang et al. 2025) as follows: y W Wowwn (SiLU(W gate: gatee o (W W.pa)). Note that the naming convention for the linear projection preceding the nonlinearity is flipped. We follow the convention that, when treating and y as column vectors, neurons are defined
vectors in W. We examine Wasserstein neurons in Wup in Pythia 70M to 12B as well as in Wgate in Llama 3.1 8B, Mistral 7B v0.3, and Qwen3 8B Base. We use Pythia better investigate training dynamies through their publicly released training checkpoints, and we use the other three models to investigate the phenomenon in more modern language models.
To compute the WD and MD of a neuron with weights w, real input text totaling N tokens is fed into the model. We use the test set of WikiText 2 (Merity et al. 2016). The distribution of input as well as the distribution of the output scalars from its dot product computation, (yi 1 [w x; } are collected. An example of this output distribution is shown in Figure [e. { is normalized to have zero mean and unit variance, and the Wasserstein distance of this normalized distribution with a unit Gaussian is calculated as WD. Unless otherwise specified, in the following sections the WD of a neuron always refers to the Wasserstein distance of its normalized output distribution unit Gaussian.
vectors into a neuron, } 1.v
To compute MD, pairs of input vectors Ty are randomly selected and their 7,2 norm is calculated, as well as the L norm of their corresponding outputs yi, 3z-- Each input pair L norm is normalized by the maximum of the set, and each output pair L norm normalized by the median of the set. An example of these pairs for a Wasserstein neuron is shown in Figure Finally, each normalized output pair difference is divided by their corresponding normalized input pair difference, and the average of these ratios is calculated as the MD) of a neuron to summarize how far apart neurons map similar inputs. Because WD and MD are proxies of each other, we primarily use WD to select for entangled neurons, and use MD when specifically targeting pairs of inputs that are mapped far apart, which will be specified in the relevant sections. We further investigate the potential confounds of the influence of asymmetry and kurtosis on WD in Section A.I
2.2 EVALUATION PROTOCOLS
We evaluate syntactic behavior using two complementary benchmark suites, BLiMP (Warstadt et al. 2020) and TSE (Marvin & Linzen 2018). BLiMP (the Benchmark of Linguistic Minimal Pairs) con- sists of 67 sub-datasets with 1,000 minimally different pairs of grammatical and ungrammatical sentence. This benchmark covers 13 broad categories spanning syntax, morphology, and semantics, such subject-verb agreement, determiner-noun agreement, and ellipsis. TSE (Targeted Syntactic Evaluation) is fine-grained challenge set of roughly 350K minimally different sentence pairs focus- ing on three families of structure-sensstive dependencies: subject-verb number agreement, reflexive anaphora, and negative polarity item licensing. TSE trres-teste specific hierarchical dependencies and complements BLiMP's broader coverage. For both benchmarks. accuracy the fraction of pairs for which the model assigns higher total log-probability to the grammatical sentence.
To assess non- -rammaaamatical performance, we utilize the Language Model Evaluation Harness (Gao et al. (024) to test model performance on ARC Challenge, ARC Easy (Clark et al. 2018). Bool (Clark et al. 2019). HellaSwag (Zellers et al. 2019), PIQA (Bisk et al. 2020), SciQ (Welbl et al., 2017), TruthfulQA (Lin et al. 2022), and WinoGrande (Sakaguchi et al.) 2021). These cover broad set of abilities, such as science (ARC Challenge, ARC Easy, SciQ), commonsense reasoning (HellaSwag, PIQA, WinoGrande), reading comprehension (BoolQ), and truthfulness (TruthfulQA). Additionally, we track perplexity on the WikiText 2 validation set. For parts of speech (POS) and dependency tagging, we the spaCy (Honnibal et al. 2020) English core web small model.
3

---

Published as a conference paper at ICLR 2026
2.3 WASSERSTEIN NEURON EMERGENCE TRACKS GRAMMATICAL ACCURACY
To better characterize Wasserstein neurons and gain an intuition for their function, we analyze their development across model sizes and over the course of training. We use the Pythia suite of language models, specifically Pythia 70M to 12B. In the Wap up of the second MLP block of each model, we compute the WD of every neuron both at the final checkpoint and across training steps. We specifically follow the same Wasserstein neurons, those with the top 1% WD as measured at the final checkpoint, across training to track their emergence.
First, we generally find that larger models tend to contain Wasserstein neurons with higher max- imum and average WD (Figure (A). Moreover, these neurons emerge very rapidly over training: their WD increases sharply within the first 25K steps, or about 50B tokens, after which the highest WD neurons are already distinguishable from the rest (Figure 2 Examining their weights across checkpoints, observe complementary patter. These neurons change more than average early in training and then undergo a period of relative consolidation (Figure 2+). We quantify this us- ing cosine dissimilarity between successive 10K-step checkpoints, normalized by the layer mean. Together, these observations suggest that high- WD neurons specialize early and stabilize thereafter.
(5) Langer models have more Vioserstam nearons
(b) Wosserstois nourons ziss
(c) Wasserstein pourons tend to scart and stop werning frotter
(2) Wasserstein neurce smergence troe lo grammattral arrurney
earry in training
331
7M5
1.40
0.5
Hoblttoo Shamne of DEAE
173-
3.00
150
17a
0.35
TSE Sccursry (36)
255
0:10
015M
2,00
25
000 035 838 033 030 9.20 8:30 658
02 63
04
Wassemcetn distance
Thiiinn step (K:11))
Twwiting Ktep (x10f)
Visserslatn distanne
Figure 2: Wasserstein neuron emergence tracks grammatical accuracy. (a) Larger models tend to have Wasserstein neurons with greater maximum WD, and more neurons with slightly greater WD.
Dotted lines indicate mean WD, and text indicates maximum WD of all neurons in layer. (b-d) share the same legend.
(b) Wasserstein neurons arise rapidly during training, within roughly 50B
tokens. The WD of the same cohort of Wasserstein neurons is calculated at each checkpoint. (c)
Wasserstein neurons tend to start and stop learning faster than other neurons, as measured by the cosine dissimilarity, normalized the layer average. between successive 10K-step checkpoints. (d)
At various checkpoints in training, the WD of the Wasserstein neuron group is compared to the model's 's performance on TSE at that time, and they strongly correlate. All neurons from the up projection in each model. Shaded bands are one standard error of the mean.
Because syntactic abilities are also known arise early in training (Duan al. 2025 Müller- Eberstein et al. 2023), we ask whether the development of Wasserstein neurons tracks grammatical competence. Indeed, across checkpoints, the aggregate WD of fixed cohort of high-WD neu- rons correlates with TSE accuracy (Figure (i). However, this correlation alone does not determine whether these neurons are uniquely tied to syntax or simply reflect general representational capacity.
We therefore explicitly pose the question that motivates the next section: is the structure that Wasser- stein neurons develop, particularly in the negative pre-activation region, causally necessary for gram- matical behavior, or merely a byproduct of global improvement? In Section 3 we address this us- ing targeted, sign-specific ablations and compare their effect on grammatical and non-grammatical benchmarks with matched- patcheed-erplexitty ablations.
3 ABLATING NEGATIVE PRE-ACTIVATIONS IN WASSERSTEIN NEURONS UNIQUELY HARMS GRAMMAR
We causally perturb Wasserstein neurons by zeroing only their negative pre-activations immediately before the nonlinearity: of = max(a 0) for l E S. dz = a4 otherwise, where a are pre-activations in the MLP gate/up projection and S contains the top 5% WD neurons per layer, with p% being on the order of 1% (Figure[A7) The model, weights, and nonlinearities are otherwise unchanged apart from this seemingly minor alteration.
4

---

Published as a conference paper at ICLR 2026
We use two control conditions. The first perturbs an equal number of randomly selected neurons per layer using the same ablation. The second is a cuplexity-matehhen control that perturbs the negative pre-activations of low-WD neurons. Specifically, for each layer we ablate the bottom m% of neurons as ranked by WD, where m is a single global percentage applied uniformly across layers, increased until the resulting Wiki'Text 2 perplexity matches that of the top- top-WD ablation. This ensures that both interventions involve comparable global degradation while differing in which neurons are perturbed.
Ableting Wasserstenn neuron negattve
1bi Matching edfect seeds more mon-eetbangiee sesomine
Wasserstela pertubbtion
Edfect Waseerstelo perturbwtion on parts of spowch
peoverd tre-ertiretton
fereissse dirives perplecily up
onripuely disruphs gremmnr
Malrol 78 stu
Monn NLL aitorcacc
Dolatoe purplesity EATTIAAA
Quee3
725
Keakee
eettory mecurve
3.30
Absalu accos
25
Part speoch
Matral
Nearons perturbed (7))
Wassenstetn nourons pertubbed (16)
1SUM!
Figure 3: Sign-specific perturbation of Wasserstein neurons disproportionately harms grammar. (a, b) share the same model colors. (a) Perplexity increases when clamping only the negative pre- activations of the top-WD neurons; random controls are much smaller. Numbers indicate starting and ending absolute perplexity. (b) Matching the perplexity increase from perturbing the entangled fraction of neurons requires an order of magnitude more non-entangled units. (c) Perturbing Wasser- stein neurons uniquely impacts grammatical capabilities, even compared to the perplexity control. In each model, the top 1% Wasserstein neurons in each layer were perturbed for the bench- mark. The least entangled 40% of neurons in Llama, 50% in Mistral, and 20% in Qwen in each layer were used as the perplexity-matcher control. (d) At per token resolution, tokens associated with syntactical scaffolding incur a much higher surprisal for the 1% Wasserstein perturbation com- pared the perplexity matched control in Llama 3.1 8B. NLL differences were mean shifted by the global difference. Randomly sampled controls were acquired over ten trials. Error bars indicate one standard error of the mean. Raw scores and CI's are in Table A2
This intervention yields a striking result: although it affects only Re 1% of neurons, and alters only their negative pre-activations, it produces disproportionately large functional damage. Perplexity increases steeply with the fraction of neurons perturbed in Llama 3.1 8B, Mistral 7B v0.3, and Qwen3 8B Base, doubling Llama and Mistral with just a 2% perturbation and in Qwen with 5%, far exceeding random controls (Figure 3p... Matching the same perplexity increase with low-WD neurons requires clamping vastly more units: the effect of perturbing just 1% of Wasserstein neurons per layer is only matched by perturbing roughly 50% of the least-entangled neurons in Mistral, 35% in Llama, and 20% in Qwen per layer (Figure 3p). Crucially, across all three models, only the 1% Wasserstein intervention, not the random perplexity- perplexity-matched matched controls, yields large drops in grammatical accuracy on BLiMP and TSE, with Llama and Mistral degrading more than Qwen (Figure 3P) Token level analysis perplexitymmathed control) to syntactic scaffolding POS classes such as determiners, punctuation, auxiliaries, and particles, but not nouns, verbs, adjectives, adverbs (Figure I1), confirming that the negative pre-activation region of Wasserstein neurons is not inert but mechanistically necessary for syntax. Additional controls validating the importance of the sign of negative pre-activations, rather than just their magnitude, can be found in Section A.4 Repeating this analysis, but selecting neurons based on their MD instead of WD, yields the same qualitative result (Figure
Llama 3.1 8B localizes the added surprisal (compared to the
To test whether the observed effects are specific to syntax rather than broad capacity loss, we evaluate the same interventions on suite of non- grammaamaaiiaal benchmarks. We focus on Llama 3.1 8B and reuse the three ablation conditions as before. In each MLP gate projection, we ablate the negative pre-activations of either the top 1% WD neurons, 1% of randomly selected neurons, or the bottom 40% WD neurons. We then evaluate the interventions on eight multiple-choice benchmarks that probe general reasoning and comprehension rather than syntactic competence: ARC Challenge, ARC Easy, BoolQ, HellaSwag, PIQA, SciQ, TruthfulQA, and WinoGrande. These tasks assess scientific and commonsense reasoning, question answering, and general reading comprehension without the potential confound of open-ended text generation.
5

---

Published as a conference paper at ICLR 2026
Wasserstein neuron perturbation is not damaging non grammatical tasks
35
1h. Wisserssbiin
Aboolate errer increase (96)
PPL mutched
Random
21
20
16
3.5
15
31
10
5-
5
-0
Gommar Any Other Aog
ILUMP
1585
ARC Chailenge ABC Eray
BoolO
NellaSweng
PIDA
Se00
TruthfulDA WiroGrande
Figure 4: Non-grammatical abilities comparatively less harmed by Wasserstein neuron perturba- tion. All benchmarks were run in 0-shot. Randomly sampled controls were acquired over ten trials. Error bars indicate one standard error of the mean. Raw scores and CI's Tables A2
A3
The performance pattern is the opposite of what we observe for BLiMP and TSE. With the ex- ception of TruthfulQA, the perplexitymmatcher ablation on the bottom 40% WD neurons produces larger error increases than the top 1% WD neuron ablation on every benchmark. On average across the non- non-grammatical tasks, the low-WD WD ablation yields z 11% absolute increase in error relative to baseline, whereas the Wasserstein ablation yields =e 4% and the random ablation produces no ap- preciable change. TruthfulQA is the only outlier, with error increasing by 5% under the Wasserstein ablation compared to 2% for the perplexity-matede ablating many low-WD neurons primarily degrades non-syntactic capabilities, while ablating tiny set of high-WD neurons has comparatively milder effect on these tasks. We replicate this non- grammatical benchmark analysis on Mistral 7B v0.3 and Qwen3 8B Base and observe consistent grammar associated degradation, with some model dependent variation (Section A.3)
control. Thus, when perplexity matched,
Taken together. these benchmarks establish double dissociation. Clamping the negative pre- activations of just 1% of Wasserstein neurons causes large drops on BLiMP and TSE but compara- tively modest degradation on non-grammatical benchmarks. In contrast, clamping the negative pre- activations of many low-WD) neurons leaves grammatical performance largely intact while substan- tially degrading general capabilities. These findings support the view that negative pre-activations in sparse set of Wasserstein neurons play critical role in syntactic processing, while more diffuse capacity for other tasks is distributed across the bulk low-WD neurons, pointing to a structured organization of negative pre-activation behavior. To localize the mechanism, move from model- level interventions to layer-level ablations in Llama 3.1 8B.
LAYERWISE ABLATIONS REVEAL EARLY LAYER ORIGINS AND CUMULATIVE ERROR BUILDUP IN SYNTACTIC STRUCTURE
To understand which layers are most crucial for particular grammatical phenomena, and to under- stand how damage compounds with depth, we split Llama 3.1 8B into eight groups of four successive layers. We perform group-wise negative pre-activation ablations to 1% of Wasserstein neurons in each of the layers in the group. We then benchmark Llama with only single group perturbed, or with all layers up to and including group perturbed. Broadly speaking, across both BLiMP and TSE, early layers are much more sensitive ablation and therefore critical for grammatical function, in line with previous works (Tenney et al. 2019 Hewitt & Manning 2019).
Specifically for BLiMP, Wasserstein neuron perturbation sharply increases error on ellipsis and sub- ject-verb agreement (Figure 5p)). two constructions requiring non-local dependencies (Merchant 2013 Franck et al., 2006). Binding and determiner-noum agreement are also affected, albeit to lesser degree. Later layer groups show much weaker effects, suggesting that Wasserstein neurons in the early network establish syntactic scaffolding upon which subsequent layers depend. When ablations are applied cumulatively, error grows monotonically, especially for ellipsis, subject-verb agreement, determiner- noun agreement, and filler-gap dependencies, demonstrating that local dis- ruptions compound across depth (Fig. 5p).

---

Published as a conference paper at ICLR 2026
(a) BLMP occur specific to layer geoups.
(3) BLIMP error buildup accoss laper qroups
(c) TSE evvor specific to layer groups
Anapiss Avanage
Aboattee WITH Increvon (i)
AMG
Arpintont Binding
NP
Determiner soun apreesuesk
SWA
sia tolo this tie z-a mol
Filler cap dependercy
Irreplar somme
AVG
TSE error buildup across loyer groups
NPI Recoxag
Seatieclate
HEP
SWA
ci sile iile 2016 X010
110
131 158 110
L. 130 1/ Layer groups
1.0
Layer groups
Layer groups
Figure 5: Individual and cumulative layerwise ablations. (a, b) share the same y-axis labels. (a) Early layer ablations yield the greatest increases in eπrоr, specifically within ellipsis and subject- verb agreement. (b) Error increases monotonically with cumulative ablation, with the strongest effects for ellipsis and subject-verb agreement. (c) TSE performance is also the most sensitive to early layer perturbation, especially for negative polarity item licensing. (d) Error for TSE grows monotonically as well. All benchmarks collected for 1% Wasserstein perturbation per layer.
TSE highlights this vulnerability even more strongly. Local ablations already produce dramatic degradation, with negative polarity item licensing suffering a striking 20% increase in error from just the perturbation of the first four layers (Fig. Sp). Furthermore, cumulative ablations raise error across all syntactic classes within TSE (Fig. 5t).
These layerwise effects mirror the earlier POS-level findings: disrupting negative pre-activations in Wasserstein neurons disproportionately harms the functional scaffolding of syntax (auxiliaries and determiners), and small early-layer hits compound across depth into broad grammatical failure. This analysis suggests an early, sign-specific mechanism that feeds many grammatical constraints. To make this concrete, we examine single Wasserstein neuron in Pythia 1.4B and examine both how it separates nearby inputs and what inputs it is separating.
5 NEGATIVE DIFFERENTIATION OF SYNTACTIC TOKENS
Returning to the notion of Wasserstein neurons input differentiators, examine particular Wasserstein neuron in Pythia 1.4B, neuron 5176, the same neuron from Figure As we feed real WikiText 2 validation data into the neuron, sample 2,000 random tokens and form 1,000 pairs We then calculate the ratio of the normalized output distance to the normalized input distance, as described previously (Section 2. and choose the pairs with the greatest ratio. We show the top ten pairs for visualization purposes in the neuron's 's input output relationship (Figure 6c). The most separated pairs are overwhelmingly grammatical, such the preposition "for" being mapped far from the determiner "the" (Figure (a)). We show additional neurons in Section A.5) finding many with interpretable pairs, such as those processing coordinating conjunctions or punctuation.
[h) Wamserstetn neurens segativn
6n Phims tokens that are separatted the forthest by the Wisserrstrin seuron
pre-activerions to sepurrenn wimller inputts
[c] Wissemettrin nemmon ID relationship
Token
Token
POS
POS
Dep 1
Dep
for
for the
The
ADP
ADP
DET
DET
prep
prep
det
det
KarS
Normalized Octppec Distaace
ADP
ADP
prep
preg
In
the
ADP
DET
prep
det
Prequemcy
Park
of
t0
ADP
ADP
prep
prep
The
to
DET
ADP
det
Put
ADP
prep
det
Put
from
the
ADP
DET
prep
det
to
DET
PART
del
aux
62
in
tbe
ADP
DET
det
00
0.8 Nornalied Tuput Distance
7ie activattiinnnno value
Figure 6: Representative example of Wasserstein neuron input differentiation. (a) The top ten pairs of tokens that are differentiated by Wasserstein neuron 5176, with the POS and dependency labeled. They are predominantly syntactically functional. (b) The output distribution of this neuron over WikiText 2, with the specific pair output values highlighted. Fight of the top ten differentiated pairs driven to two very distinct negative values, rather than the perhaps more expected positive and negative value pair. (c) The top ten differrnntated pairs are all fairly similar as input vectors, but are mapped very far apart by this neuron.

---

Published as a conference paper at ICLR 2026
Counterintuitively. these most differentiated pairs not mapped far apart because output value is positive and the other is negative. Rather, both elements of each pair are driven negative, but to different depths (Figure/Op). accounting for eight out of the ten pairs. This "negative differentiation" utilizes the negative tail of the distribution heavily. Even following the GELU nonlinearity, an appreciable difference remains in their values the very negative values are driven close to zero but the less negative values remain. This sign-specific separation concentrates on contexts of functional words and consistent with the early-layer syntactic scaffolding implicated by our ablations.
Taken together, this single-unit case study shows that Wasserstein neurons can enforce large sepa- rations among nearby inputs by pushing both items into the negative region to different degrees- sign-specific mechanism that targets functional contexts. Having established this intuition at the neuron level, we now scale up: we quantify how common this negative differentiation is across neurons and layers, and how it evolves during training and across model families.
6 NEGATIVE DIFFERENTIATION CONCENTRATES EARLY
Having seen negative differentiation in a single unit, we now ask how the sign pattern of differen- tiated pairs -eggnnie--egaaliieeeeeee (NN) for two negative pre-activation values, positive-negative (PN) for one of each, and positive-positive (PP) for two positive values-varies across layers and over training. In each layer of Pythia 1.4B, the top 5% entangled neurons are selected by MD, and their top 100 most differentiated pairs out of 1000 are analyzed as before. We label each pair according to the signs of the two output actvvtvvasi
Oul Wiserocain ain neumnn risiclly NM
(c) worer vs. Wissarrdein during traisting
(4) erree os. Wespendaain during training
Layerwise proportion pair types
25, Womorooiis
Proprtan (80)
Sis Runien
0.064 It' 0.139
12-
a.8 6.12 0.1
314
E.11
3.5 66e 030 ata 0.6 6.16
Layyer
Training step (x10%)
Wassemstetn distance
Wassemsten distence
Figure 7: Negative differentiation emerges early and persists across depth in Pythia L.4B. (a, b) share the same legend and y-axis label. (a) Layerwise composition of the most separated input pairs for the top 5% MD neurons, tracking the top 100 tokens out of 1000 per neuron. Proportion of NN, PN, PP pairs across layers reveals widespread usage of negative differentiation across layers, with the most practitioners belonging to layer 2. (b) In layer 2, the prevalence of NN output pairs in the most differentiated pairs rapidly arises with training in entangled neurons, indicating early specialization into negative differentiation. (c. d) share the same legend and y-axis label. (c) At each checkpoint, the same 5% of Wasserstein neurons are perturbed, and the resulting BLiMP error correlates very strongly to the WD of the cohort that checkpoint. (d) Analysis from (c) repeated for TSE. For (c, d), neither the random nor the perplexity-matcheee correlate with error. Data acquired from the top 5% of Wasserstein neurons in Pythia 1 1.4B. Shaded bands are one standard error of the mean.
controls (plotted at the same WD for reference)
We find that NN pairs are highly prevalent in early layers, especially layer but that all layers have significant proportion of these pairs (at least 30% with the exception of layer 1) while PP pairs are comparatively rare (hi This indicates that Wasserstein neurons in Pythia frequently separate similar inputs to negative values of varying degree, rather than always two values of different signs. To better investigate this in layer 2, we tracked the same most entangled neurons over training checkpoints, and measured the properties of the pairs they most differentiated as training progressed. We strikingly find that there is rapid specialization: while NN and PP pairs both occupy a very small fraction of the most differentiated pair type at the onset of training, NN pairs rapidly become prevalent, unlike PP pairs. PN pairs correspondingly decrease in frequency as NN pairs rise (Figure Zp). Thus, negative differentiation is early emerging and sustained strategy.
To relate this behaviorally to grammar causally rather than by correlation (Figure 21), at each checkpoint we clamp only the negative pre-activations of 5% of Wasserstein neurons in Pythia
8

---

Published as a conference paper at ICLR 2026
1.4B and measure the resulting accuracy drop BLiMP and TSE, compared the random and perplexity-matche controls. Plotting the error increase against the mean WD of the cohort at that checkpoint shows strong positive relationship: as Wasserstein neurons' distributions become more non-Gaussian, ablation yields larger grammatical damage (Figure 77. d). Random and perplexity- matched controls, plotted at the same WD for reference, show little to no correlation. Together, these results indicate that as Wasserstein neurons mature, the model increasingly relies upon them for grammar, and so error increases as these neurons' pre-activations diverge from a Gaussian. Addi- tional experiments observing this phenomenon in Llama 3.1 8B can be found in Section AA 6. Taken together, our results show that negative pre-activations are mechanistically salient rather than inert: they are leveraged by sparse set of entangled neurons to differentiate syntax.
7 RELATED WORK
7.1 ACTIVATION FUNCTIONS
Modern transformers largely replace ReLU with smooth activations such as GELU and SiLU to case optimization, avoid dead neurons, and empirically improve performance (Hendrycks & Gimpel 2016 Elfwing et al. 2018 Shazeer 2020 Lu et al. 2019).. with contemporancous work examining sign-conditional effects in gated MLPs (Gerstner & Schuetze, 2025). Our contribution is show that models actively leverage the negative pre-activation region beyond simply for training: selec- tively clamping only negative pre-activations of high-WD neurons impairs perplexity and grammar far beyond random or perplexity-matched controls, revealing a sign-specific computation that prior coamiizaaiioncentrri discussions did not examine.
7.2
GRAMMAR ACQUISITION
A long line of work links internal representations to grammatical phenomena using probes, causal tracing, and targeted evaluations (Marvin & Linzen 2018; Warstadt et al. 2020; Saphra & Lopez
2018 Tenney et al. 2019 Mueller et al. [022] Müller Emer--eeeennnnnn et al. 2023 Duan al. 2025 Chen
ct al. 2023). Probing studies have shown that syntactic structure is often encoded in middle trans-
former layers, especially in attention mechanisms (Hewitt & Manning 2019). Recent approaches with sparse autoencoders recover interpretable, grammar- grammar-releyant features from hidden states (Cun-) ningham al. [023 Jing et al. 2025 Brinkmann et al. 2025). Together, these studies suggest that grammatical competence emerges early in training and accumulates across depth, and have largely framed syntax as residing attention patterns or dense hidden subspaces. Our results reveal com- plementary avenue through which grammar is computed, especially compared to ReLU models that must necessarily learn grammar through different mechanism (Sinha et al. 2023). We identify a sign-specific mechanism in the MLP, where early layers use negative differentiation in a small group of entangled neurons, and show that ablating negative pre-activations in these neurons causally de- grade grammatical performance, linking emergence, layerwise structure, and behavior at the level of individual neurons.
7.3 INTERPRETABILITY
Interpretability in modern machine learning remains difficult in part because many neurons are pol-
ysemantic, responding to different features (Arora et al., 2018: Mu & Andreas) 2020 Olah et al.,
2020
Goh al. 2021 Jermyn et al. 2022 Gurnee et al. 2023 Templeton] 2024; Gurnee et al., A central driver of this is superposition: features are compressed onto shared directions,
2024
allowing networks to represent more features than neurons, creating entanglement (Elhage et al.) Lecomte al. 2023 Adler et al.) 2025). Prior work identified orthogonal form of en- langlement in which certain neurons separate highly similar inputs (Sawmya et al., 2025), but only showed that these neurons are sensitive weight sparsification, without interpreting their role or linking them to syntax. Here, we demonstrate that these Wasserstein neurons specifically leverage the negative pre-activation region implement syntactic differentiation, aspect underexplored in analyses that implicitly treat positive pre-activations as the sole carrier of signal.
2022

---

Published as a conference paper at ICLR 2026
8 CONCLUSION AND DISCUSSION
We have shown that the negative pre-activation region of smooth nonlinearities is actively used to support syntax in large language models. This usage is most readily observable in Wasserstein neu- rons. Such neurons emerge and stabilize early in training, tracking the development of grammatical competence. Causal, sign-specific ablations that zero only their negative pre-activations sharply disrupt BLiMP and TSE performance with comparatively less impact on non graa-mammaaaaal tasks, whereas applexxty-matcher ablations of many more low-WD neurons largely spare grammar but cause greater degradation the complexity of the negative pre-activation space. Together, these results indicate that negative pre-activations are not inert byproduct of GELU SiLU, but a functional substrate that certain neurons leverage to implement grammar. This reframes common ReLU-era intuitions that equate "acttvity"" with positive pre-activations and underscores that interpretability methods must attend to the full activation landscape, including negative regions where crucial computation can reside.
other benchmarks, yielding a double dissociation that demonstrates
LCKNOWLEDGMENTS
The authors would like to extend their gratitude to many members of the interpretability community for fruitful discussions and insightful comments. In particular, thank Ben Wu, Georg Lange, Can Rager, Josh Engels, and Alessandro Stolfo for their thoughtful input. We are also grateful to Jacob Andreas, Dan Alistarh, Dan Gutfreund, Alexandre Marques, Tony Wang, and Thomas Athey for providing valuable feedback and constructive guidance as the project developed.
This project was supported by an MIT-IBM Watson AI Lab grant and an NIH Brains CONNECTS U0I grant. Computational resources were provided by Red Hat AI and AMD's AI & HPC Fund.
Finally, we dedicate this work to the memory of Shashata Sawmya, whose friendship, insight, and curiosity are deeply missed.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Micah Adler, Dan Alistarh, and Nir Shavit. Towards combinatorial interpretability of neural com- putation. arXiv preprint arXiv:2504. 08842, 2025.
Badr AlKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. The Ilm language net- work: A neuroscientific approach for identifying causally task-relevant units. arXiv preprint arXiv:2411.0P2 1. 1.02280, 2024.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraie struc- ture of word senses, with applications to polysemy. Transactions of the Association for Compu- tational Linguistics, 6:483-499 2018.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Jannik Brinkmann, Chris Wendler, Christian Bartelt, and Aaron Mueller. Large language mod- els share representations of latent grammatical concepts across typologically diverse languages. arXiv preprint arXiv:2501.06344 2025.
Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew Leavitt, and Naomi Saphra. Sud- den drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms. arXiv preprint arXiv:2309.073/11 2023.
Christopher Clark, Kenton Lee, Ming-Was wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1905 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. 2018.
arXiv preprint
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.0888 08600, 2023.
Xufeng Duan, Zhaoqian Yao, Yunhao Zhang, Shaonan Wang, and Zhenguang G Cai. How syntax specialization emerges in language models. arXiv preprint arXiv:2505. 19548, 2025.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighte linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hodse Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposi- tion. arXiv preprint arXiv:2209, 10652, 2022.
Julie Franck, Glenda Lassi, Ulrich H Frauenfelder, and Luigi Rizzi. Agreement and movement: A syntactic analysis of attraction. Cognition, 101(1):133-216, 2006.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL /tt:://////o..ooo org/ wwwgg//ecoroo recor ds 002600260000
Sebastian Gerstner and Hinrich Schuetze. Weakening neurons: A newly discovered read-write functionality in transformers with outsize influence. OpenReview, 2025. URL hntpp::// opeenreviww htt.://w.nnnt/ooomm//////s/aa.a

---

Published as a conference paper at ICLR 2026
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3)=e0,, 2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint urXiv:2407.21783 2024.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bert- simas. Finding neurons in haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. arXiv preprint 10Xi::2401.12181, 2024.
Dan Hendrycks and Kevin Gimpel. Gaussian error
linear units (gelus). arXiv preprint
arXiv:1606.0844 08415, 2016.
John Hewitt and Christopher Manning. structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pp. 4129-4138, 2019.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python. 2020. doi: 12928//zenod..22233000
Adam S Jermyn, Nicholas Schiefer, and Evan Hubinger. Engineering monosemanticity in toy mod- els. arXiv preprint arXiv:2211.09166 2022.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023.
Yi Jing, Zijun Yao, Lingxu Ran, Hongzhu Guo, Xiaozi Wang, Lei Hou, and Juanzi Li, Sparse auto- encoder interprets linguistic features in large language models. arXiv preprint arXiv:2502.20344 2025.
Linghao Kong, Angelina Ning, and Nir N Shavit. Input differentiation via negative computa- tion. 3rd Workshop on High-dimensional Learning Dynamics, ICML, 2025. URL https: //openzeview.co wwwpeevvwwwweee..ourataa...a http://oooooopoo/ddecccell
Victor Lecomte, Kushal Thaman, Rylan Schaeffer, Naomi Bashkansky, Trevor Chow, and Sanmi Koyejo. What causes polysemanticity? an alternative origin story of mixed selectivity from incidental causes. arXiv preprint arXiv:2312. 1330996 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 3214-3252, 2022.
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Kamiadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint wxXiv:193.006733 2019.
Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. arXiv preprint arXiv: 09031, 2018
Jason Merchant. Voice and ellipsis. Linguistic Inquiry, 44(1):77-108, 2013.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arxiv preprint 16X1v:1609.07843, 2016.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Informa- fion Processing Systems, 33:17153- 2020.
12

---

Published as a conference paper at ICLR 2026
Aaron Mueller, Yu Xia, and Tal Linzen. Causal analysis of syntactic agreement neurons in multilin- gual language models. arXiv preprint arXiv:2210.14328, 2022.
Max Müller-Eberstein, Rob Van Der Goot, Barbara Plank, and Ivan Titov. Subspace chronicles: How linguistic information emerges, shifts and interacts during language model training. arXiv preprint arXiv:2310. 1.iv:2310.16484, 2023.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814,
2010.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 503):,00024-001, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with sveca. arXiv preprint arXiv:1811.0022. 2018.
Shashata Sawmya, Linghao Kong, Ilia Markov, Dan Alistarh, and Nir N Shavit. Wasserstein dis- tances, neuronal entanglement, and sparsity. In The Thirteenth International Conference on Learning Representations, 2025.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.052022 2020.
Koustuv Sinha, Jon Gauthier, Aaron Mueller, Kanishka Misra, Keren Fuentes, Roger Levy, and Adina Williams. Language model acceptability judgements are not always robust to context. In Proceedings of the 6t Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 6043-6063, 2023.
Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models. arXiv preprint arXiv:2406.1623 16254, 2024.
Adly Templeton. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Anthropic, 2024.
lan Temney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint arx:: 0ww..9955009990 2019.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377-392, 2020.
Johannes Welbl, Nelson F Liu, and Matt Gardner, Crowdsourcing multiple choice science questions. arXiv preprint 1707.0627700099 2017.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 30Xv:22505009388. 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv:1905.07830 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et Opt: Open pre-trained transformer language models. arXiv preprint arXiv: 2022. 2022.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 VALIDATING THE WASSERSTEIN DISTANCE AS A PROXY OF MAPPING DIFFICULTY
A.1.1 WASSERSTEIN DISTANCE VS. MAPPING DIFFICULTY FOR EVERY NEURON
To demonstrate the association of a neuron's Wasserstein distance and its mapping difficulty, we calculate every neuron's WD and MD in each gate projection layer in Llama 3.1 8B. In each layer, the two metrics track one another closely.
Mapping diffiulty Wassecstoon distance different loyees Llama
6B
Figure A1: The Wasserstein distance of neuron closely tracks its mapping difficulty. Neurons from every gate projection Llama 3.1 8B. Data used to calculate metrics from WikiText 2.
14

---

Published as a conference paper at ICLR 2026
A.1.2 WASSERSTEIN DISTANCE MORE CLOSELY ASSOCIATES WITH MAPPING DIFFICULTY THAN WITH ASYMMETRY AND KURTOSIS
To investigate the possible confounds that skewness or kurtosis may cause in the calculation of WD, we calculate the WD. asymmetry 1/1/ and excess kurtosis |K| of the output distribution for each neuron in each gate projection in Llama 3.1 8B. We then compare these metrics between themselves and with MD. Because our analysis utilizes the most entangled neurons, we calculate the Jaccard index between the top 1% of neurons ranked by each metric and measure agreement, with Jaccard index of indicating no agreement and indicating complete agreement. Across all layers, the agreement of the top 1% of neurons calculated by WD and MD is the highest, compared to WD and asymmetry, WD and kurtosis, MD and asymmetry, and MD and kurtosis (Table Al).
Table A1: Jaccard index of the top 1% of neurons as ranked by Wasserstein distance, mapping difficulty, asymmetry, and kurtosis in each gate projection layer of Llama 3.1 8B. Greatest values in each layer are bolded.
Layer
J(WD, MD)
J(W
J(WD,
J(MD, 7
J(MD, [R]
0.713
0.294
0.083
0.222
0.051
0.713
0.452
0.217
0.349
0.153
3
0.810
0.474
0.202
0.437
0.182
0.799
0.497
0.283
0.452
0.254
5
0,810
0.474
0.197
0,459
0.192
0.833
0.529
0,238
0.513
0.233
0.810
0.423
0.207
0.423
0.202
0.673
0.474
0,222
0.474
0.254
0.733
0.521
0.192
0.563
0.212
10
0.702
0.490
0,202
0.529
0,243
0.755
0.505
0.227
0.505
0.238
12
0.692
0.474
0.254
0,482
0.277
0.776
0.482
0.233
0.497
0.254
14
0,673
0.521
0.277
0.513
0.294
0.744
0.437
0.207
0.490
0.222
16
0.799
0.409
0.227
0.368
0.197
0.787
0.416
0.172
0.416
0.187
18
0.787
0.416
0.192
0.409
0.172
19
0.733
0.388
0.172
0.368
0.177
20
0.744
0.444
0.177
0.416
0.167
21
0.682
0.416
0.144
0.375
0.153
22
0.755
0.474
0.197
0.423
0.187
23
0.702
0.444
0.182
0.452
0.207
24
0.776
0.474
0.197
0.474
0.212
25
0.733
0.505
0.207
0.452
0.217
26
0.755
0.474
0.233
0.459
0.217
27
0.755
0.505
0.217
0.529
0.249
28
0.810
0.529
0.254
0.529
0.254
29
0.755
0.505
0.217
0,490
0.222
0.755
0.388
0.187
0.402
0.192
31
0.776
0.324
0.109
0.288
0.092
0.810
0.243
0.092
0.197
0.067
15

---

Published as a conference paper at ICLR 2026
We also show the visual correlation between each the WD and MD, WD and asymmetry. WD and kurtosis, MD and asymmetry, and MD and kurtosis for representative gate projection layers in Llama 3.1 8B. We find that MD and WD track each other more closely than either metric with either asymmetry or kurtosis.
Laper
Loger & MD Aupuratty
Layer WD Karowa
Laper Aguanetry
Layer & Nattods
Bilacc
Tie
MB
17:
Layer 12, MD Kartedt
Ancacestt
Lover 12, Harbera
Auytunnty
Layer Karteds
Sager Aspucaiity
Loer
Figure A2: Wasserstein distance and mapping difficulty correlate more closely than asymmetry or kurtosis. Neurons from specified gate projection layers in Llama 3.1 8B. Data used calculate metrics from WikiText 2.
16

---

Published as a conference paper at ICLR 2026
A.1.3 USING MAPPING DIFFICULTY DIRECTLY RATHER THAN THE WASSERSTEIN DISTANCE PROXY YIELDS THE SAME QUALITATIVE RESULT IN CAUSAL ABLATIONS
To validate the choice of WD as proxy of MD, we also directly use MD as a selection criteria for the most entangled neurons, rather than WD. Repeating the experiments in Figure B for Llama 3.1 8B yields the same qualitative result: the negative pre-activation is highly sensitive for these neurons and is implicated directly in grammatical function.
Ablating Wasserstein nogative pew-rttiattons ditives perpletily up
(bo Matching effect seeds more non entangled semmons
Wasserstein pertubation
Effect Wasserstii perturbation on parts of speech
Intiguely disriphs grammar
entargged pecturtend
diterracco
Rundon
coulted
perpleatty
230
30
Kiath Lit accer
PArT sprarh
Neurons perturbed (9)
Wasserstein neurons perturbed. (%6
DI.MP
TSB
Figure A3: Sign-specific perturbation of Wasserstein neurons disproportionately harms grammar when using MD instead of WD as the selection metric as well. (a) Perplexity increases when clamp- ing only the negative pre-activations of the top-MD neurons; random controls are much smaller. Number indicates starting and ending perplexity in absolute terms. (b) Matching the perplexity in- crease from perturbing the entangled fraction of neurons requires order of magnitude more entangled units. (c) Perturbing Wasserstein neurons, as chosen by MD, uniquely impacts grammat- ical capabilities, even when compared to the perplexity-matched control. The top 1% Wasserstein neurons in each layer, as chosen by MD, were perturbed for the benchmark. The least entangled 30% of neurons, as chosen by MD, were used the perplexity-matched control. (d) At per token resolution of the WikiText 2 validation dataset, tokens associated with syntactical scaffolding expe- rience a much higher surprisal for the 1% Wasserstein perturbation, chosen by MD, compared to the perplexity matched control. NLL differences were mean shifted by the global difference. Ran- domly sampled controls were acquired over ten trials. Error bars indicate one standard error of the mean. Data from Llama 3.1 8B.
17

---

Published as a conference paper at ICLR 2026
A.2
RAW BENCHMARK SCORES
Table A2:
Model performance on BLiMP and TSE with the applied perturbations. Data used in Randomly sampled controls acquired over ten trials. Lowest value in bold.
Figure 3
Model
Benchmark
Baseline
1% Wasserstein
PPL Matched
1% Random
Llama 3.1 8B
BLiMP
81.2 + 0.2
77.5 + 0.2
81.5 + 0.2
81.1 + 0.2
TSE
84.4 + 0.1
55.2 + 0.1
85.9 = 0.1
84.1 0.1
Mistral 7B v0.3
BLiMP
83.3 0.1
77.9 + 0.2
82.8 + 0.1
83.2
TSE
8.1
54.6 + 0.1
872
84.5
Qwen3 8B Base
BLiMP
0.1
80.6 + 0.2
0.2
82.4
TSE
8.1
78.6 + 0.1
82.4
84.7 0.1
18

---

Published as a conference paper at ICLR 2026
Table A3: Llama 3.1 8B performance on a suite of non-grammar benchmarks with the applied perturbations. Data used in Figure Randomly sampled controls were acquired over ten trials. Lowest value bold.
Benchmark
Baseline
1% Wasserstein
PPL Matched
1% Random
ARC Challenge
53.4 + 1.5
48.6 1.5
38.1 - 1.4
53.3 = 1.5
ARC Easy
81.1 = 0.8
76.2 = 0.9
60.4 = 1.0
80.7 = 0.8
BoolQ
82.1 + 0.7
74.9 + 0.8
67.0 + 0.8
81.9 + 0.7
HellaSwag
78.9 + 0.4
75.3 0.4
69.4 0.5
78.6 + 0.4
PIQA
81.2 0.9
78.8 1.0
76.6 1.0
81.4 + 0.9
SciQ
94.6 0.7
95.4 = 0.7
8.13
94.5 = 0.7
TruthfulQA
45.2 + 1.4
39.8 = 1.4
4.5
46.0 = 1.4
WinoGrande
73.9 1.2
70.4 1.3
66.3 1.3
74.7 = 1.2
Table A4: Mistral 7B v0.3 performance on a suite non-grammar benchmarks with the applied perturbations. Data used in Figure A4 Randomly sampled controls were acquired over ten trials. Lowest value bold.
Benchmark
Baseline
1% Wasserstein
PPL Matched
1% Random
ARC Challenge
52.3 1.5
52.4 1.5
43.5 1.4
52.7 = 1.5
ARC Easy
78.2 0.8
77.0 0.9
65.2 + 1.0
78.5 + 0.8
BoolQ
82.1 0.7
79.8 0.7
71.4 4 0.8
81.9 + 0.7
HellaSwag
8.4
7.4
72.5 0.4
80.3 0.4
PIQA
82.3
0.9
1.0
81.9
SciQ
95.1 0.7
8.2
94.2 + 0.7
TruthfulQA
42.6 12.61.4
41.4 1.4
46.1 1
42.7 = 1.4
WinoGrande
73.9 1.2
71.1 1.3
61.6 1.4
73.6 + 1.2
Table A5: Qwen3 8B Base performance on suite of non non-grammar benchmarks with the applied Randomly sampled controls were acquired over ten trials.
perturbations. Data used in Figure A5 Lowest value bold.
Benchmark
Baseline
1% Wasserstein
PPL Matched
1% Random
ARC Challenge
56.9 1 1.4
51.5 1.5
54.2 + 1.5
56.8 + 1.4
ARC Easy
80.0 1 0.8
75.5 0.9
78.0 0.8
80.4 = 0.8
BoolQ
83.0
81.9 = 0.7
82.2 + 0.7
83.2 + 0.7
HellaSwag
78.6
78.8
74.7 0.4
78.6 + 0.4
PIQA
79.3
79.1 0.9
78.6 1.0
79.3 + 0.9
SciQ
96.1 + 0.6
96.0 0.6
94.7 0.7
95.9 + 0.6
TruthfulQA
02.31..
50.9 1.5
54.8
52.6 + 1.5
WinoGrande
72.8 + 1.2
69.5 1.3
68.6
71.9 + 1.3
19

---

Published as a conference paper at ICLR 2026
A.3
NON-GRAMMATICAL BENCHMARKS FOR ADDITIONAL MODELS
Mistral 7B v0.3: Wasserstein neuron perturbation not as damaging on non- grammatical tasks
1% Waswersbiin
Aboolute error increato (%)
PPL matched
11
10-
-0
Grammma Aug Other
BLME
TSE
ABC Challenge ABC Easy
BuolQ
HullaSwaa
PIQA
Seii
TrutifalQA WiseGrande
Figure A4: Non con-grammatical abilities less harmed by Wasserstein neuron perturbation in Mistral 7B v0.3. All benchmarks were run in O-shot. Randomly sampled controls were acquired over ten trials. Error bars indicate one standard error of the mean. Raw scores and CI's are in Tables A2
A4
Qwen3 BB Base: Wasserstein nouron perturbation disproportionately harms grammatical benchmarks
Absolute error increase (%)
3
2
Grammnr Other
BLIMP
ABC Challenge
BeoIQ
Hellowwog
PIQA
TrutzuuoA WinoGraade
Figure A5: Non- con-grammatical abilitics are comparatively less harmed by Wasserstein neuron pertur- bation in Qwen3 8B Base. Although the Wasserstein neuron perturbation causes damage in both grammatical and non-grammatical tasks, it still causes more relative and absolute damage in the former. All benchmarks were run in 0-shot. Randomly sampled controls were acquired over trials. Error bars indicate one standard error of the mean. Raw scores and CI's are in Tables A2 A5
affect Wasserstein perturbations models
-00
Wisserstein PPL matched
Absolute esvor incresse (%)
1% Random
20
15-
12
Aug Other Aoe
BLMM
Challunge Easy
BeelQ
Hullusooaa
PIQA
SeiQ
TrutzfalOA WineGrande
Figure A6: Average effect across Llama 3.1 8B, Mistral 7B v0.3, and Qwen3 8B Base. While indi- vidual models show varying degrees of grammar selectivity, on average, grammar benchmarks are substantially more sensitive to targeted Wasserstein neuron perturbations, whereas con-grammatical benchmarks are more sensitive to broad low-WD ablations at matched perplexity. All benchmarks were run in 0-shot. Randomly sampled controls were acquired over ten trials model. Error bars indicate one standard error of the mean. Raw scores and CI's are in Tables
A4
A5
20

---

Published as a conference paper at ICLR 2026
A.4 ADDITIONAL ACTIVATION FUNCTION ABLATIONS REVEAL IMPORTANCE OF THE NEGATIVE SIGN ITSELF
We further study additional modifications the effective activation function and therefore the pre- activation space, revealing the importance of negative sign itself rather than magnitude. We compare four ablations:
zeroing negative pre-activations, as was done in Section
// = (z) if x > 0, y = 0 otherwise
2. setting positive pre-activations
use the ReLU activation function rather than SiLU
y/ = (z) if æ < 0, y = z otherwise
3. zeroing positive pre-activations
y/ = (z) if x < 0, y = 0 otherwise
4. flipping the sign of negative post-activations
y = (a) if æ > 0, y = SiLU(z) otherwise
For each control, as in Section 3] we modify the activation function for the top-WD neurons each layer and compare that modifying the activation function in the same way for an equal number of randomly chosen neurons in each layer.
As initial confirmation, the curvature of the SiLU in the positive is particularly salient, as replacing that with ReLU yielded very little change to model performance. By contrast, zeroing the positive signal of Wasserstein neurons is substantially more damaging than applying the same intervention to random neurons and also more damaging than zeroing the negative signal, consistent with the positive branch carrying the bulk of the model's overall information.
Strikingly, reversing the sign of negative post-activations yields substantial model damage, moreso than even zeroing out positive activations 2% ablation. Indeed, with just reversal of the negative signs of the top 2% WD neurons, perplexity climbs from 6.50 to 2277. Crucially, this intervention preserves magnitude the negative responses and thereby largely preserves their contribution to RMS/LayerNorr statistics, while inverting only the sign. Together, these controls show that the sign of negative activations in Wasserstein neurons carries essential information, and inverting is more harmful than removing it altogether.
21

---

Published as a conference paper at ICLR 2026
(a) Ablate negative pre-artivations
18-
1% Whsserstain Random
Perplexts
12
8-
Effective activation fanction
Input
Neurons perturbed (%)
Set positive pre-activation
ReLU
6.67
1% Wasserstein
Randio
6.050
Perplextry
8.000
8.375
0.350
SLU
Offective activation function
Ioput
Neurons perturbed (96
Ablate positive pre actavations
2000
1% Wasserstein
Random
Porplexity
300
SIUU
Effecivve activation function
0-
Iapul
Neurons porturbed (%)
(d) Flip sign negative pre-activations
35600
1% Wasserstain
30000
Fandom
25000
Output
Perplexity
15000
10000
SILU
Fffective activation function
0-
Input
Neurons parturbed (%)
Figure A7: Additional controls reveal importance of negative sign itself rather than just its magni- tude. For each control, in Section 3] we modify the activation function for the top-Wi WD neurons in each layer and compare that to modifying the activation function in the same way for an equal number of randomly chosen neurons in each layer. For each modification, the effective activation function is shown in the left column, and the effect on model perplexity on the WikiText 2 validation dataset is shown in right column. (a) Zeroing negative pre-activations, was done in Section (b) Setting positive pre-activations to use the ReLU activation function rather than SiLU, (c) Zeroing positive pre-activations. (d) Flipping the sign of negative post-activations.
22

---

Published as a conference paper at ICLR 2026
A.5 FURTHER CHARACTERIZATION OF WASSERSTEIN NEURONS AND PROSPECTS FOR AUTOMATION
To probe how interpretable the differentiation of tokens is among Wasserstein neurons, we extend the analysis of Figure 6 to additional neurons in the second up projection layer in Pythia 1.4B. For each neuron, we collect the top output pairs that have been mapped the furthest given their input distance, and observe the underlying tokens that these inputs correspond to. Doing so reveals further clear, neuron-specific patterns. For example, neuron 1 1168 strongly separates the token "and" from variety of other tokens, neuron 4093 preferentially differentiates past-tense verb forms, neuron 4457 separates adpositions from determiners, neuron 4606 distinguishes punctuation from other tokens, neuron 5776 isolates numerical tokens from other vocabulary items, and neuron 6984 is selective for determiners (Figure A8).
These qualitative patterns reflected quantitatively in the POS statisties of the top differentiated pairs. Considering the top 50 token pairs per neuron, we find that specific POS categorics (determin- ers, adpositions, punctuation, numerals, verbs) are heavily enriched among the differentiated tokens compared to their base rates in Wiki'Text 2. This suggests that many Wasserstein neurons implement relatively coherent, fine-gr mne-grained syntactic distinctions. While stop short of building fully auto- mated mapping from neurons to syntactic features, these POS enrichment statistics provide a natural starting point for such system in future work.
23

---

Published as a conference paper at ICLR 2026
Neuron 1168, XD 3,48
Nar of tokens Ust sqscried fached
(U PDS interiutio
PC5
Dhp
Usp
Hd
0750
203
P.307
Neuron 4093, MD 2.36
Marr of Sakens tht seperosl the furtheat
(to POS (ustritattn 10 pairs
Exncemmnnto eacs
Tokao
USQUES
EUS
DED
12-0
2014
1014
Thay
2014
Neuron 6457. MD 2.32
2a Pirrs of sakees thet seperoes the Burthea
the PSS cistritttonn of 50 poirs
Banchmentto eads POS
Tokonn
FOS
18
Neuron 1606. MD 2.51
24 Piire of salmee that repereeet the Burthwed
(he 99S riartrettnnn of 50 poirt
Pencheevntoo ands POS.
T0R00
SOION
FUIS
EXS
1002
COPE
100
112
20001
4.24
Neuron 5778, MD 3.54
(41 Pair of tckeec that oepernted the nembee
(N) POS diartiectinn oo 50 paint
50 Baridhewerr of PoS
To
NOM
2000
commooung
Neuron 8904, MD 3.10
24 Patre of sabees that orgareedd the Burtaee
(h) POS clartiatiinn oo 50 point
12) Df POS
Tokon
R0S
121
221
CORE
1000
0000
6033
Figure A8: Additional examples of interpretable Wasserstein neurons based on the tokens they differentiate. For each neuron, the top 10 pairs of tokens that are most differentiated are visualized in (a). The distribution of each POS within the top 50 pairs that most differentiated is shown in (b). These distributions are compared to the underlying POS distribution for the entire Wiki 2 dataset, and the relative enrichment is visualized in (c). The relative enrichment for each POS is calculated as the ratio between the proportion of that POS in the top 50 pairs and the proportion of that POS in the Wixi dataset as a whole, minus 1.
24

---

Published as a conference paper at ICLR 2026
A.6 LAYERWISE NEGATIVE DIFFERENTIATION PREDICTS GRAMMATICAL ERROR IN LLAMA
We repeat the pair analysis and sign-specific ablations presented in Section 6 for Llama 3.1 8B. In this case, we the top 0.5% MD of neurons, with 50 out of 1000 pairs each increase specificity. Compared to Pythia, Llama exhibits a lower overall rate of NN pairs, with PN dominating across depth. Nevertheless, early layers still show an elevated NN proportion compared later layers (Figure A9 To test whether this residual NN usage is behaviorally meaningful, we perturbed the top 1% Wasserstein neurons one layer at time and measured the induced error on BLiMP and TSE.
Across layers. the fraction of NN pairs in the most separated set positively correlates with error under the perturbation, while PN and PP proportions show weak or opposite trends. Both BLiMP and TSE show this pattern (FigureAA) c). Thus, even in an architecture that uses NN differentiation less overall, layers that rely on suffer the most when negative pre-activations are suppressed.
Layerwiiee proportion types
BLIM type proportien
(c) TSE type propection
(di Layervise NN proportion and
Abollde WITHrr In sereewwngg Ph)
Fovarisa (N)
in
Normaltned poir
Normslized
Figure A9: Llama uses less negative differentiation overall, yet early-layer NN still predicts gram- matical fragility. (a) Layerwise composition of the most-separated pairs (top 50 of 1000 per neuron) for the top 0.5% MD neurons in Llama. PN dominates, but NN remains elevated in early layers. Log scale used for y-axis to better show differences in trends. (b, c) share the same legend. (b) The induced error in BLiMP from a layerwise 1% Wasserstein neuron negative pre-activation ablation correlates with the proportion of NN pairs in that layer. (c) Analysis from (b) repeated for TSE. (d) Normalized overlay of NN proportion, BLiMP error, and TSE error over layer depth highlights a shared early layer peak, indicating that layers with more NN differentiation are those whose perfor- mance degrades most under negative clamping. Shaded regions are one standard error of the mean.
Finally, overlaying the normalized NN proportion with the normalized BLiMP and TSE error reveals closely aligned peaks in the earliest layers (Fig. [9 This aligmment supports general picture: negative differentiation is an early-laye mechanism that downstream computation depends on, even when its global prevalence is modest.
However, the interpretation of NN differentiation limited by a structural confound in our abla- tion method. Because our intervention zeroes only negative pre-activations, it disproportionnttly collapses NN pairs relative to PN or PP pairs. As a result, correlations between the prevalence of NN pairs and grammatical vulnerability under ablation may partly reflect the mechanics of the intervention itself, rather than intrinsic grammatical importance. We therefore treat this analysis suggestive rather than causal, and focus our main claims the sign-specific causal ablations. Our sign-flip experiment (Section A.4) partly mitigates this concern by modifying the sign of nega- tive activations while preserving magnitude, producing even larger degradation. Nonetheless, fully controlled comparisons between NN, PN, and PP specific perturbations remain an open direction.
25


Reviews:
[{"review_id": "utePo1nBvV", "reviewer": "Reviewer_6mmb", "summary": "This paper argues that a tiny, entangled subpopulation of MLP \"Wasserstein neurons\" in LLMs computes in the negative pre-activation region of smooth activations, and that this sign-specific behavior is mechanistically important for syntax. Evidence: (1) WD/\"mapping difficulty\" characterization across models; (2) sign-specific ablations that zero only negative pre-activations for the top-WD neurons cause large PPL increases and large drops on BLiMP/TSE, unlike random or perplexity-matched controls; (3) effects localize to functional POS tokens and early layers, compounding across depth; (4) training-dynamics: Wasserstein neurons emerge early and the same sign-specific ablation harms grammar more as training proceeds. ReLU models (OPT) lack the negative-region structure, consistent with clamping.", "questions": "1. In training dynamics, if you freeze the early high-WD cohort vs. others, do grammar curves diverge as predicted?\n\n2. Could you report CIs/seeds for BLiMP/TSE?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Clear, testable mechanistic claim (negative pre-activation computation) linked to syntax; the sign-specific intervention is surgical and well-motivated for non-ReLU models.\n\n2. The paper also provides compelling causal evidence. That is, clamping negatives of top-WD neurons yields large BLiMP/TSE drops even when perplexity is matched by ablating many low-WD neurons; random controls are much smaller.\n\n3. The paper demonstrates cross-model coverage (Pythia, Llama-3.1-8B, Mistral-7B-v0.3, Qwen-3-8B) and ReLU vs non-ReLU comparison.", "weaknesses": "1. Zeroing negatives changes the neuron’s transfer function (effectively ReLU-izing selected channels), which can alter residual/RMSNorm statistics and induce broad PPL rises unrelated to \"syntax-specific\" computation. My worry is that the perplexity-matched control (many low-WD channels clamped) helps, but it does not fully control for distributional/variance changes in the same high-WD channels.\n\n2. The figure-level observation that OPT (ReLU) lacks negative-region structure is interesting; strengthen by running the same sign-specific ablation (no-op in ReLU) plus a mirror positive-only clamp in non-ReLU models to show asymmetry is truly about negative region utility rather than general clipping.  \n\n3. The negative-negative (NN) separation result is striking; however, the pipeline samples top MD pairs among similar inputs and counts sign patterns. To exclude lexical/position confounds, you can do better control for token frequency, position, and subword overlap.\n\n4. Grammar drops are strong; it would be great if you could also report non-syntactic tasks (e.g., TruthfulQA/QA/calibration) to establish specificity: are you removing general capacity or truly syntax-critical computation? The POS analysis suggests scaffolding specificity, but a broader behavioral panel would solidify the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3cPtLOsun1", "reviewer": "Reviewer_abiD", "summary": "The authors study \"Wasserstein\" neurons, which exhibit strongly non-Gaussian distributions of preactivations. They show that for models with non-ReLU activation functions, negative preactivations are functionally important through sign-specific ablations, developmental analysis across checkpoints, and other experiments. The authors provide additional evidence for these neurons being especially implicated in predicting grammatical techniques by targeting their analysis to grammatical benchmarks.", "questions": "- Besides the fact that WD is more neuron-focused than MD (lines 127–129), is the main reason to favor WD over MD that WD is more computationally efficient? It seems like it might otherwise be better to compute aggregate MD scores for each neuron since this would get closer to the question of entanglement. \n\nMinor improvements:\n\n- (This is related to weakness A) The definition of Wasserstein neurons and \"non-Gaussian\" structure could be improved in Sec 2.1 and Fig 1. Phrasing such as (lines 76–78) \"the non-Gaussian structure of Wasserstein neurons appears in the positive region ... [for GELU] the non-Gaussian structure is concentrated in the negative pre-activation region\"  is confusing because the distribution as a whole is clearly non-Gaussian. I believe the authors mean to say something like \"the bulk of preactivations occupies a large Gaussian peak. The mode of this peak is concentrated in the negative/positive region respectively for ReLU and non-ReLU networks.\" It would be clearer if the authors discussed this purely in terms of the largest mode of the distribution.\n\t- Line 63–66 \"the deviation from Gaussianity concentrates in the negative region of the pre-activation space ... ReLU-based models ... do not exhibit comparable structure in the negative space\" is clearly wrong as stated since the ReLU-based neurons also have clear non-Gaussian structure across the positive and negative range of activations. \n- Lines 55–57 \"Nevertheless, many interpretability workflows retain the ReLU-era heuristic, equating activity with positive pre-activations and implicitly treating the negative region as inert.\" I question whether this is true. I believe many workflows like SAEs are sensitive to negative activations. Maybe the authors believe that the field still implicit acts with this expectation (which would require weakening the language) or they can provide some references to examples of real-world workflows making this assumption. \n- Lines 16–17: \"We show that this negative region is functional rather than simply favorable for optimization.\" What does this mean?\n- Line 19 \"entangled neurons\" is not defined until line 38\n- Line 161: \"mildly more expressive in their distributions\" what does this mean?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Well-motivated**: The introduction is clearly written. The discussion on the need to reexamine the distribution of negative pre-activations for non-ReLU is strong.\n- **Diverse set of models**: I appreciate that the authors have included a diversity of LLMs. \n- **Self-explanatory figures**: The figure captions are clearly written and self-explanatory, making this easy to follow. \n\nFull disclosure: I reviewed an earlier workshop version. From memory, this revision is a substantial improvement in cohesion.", "weaknesses": "A. WD may act as a proxy for skew/kurtosis rather than \"entanglement.\"  \n\nIn many places, the authors treat \"high WD from a Gaussian baseline\" (\"WD score\") as equivalent to \"highly multimodal\" or \"strongly entangling.\" This is implicitly justified via the observation that WD and mapping difficulty (MD)  are highly correlated: i.e., neurons with high WD score are likely to map similar input vectors to widely separated preactivations. However, \"high WD score\" is also compatible with \"non-negligible higher moments.\" This is not discussed, which is a source of confusion and leads to implicitly overstating the strength of the Wasserstein neurons as \"entangled neurons\" hypothesis.\n\nMinimum fix: Some additional comment clarifying the alternative hypothesis (high skew/kurtosis/higher moments) would be enough to resolve this concern. The authors could mention as a limitation that the methodology cannot pick up the distinction between heavy tails and multimodality and that this weakens the \"entanglement\" interpretation.\n\nMaximum fix: Run an additional comparison between WD and skewness/kurtosis. \n\nPoint of clarification: Section 2.1 defines MD by input‑difference / output‑difference averages. If that is indeed the implemented metric, high MD means _low_ output separation for similar inputs. This is a bit confusing because the text uses (lines 126–127) \"MD of a neuron to summarize how far apart neurons map similar inputs.\"\n\nB. The argument for WNs affecting grammar over other kinds of structure is weak(ly presented). \n\nThe grammar hypothesis is introduced in section 2 by demonstrating that WD correlates with TSE (a specific grammatical benchmark) performance over training. In this section, the authors never discuss any alternative explanation (e.g., WNs are helpful for general-purpose natural language processing). Towards the end of the section the authors ask (lines 195–198): \"is [this] structure merely a byproduct, or is it causally necessary for grammatical behavior?\" This is a sudden and unexpected jump.\n\nEvidence for the grammar hypothesis is presented in section 3 and Fig 3c & d and section 3. I find this mostly convincing, though I would like to see an additional comparison like Figure 3c against a non-grammatical control benchmark (from my current understanding, the authors have not yet ruled out the possibility of WNs being important for grammar *as well as* being important to other roles – though figure 4d does point in this direction). \n\nLine 275–278 \"Cumulative ablations raise error across all syntactic classes, consistent with a role for Wasserstein neurons as more general-purpose carriers of grammatical information rather than specifically detecting individual phenomena\" There is no discussion of any control assessing the effects across non-syntactic categories. Perhaps this evidence is implicitly presented in figure 4, but it is not clear to me where to look for this. I would appreciate if the authors can comment on this. \n\nMinimum fix: The paper would be much easier to understand if the grammar hypothesis is introduced more naturally. Another option would be to simply foreshadow the grammar hypothesis in section 2 and promise that it will be justified later. \n\nMaximum fix: The paper would benefit from repeating 4c with additional non-grammatical benchmarks, and providing further detailed examples illustrating the effects of ablations on individual samples in Fig 4d. This could be limited to an appendix. \n\nSmall additional note: when comparing to perplexity‑matched controls, please state clearly whether the layer distribution and number of perturbed units per layer are matched. If not, a sentence explaining that the current matching is still incomplete would pre‑empt reader confusion.\n\nC. Treatment of negative-negative separation is relatively weak. \n\nThere's an obvious confounder in the analysis of NN vs PN vs PP differentiation. Zero-clamping negative preactivations collapses the distinction between two samples that both have negative preactivations that are only distinguished in magnitude. The same is not true for two samples that differ in preactivation sign or where both inputs have negative preactivations. As such, it is very unclear to me whether the observed higher correlations between NN ablations and damaged performance tell us anything about whether NN-differentiated samples are more important for grammar. The correct baseline would also involve ablating the particular positive activations for the PN or PP pairs. \n\nMinimum fix: Mention the confounder explicitly and point to it as a limitation of the current design. \n\nMaximum fix: Run additional experiments. For example, you could run (i) a matching positive‑clamp for the PN/PP baselines in the same layers, or (ii) sign‑flip/abs interventions to test whether sign vs. magnitude drives the effect.\n\nSuggestion: I find the NN/PN/PP differentiation analysis to be relatively weaker than the rest of the text. I would recommend moving section 7 (and 6?) to the appendix (along with a clearer mention of confounders), and using the remaining space to strengthen the case for the preceding sections (esp. the grammar hypothesis). \n\nIf the authors implement all of my minimum fixes and one or two of the maximum fixes, I will raise my score to an accept. I think including all of the maximum fixes is unlikely to further raise my score to a strong accept, but I would still recommend the authors consider these additional changes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KaaKwSWaxm", "reviewer": "Reviewer_9wYS", "summary": "The paper shows the importance of wasserstein neurons within LLMs for syntactic processing. Precisely, wasserstein neurons are shown to map similar input vectors to different negative output values in LLMs equipped with smooth activation functions. Such neurons emerge very early during retraining and are more frequent in larger models. Ablating wasserstein neurons results in degraded syntactic capabilities, as reflected by the BLIMP and TSE benchmark scores. Furthermore, wasserstein neurons are shown to affect most the LLM’s performance on ellipsis and subject agreements, most crucially at early layers. Lastly, negative differenntiation is shown to be a mechanism for syntactic processing, forming negative-negative pairs.", "questions": "* When computing WD with respect to the unit Gaussian, where is this gaussian centered? Does this not modify the WD?\n* Figure 2d, what is “syntactical scaffolding experience”? Why are determiners, punctuation … more important for syntax?\n* It is quite surprising that syntactic processing happens at early layers within the MLP blocks and not in middle layers in attention layers (as previously suggested by probing works)? Is there any explanation for this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is creative and shows a large set of results.\n* The paper links elegantly syntax and single neurons, which is a unique and well executed finding.\n* The writing and the figures are clear and precise.", "weaknesses": "* The correlation between the pretraining emergence of Wasserstein neurons and TSE is not convincing alone, there could be a lot of possible confounds during pretraining.\n\n* There is no link with the mechanistic understanding of syntactic processing. For example, understanding *how* these neurons combine to  parse syntactic trees.\n\n* There is no automated method for knowing what syntactic features drive each neuron, this would allow to list all wasserstein neurons and their associated syntactic pattern.\n\n* The proposed mechanism is specific to LLMs with smooth activation functions (GELU/SiLU), but, is this condition necessary to acquire syntax? However, ReLU-based LLMs such as OPT also exhibit clear syntactic competence, suggesting that smoothness in the negative regime is not necessary for syntax acquisition. This raises the possibility that negative differentiation is one implementation path rather than a universal requirement for syntactic encoding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ugHs77yOj", "reviewer": "Reviewer_LJm9", "summary": "This paper examines the role of so-called 'Wasserstein neurons' in transformer networks, emphasising their use of negative activations to encode syntactic information. Building upon prior work on Wasserstein neurons, the authors analyse how clamping these negative activations to zero affects model performance.\nExperiments demonstrate that eliminating negative activations in Wasserstein neurons significantly degrades model performance, whereas the same intervention has minimal impact on non-Wasserstein neurons.", "questions": "Specifically, the \"CONCLUSION AND DISCUSSION\" part reads to me like all it could be logically derived from the statement:\n\"Wasserstein neurons are important and modifying them disrupts model performance.\"\nCould the authors elaborate on key findings that are rigorously analyzed with e.g., ablation studies that are not in very close relation to this?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The concept of Wassestein neurons is interesting, and this work tries to explain why they are important.\n\nImportant plots have error bars.\n\nThe importance of Wasserstein neurons for grammar in LLMs is rigorously shown.", "weaknesses": "1) While the findings are interesting, the work largely extends ideas already presented in “Wasserstein Distances, Neuronal Entanglement, and Sparsity.”\nI understand the main experimental results as follows: Suppressing activations in neurons identified as important (from previous work) harms model performance more than suppressing other neurons, which is intuitive and expected from the original discussion. \n\n2) The part of specifically attributing grammar to these neurons is not convincing.  \nIs it possible that grammar is simply the first thing that is lost when a model degenerates? \nHow does performance decrease on non-grammar benchmarks when disturbing Wasserstein-Neurons? Perplexity indicates that changes could be large, too, which would contradict a specific attribution.\n\n**Minor remarks:**\n\nFigure 3 states: \"(a,b) share the same legend.\" However, they have different x and y labels and different numbers on the axis.\n\nThe text has an abundance of -- [some text] -- mid-sentences, which disturbs the flow of reading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)