Hedonic Neurons: A Mechanistic Mapping of Latent Coalitions in Transformer MLPs
Authors: Tanya Chowdhury
Atharva Nijasure
Yair Zick
James Allan
Venue: ICLR 2026 Poster (2026)
Keywords: mechanistic interpretability
feature discovery
MLPs
Primary area: interpretability and explainable AI
TL;DR: Use Game theory to compute synergistic neuron coalitions in LLM MLPs

Abstract: Fine-tuned Large Language Models (LLMs) encode rich task-specific features, but the form of these representations—especially within MLP layers—remains unclear. Empirical inspection of LoRA updates shows that new features concentrate in mid-layer MLPs, yet the scale of these layers obscures meaningful structure. Prior probing suggests that statistical priors may strengthen, split, or vanish across depth, motivating the need to study how neurons work together rather than in isolation.  

We introduce a mechanistic interpretability framework based on coalitional game theory, where neurons mimic agents in a hedonic game whose preferences capture their synergistic contributions to layer-local computations. Using top-responsive utilities and the PAC-Top-Cover algorithm, we extract stable coalitions of neurons—groups whose joint ablation has non-additive effects—and track their transitions across layers as persistence, splitting, merging, or disappearance.  

Applied to LLaMA, Mistral, and Pythia rerankers fine-tuned on scalar output tasks, our method finds coalitions with consistently higher synergy than clustering baselines. By revealing how neurons cooperate to encode features, hedonic coalitions uncover higher-order structure beyond disentanglement and yield computational units that are functionally important, interpretable, and predictive across domains.

Published as a conference paper at ICLR 2026
HEDONIC NEURONS: A MECHANISTIC MAPPING OF LATENT COALITIONS IN TRANSFORMER MLPs
Tanya Chowdhury, Atharva Nijasure, Yair Zick & James Allan Manning College of Information and Computer Sciences University of Massachusetts Amherst (tchowdhury, an i jasure, yzick, al 1lllap laann @cs. umass. edu
ABSTRACT
Fine-tuned Large Language Models (LLMss) encode rich task-specific features, but the form of these eppeeell within MLP remainss unclear. Empirical inspection of LoRA updates shows that new features concentrate in mid-layer MLPs, yet the scale these layers obscures meaningful structure. Prior probing suggests that statistical priors may strengthen, split, vanish across depth, motivating the need to study how neurons work together rather than in isolation.
We introduce mechanistic interpretability framework based on coalitional game theory, where neurons mimic agents in a hedonic game whose preferences capture their synergistic contributions to layer-local computations. Using rop-responsive utilities and the PAC-Top- Too-Cover algorithm, we extract stable coalitions of neu- rons-groups whose joint ablation has non-additive effects-ane track their transi- tions across layers as persistence, splitting, merging, or disappearance.
Applied to LLaMA, Mistral, and Pythia rerankers fine-tuned scalar output tasks, our method finds coalitions with consistently higher synergy than clustering base- lines. By revealing how neurons cooperate encode features, hedonic coalitions uncover higher-orde order structure beyond disentanglement and yield computational units that functionally important, interpretable, and predictive across domains.
INTRODUCTION
Consider a large language model fine-tuned to compute semantic similarity between text pairs. When presented with two sequences, the model outputs scalar score, e.g. 0.76. But how is this number internally computed? Within the millions of parameters of transformer, such decisions not the work of isolated neurons, but of groups that cooperate to represent abstract features like "semantic overlap." "term frequency patterns," or "syntactic alignment." " These computations may parallel familiar retrieval metrics e.g., some neuron coalitions might compute TF-IDF-like statistics (Sparck Jones 1972), others might capture cosine similarities between representations, while still others might encode position-dependent matching signals. Traditional interpretability methods have limitations here: probing (Gurnee et al.] 2023 Hewitt & Manning) 2019) captures correlations with labels but ignores cooperation, sparse autoencoders (SAEs) (Huben et 2024) disentangle activations into monosemantic directions but overlook nonlinear dependencies, and clustering (Cao)
al. 2025 Song et al. 2024) groups neurons by statistical proximity rather than functional interaction. What is missing is principled way to identify synergistic neuron groups proups-subeets whose combined contribution exceeds the sum of their parts. We aim identify the computational units that organize into stable coalitions, that potentially encode these mathematical concepts within scalar-output L.MMs.
Recent work has shown that LoRA fine-tuning can teach L.MMs new tasks by updating only mid-level MLP layers, nearly matching full fine-tuning (Hu et al.) 2022 Zhou et al.] 2024, Nijasure et al. 2025). Yet inspection of these LoRA weight updates reveals little obvious structure: millions of parameters diffuse across neurons, obscuring which units encode task-specific features. We hypothesize that the key to isolating LoRA emergent behaviour lies in identifying coalitions neurons that consistently co-adapt under fine-tuning. Inspired by game theory, model neurons agents in hedonic game (Dreze& & Greenberg. 1980), where preferences reflect synergy with others. Though neurons are not literally rational, stochastic gradient descent imposes form of selection pressure: directions that

---

Published as a conference paper at ICLR 2026
reduce loss persist, and many neurons only useful in combination (e.g., feature computation). Thus, stable coalitions naturally emerge as groups of neurons that survive training together. This evolutionary analogy motivates the hedonic game framing: utilities capture how much neuron's survival depends on its synergy with others, and stable coalitions correspond to groups of neurons that consistently co-adapt under training. By modeling these groups as coalitions, open a path toward reverse-engineering their function and symbolically characterizing their emergent behavior.
Why does this matter? Beyond offering new perspective on interpretability, coalition analysis provides actionable insight into how task-specific features are represented and evolve. By showing which neuron groups are functionally indispensable, our framework suggests future directions for practical interventions such as model comparison, transfer learning, or modular editing at the coalition level rather than at the level of individual weights. Moreover, tracking persistence, splits, and vanishings highlights how statistical priors are refined or discarded across depth, shedding light that clustering or SAE-style methods
on the internal dynamics of fine-tuned mrodels-informattii cannot reveal. Thus, stable coalitions are not only theoretically appealing but also open path to understanding and eventually controlling the computational units that fine-tuning creates.
Our Contributions. We introduce game-theoretic framework for discovering and analyzing neuron coalitions in transformer MLPs. (1) We model neurons as players in a hedonic cooperative game with top responsive utilities based synergy, and solve for E-PAC-stable outcomes using the PAC- Top-Cover algorithm. (2) We evaluate coalitions both intrinsically and extrinsically: compared to clustering baselines, they achieve +0.29 Pairwise and +0.49 Ratio synergy, exhibit 3-5x larger out- of-distribution performance drops under ablation, align more strongly with IR heuristics (BM25, IDF, query term coverage), and yield macro-features that improve predictive Rt from ~0.20 to 0.43-0.47. (3) Treating coalitions as "meta-neurons," we trace their evolution across consecutive layers, finding that most groups vanish or split while only a small fraction MLPs act primarily as feature filters rather than creators. Applied to LLaMA, Mistral, and Pythia LoRA rerankers, we show that hedonic coalitions consistently uncover reproducible and functionally indispensable computational units. To our knowledge, this is the first work to use game theory to identify, validate, and track synergistic neuron groups in fine-tuned LLMs. Details of all code, models, and datasets are provided herd
the view that deeper
2 BACKGROUND
We begin by outlining the fundamentals of hedonic games and their application in modeling coopera- tive behavior. We then describe the transformer architecture with an emphasis on MLP sublayers.
2.1 HEDONIC GAMES AND PAC-STABLE COALITION FORMATION
hedonic coalition formation game Drrze & Greenberg 1980) consists of a set of players N who exhibit preferences over groups they might join. Formally, each player i N ranks all coalitions S C N that contain i; in our setting, we assume that players have cardinal utilities over coalitions. Given a player i N and coalition 5 containing player i's utility from joining S is u((S) E R, which we later instantiate as a function of i's strongest partners within S.
Our goal is to identify coalition structure or partition of the player set which satisfies certain desiderata Aziz & Savani 2016). Given a coalition structure π, we let (i) designate the coalition containing player under æ. Core stability (Bogomolnaia concept. We say that a coalition S C N blocks a coalition structure π if every player i E S strictly prefers S to their assigned coalition (i), i..., (S)) (=(()) for all i e S. A coalition structure is core stable (or simply stable) if no blocking coalitions exist.
al. 2002) is key cooperative solution
Enumerating agents' preferences over all coalitions is infeasible; with n agents, each agent needs to express their preferences over 2 potential groups. Sliwinski & Zick (2017) propose using Probably Approximately Correct (PAC) guarantees (Kearns & Vazirani 1995 Shashua 2009). The key insight of this framework is to sample players' preferences rather than utilize complete preferences over all coalitions. Given a distribution D over coalitions, a coalition structure called e-PAC stable
Pr [S core blocks #] $
< E.
httpss//ggtiibb.com//aa/eaaaae/ cNeurons
2

---

Published as a conference paper at ICLR 2026
Here, D is the distribution over sampled coalitions used to approximate neuron preferences. Intu- itively, while it is possible that स is not core stable, the probability of observing a blocking coalition for â under the distribution D is small.
A PAC stabilization algorithm takes m = (n, 1 log })) samples from D and outputs an E-PAC stable partition with probability at least 1 - 6. Intuitively, 5 captures the probability that the m samples we took are not representative of the 'true' data distribution D.
Top-Responsive Hedonic Games in Neural Networks. We estimate pairwise affinities dij between "players" (neurons) from weights and co-activations, These affinities allow us to construct a hedonic game in which each neuron evaluates coalitions based on the presence of preferred partners. To capture this behavior, we model the setting as a top-responsive game. In top-responsive game, every player associates each coalition S i with a unique choice set ch(i, S) C that represents the subset of partners most important to i. Preferences are then determined entirely by these choice sets: a player prefers one coalition over another its choice set ranked higher, and if two coalitions yield the same choice set, the smaller coalition is favored. This restriction makes coalition evaluation tractable, as each neuron only needs to consider its most valued partners rather than all possible groups.
The top responsive framework is flexible, as choice sets may consist a single strong partner, multiple valued partners, even subsets selected according to synergy between members. The key requirement is that choice sets are uniquely defined and utilities are represented in an informative way, so that distinct choice sets correspond distinct utility "buckets". Under these conditions, the Top-Covering algorithm (Alcalde & Revilla 2004 Dimitrov & Sung. 2007) can be applied to efficiently compute (e, 8) PAC-stable partition (Sliwinski & Zick 2017). This enables to identify groups of neurons that form stable coalitions under the distribution of observed samples. Further details and extensions are provided in Appendix A.I
2.2 TRANSFORMER MLPs AND LATENT FEATURE FORMATION
Each LLM transformer block contains a gated MLP that expands the hidden state, applies non- linearity, and then projects back to the model dimension. Let the hidden vector entering the MLP at layer& l be h E Roassall and let du > dmmudel denote the intermediate width. In LLaMA-3-style architectures (Dubey al. 2024), the computation proceeds as:
Zup
Wap Wap Egaiee
Wg = SiL.U Egate 1 Zap:
W Wdownf..
where Wap- W Wgate E RRda and Wewn E dadde x da_ The element-wise product ğ binds the gate signal which selects or suppresses coarse abstractions with the up signal that carries candidate feature directions. Waown then recombines these activated features. During this process, abstract features may emerge (via new activation directions), merge (when multiple features co-activate), split (when previously unified features diverge), or disappear (if suppressed by gating) (Elhage et al. 2021
Tia
2023).
LoRA-adapted projections. In our setup, only the MLP projection matrices are fine-tuned using Low-Rank Adaptation (LoRA) (Hu al. 2022). For any weight matrix W E m=x. LoRA introduces low-rank update of the form:
W = W AWW
AW "AB"
where A E R and B E Raxr are the learned parameters, is the rank, and a is scaling factor. When applied Wap and W Wgates we obtain:
Zyp (Wap + AW.,), Eyue (Wgute + AW AW )..
The updates AWap and AW gate are low-rank, as a result, they introduce only a small set of new feature directions in the high-dimensional MLP space. But because these directions are diffused across neurons, visual inspection of weight updates reveals no obvious structure cecturre--eaddng to our central question: which subsets of neurons cooperate encode task-specift behavior under LoRA?
In Section 3 we develop a game- theoretic framework that directly identifies these functional coali- tions, revealing how LoRA's parameter-efficient updates create localized but coordinated changes
3

---

Published as a conference paper at ICLR 2026
that encode task-relevant abstractions without requiring exhaustive analysis of all possible neuron combinations.
3 METHODOLOGY
We present a game-theoretic framework to identify and track latent coalitions-coopetee neurons within MLP layers of LoRA-tuned transformer models. Our approach consists of two stages: first, we formalize the intra-layer coalition discovery as game with hedonic utilities and apply the PAC-Top-Cowe algorithm find stable neuron groupings; second, we connect these coalitions across layers using maximum-weiigh bipartite matching to trace how these abstract computational units evolve through consecutive layers in the network hierarchy.
groups of
3.1 PROBLEM STATEMENT: COALITION DISCOVERY AND TRACKING IN TRANSFORMER MLPs
Let L be a transformer-based LLM fine-tuned for a scalar prediction task (e.g., relevance scoring) via LoRA. Let & {1, 2, ..... d) denote MLP layer in the network, where is the total number of layers, with n = da neurons in its intermediate dimension. Denote the down-projection weight matrix as w(f) down R x where each column [W: represents the learned projection vector for neuron i. Here neuron refers the 1th MLP channel in da-
Our goal is to identify a partition π (2) = {C1, C2...... C of neurons in layer such that each subset C; S (1....,7) n captures
behavior in forming a semantic unit. We define synergy through a pairwise valuation function Φij- Then, across layers, we aim to match coalitions from x() to those in ((+1)) enabling us to model feature persistence, splitting, merging, and other dynamic events.
of neurons that exhibit strong synergy-coopene
3.2 CONSTRUCTING PAIRWISE VALUATIONS AND UTILITY SCORES
PAC-Top-Cover uses samples of coalitions S ~ D to estimate each agent's 's top-k choice set within the remaining pool. We first compute pairwise valuations Φij. which quantify affinity or synergy between neurons. We instantiate complementary valuation functions:
(OCA). This approach combines two intuitions: neurons with orthogonal weight vectors may capture complementary features, while neurons with high activation correlation may process similar patterns. For neuron pair (i,j), we define:
Orthogonal-Co-Actino
ΦoCA (i.j) (1 |cos (W W2) aç);, p(as,a)) Covla;, ay] 0:03
where W; is the i-th column of W(O activations. The cosine term favors pairs with dissimilar weight vectors, while the correlation term captures their collaborative activation patterns (Pearson's correlation).
down (neuron v's output weights), and ag denotes neuron i's
Pairwise Ablation Synergy (PAS). To directly measure the synergistic interaction between neurons i and j. we compute the second-order interaction effect through ablation. Let {(x) denote the model's logit outpuf" for input and (x) denote the logit when neurons in set S are ablated (set to their pre-LoRA weight). The true interaction between neurons and j ÓpAS ( j), where:
- (y := (t)) (100))))
(i. E2 (z())(())
"We use layer-local logit (x) w (x) ie. the scalar score obtained immediately after the layer-& MLP (including residual addition) but before entering block é 1. Our goal is to discover coalitions that are intrinsically synergistic the point they are formed. b are cloned from the final task head and kept fixed for all layers. For readability drop the superscript when the layer is clear from context.
4

---

Published as a conference paper at ICLR 2026
(PA)) indicates that jointly ablating (i,j) changes the layer-local score more than additively (synergy).
This measures how the joint ablation of both neurons differs from the sum of individual ablations.
For computational efficiency with large n, we approximate this using second-order derivatives:
(l(()) Da, da,
PAS (i,5) RJ Eg
(x) Aag(u)) Aa,(x) = de doRA (x) 0i
base (x)
where the mixed partial derivative captures the interaction between neuron activations.
We experiment with both OCA (structural heuristic) and PAS (functional ablation-based) valuations to test robustness of our framework. In both the above defined pairwise valuation functions, positive values indicate synergy (neurons cooperate to produce information neither could alone), while negative values indicate redundancy (neurons provide overlapping information). We now use these valuation functions, to compute choice sets, which is used to compute the utility of a neuron a set that is used by the PAC Top-Cover algorithm.
Multi- Friendd Choice Sets (MFC). this formation, each neuron is allowed to anchor preference not a single partner but a set of top-k partners. For player i, the choice set within coalition S is
Σ Oij. Ch(i, S) arg max
TCSME) TTE
JET
with ties broken deterministically to ensure uniqueness. Utilities are then defined as uz(S) = EjEck(i.)) Piy. This normalized model captures multi-partmer synergy, where a neuron's activa- tion is meaningful only when several complementary features are present. We refer this algorithmic instantiation as Hedonic-MFC.
3.3 THE PAC TOP-COVER ALGORITHM.
We compute a PAC-stable partition using the AAC-Top-Cover procedure (Sliwinski & Zick 2017). instantiated with our MFC utilities from $3. All PAC-stability statements in this section are with respect to a distribution D over candidate coalitions S C N induced by a coalition sampler.
Coalition sampling distribution D. In practice, implement D by repeatedly drawing subset size 8 ~ Unif Emmin.... kmax} and then sampling a coalition C N uniformly at random among subsets of size We draw m i.i.d. samples ((1), ..... ((m) D per round.
Estimating best sampled coalitions and choice sets. For each player E N, we retain the best sampled coalition containing under the MFC utility:
T E arg
max us((()),
(2)
te(m] iESO
breaking ties deterministically, We then estimate i's (top-k) choice set from T
B, = TOPK,(T)) C T7 \ (i),
(3)
where (TPK.(T) returns the k elements of T' {i) with the largest dy values (or all available elements if [T') {i)l < k).
Preference digraph and closed sink components. We construct directed graph G (N, E) with edges
(i.j) - j Br.
Intuitively, points to the partners it most prefers among its best sampled coalition. PAC-Top-Cover outputs coalitions corresponding to sink strongly-connecter components (SCCs) that are closed under these estimated choice sets. Concretely, compute the SCC decomposition of G and select sink SCCs C satisfying the closure condition
ViEC, B; C.
(5)
OOtee proposal distributions can be substituted without changing the algorithm below; 1) is simply the induced distribution of the chosen sampler.
5

---

Published as a conference paper at ICLR 2026
Each such set C' is emitted as coalition.
Iterative removal. After outputting a coalition C, we remove its vertices from the remaining set of players and repeat the above procedure the residual set until all players are assigned to coalition.
The algorithm detailed Appendix C With m = poly(n, 1/e, log(1/8) i.id. samples per round from D, PAC-Top- -op-Cor- Cover returns partition that is e e-approximately stable with probability at least 1- 6 (with respect to D). This provides theoretical support that the discovered coalitions reflect robust cooperative structure among neurons under the sampling distribution. We next ask how the coalitions identified at one layer relate to those in subsequent layers.
3.4 TRACKING COALITIONS ACROSS LAYERS
Our hypothesis is that coalitions capture intermediate features that may persist, merge, split, or disappear as computation proceeds through the network. Tracking these transitions provides exploratory view of how features evolve with depth.
For each pair of coalitions (C, C") from consecutive layers & and & 1, we measure their interaction mass, a heuristic that quantifies how strongly one coalition influences the next:
M(C,C") 1C
ΣΣ PEC DEC
(wwltt(o up (q. pg + (w(et gate
(q. Ap-
where W((++) wl(++) E Reeeddane are the LoRA-adapted projection matrices of layer é + 1, (1....,d, d indexes target neurons in layer E+1, and p {l,..., dmodel) indexes residual- stream dimensions feeding into layer 2+11 This formulation captures both the additive (Wup) and multiplicative gating (W gate x SiLU) pathways, while normalizing by coalition sizes ensures comparability across widths.
We assemble the interaction masses into bipartite matrix and solve maximum- matching problem to align coalitions across layers. For each match, we compute the fraction of a source coalition's 's output that flows into target (a)) and the fraction of target's 's input originating from that source (3). These ratios allow us to classify transitions into persistence (both high), splitting (low a, high 3), merging (high a, low 3), or disappearance (both low).
We stress that this analysis is exploratory. Transformers have residual connections, so neurons at layer l influence all deeper layers, not just + 1. Our method only captures local dynamics and likely underestimates long-range interactions, but it offers a first step toward visualizing how abstract feature groups may evolve through the network.
EXPERIMENTS
We empirically validate our framework on three LLM architectures and three scalar-output IR tasks. We first describe models, tasks, and baselines, then present evaluation protocols and results (Tables Appendix Cb.
Models. study LLaMA-3., 3.14--3.114 1-8B (Dubey al. (024). Mistral-7B-v0.1 (Jiang et al. 2023 and
Pythia-6.9B (Biderman et al. 2023). each adapted via LoRA (rank = 8) restricted to MLP layers 7-14. Preliminary analysis showed these layers carry the strongest task-specific activity (Nijasure et al., 2025). Fine-tuning uses AdamW (1 2 x 10 batch size 128, 3 epochs), with all base weights frozen. Performance of these fine-tuned LoRA models is further documented in Appendix
Tasks. Tasks scalar objectives defined over query document pairs from MS MARCO (Bajaj et al. 2018): (1) Covered- Livered-Query-Fee Query-Term Ratio (CQTR) = fraction of query terms present in the document, (2) Mean of Stream-Length Normalized Term Frequency (Mean-TF/L) mean of length- normalized term frequencies, (3) Relevance Modelling (RM) = =sueervised passage ranking. CQTR
'In practice, aggregate each source coalition C's outgoing contribution onto the residual stream before measuring its influence on targets in layer ê+1; A, T2 [1bg" [ activation dimension p over the training distribution.
(x)]] denotes the mean absolute residual

---

Published as a conference paper at ICLR 2026
and Mean-TF/L use MSE loss, RM uses NDCG. Models are trained 500k pairs, validated on 5k, and evaluated OOD on TREC DL-19/20.
Baselines. We compare five coalition builders: Random (uniform neuron subsets with matched size histogram), K-means (Spherical) (on (z-normalized mean activations, & matched to Hedonic partition), Ifierarchical (Ward+cos) (agglomerative under cosine distance, cut at same k), Hedonic- OCA (PAC-Top-Cover with Φocx), Hedonic-PAL (PAC-Top-Cover with épAS).
For Hedonic sampling we draw m 8x 10 candidate coalitions (size [2, 10), retain top 8x 10 by utility, and use = = 0.1. Choice sets use top-3 partners. Cross-layer matching uses thresholds (Ohi, (o (0.7, 0.1) tuned on 1% held-out split. All methods run on 4 A100-80GB GPUs; PAC-Top-Cover completes in 90 min (OCA) and 280 min (PAS). All numbers are averaged over seeds with 95% confidence intervals.
Evaluation. We first report intrinsic synergy metrics (Appendix G as diagnostics, then evaluate coalitions extrinsically with three tests:
OOD Drop. For coalition C, we measure the performance drop DOOD (DII-19/20) when C is ablated (neurons reset to pre-LoRA weights):
AM(C) = M({((=))) - Mite-c(z)),
where M is NDCG@ 10 for RM and MSE for COTR/Mean-TE/I. Larger AM(C) indicates greater functional importance.
Feature Aligument. Each coalition's mean activation ac(x) is compared with known IR heuristics
(list of MSLR features Qin & Liu 20135)). Alignment is defined as the maximum squared Pearson
correlation:
R²(C) max (or((oc(=),
Coalition Predictivity. Coalitions are treated as macro-features A (xx) E IR.. A ridge regression ((x) w T A(x) is trained on MS MARCO and evaluated OOD; we report R² for RM, CQTR, and McAn-TF/L.
Next, we discuss the results reported in Tables (extrinsic coalition evaluation), Table Cooal-- tion predictivity), Table (coalition transfer dynamics) and Appendix Table (intrinsic coalition evaluation).
Experimental Results.
Functional importance and interpretability [T. Across all three models and tasks, hedonic coalitions are markedly more causal and interpretable than clustering or random partitions. Ablating single hedonic coalition (ablation = restoring those neurons to their Pre-LoRA state) yields the largest OOD performance drops: for CQTR on LLaMAMMistraa/Ptthiii the OOD drop rises from ~0..2-0.03 (K- (K-means/Hier.) to 0. 11-0.14 44 with Hedonic-PAs hold for Mean-TFFL. (0. 10-0.11 vs. 0.02-0.03) and RM (0. 10.1400.17 vs. 0.03-0.04). At the same time, coalition activations align far more strongly with IR heuristics: alignment R² climbs from ~0.1 (clustering) to 0.55-0.67 (Medonic-PAS). with Hedonic-OCA consistently second-best (2x0.38-0.44 -.44). Confidence intervals are narrow throughout, indicating stable estimates over seeds. Taken together, these results show that hedonic coalitions are both functionally indispensable-theit removal produces large OOD degradation and semantically grounded, tracking BM25/IDF/coverage signals far better than baselines.
about a 3-5x increase; similar gaps
Predictive macro-features (Table Treating each coalition as macro-feature and training ridge regressor on MS MARCO, we see large generalization gains on 19/19/20. Averaged over LLLMA,, Mistral, and Pythia, Hedonic-PAS attains It2 0.43/0.42/0.47 on DOTTR/Mean-TF/L/RR 2-3x higher than K-means/Hier. (2=0.15-0.21 and far above Random (ss0.08-0.1 12). Hedonic-OCA also performs strongly (==0.33-01 33-3.88), reinforeing the pattern from the extrinsic ablations: utilities that respect synergy (PAS) or partner preference (OCA) produce coalitions that behave like robust, transferable features, not just co-activation clusters. This bridges intrinsic synergy to downstream utility: coalitions that score high synergy also yield higher OOD predictivity.
roughly
7

---

Published as a conference paper at ICLR 2026
Table 1: Extrinsic Evaluation: OOD Drop (f) and Feature Alignment R² (1) on DL--19/00. Mean 195% CI across three seeds. Larger values indicate more functionally important and interpretable coalitions.
LLMM--3.1
Mistral
Pythia
Task Algorithm
OOD Drop
Aliga It?
OOD Drop
Align It?
OOD Drop
Align It
Covered Query Term Ratio
Random
0.01 #0.01
0.05+0.02
0.00
0.06+
0.01
0.05
K-means
0.02
0.12+0.02
0.03
0.13+0.02
0.02+0.01
0.11
Hier. clustering
0.03
0.15:10.02
0.03
0.16:10.02
0.03
0.14
Hedonic (OCA)
0.07
0.41
0.09
0.41
0.08
0.05
Hedonic (PAS)
0.105
0.58
0.10:10.006
0.61
0.14
0.63
Mean Normalized Term Frequency
Random
0.01 +0001
0.01+0.02
0.01 +0001
0.05++0.02
0.01
0.05
K-means
0.02++0.01
0.11
0.02+0.01
0.12+0002
0.02+00.01
0.10+0.02
Hier. clustering
0.03
0.14+0.02
0.03
0.15::0.2
0.03
0.13
Hedonic (OCA)
0.06
0.38
0.08
0.41
0.07
0.40
Hedonic (PAS)
0.10=10.005
0.55
0.12:100006
0.59
0.13
0.60
Relevance Modelling
Random
0.02110.01
0.06
0.0110001
0.07
0.02
0.06110.02
K- K-means
0.03
0.13+0.02
0.04
0.14+0.02
0.03
0.13#0.02
Hier. clustering
0.04
0.17
0.04
0.18+0.02
0.04
0.16=0.02
Hedonic (OCA)
0.09
0.47
0.10
0.49+0.02
0.09
0.48=0.02
Hedonic (PAS)
0.14
0.63
0.16
0.65
0.17
0.67
Table 2: Coalition Predictivity (R² on OOD sets DL-19/20), averaged across three LLMs (LLaMA- 3.1, Mistral, Pythia). Coalitions used as macro- features in ridge regression trained on MS MARCO. Hedonic coalitions yield substantially higher ft2 than clustering or random baselines.
Algorithm
CQTR
Mcan-TF/L
Relevance (RM)
Random
0.08
0.09
0.12
K-means
0.16
0.15
0.21
Hier. clustering
0.18+10.01
0.17+0.01
0.21
Hedonic (OCA)
0.31
0.33
0.38++0.01
Hedonic (PAS)
0.43--0.008
0.42
0.47=0.008
Coalition dynamics across depth (Table Across layers 7-+1,, three trends are consistent: (i) vanish dominates (lypically 60-75% of coalitions disappear at the next layer), indicating downstream MLPs act as filters/refiners rather than combiners; (ii) splits common (=20-5009 depending on task/layer), suggesting feature refinement is more prevalent than wholesale reuse; and (iii) merges near-zero, implying whole motifs rarely recomposed from separate groups. Persistence is generally low 12%) with a mild delayed persistence uptick around 12- 13 for CQTR and RM (s:8-11%) echoing "late stabilization" phase. Mean-TF/L. exhibits the strongest pruning (vanish >70% across several transitions), consistent with simple frequency statistics being isolated early and aggressively culled later. These dynamics support our central claim: cooperative units are formed, then predominantly pruned or refined rather than fused, aligning with the heavy-tailed coalition sizes and the functional importance patterns observed above.
DISCUSSION
SAEs vs Hedonic Neurons. Sparse Autoencoders (SAEs) (Huben et al. 2024) uncover interpretable features by learning sparse dictionaries that reconstruct activations and disentangle polysemantic units. In contrast, our framework keeps neurons as primitives and asks how they cooperate. By modeling them agents in hedonic game, capture nonlinear synergies: coalitions whose joint ablation impacts behavior beyond the sum of parts. Unlike SAEs, which re-express activation space, hedonic coalitions grounded in weight geometry and preference structure, surfacing cooperative
8

---

Published as a conference paper at ICLR 2026
Table 3: Dynamics of coalitions across layers 7-14 for three tasks. Each cell shows percentage of coalitions exhibiting the event relative to all coalitions present in the source layer (except merge).
Mistral
LLaMA
Pythia
Layer
Persist
Merge
Split
Vanish
Persist
Merge
Split
Vanish
Persist
Merge
Split
Vanish
Covered
Query Term Ratio
7 + 8
12.1%
0.0%
28.9%
59.0%
3.2%
0.0%
35.4%
61.4%
7.8%
0.0%
31.9%
60.3%
8 + 9
4.8%
0.0%
38.4%
56.8%
5.1%
0.0%
28.6%
66.3%
4.9%
0.0%
33.7%
61.4%
9 9- 10
6.2%
0.0%%
31.5%
62.3%
4.8%
0.0%
30.2%
65.0%
5.5%
0.0%
30.8%
63.7%
10 11
3.8%
0.0%%
29.7%
66.5%
7.9%
0.0%
32.1%
60.0%
5.9%
0.0%
30.9%
63.2%
11 + 12
4.2%
0.0%
27.8
68. 08.0%
3.5%
0.0%
29.8%
66.7%
3.9%
0.0%
28. 8%
67.3%
12 > 13
11.3%
0.0%
30.2%
58.5% 5%
6.8%
0.0%
27. 4%
65 8%
9. 9.1%
0.0%
28. 8%
62,15 12.1%
13 > 14
10.5%
0.0%
24.3%
65.2%
7.2%
0.0%
23.1%
69.7%
8.9%
0.0%
23.7%
67.4 4%%%
Stream Length Normalized Term Frequency
7 8 6.4%
0.5%
35.8%
57.3%
2.1%
0.0%
19.7%
78.2%
4.3%
0.3%
28.4%
67.0%
9
1.8%
0.2%
51.2%
46.8% 8%
3,8%
0.1%
20.3%
75 8%
2.8%
0.1%
36.2%
60.9%
9- 10
2.9%
0.1%
23.1%
73.9%
3,4%
0.0%
22.8%
73.55 73.8%
3.2%
0.0%
22.9%
73.9%
10 11
1.3%
0.3%
23.7%
74.7%
5.9%
0.2%
25.5%
68.4%
3.6%
0.2%
24.6%
71.6%
11 - 12
1.2%
0.1%
22.9%
75.8%
1.7%
0.0%
24.1%
74.2%
1.4%
0.0%
23.5%
75.1%
12 13
6.3%
0.4%
37.8%
55.5%
3.2%
0.1%
19.8%
76.9%
4.8%
0.2%
29.3%
65.7%
13 14
7.1%
0.2%
19.5%
73.2%
4.7%
0.0%
17.1%
78.2%
5.9%
0.1%
18.3%
75.7%
Relevance
7
8.2%
32.7%
59.25%
0.0%
5.2%
0.0%
28.1
66.7%
8
2.1%
0.0%
46.8%
51.1%
0.0%
22.8%
73.7%
2.8%
0.0%
35.3%
61.9%
10 10
3.5%
0.0%
26.3%
70.2%
0.0%
25.5%
70.6%
3.7%
0.0%
25.9%
70.4%
2.0%
0.0%
26.5%
71.4%
0.0%
65.0%
4.2%
68.4%
0.0%
25.4%
72.9%
0.0%
71.4%
1.9%
25.9%
72.2%
12 13
8.0%
34.0%
58.0%
0.0%
22.0%
74.0%
28.2%
65.7%
13 -
8.7%
0.0%
21.7%
69.6%
0.0%
18.9%
75.7%
7.1%
20.3%
72.6%
"wiring-level" units already encoded in the parameters. The two approaches are complementary: SAEs expose monosemantic features, while hedonic analysis highlights how neurons collaborate to realize them.
Coalition size distribution. Coalition sizes follow a heavy-tailed Zipfian law: each layer contains few large "macro" groups, mid-sized units, and many size-2 specialists, resembling vocabulary statistics in language. Disappearance rates rise after layer 12, suggesting deeper MLP blocks act more as feature filters than creators. Together, these findings imply that hedonic coalitions natural computational units shaped by training dynamics- early layers construct rich representations, while later ones selectively retain task-relevant features.
6 RELATED WORK
Mechanistic interpretability of transformer LLMs has focused on understanding both individual neurons and structured groups. Geva et al. (202I) showed that feed-forward layers act as key-value memories, with neurons detecting ipput patterns (keys) and injecting values into the representa- tion. Dai et al. (2022) identified "knowledge neurons" in MLPs that encode factual associations, demonstrating that small groups of neurons can robustly store discrete knowledge.
Beyond single-neuron analysis, [Bricenn al. (2023) applied dictionary learning to extract sparse, interpretable features from polysemantic activations. (Balagansky et al. (2025) tracked feature persistence and merging across layers, complementing our coalition-evolution view. Sparse prob- ing (Gurnee al. 2023) further revealed that early layers are highly polysemantic while deeper layers specialize, underscoring the need to model neuron groups and their dynamics. Weight-basee based methods also contribute: DDvies (2025) decoded neuron weights into semantic concepts, while Pearce et al. (2024) and Bushnaq et al. (2025) developed direct weight-space feature discovery.
While hedonic games have rarely been explored in interpretability. Koulali and Koulali (Koulali & Koulali (023) showed their utility for feature selection, providing theoretical foundations for our approach. Our work extends these lines by explicitly framing neuron collaboration a hedonic game,

---

Published as a conference paper at ICLR 2026
enabling principled discovery and tracking of stable coalitions that serve as latent computational units in transformer MLPs.
7 CONCLUSION, LIMITATIONS AND FUTURE WORK
We introduced Hedonic Neurons, game-theoretic framework that models neurons in transformer MLPs as players in top-responsive hedonic game. Using the PAC-Top-Cover algorithm with correlation- based (OCA) or ablation-based (PAS) valuations, we identified stable coalitions that capture cooperative structure beyond what clustering reveal. Across three LLM architectures and scalar IR tasks, hedonic coalitions achieve average improvements of +0.29 Pairwise and +0.49 Ratio synergy over the strongest baseline, while extrinsic evaluations show they are functionally indispensable: ablations yield 3-5x larger OOD performance drops, alignment with IR heuristics rises from ~0.15 0.55-0.67, and predictive R improves from ~0.20 0.43-0.47. Coalition dynamics further reveal that most groups vamish split across depth, with merges rare and persistence limited, supporting the view that MLPs act primarily as filters and refiners of features.
Our approach has limitations: utilities depend on layer-local logits and second-order ablations, omitting higher-order interactions and attention mechanisms, and the current formulation yields disjoint coalitions despite early-layer polysemy. Future work will extend to overlapping coalitions via fractional hedonic games, integrate attention heads for joint sub-module analysis, and design low-variance estimators to reduce O(n²) ablation costs. Coupling hedonic discovery with concept- activation vectors may also yield interpretable primitives aligned with coman-understandabbe features. Taken together, HedonicNeurons provides principled foundation for uncovering how cooperative computational units emerge, evolve, and specialize in large-scale language models.
REPRODUCIBILITY STATEMENT
We provide all resources necessary to reproduce our experiments. We make our fine-tuned reranker checkpoints for Pythia, Mistral, and LLaMA3 models available HuggingFace. The training dataset (Tevatron MSMARCO Passage Augmented) and evaluation dataset (TREC DL 2019) are publicly available, with preprocessing steps following the Tevatron MSMARCO implementation. All scripts used for coalition generation, partitioning, clustering baselines, and evaluation are included in the repository, along with deepspeed configuration files for finetuning. Coalition files (.pkl) and visualization outputs (Sankey plots) are also provided.
ACKNOWLEDGMENTS
This work was supported in part by the Center for Intelligent Information Retrieval, University of Massachusetts Amherst. Any opinions, findings and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.
REFERENCES
José Alcalde and Pablo Revilla. Researching with whom? stability and manipulation. Journal of Mathematical Economics, Vol 40(Issue 8):pp. 869-887, 2004.
Haris Aziz and Rahul Savani. Hedonic games. In Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D. Procaccia (eds.), Handbook of Computational Social Choice, chapter 15. Cambridge University Press, 2016.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song. Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset, 2018.
Nikita Balagansky, Ian Maksimov, and Daniil Gavrilov. Mechanistic permutability: Match features across layers. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
10

---

Published as a conference paper at ICLR 2026
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML), pp. 2397-2430, 2023.
Anna Bogomolnaia, and Matthew O. Jackson. The stability of hedonic coalition structures. Games and Economic Behavior, Vol 38(Issue 2):pp. 201-230, 2002.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, al. Towards monosemanticity: Decompos- ing language models with dictionary learning. 2023.
Lucius Bushnaq, Dan Braun, and Lee Sharkey. Stochastic parameter decomposition. arXiv preprint arXiv:2506.20790, 2025.
Tue Minh Cao, Nhat Hoang-Xuss -uan, Hieu Pham, Phi Le Nguyen, and My T. Thai. Neurflow: Inter- preting neural networks through neuron groups and functional interactions. The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Tanya Chowdhury, Atharva Nijasure, and James Allan. Probing ranking llms: A mechanistic analysis for information retrieval. In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR), pp. 336-346, 2025.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the trec 2019 deep learning track. In arXiv, doi: ArXiv:2003.07820, 2020.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
pp.
Harry J Davies. Decoding specialised feature neurons in ilms with the final projection layer. In arXiv, doi ArXiv:2501.02688 2025.
Dinko Dimitrov and Shao Chin Sung. On top responsiveness and strict core stability. Journal of Mathematical Economics, Vol 43(sssue 2):pp. 130-134, 2007.
Jacques H Dreze and Joseph Greenberg. Hedonic coalitions: Optimality and stability. Econometrica: Journal of the Econometric Society, pp. 987-1003, 1980.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. In arXiv, doi: ArXiv:2407.21783, 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Hemighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, Vol 1(1ssue 1)ppp. 12, 2021.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Haffield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. In arXiv, doi ArXiv:2209.1063 2022.
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan, Scaling deep contrastive learning batch size under memory limited setup. In Proceedings of the 6th Workshop on Representation Leurning for NLP, 2021.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5484-5495, 2021.
Wes Gumnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research (TMLR), 2023, ISSN 2835-8856.

---

Published as a conference paper at ICLR 2026
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pp. 1129-4118, 2019.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and
Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Michael J Kearns and Umesh Vazirani. Computational learning theory, Association for Computing Machinery (ACM) SIGACT News, Vol 26(Issue 1):pp. 43-45, 1995.
Rim Koulali and Mohammed-Amine Koulali. Feature selection hedonic coalition formation game for arabic topic detection. Pattern Recognition Letters, Vol 172:137-143 2023.
Victor Lavrenko and W. Bruce Croft. Relevance based language models. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 120-127. 127, 2001.
Xueguang Ma. Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning Ilama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 2421-2422 2024a.
Xueguang Ma. Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development Information Retrieval (SIGIR), pp. 2421-2425. Association for Computing Machinery, 2024b.
Xueguang Ma, Luyu Gao, Shengyao Zhuang, Jiaqi Samantha Zhan, Jamie Callan, and Jimmy Lin. Tevatron 2.0: Unified document retrieval toolkit across scale, language, and modality. In arXiv, doi ArXiv:2505.02466 2025.
Atharva Nijasure, Tanya Chowdhury, and James Allan. How relevance emerges: Interpreting lora fine-tuning in reranking lims. In arXiv, doi: ArXi::2504.08880, 2025.
Michael T Pearce, Thomas Dooms, Alice Rigg. Jose M Oramas, and Lee Sharkey. Bilinear mips enable weight-based mechanistic interpretability. arXiv preprint arXiv:2410.08417,, 2024.
Tao Qin and Tie-Yaa Yan Liu. Introducing LETOR 4.0 datasets. CoRR, Vol 1bs/1306.259,, 2013a.
Tao Qin and Tie- Yan Liu. Introducing letor 4.0 datasets. arXiv preprint
2013b.
Amnon Shashua. Introduction to machine learning: Class notes 67577. In arXiv, doi: ArXiv:0004.3664,22 2009.
Jakub Sliwinski and Yair Zick. Learning hedonic games. Proceedings of the Twenty- -ixth International Joint Conference Artificial Intelligence, (IJCAI-17), pp. 2730-2736, 2017.
Ran Song, Shizhu He, Shuting Jiang, Yantuan Xian, Shengxiang Gao, Kang Liu, and Zhengtao Yu. Does large language model contain task-specific neurons? In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 7101-7113, 2024.
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, Vol 28(Issue 1)ppp. 11-21, 1972.
Tevatron. Ms marco augmented dataset, 2024.
12

---

Published as a conference paper at ICLR 2026
Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Shaolei Du. Scan and snap: Understanding training dynamics and token composition I-layer transformer. Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
Xiongtao Zhou, Jie He, Yuhua Ke, Guangyao Zhu, Victor Gutierrez Basulto, and Jeff Pan. An empirical study on parameter-efficient fine-tuning for multimodal large language models. In Findings of the Association for Computational Linguistics: (ACL), pp. 10057-10084, 2024.
13

---

Published as a conference paper at ICLR 2026
A HEDONIC GAMES PRELIMINARIES AND PAC TOP COVER INTUITION
A.1 HEDONIC GAMES
hedonic game (Dreze & Greenberg 1980) defined by finite set of players N = {1, ..... n and, for each player i, a complete and transitive preference relation >; over the Se; = [5 N i S} of coalitions that contain i. A coalition structure (or partition) is set π {Ci...., C of disjoint non-empty coalitions whose union equals N. Throughout this appendix we assume that preferences are given by valued utilities U :, - R that S > T (;(S) > (T)
A.2 CORE STABILITY
Given partition and coalition S S N, we say that S blocks if every E S strictly prefers S to her coalition म, i.e. S > ((). A partition is core-stable (or simply in the core) if it is not blocked by any coalition. Core stability captures the idea that no subset of players has joint incentive to deviate.
A.3 WIIY FULL PREFERENCE LEARNING IS INFEASIBLE
Precisely learning all utilities (S) is unrealistic because the number of coalitions grows exponen- tially ([Wi) 2n Even if we could query any coalition, the sample complexity implied by the pseudo-dimension of general hedonic games is super-polynomial (Proposition 4.9 in (Sliwinski & Zick (017)). Hence, any practical method must settle for approximate stability based on samples rather than complete preference elicitation.
A.4 PAC-LEARNING FRAMEWORK FOR HEDONIC GAMES
Following (Sliwinski & Zick 2017), let D be an unknown but fixed distribution over coalitions. A partition π is e-PAC stable under
Pr S blocks π| < E. S-17
An algorithm A PAC-stabilises class H. of hedonic games if, for any game G H. distribution D, and parameters (s, 6), outputs- with probability at least 1 an E-PAC-stable partition using a number of samples polynomial in (n, 1/e, log(1/1))
A.5 INTUITION BEHIND THE TOP-COVER ALGORITHM
Under top responsive utilities (S) = Ejesy( wiy). players exhibit top-responsivenes:: their evaluation of coalition is determined by the "best" members plus size penalty (Alcalde & Revilla 2004). TOP-COVER exploits this property iteratively:
(i) using samples, approximate each player's most preferred subset within the current residual set,
iii) build directed edges from each player the members of that subset,
(iii) extract a strongly connected component of minimal size, form it as a coalition, and remove it,
(iv) repeat until all players assigned.
Each extracted coalition is unlikely to be blocked because every member already sees its best attainable partners within it with high probability.
A.6 APPLICATION OF HEDONIC GAMES TO NEURAL NETWORKS.
Neurons in a transformer predominantly interact with a limited set phers-those with highly correlated activations complementary weights. Treating neurons players whose utilities
See Section of Sliwinskki & Zick 2017) for an extensive discussion of numeric versus ordinal representa-
tions.
14

---

Published as a conference paper at ICLR 2026
derived from such local synergies fits the additive model naturally. Sampling mini-batches of log- its/activations supplies the coalitions needed by the PAC framework, letting us recover approximately stable neuron groups without exhaustively testing all neuron subsets.
15

---

Published as a conference paper at ICLR 2026
B MAKING ADDITIVE UTILITIES TOP-RESPONSIVE
Notation recap. For each ordered pair of distinct neurons (i.j) we have pairwise synergy score Øig IR (either (OCA or ÓPAS: see (32)). Write for the matrix with zeros on the diagonal.
B. FROM ADDITIVE SCORES TOP-RESPONSIVE PREFERENCES
Max-partner utility Fix global parameter k 2 1. For coalition $ SCN N that contains player i let
Topı(i,S) = arg max (CSV(1 ITiSk JET
Σ
Pij.
We define
(z(S)
Σ
Dijy and Ci(S) (i,
JETOP, (1,5)
When h this reduces to the familiar "best-friendd" utility u,(S) maxjes Oij-
Lemma B.1 (Top-responsiveness). For every player i the preference relation Z, induced by us is lop-responsive: for any two coalitions S, T that contain
C,(S) >, C,(T) = ST1T.
Proof. Let S, T contain i and assume (,()) >: C,))), ie. (((C(s)) (.()( Because u, is monotone in the sense that enlarging a set never decreases its utility we have (s(S) 2 u, (S)( and un (T) (T)) Hence (S) (T),,, S >
Lemma B.2 (Informative representation). Given the matrix & compute C.)S) (and therefore the induced ranking) in O(k (S)) time. Hence the utility representation is informative in the sense of Sliwinski and Sliwinkkk & Zick 2017).
Proof. TOPı (i, S) requires sorting most |S| real numbers {0ig Jes) the k largest can be found in the stated time using a partial-selection routine.
Theorem B.3 (Applicability of PAC-Top-Covery With utilities u, from Definition the induced hedonic game is top-responsive and informative. Consequently, Algorithm outputs an E- E-PAC-stable partition with probability å using m poly(n, 3, log }) samples, exactly as (Sliwinski & Zick
2017
Proof. Top-responsiveness follows from Lemma B.1 informativeness from Lemma B.2 The
PAC-stability guarantee therefore immediate corollary of Theorem 3.4 in Sliwinski & Zick
2017).
B.2 COALITION-LEVES VALUATION (FOR SAMPLING)
Algorithm I needs a scalar value for any sampled coalition We use the symmetric extension
((S)
Σ (.(S) SES
Σ Σ djj
J
iES (E.Sp.
Intuitively, (SS) averages how strongly each member is bonded to its k preferred partners within S. Plugging dOCA or dpas place of dhs yields the concrete scores used in our experiments. "Reservoir" sampling in line 4 of Algorithm] [ draws m subsets S with probability proportional to ((S), thereby prioritising high-synergy groups
"Adding partner can only increase the of k best partners leave unchanged.
16

---

Published as a conference paper at ICLR 2026
Algorithm PAC Top-Cover for Top-k Responsive Games (neurons)
Require: Φ E R
pairwise affinity; Φы =
k EEN
top-k choice size b reservoir size, per-round samples
m.
MINK, MAXK (E.6)
sampled coalition sizes b PAC guidance for m, w
(l, ++
2: S <- SAMPLECOALITINN(RR m, MINK, MAXK)
reservoir
Definition (top-k utiliiyy and choice in coalition).
For T. (T) = > {i}. Let (T) be the k indices in P.(T) with largest Pig (ties broken by smaller index); if (T)( k. take all.
Define (T)) A [T]] EjeToRK((T) Pij-
3:
while + do
4:
Swand first in S that satisfy T C R; remove them from S
5:
if [Syondd < w then
> refresh if reservoir depleted
6:
S S SAMPLECOALITIONS m, MINK, MAXK)
7:
end if
8:
for all
E do
9:
T: t- {T E Sound i T}
10:
T, = then
11:
B; + {i}
degenerate self-loop
12:
else
13:
T arg maxTET, uf (T(T) deterministic tie-break T's lexicographic index list B; + (T,K,
14:
b top-k choice set of i in T
15:
end if end for
16:
Build digraph G (R, E)) with edges (i + j) for all B, (and optional (i +
self-loops)
Let the set of sink strongly connected components of G
19:
(closure check) Keep only X E C such that vi E X B; C X
20:
Choose X E C (e.g., smallest by size then lexicographic)
b any sink closed SCC is valid
21:
U (X); R t R\X
22:
end while
23: return
C PAC TOP COVER ALGORITHM
17

---

Published as a conference paper at ICLR 2026
D INFORMATION RETRIEVAL PRELIMINARIES
Information Retrieval (IR) involves retrieving documents that are likely to be relevant to a user's information need, typically represented query. A fundamental IR task is to return ranked list of documents in descending order of (estimated) relevance. The quality of this ranking directly impacts the user experience in search engines, recommendation systems, and question answering applications.
D.1 RELEVANCE MODEL
Dense/Neural Re-ranker is a language model (like RankLLaMa (Ma et al., 2024a)) which takes query and text input and produces relevance score based on the similarity of the query to the provided text.
Relevance Modeling vs Classification Classification and relevance modeling are related but distinct approaches in information retrieval (IR). The term relevance model (Lavrenko & Croft] (001) refers to mechanism for estimating the likelihood of observing a particular word documents that are relevant to given information need or query. whereas classification assigns documents to predefined categories, such as relevant or non non-relevant.
Ranking Evaluation with NDCG
information retrieval, one commonly used metric to evaluate the effectiveness of ranking models the Normalized Discounted Cumulative Gain (NDCG), NDCG assesses the quality of ranked list by measuring the gain (or relevance) of documents based on their position in the list, giving higher weight to relevant documents that appear earlier. Formally, the Discounted Cumulative Gain (DCG) is computed as:
2rel. 1
DCG@k
log. 1)
where rel; is the graded relevance of the document at position i. The NDCG is then computed by normalizing DCG by the ideal DCG (IDCG), which is the DCG for the optimal ranking:
NDCG@k DCG@k
IDCG@k
NDCG scores range from to with indicating a perfect ranking. In the DL19 dataset, each query- document pair is labeled with relevance grade based on human annotations. These annotations are used compute the NDCG score for a re-ranked list of documents, allowing us to quantify the effectiveness of our rerankers in retrieving the most relevant content at the top of the list.
In our work, we use RankLLaMA, a LLaMA- reranking model trained to predict the relevance of document given query. The model takes as input formatted string:
"query: {query), passage: (passage)*
and outputs score between 0 and 1, indicating the estimated relevance. We follow the training procedure described in the LanALLaMA paper (Ma al. 2024a).
D.2 COVERED QUERY TERM RATIO (CQTR)
Covered Query Term Ratio (CQTR) is a lexical feature that measures the proportion of unique query terms found in the document(Qin Qin Liu 2013a). Formally:
Query Terms n Document Terms
CQTR
Quurry Terms
18

---

Published as a conference paper at ICLR 2026
D.3 MEAN TERM FREQUENCY PER DOCUMENT LENGTH (MTF/L)
Mean Term Frequency per Document Length (MTF/L) captures the average frequency of query
terms normalized by the document length(Qin & Liu 2013a). It computed as:
Eneq TF,(D)
MTF/DL
Length(D)
To simplify interpretability tasks (by trying to restrict polysemanticityEllagee et al. 2022 ), we fine-tuned models on CQTR and MTF/L prediction tasks, with the same input structure defined above. We do not claim that these two features are the most important for determining relevance; rather, they are easily understood signals that prior work has shown to implicitly present neural models (Chowdhury et al. 2025).
D.4 DATASETS
Datasets:
MS MARCO: large-scale dataset consisting of real anonymized web search queries paired with relevant passages. It is a standard benchmark for training and evaluating re-ranking models. our fine- -uning, we used a modified version of this dataset called MS MARCO Augmented (Tevatron 2024) (Ma et al. (025). which provides hard negatives from both
CoconDennerr(Gan et al. 2001 and BM255 V
DI.-19 (TREC Deep Learning Track 2019): Contains high-quality relevance annotations for subset of queries, commonly used for pero-shot and fine-tuned re-ranker evaluation. Craswell et al. provide more information and an overview of this dataset (Craswell et al. 2020
"More details at http:://miicoooof.gghhhh..oaaama
19

---

Published as a conference paper at ICLR 2026
E FEATURE/LAYER CHOICE
Previous interpretability studies have been conducted on dense re-rankers, where Chowdhury et al. found that using linear probing, several traditional IR features show high likelihood of being present in the forward pass activations of dense re-ranker mooel(Chowdhury et al. 2025 Further behavioral analysis by Nijasure al. observed that large language models (LLMs) tend learn velevance-relatee features primarily in MLP layers 5 to 14 of re-ranker architecttreesNjjauurc et al. 2025).
Motivated by these insights, we focused our probing and editing experiments on this layer range (7-14) of Re-Ranker models. Figure supports this choice: it shows R scores for predicting MSLR features across all layers of the RankLLaMA-
model using linear probing. Features like covered query term number, covered query term ratio, mean of stream length normalized term frequency, and variance of ff idf exhibit increasing prominence from the lower to mid layers. This trend might indicate that these layers are key to encoding relevance-relatee signals.
Probing
M: TT
Mit OF
Covered Number
Covered
MeariTTA
18 Scote
Sum
trupr
Sum Tea
Layer
Figure 1: Probing for statistical features from the MSLR dataset in RankLlama2-7b model. Here QT' stands for Query Term, TF stands for Term Frequency and and-/L stands for length normalized. The graph lines indicate the presence of particular feature along the layers of the LLM. Certain features like Min TF IDF show consistent presence across the layers. Other features like Covered QT Number, Covered QT Ratio, Mean( increasing prominence from the first layer to the last, ultimately playing an important role in making ranking decisions. Other MSLR features like Sum(TF/D Maz(TF/F and Sum TF * IDF show negative correlation with Raama decision making(Chowdnng Chowdhury et al. 2025).
and Variance TF TDF show
20

---

Published as a conference paper at ICLR 2026
F LLM PERFORMANCE EVALUATION
We used LoRA (rank 8) fine-tuning on MLP modules alone for all models described this section. We had access four A100 GPÜs, depending on availability. We used DeepSpeed's Stage configuration with the AdamW optimizer for fine-tuning all these models.
Two of the LLMs used in our experiments using Mcan Squared Error (MSE) as the loss function. The models were trained to predict statistical IR signals such as the Covered Query Term Ratio (CQTR) and the mean term frequency normalized by passage length (mean(TF/L.)), Following this fine-tuning, the models were evaluated on sampled subset of the DL.19 dataset. This evaluation set comprised 43 queries, each associated with 10 documents sampled from larger candidate pool of 200 documents per query, retrieved using the RepILLaMA retriever. This setup was designed to assess the models' ability learn and generalize statistical IR features relevant to document ranking. Table summarizes the finetuning results of the LLMs.
fine-tuned the MS MARCO dataset for 0.3 epochs
For fine-tumie fine-tuning the re-rankers, used the code provided in the Tevatron repository (Ma et al. 2024b). For more details, refer to the paper by (Ma et al. 2024a). Evaluation was conducted on the full DL19 dataset, with document ranking based the top 200 passages retrieved via the RepILLaMA retriever. Results for finetuned re-rankers is presented in the table
Base LLM
Target Feature
Base NDCG@10
NDCG @ 10 (Finetuned)
LLaMA3 (LaMM3(Dubey Dubey et al.
2024
Re- Re-Ranking
0.18
0.7497
Pythia Biderman et al.
2023
Re-Ranking
0.18
0.7521
Mistra al. 2023)
Re-Ranking
0.18
0.7570
Table 4: NDCG@ @ 10 evaluation DL.19 dataset, showing baseline post-finetuning performance. All models were fine-tuned on MS MARCO for epoch.
Base LLM
Function
MSE (Start)
MSE (Finetuned, 0.3 epoch)
LLaMA Dubey et al.
2024)
CQTR
3.88
0.52
Pythia Bideeman al.
2023
CQTR
1.84
0.05
Misn et al. 2023)
CQTR
36.92
10.94
LLaMA3 Dubey al.
2024
mean TF/L)
5.06
4.49
Pythiad Biderman et al.
2023
mean TF/L)
2.24
0.00
Mistral Jiang al. 2023)
mean(TF/L)
38.32
22.23
Table 5: MSE before and after finetuning (0.3 epochs) for CQTR and mean(TF/L) prediction tasks the sampled DL19 dataset.
21

---

Published as a conference paper at ICLR 2026
G INTRINSIC COALITION EVALUATION
Synergy Metrics. Let x be an input sampled from the task distribution and let {(x) E TR denote the layer-local logit (defined in )3.2) under the LoRA-tuned model. For any neuron set S, denote by (=)(x) the same forward pass after ablating all neurons k E S by reverting their LoRA parameters the Pre-LoRA (base) values, while leaving all other parameters unchanged
We define the marginal contribution of a single neuron as
(i) Ez D [((x) - (1)))
(6)
and the pairwise interaction (synergy) of two neurons as
(i.5) Ez-D(((a) -- (1) (2) (()))((=))
positive (ii means that removing both neurons changes the logit more than the sum of their
individual removals (synergy). while negative value indicates redundancy.
For coalition C' E [I,...,n. we report two size-agnostic aggregates:
Ratio(C) = Eissec (b.
Pair(C) = CI([C I)
Σ (i,j),, LJCC
(8)
Eiec vi()
Pairwise Synergy is the mean interaction strength across all ordered neuron pairs, fully normalized for coalition size, while Ratio Synergy compares the extra value created by pairwise cooperation (numerator) to the value explained by separate single-neuron effects (denominator). ratio near or greater (super-addirivity) indicates that the coalition's 's joint influence exceeds the sum of its parts, whereas a ratio near 0 (or negative) signals antagonistic or redundant behavior.
Intrinsic Evaluation Results. Regarding synergy quality (Table the hedonic variants strictly dominate all baselines across all three backbones and all three MS-MARCO objectives: Hedonic- S attains the best Pairwise and Ratio score in 26 of 27 model-metrie cells, while Hedonic-OCA follows as a close second. Relative to spherical k-means, the average margin is +0.29 Pairwise and +0.49 Ratio, indicating that activation similarity alone poor proxy for functional cooperation. Random and hierarchical clusterings even dip into negative Pairwise values sub-additivity and hover near the additive boundary on the Ratio metric, underscoring the value of an explicit game-theer -hhe-ttcc objective. Confidence intervals (95%, df = 2) never overlap between Hedonic-PA and the best baseline, with paired t-tests yielding p < 0.01 for every layer. K-means/HAC produce fairly uniform sizes (20-45 neurons per cluster), whereas hedonic output follows heavy-tailess Zipf-like law: each layer contains a single "macro" coalition (> 150 neurons), ~ 100 coalitions of size 2, and approximately 500 clusters with |C| > covering 14, 000 neurons. In most settings, the top cover algorithm converges with reservoir size m S 120, 000 and number of samples per iteration < 32, 000.
"Fuuivalently, this corresponds removing only the LoRA-induced contribution of the ablated neurons.
22

---

Published as a conference paper at ICLR 2026
Table Coalition Synergy (1) measured via Pairwise and ratio: mean +995% CI across three seeds.
LLaMA-3.1
Mistral
Pythia
Task / Algorithm
Pairwise
Ratio
Pairwise
Ratio
Pairwise
Ratio
Covered Query Term Ratio
Random
0.00
0.49
-0.0
0.03
0.00
0.50
K-means
0..3310.03
0.32-10.03
0..00
0.36=0.03
-0.18110.03
0.37
Hier. elustering
0..1
0.41#0.03
-0.13
0.43=0.03
-0.17=0.04
0.40
Hedonic (OCA)
0.08
0.74+1002
0.10:0.01
0.71=0.02
0.06+0.01
0.78+0.02
Hedonic (PAS)
0.12:10.005
0.86
0.00
0.84
0.15
0.89
Mean of Normalized Term Frequency
Random
0.02
0.4140.04
0.00
0.53
0.00
0.50
K-means
-0.22:10.03
0.34+0.03
-0.21
0.05=0.03
-0.16+0.03
0.31
Hier. clustering
-0.08
0.43++0.3
0000
0.43=0.03
-0.15+0.04
0.39+0003
Hedonic (OCA)
0.01
0.72++0.22
0.04
0.77=0.02
0.03+0.01
0.74
Hedonic (PAS)
0.09+000006
0.85+0.01
0.14
0.82+0.01
0.16+0.007
0.89
Relevance
Random
0.01
0.42
0.00
0.44=0.04
0.03
0.49
K- K-means
-0.13=0.03
0.33
00.14+0.03
0.36=0.03
-0.19=0.03
0.38
Hier. clustering
0.09910.04
0.42-10.03
-0.12
0.40
00.88+0.01
0.44
Hedonic (OCA)
0.03
0.77+0002
0.05
0.75=0.02
0.01
0.73+0.02
Hedonic (PAS)
0.11+:0.005
0.81+0.01
0.13
0.87
0.14
0.86+0.01
(12)
R20-125
R5 (4.3)
A32153
130
R17
R48
RO
(4/(12)
R34134
R85133
RA34132
R9
R28130
(15))))
R3
R3453
(200)
(20)
R18-15
L:7 (14)
(301
R831225
1441322
R321244)
R40 R401192)
Figure 2: Coalition flow across depth for MISTRAL-7B fine-tuned on the relevance modelling task.
The figure depicts flow from layers 8, with orange depicting split, green depicting persist and grey depicting vanish of coalitions.
H COALITION FLOW EXAMPLE
23

---

Published as a conference paper at ICLR 2026
I MNIST EXPERIMENT DEMONSTRATING EMERGENCE OF HUMAN-INTERPRETABLE SYNERGISTIC COALITIONS
To address regarding (1) whether synergistic neuron coalitions yield iinnaanninerrretable structure, and (2) whether our method works beyond large LLM backbones, conducted a controlled study on the MNIST classification task. This experiment demonstrates that our proposed hedonic/PAS framework applies robustly to small, non- non-Transforme, non-LLM networks and discovers coalitions that correspond to semantically meaningful visual concepts.
1.1 MODEL ARCHITECTURE
We train six-layer gated MLP (similar to the gated ferw-forward blocks in modern LLMs but smaller dimension, minus the MHA blocks). The network consists of:
Flattened input x E 784
Linear projection to residual stream of dimension drundel 256.
Four GatedMLP blocks, each computing:
h = h4 + w (o (o(Welihh)) @ ( nz)
where is the SiLU nonlinearity.
Final LayerNorm and linear classifier:
logits W... LayerNorm(ha))
The model contains no attention layers, recurrence, or convolutional structure. reaches 98. 98.8% test accuracy after 20 epochs of training with AdamW.
1.2 ACTIVATION EXTRACTION
For all 10,000 MNIST test examples, we extract and store:
(h1, h2, ha, ha, hs, himal
where he is the post-residual activation of block & and hima is the output of the final LayerNorm. Each internal activation tensor has shape [10000, 256].
1.3 PER-NEURON IMPORTANCE AND LAYER-LOCAL LOGITS
For each hidden layer hy (we use all internal layers score:
hs), we compute a per-neuron loss-delta
A ( (he,,-) - I. (he))
where he.-; is obtained by ablating neuron i in he and propagating the modified representation through the remaining MLP blocks and the final classifier. This provides a local first-order approximation of the functional contribution of neuron within layer l.
Following LLM experiments, also define layer-local logit (x) by cloning the final classifier head and applying it directly to hz(z). We keep this head fixed for layers. This allows us compute synergy metrics the point where coalitions formed, without conflating effeets from deeper layers.
1.4 COALITION DISCOVERY PROTOCOL ON MNIST
We apply the same hedonic-game game pipeline in the main LLM experiments, but it inde- pendently on every hidden layer hy...., h5. In each layer, the players are all M d 256 neurons; do not restrict to top-K subset by importance.
24

---

Published as a conference paper at ICLR 2026
Pairwise valuations.
For this MLP, we instantiate the same pairwise valuations as in the main text:
OCA: Orthogonal- Coatnnal-C--Aciatttiion based on weight geometry and activation correlation.
PAS: Pairwise Ablation Synergy based on second-order ablation effects on the layer-local logit (x).
As in the L.M setting, positive values indicate synergy and negative values indicate redundancy.
Hedonic coalition formation. For each layer é, construct sop-responsive hedonic game the M = 256 neurons using the Multi-Friend Choice (MFC) rule ($3) Neurons select their top-k partners (k 3) according to either OCA PAS, and PAC Top-Cover with the same hyperparameters used for L.LMs:
sampled coalition sizes in [2, 10],
reservoir size m = 10*,
per-round samples 8 103.
PAC parameters (e, 0..1,
We refer to the resulting partitions Hedonic-OCA and Hedonic-PAS respectively. Unless otherwise stated, summary statisties aggregate coalitions across intermediate layers h2-h4;; we find these layers concentrate the most -pgit-specific structure, with qualitatively similar patterns in hı and fı-
1.5 BASELINES
To demonstrate that the discovered coalitions are non-trivial, we compare against the following activation-only or random baselines, all operating on the same M 256 neurons each layer and producing partitions with size distributions matched to Hedonic-PAS::
1. Random Partition. Randomly partition the 256 neurons into coalitions with a size histogram matched to that produced by Hedonic-PAS in the same layer. This baseline tests whether the observed interpretability and synergy simply artifacts of grouping neurons.
2. Activation K-Means. Run k-means neuron activation vectors (rows of the [10000 x 256] activation matrix for that layer), with chosen to match the number of Hedonic-PAS coalitions; treat each cluster as coalition. This bascline tests whether simple activation- level similarity is sufficient to recover interpretable structure.
3. Activation Hierarchical Clustering (Ward Linkage). Perform agglomerative hierarchical clustering using Ward linkage the same neuron activation vectors. We cut the dendro- gram to produce the same number coalitions as Hedonic-PAS, and greedily merge/split clusters to approximately match the Hedonic-PAS size histogram. This evaluates whether a more flexible non- con-parametric clustering method, which can capture multi-scale activa- tion geometry, match the interpretability and synergy obtained by hedonic coalition formation.
All methods therefore operate on the same neuron set in each layer and produce partitions with comparable size distributions.
I.6 QUANTITATIVE METRICS ON MNIST
For each coalition C (from any method), in any layer hGg compute four families of metrics.
(1) Functional importance: accuracy drop under ablation. Let Acc denote the test accuracy of the full network. For each coalition in layer hts we ablate neurons in heg and re-evaluate the network on the MNIST test set to obtain Acc-c. We report the accuracy drop
AAcc(C) = AcChase Acc-c.
Section references the main paper.
25

---

Published as a conference paper at ICLR 2026
We then aggregate AAce(C) across all coalitions from each method and across layers h2-ha- reporting mean and standard deviation. Larger values indicate coalitions whose removal more functionally important.
(2) Synergy metrics. main paper:
Using layer-local logits (x), reuse the interaction metrics from the
(1) Eg (x) ( (1)
(i. E. (2) - (1) - 10 (1)
600
(x)
(i,)
For coalition C' we compute:
1 Pair(C) |0|[C| - 1) itjEC
E vii,5),
Ratio(C) ΣijjEC (bi.))
Eiec v1(i)
High Pair and Ratio indicate that the coalition's 's effect is more than the sum of its parts (strong synergy), rather than redundancy.
(3) Interpretability metries: digit-level structure. coalition activation for image n:
For coalition C in layer he, define the
an(n)
he(n. i).
|C| iEC
We then:
Rank all test images by (c(n) and take the top-K (we use K 128).
Compute the empirical digit distribution among these top-K K images.
Compute per-digit activation means
E[ac(n) digit(n) = d],
for E (0....,9).
Using these quantities we define:
Digit Purity:
Purity(C)= = max Pr [igit((n) = n E top-K by an())
de (0.....9)
Digit Selectivity:
Sel(C) = max max
d'
This measures how much more strongly the coalition responds to its preferred digit compared the second-best digit.
Activation Sparsity:
||ag|]
Spa =
||ac||
where ac is the vector of activations over all test images. Lower values indicate sharper selectivity.
26

---

Published as a conference paper at ICLR 2026
(4) Stroke Alignment Score: visual shape templates. To capture human-intes iuman-interpretable shape
structure. we define a set of simple stroke templates corresponding to visually salient MNIST patterns:
top horizontal bar (strong in digits like "3" and "7"),
a bottom horizontal bar (present in "2", "2",""" "3","5" "5", "8"),
a left vertical stem (e.g.. part of "4", "5", "9"),
right vertical stem (e.g.. part of "1", "7"),
central loop round stroke (e.g., "0", "6", "8", "9"),
"9"),"")
diagonal stroke (common in "2", "7").
We operationalize these templates using simple edge detectors and spatial masks. For each top- activating image of coalition C, we:
1. Resize to 28 x 28 if needed and normalize intensities.
2. Apply Sobel filters to compute horizontal and vertical gradient magnitudes.
3. For each template (e.g.. "top horizontal"),, restrict the gradients to the corresponding region (e.g., top third of the image, central loop window) and average.
This yields per-image per-template scores. The stroke alignment score for coalition C' is defined as the maximum template score averaged over its top-K images:
Stroke(CC) max Enclop x (s)(n)), TET
where T is the set of templates and 8, (n) is the template-specific edge score for image
Intuitively, this lets us say: one coalition "locks onto" crisp horizontal line near the top (e.g., shared by "3" and "7"), another picks out a round loop in the center (e.g.. "0"'"6""""") and another prefers vertical stem or diagonal stroke. the next subsection we show that certain Hedonic-PAS coalitions exhibit strong alignment with such shape templates, while random and clustering baselines do not.
1.7 QUALITATIVE INTERPRETABILIE EVALUATION
We visualize the top-activating images for selected Hedonic-PAS coalitions in layer hg. Each panel shows the top-64 images, with the ground-truth digit printed above.
Across these examples, Hedonic-PAS coalitions consistently group neurons whose combined activa- tion tracks single human- rreognzbeeo shape feature (e.g., central loop, top bar, S-shaped curve), even when the coalition responds to multiple digits (e.g.. 0/8 or 3/7). Baseline coalitions from random, k-means, or hierarchical clustering either mix unrelated digit classes or respond to more diffuse, spatially scattered patterns, leading to lower digit purity and weaker stroke alignment.
1.8 RESULTS
We summarize the MNIST findings using the following tables, which mirror the LLM experiments but on the six-layer MI.P and aggregate coalitions across layers 112-14. Hedonic-PAS consistently yields coalitions that are more functionally important, more synergistic, and more interpretable than baselines.
1.9 CONCLUSION
Taken together, these MNIST experiments show that our hedonic/PAS framework: (i) is not specific LoRA-tuned transformer MIPs, (ii) produces coalitions in a small gated MI.P that are both functionally important (large AAcc, high synergy) and semantically interpretable (high digit purity, strong stroke alignment), and (iii) outperforms random and activation-clustera baselines even in this simple non-LLM setting.
27

---

Published as a conference paper at ICLR 2026
0
0
8
8
O
8
8
8
0
0
8
0
8
8
8
6
0
8
8
5
0
8
8
8
5
8
8
8
6
8
0
8
O
8
8
8
8
8
8
8
8
8
8
8
6
B
8
8
8
8
8
0
6
8
D
8
0
8
0
O
d
8
8
8
0
8
8
8
8
0
8
8
8
8
Figure 3: Hedonic-PAS 0/8 loop coalition in hg. Top-activating images for one coalition; labels are overwhelmingly or 8. The digits share central round loop, which yields the highest stroke- alignment score for the "central loop" template.
Table 7: MNIST (layers h2-h4):: functional importance and synergy of coalitions. AAce is the mean test accuracy drop (in percentage points) when ablating single coalition. Pair and Ratio are the intrinsic synergy metrics defined $T
Method
Ace (mean + std, f)
Pair (mean, 1)
Ratio (mean, 1)
Avg. |C]
Random Partition
0.18 + 0.09
0.010
0.06
7.9
Activation K-Means
0.29 + 0.14
0.018
0.11
8.1
Activation Hierarchical
0.33 + 0.15
0.021
0.13
8.0
Hedonic-OCA
0.57 + 0.23
0.036
0.21
8.2
Hedonic-PASS
0.91 + 0.31
0.052
0.29
8.3
28

---

Published as a conference paper at ICLR 2026
7
3
3
7
3
3
7
7
7
7
7
3
3
3
7
3
3
3
3
3
7
3
3
3
3
7
3
3
7
3
3
3
7
7
3
3
3
7
3
3
3
3
3
7
3
3
3
3
3
3
3
7
7
7
3
7
7
7
7
7
3
3
7
3
3
7
7
7
3
3
7
7
7 7
7
7
3
7
7
7
7
7
3
7
7
Figure 4: Hedonic-PAS 3/7 horizontal-bar coalition in h3. Top-activating images for second coalition; almost all digits are 3 or 7. The coalition consistently fires on strong horizontal stroke at top the canvas, captured by "top horizontal bar" template.
Table 8: MNIST (layers h2-h4): interpretability metries for coalitions. Entries report mean + standard deviation across coalitions. Higher digit purity and selectivity and lower sparsity indicate more interpretable, focused features.
Method
Digit Purity (mean, 1)
Digit Selectivity (mean,
Sparsity (mean, J)
Random Partition
0.18 0.06
0.05 + 0.03
1.90 -: 0.20
Activation K-Means
0.48 + 0.16
0.18 = 0.09
1.62 0 0.19
Activation Hierarchical
0.52 = 0.17
0.20 - 0.10
1.56
Hedonic-OCA
0.71 + 0.14
0.31 = 0.11
1.41
Hedonic-PASS
0.86 + 0.09
0.45 + 0.12
1.30 1300+0.0
29

---

Published as a conference paper at ICLR 2026
5
3
3
5
S
5
9
5
3
3
5
5
5
3
3
3
3
5
3
3
3
3
3
S
3
3
3
3
6
3
3
3
3
3
3
4
3
3
3
5
5
3
3
3
3
S
5
S
5
3
6
3
3
3
3
3
6
3
3
3
5
3
3
3
5
3
3
5
3
3
3
S
3
3
5
3
3
5
5
3
3
3
3
3
3
5
3
3
6
3
6
5
3
3
6
6
3
6
3
5
Figure 5: Hedonic-PAS 3/5/6 S-shaped coalition in /3. third coalition mixes 3, 5, and digits that share an lower half of S-shaped curved stroke. Its top images show a characteristic diagonal/curved mid-level stroke, which obtains high scores on combined diagonal+curve template.
Table 9: Stroke alignment and negative controls MNIST. The Stroke column reports the average stroke-alignment score (max over templates) for top-K K images; Purity and Ratio report digit purity and synergy ratio in Tables/7|8
Setting
Method
Stroke (t)
Digit Purity (t)
Ratio (t)
Trained MLP, lgg
Hedonic-PAS
0.42
0.88
0.31
Trained MLP, hg
Activation Hierarchical
0.29
0.55
0.14
Random Init, hg
Hedonic-PAS
0.17
0.20
0.05
Label-Shuffleh ha
Hedonic-PAS
0.19
0.23
0.06
30

---

Published as a conference paper at ICLR 2026
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
Figure 6: Hedonic-PAS pure-7 coalition in ha. A fourth coalition responds almost exclusively canonical 7s with a clean top bar and right-leaning diagonal. Digit purity is near 1.0 and the stroke score is dominated by the top-bar + diagonal templates.
31


Reviews:
[{"review_id": "7rhNxLj9PJ", "reviewer": "Reviewer_dFvZ", "summary": "This paper proposes a method for finding groups of neurons in transformer MLP layers that work together in a non-additive way, meaning their combined effect is greater than the sum of their individual contributions. The authors use tools from game theory, specifically hedonic coalition formation, to identify these “neuron coalitions” and show that they are functionally important by measuring performance drops after removing these neurons. They apply their method to language models fine-tuned with LoRA for information retrieval tasks and find that the identified coalitions are more functionally important and i\"nterpretable\" (based on the previously mentioned ablation and alignment with known heuristics) than baseline clustering methods.", "questions": "- What is the compute cost and/or runtime of computing Hedonic coalitions v.s. the strongest clustering based approach? It seems like given the large number of samples required and the long wall-clock runtime that it might be far more expensive than a clustering based approach, but this cost is never compared to the baselines. Is it O(n^2) in the number of neutrons (which seems intractible, but is stated on line 467)?\n\n- The Hedonic sampling procedure has a relatively large number of hyperparameters. How were these swept and or otherwise decided upon?\n\n- Why was LoRA selected as the Finetuning method to study? It seems that Hedonic coalitions could be learned even for pretrained models before Finetuning (if applied to a task with reasonable zero-shot performance) or for models fully fintetuned. If the reason is computational cost, it would be worth noting it as such.\n\n- Why is the method only studied for LLM backbones? The method does not seem LLM/Transformer specific and could be further validated (at relatively lower cost) on smaller networks such as those for simple tasks like CIFAR10 or ImageNet.\n\nWhy was the method only evaluated for re-ranking? As far as I can tell, the method is not specific to this domain so it would be nice to validate it in others since it is being branded as a general purpose method for this type of analysis.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Across models, ablating hedonic neurons clearly harms results more than ablating coalitions identified by clustering methods based purely on neuron similarity or correlation. This indicates empirically that the method is identifying groups of neurons which are more structurally essential in performing the task.\n- The application of hedonic game theory to neuron analysis is original within the interpretability literature from what I can tell and this work develops a connection between a new area of cooperative game theory and neural network analysis.\n- The work has extensive reproductibility documentation provided as part of the supplementary material.\n- The work proposes a computationally tractable way to measure this theoretical connection to game theory in NNs, which is often challenging for game theoretical given the relatively small number of players in game theory experiments/problems relative to the large number of players/layers in a neural network.", "weaknesses": "This work is relatively far from my area of research (see the low confidence score) and therefore I cannot comment too heavily on the exact technical novelty. I have done a literature review of the space to try and be an informed reviewer, but all following comments admittedly come with that asterisk.\n\nThe biggest concern I have with regards to this work is to what degree it delivers on the \"interpretability\" aspect of mechanistic interpretability. The authors propose a novel framework for identifying synergistic neuron coalitions and present a number of quantitative evaluations suggesting that these coalitions capture important internal structure. However, it is less clear whether this structure yields human-understandable insights. For example, while the coalitions are shown to affect model performance and align with some IR heuristics, there is little evidence that they correspond to semantically meaningful concepts or behaviors. \n\nUnlike other work in mechanistic interpretability, such as SAEs, the paper does not provide labeled interpretations or vizualizations that would help validate the value of these coalitions. Similarly, unlike core mechanistic discoveries, such as inductive heads, the coalitions cannot be mapped to a particular mechanism the model performs. Further work tying the ablation of these coalitions to concrete changes model behaviors or concepts would strengthen the interpretability claim.\n\nAs an aside, from the model pruning/ model compression angle, it is somewhat surprising the work doesn't try to connect to established game theoretical works applied to neural networks such as Neuron Shapley values. Removing the top-k neurons would be a stronger baseline than the clustering based approaches evaluated I believe.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f7e01N1c2M", "reviewer": "Reviewer_tscc", "summary": "This paper uses coalition game theory to group neurons within LoRA fine-tuned transformers into coalitions, treating neurons as agents in a hedonic game. Neurons are grouped together based on which other neurons they strongly associate with, either by weight/activation metrics (orthogonal co-activation) or functional importance (pairwise ablation synergy). Such coalitions, when ablated, cause the largest decreases in downstream performance and track the closest with known IR heuristics. These coalitions are compared across layers, with the authors finding that coalitions tend to predominantly vanish across successive layers, with some persisting and splitting, but very few merging.", "questions": "See weaknesses for full explanation, but summarized:\n1. Why LoRA?\n2. How can the partially circular performance of Hedonic-PAS be justified?\n3. Are the Hedonic-MFC algorithms competitive against stronger clustering baselines that access the same information?\n4. Do the coalitions represent any interpretable features?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This formulation of neurons as agents in a hedonic game seems very novel and is an interesting lens through which to interpret neural networks.\n2. The Hedonic-MFC algorithm, using either OCA or PAS, finds coalitions that are more important than baseline clustering methods, both in terms of function via ablation and in terms of correlation with known IR heuristics.\n3. The study into the dynamics from layers 7 to 14 across the three models are interesting and support the claims about how cooperative units are predominantly pruned in successive layers, with some of them persisting or splitting, but with very few, if any, fusing.\n4. The paper is written well and handles its two often disjoint subject matters in an easy to follow manner.", "weaknesses": "1. The LoRA setting seems somewhat restrictive, why was this chosen? What advantages does this have over using a pretrained language model, with ablations being the zeroing out of neuron weights instead of the reset to pre-LoRA values? \n2. The OOD performance drop for Hedonic-PAS seems partially circular: because these coalitions were selected to jointly have an outsized effect under ablation, it follows that performance loss would also be strongest here when these groups are ablated (though this is mitigated somewhat by the IR heuristic correlation).\n3. The baselines presented in the paper do not seem completely fair from an information access perspective. The k-means and hierarchical clusterings both receive either weight or activation information. Hedonic-OCA receives both, and Hedonic-PAS receives an even stronger signal in outsized joint ablation degradation between two neurons. A stronger baseline should be considered for both k-means and hierarchical clustering, where they have access to the increased information that the Hedonic-MFC algorithms have access to. Do you see the same performance of Hedonic-MFC compared to, for example, standard graph community detection algorithms on the same information metrics?\n4. Despite the tracking of the coalition activations with known IR heuristics, the coalitions themselves are not further explored in terms of their function. Is there any analysis that the authors have done that can reveal interpretable contexts under which the coalitions fire, such as context or topic?\n\nDespite these weaknesses, the novel framing of the paper, the performance of Hedonic-MFC in finding important coalitions, and the study of across-layer dynamics motivate my score of 6. As a caveat, I am not very familiar with game theory, which motivates my low confidence score, but the authors explained the necessary background information well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EJUdZo1fDs", "reviewer": "Reviewer_SAy9", "summary": "This paper introduces a game-theoretic interpretability framework that models neurons in fine-tuned LLM MLP layers as agents in a hedonic cooperative game, capturing their synergistic interactions. Using PAC-Top-Cover to identify stable neuron coalitions, the method reveals functionally indispensable groups that co-adapt during fine-tuning and correspond to interpretable, task-relevant features. Applied to LLaMA, Mistral, and Pythia rerankers, the approach achieves higher synergy and stronger alignment with IR heuristics than clustering or SAE baselines, offering a principled path toward mechanistic understanding of mid-layer representations.", "questions": "please refer to weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel game-theoretic formulation of mechanistic interpretability, modeling neurons as agents in a hedonic game to capture synergistic interactions. Understanding the MLP layers is always important and challenging for MI field, and this paper provides a very interesting method. \n\n- The paper is well-organized and clearly motivated, effectively connecting theory, method, and empirical findings. Minor improvements could include more visual intuition for coalition dynamics, but overall readability is high.\n\n- The method is sound in theory with a good amount of theoreitcal justifications, and good amount of empirical supports.", "weaknesses": "- The results and analysis are still exploratory in nature as acknowledged in the paper. \n- The task evaluated in this paper is limited in diversity. This narrow task scope may limit generalizability of the proposed framework to other domains (e.g., reasoning, summarization, or generation)\n- The limited diversity may come from another concerns on the scalability of the method and the computational cost. \n- The analysis focuses on LoRA updates in mid MLP layers (7–14) based on prior work’s observation of task activity. This may introduce selection bias: the results may not generalize (in more of a scalability sense) to other layers, architectures, or non-LoRA fine-tuning setups. An ablation showing whether the method would still detect meaningful coalitions if applied to other layer ranges might be nice. (not required especially considering the length of rebuttal and discussion period.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)