Structural Inference: Interpreting Small Language Models with Susceptibilities
Authors: Garrett Baker
George Wang
Jesse Hoogland
Vinayak Pathak
Daniel Murfet
Venue: ICLR 2026 Poster (2026)
Keywords: Interpretability
Statistical Physics
Singular Learning Theory
Primary area: interpretability and explainable AI
TL;DR: Introduces susceptibilities to study the internal structure of language models

Abstract: We develop a linear response framework for interpretability that treats a neural network as a Bayesian statistical mechanical system. A small perturbation of the data distribution, for example shifting the Pile toward GitHub or legal text, induces a first-order change in the posterior expectation of an observable localized on a chosen component of the network. The resulting susceptibility can be estimated efficiently with local SGLD samples and factorizes into signed, per-token contributions that serve as attribution scores. We combine these susceptibilities into a response matrix whose low-rank structure separates functional modules such as multigram and induction heads in a 3M-parameter transformer.

Published as a conference paper at ICLR 2026
STRUCTURAL INFERENCE: INTERPRETING SMALL LANGUAGE MODELS WITH SUSCEPTIBILITIES
Garrett Baker*"
George Wang"* Jesse Hoogland
Vinayak Pathak
Daniel Murfet"
ABSTRACT
We develop a linear response framework for interpretability that treats a neural network as a Bayesian statistical mechanical system. A small perturbation of the data distribution, for example shifting the Pile toward GitHub or legal text, induces a first-order change in the posterior expectation of an observable localized on a chosen component of the network. The resulting susceptibility can be estimated efficiently with local SGLDD samples and factorizes into signed, per-token contri- butions that serve as attribution scores. We combine these susceptibilities into a response matrix whose low-rank structure separates functional modules such multigram and induction heads in a 3M AM-parrmeter transformer.
INTRODUCTION
The microscopic organization enabling the complex behaviors neural networks remains poorly understood. This paper introduces susceptibilities, novel interpretability framework rooted in statistical physics, to probe this internal structure. We treat a neural network as a Baycsian statistical mechanical system (Balasubramanian 1997 LaMont and Wiggins 2019) where an infinitesimal, controlled perturbation to the data distribution induces a first-order linear response in the expected behavior of chosen network component, such attention head. This response, the susceptibility. quantifies the component's 's sensitivity to the specific data shift (Section 2 and further related to geometry and generalization within singular learning theory Watanabe (2007).
Contributions. Our main contribution the development of a new interpretability paradigm derived from Bayesian learning theory and statistical physics. In particular:
We derive the theoretical framework of susceptibilities for quantifying how model com- ponents respond to changes in the data distribution. This provides a principled link between structure in data and model internals.
We introduce the methodology of structural inference for discovering internal structure in neural networks and attributing that structure to patterns in data. This yields new insight into how models balance expression and suppression.
Applying this methodology demonstrate that attention heads exhibit meaningfully differentiated susceptibilities to various data shifts. Our structural inference approach successfully distinguishes and separates known functional circuits from Hoogland et al. (2025) Wang et al. (2024) such as the induction circuit. This work thus provides principled and empirically validated tool for dissecting the functional organization of neural networks, offering insights that align with and extend prior mechanistic studies.
3MM-arameter transformer trained on the Pile (Section 4.2
2
THEORY
2.1
SETUP
We consider the modeel-rutt-prior triplet (p(ylæ, w), q(x, 3). ((w)) where ((,, u) = ((vl)){(x) is the true data-generating mechanism, p(y w) is the posited model of the conditional distribution
Timaeus
Equul contribution

---

Published as a conference paper at ICLR 2026
with parameter W C Rd representing the neural network weights, and o is prior We assume that q( y) > 0 and p(y|æ, w) > 0 for all (z, y) E X x and w E W. We assume is compact. Let the sample spaces X, Y be measure spaces so that q y) is probability density with respect to the measure on X x Y. In our intended application is the set of sequences of lengths K in some alphahet Σ of tokens and Y Σ, with the counting measure.
1 S k
Given a dataset D, = (1),,:)))==> drawn i..d. from ((2, y), we define the sample negative log-likelihood function as (w) = - E logp p(ys 24 w) and its theoretical counterpart, the average negative log- likelihood population loss, L(w) (g(,,)) log p(ylæ, w)). The annealed posterior at inverse temperature 3 and sample size n is
(w)
expl cxp(-n3L((w))p(u) where 2 =
/ (sp((nnL((w)))de dw.
(1)
Z:2
2.2 VARIATION IN THE TRUTHI
We consider a one- parameere variation of the data distribution of the following form. Let H C R be some open interval containing 0 and for h H let qa (2, y) be a probability density with respect to the measure μ on X x Y. We assume that h + Qn is differentiable as map from H to (X x Y,).. We write dædy for dp(x, y). The average negative log-likelihood for qb L*(w) - E (x.0) log p(yla, w) and the annealed posterior is
/
2 (w(h)) 3. expl-n313 (w)) (w)))))
where Z =
du. (2)
The simplest kind of me-parameter variation in the data distribution is a mixture. Let g' be another probability density on X Y. Given E [0, 1] we define
h) + hg'
qh
Then the (one-sided) derivative exists and is the function gn (x. v) = q'(x, - ((x,)).
2.3 SUSCEPTIBILITIES
response function measures how some observable changes under a small external perturbation or variation of a controlling parameter. In our Bayesian setting, suppose we pick an observable (a function or generalized function (Gelfand and Shilov 1977) on parameter space) d(w). We then consider its (annealed) posterior expectation, where we denote by some hyperparameter
p.1 = f (fw)))(())) (w()))d.
If we drop h from the notation it means we are computing expectations with respect to the umperturbed (annealed) posterior (i.e. h = 0). A small change and some aspect of this shift is captured by the shift in the expectation value (d)
induces a shift in the posterior as a distribution,
Definition 2.1. The susceptibility of & to the perturbation qu at inverse temperature 3 is
0
X
(0)
(5)
nB 0h
h
Lemma 2.2. The susceptibilityy for an observable & is computed by
X -Cov (0, AL]
(6)
where Cov AL] = (BAL), (AL) and AL
81 Oh
h=0
Proof. See Appendix A
Lemma 2.3. For the (data-mixture) susceptibility, qr is the variation in 33) and we have AL(w)
(www)) - L(w) where L (w) and L(w) are (ww) with h = 1 and respectively.
Proof. See Appendix
2

---

Published as a conference paper at ICLR 2026
From the lemma we learn that the susceptibility can be defined in terms of the unperturbed posterior. Next we define the primary class of observables to be considered in this paper. We consider models p(y|æ, w) parametrized by neural networks with weights w E W. By component, mean a subset C' of these weights (e.g., an attention head transformer). Formally, a component is a product decomposition W U C. with U the parameters not in the component.
Definition 2.4. Given component C' with W U X C' and chosen parameter w- (u", v*)
write w (u, v) and define
Φc(w) 8(u - u²) [L(w) L(w"))
(7)
The motivation for using the observable de in the setting of susceptibilities that its expectation
value is related refined learning coefficients introduced in WWag al. (2024). This means that
we can think of the susceptibility associated to this observable as a kind of "rate of change" of the
local learning coefficient, a geometric measure of model complexity (Appendix D.3). This provides
principled link between susceptibilities, generalization error in Bayesian statistics and changes in the geometry of the loss landscape with changes in the data distribution (Watanabe) (2009.
Definition 2.5. The per-sample susceptibiliity of C for (x, E X Y is
X(a.y) == CCo ((2,)) L); (w) log
(8)
The notation is justified by the fact that the expectation of X(z.w) under the probe distribution q' (1r, y) the susceptibility x of Φc (see Appendix A
our applications x is sequence of tokens, y is single token and so we also refer to ((z.y) as the per-token susceptibility.
C
3 METHODOLOGY
In condensed matter physics, one way to define internal struciure is that it is what causes the patterns that manifest in the responses of a system to external perturbations, as measured by natural observables (Altland and Simons 2010 $7). In the previous section, adapted this perspective to neural networks: changes in data are the external fields or perturbations we use to probe the system, changes in component-wise loss éc the observables, and the susceptibility is the response.
we develop the local susceptibility, which enables us to practically estimate sus- ceptibility matrices for individual neural network checkpoints. We then discuss how to interpret positive and negative susceptibility values as suppression and expression, respectively (Section 3.2). Finally, in Section 3.3] we introduce structural inference, framework for identifying the patterns that manifest across set of susceptibilities or, equivalently, for discovering internal structure.
In Section 3.1
3.1 ESTIMATING THE I.OAAL SUSCEPTIBILITY
In practice, we are concerned with the properties of specific neural network checkpoints trained via stochastic optimizers, rather than ensembles drawn from the full Bayesian posterior. Moreover, sampling from global posterior is computationally infeasible. To circumvent these issues, we introduce local susceptibilities, which make it possible to estimate susceptibilities for individual models with samplers like Stochastic Gradient Langevin Dynamics (SGLD; Welling and 2011).
We "localize" the posterior by replacing the prior &p with a Gaussian prior centered at w", local minimizer of L(w). This ensures that sampling remains in small neighborhood of w", We define a local Gibbs posterior (Bissirí al. 2016) with inverse temperature / > 0,
p(w; w", ,),7) x exp { (n,L,, - w" w²1
(9)
as well local annealed posterior,
p(w; w", BB,,,) x exp (n33I (
7 2w
(10)
Given an observable o, we write the associated local (annealed) expectation as (d: w²) ) which is the integral against this distribution ((w)p( w: w", ,,, h)dw.
3

---

Published as a conference paper at ICLR 2026
Per token Susceptihility Xj
Probe: Upweight tolen in dataset J.
Observable Measar change expented les.
Component: Restrict to altenrion head
weges
abbreg.
uno 500
Lin
(riig Azt
2h
Nhen only
sodium tol vedea
Eonete
Fe Se
Sabbevittee
to Na PTC bere E
baving
Art
Mhen only
sodEum
to1 penee
abbrevs
Na-p1'
Azt
Mhen
only
Rbbret
having
When
When
om
eneau Fomaze (Eo begaz A2prwy
uec Loce
(bo
When only
50d km
butyy naph
halonns Fonate (to be A7brey
RA-BNS
corcinal ftar)
Nhon
uocium
butyinaphene.
Eonatee
Ma-BNG cereinetted
Nhea
on
Winaptes E3 530e Lonase [to
Hopp
When
Se onnked (to
Nisan omay
nagt wnw.ulnonno Constee to be degeeben
baving
99b
Eaving
Nhwn
on 1y
nigh
eueke Eonate [6o be Abarewi
Ne Termenal Prrinnffee Fir)
have Ing
(rrior
Art
49
When
only
sod
naghtha preeeseeness eriesll Conate Nto be abbrevin Sorvviate to Ne hecelnn Etecl
having
160
hare CLA0
onag
tto
to Mamtt Eare
havling
Negative Susce
Expression
Positive Suscepibilitt Suppression
Source: USPTO BADAGROUNDS
Figure I: Per-token susceptibilities reveal patterns of expression and suppression. In this context, the beginning of three lines are shown. Each line is repeated six times, and each repeat is divided into three color bars. These bars correspond to individual heads 0:00-0:7 in layer 0 and 1:01-1:7 in layer 1, as shown on the left. The susceptibility values are standardized for each head, with solid red representing z-score of -3 and solid green representing z-score of +3. We see that the susceptibility increases for some heads on repeated token sequences like Prior Art These positive susceptibilities are examples of the suppression of induction patterns.
Definition 3.1. We define the local susceptibility at inverse temperature 3 to be
1
x(w*) = n1 ah (Ø: w ),,1
(11)
h=0
By Lemma 2.2 we can write
x(w") = ( w²) + (6: w
(AL::
(12)
We fix mixture percentage öh E (0, 1) and, as in Remark D.I
approximate AL by
(L) = 8h (w) Ln (w)
(13)
for a dataset size n. We also replace the expectations in (2) over the annealed posterior (10) with expectations over the ordinary localized posterior (9) involving L,, We obtain estimator for the per-token susceptibility X(z.y) by replacing 6/ with ((a)) (see Appendix C.4
We compute T., (w: by drawing sample from the original data distribution g, and (w) by drawing a set of samples of size n, consisting of a mixture of samples from the unperturbed data distribution and samples from the perturbed data distribution (the mixture being controlled by 8h). Given approximate samples {wr]=z from the (unperturbed and un-annealed) localized posterior with parameter 33 we compute our estimate of the susceptibility to be
x(w') = - Σ [d((:)) (l(w))ma
Σ (wr)
[
Ain (wx)) t=]
(14)
We further average this quantity over multiple chains {wr Hr For the precise expression that we implement in code in our experiments, see Appendix
3.2 INTERPRETING SUSCEPTIBILITIE AS expression AND suppression
The per-token susceptibility is defined as the first-order response of expectation value to change in the data distribution and as such, it is intrinsically meaningful. However there is a more concrete interpretation which is helpful in interpreting the empirical results in this paper.
4

---

Published as a conference paper at ICLR 2026
Note that by linearity of the covariance
X(e.)) 8 o(u u²) u²) L(w")[ (m..)))))))))))))))))))))))))))) [(w)]
6(u -ov8 u² ('))( ((a + Cov3 j(u - u² (w),, ((w)
Does not depend on
(F.a)
we get decomposition of the per-token susceptibility into part (a.y) which depends on both C and the token sequence zy and part that depends only on C. In our analysis we will form matrices of per-token susceptibilities and then standardize them by subtracting the mean across rows which are indexed by pairs (z. y). This process of standardization cancels off the second term above and this means that our analysis ultimately depends only on the first summand ((a.)))
We call this the standardized susceptibility and now offer an interpretation of the sign of this term.
The standardized susceptibility measures how 6(u u² [(w) and (w) covary when we perturb away from w", with perturbations being more likely according to their probability in the annealed posterior (that is, perturbations which increase the population loss (ww) are exponentially suppressed). A variation in which only changes L(w) by small amount may nonetheless increase {(x,))(w) (w) for some tokens and lower it for others, and this correlation is what we care about.
Negative standardized susceptibility means that variations w w which increase ((x,)) (that is, make y less probable in context æ) tend to be perturbations in the weights of which increase the loss overall. This makes sense if (Ir y) follows a pattern that C' is involved in predicting, mechanistically. Thus we associate negative susceptibility with the component C' expressing that should follow x.
Positive standardized susceptibility means that variations wt more probable in context .) tend to be perturbations in the weights of C' which increase the loss overall. This makes sense if (ır, y) follows pattern that C' is involved in "opposing", mechanistically. It could be predicting an alternative completion, or just decreasing the probability of this one. Thus we associate positive susceptibility with the component suppressing the continuation of x by y.
which lower (a,,)) (that is, make
Sign of vb
Interpretation
Expression
Variations in C' which decrease loss, also raise p(y) (ar, w). Variations in C' which decrease loss, also lower p(y) (zr, w).
Suppression
visualization of the pattern of expression and suppression in natural language is given in Figure For an analogy with magnetic susceptibility Appendix [.I]]
natural question is how this susceptibility-centeedd notion of expression and suppression relates to existing work prediction and suppression neurons in mechanistie interpretabilityy characterized through direct effects on logits (Gurnee et al. 20248) (Lad et al.) 2025). Our preliminary investigation
does not reveal any simple relation (Appendix H
3.3 SUSCEPTIBILITIES FOR STRUCTURAL INFERENCE
Our main application of susceptibilities is discovering internal structure in neural networks. In our approach we start with two inputs: finite set of components (Cj)eи with associated observables
Φ := éc; (e.g. attention heads) and a finite set of data distributions {ad) dEED with associated variations from q by taking mixtures (e.g. Pile subsets like GITHUB). We refer to this as a probe set and the individual data distributions probe distributions. The responses are by definition the entries of the |D] x [H] data matrix or response matrix
=(xx))
( dED,jEN
(15)
where X,d is the estimated susceptibility observable Φc, with respect to the variation of q in the direction of qº Alternatively we can perform the analysis at the token level using variations in the data distribution which upweight continuations y in contexts where some number of samples
5

---

Published as a conference paper at ICLR 2026
(z, y) are taken from each q" and use data matrix containing per-token susceptibilities for these samples (see Section 4.2 below).
In this view internal structure in the network means the linear algebraic structure of X (to first- order). In Appendix D.4] we explain how for a particular formal definition of "patterns" in the data distribution (which we call modes, following Chen and Murfet (025 can factor X product X = CP where C consists of coupling constants between the probe distributions and modes, and P couples observables and modes.
This motivates applying PCA to the data matrix and identifying the principal components (left singular vectors multiplied by singular values) with patterns in the data and the loadings (right singular vectors) with structures in the model. We call this approach, where we infer the internal structure in models via data analysis of matrices of susceptibilities, structural inference.
RESULTS
In this section, we apply our framework to study structure in small language model. Our analysis automatically identifies attention heads' different functional roles, specializations, and higher-level structure like the "induction circuit." Following Elhage et al. (202T), Olsson et al. (2022). Hoogland
itt al. (2025) and Wang and/Wang et al. (2024) we study two-layer eitention-only (without MLP MLPlayers layers) trans- formers trained on next-token prediction on a subset of the Pile (Gao et al. 2020 Xie et al. 2023).
For architecture and training details see Hoogland et al. (2025). TTroughout, we refer to attention
head h E .0.....77 7} in layer E {0, 1} by the notation (:h). The empirical loss for the transformer is defined in the usual way (see Appendix
We denote by Σ the set of tokens. For tokenization, we used a truncated variant of the GPT-2 tokenizer that cut the original vocabulary of 50,000 tokens down 5,000 (Eldan and Li| (023). We denote token sequences as follows: wa vel ength is a sequence of three tokens. We set X to be the disjoint union X = K-1 -k I and Y = E, both with the counting measure. We assume given a probability distribution q (z, v) on Σ* x E for 1 S h < K and define q(z, v) K-1 9 v) for Z E X, y Y with of length k. The conditional distribution p(y|zz w) is parametrized by the transformer in the usual way (Phuong and Hutter 2022).
1
4.1 SANITY CHECKS
We performed range of checks validate that our estimates of susceptibilities non-trivial. While some of these are of independent interest, we have relegated them to appendices which we now summarize: (Appendix FF Across heads and datasets there is significant variation in the susceptibilitiess (so there is signal); (Appendix E The correlation of per-token susceptibilities with per-token losses and loss difference after ablation is very small (so the signal is not redundant), (Appendix E.2) Per-token susceptibilities tend to increase with context length especially for some layer heads (so the signal depends also on æ, not just y); (Appendix E.3) Per-token susceptibilities explain variation in overall susceptibilities (which justifies their use); (Appendix E.4) The same token y can appear with positive or negative susceptibility depending on context.
4.2 FINDING INTERNAL STRUCTURE WITH SUSCEPTIBILITIE
We adopt the hypothesis from Wang al. (2024) that internal structure in models reflects structure in the data distribution (Harris 1954 Rogers and McClelland, 2004 Saxe et al. 2019).. We therefore organize our results around six kinds of patterns, by which we mean a property of a token sequence examples in Figure
ay (full definitions in Appendix C.8
Word start: a token that decodes to a space followed by lower or upper case letters.
Word part: a non-word-end token that decodes to sequence of upper or lower case letters.
Word end: token that decodes to sequence of upper or lower case letters followed by formatting token (see Appendix C.8 for an exhaustive list) delimiter or space.
Induction pattern: a sequence of tokens wv ue where U a sequence of any length, u, v are individual tokens, and we is not common bigram, ((ulu) & 0.05.

---

Published as a conference paper at ICLR 2026
Patterns in Data
Word Start
Word Part
Word Rnd
Spacing
Induction Pattern
Delimiters
Data-laadings of the
wa vel ength
wa Ivel ength
wa
(ength
suseeptibilliies mitrix
differe
differe
differe (nces]
Anin
30
2.6
27.5
40.8
35.7
75.8
25
60
Top 10% Posttive
46
169
10.2
54
103
15.6
20
16.8
161
269
21
23.8
21
50
02
38.5
43
52
32
Top 10% Naqative
31.5
16.3
7.1
31.7
10.7
23.0
115
5.3
14.7
5.5
5.1
14.3
17.1
20
2.4
12.6
10.7
2.5
11.3
27.3
30
110
115
Tap 1% Nogotive
29.8
19:0
4.0
30.0
11.4
10.2
5
10.3
14.8
3.2
8.8
23.4
17.0
10
5.0
11.2
2.5
4.1
37.8
41.7
Induction
Spacing
Right Delimiter
Top 1% Posttive 20.9
10
72.5
45.7
3.6
20
2.8
0.7
0.0
0.1
0.0
0.1
1.5
13.7
1.7
0.1
22.9
0.1
0.2
20
-30
Tap 10% Rostiie 24.6
78
35.2
384
5.6
24.3
50
1.8
31
0.4
2.0
0.2
2.0
6.1
2.3
0.6
5,3
9.9
1.1
Top 10% Negative
3.2
53.
26.1
18
50.1
8.8
36.0
1.7
16.4
37.3
5.8
1.4
-5
0.2
4.6
8.7
0.2
5.2
6.8
10
-10
Negative
-10
PC1
PC2
PC3
PC1
PC2
PCS
PC1
PC2
PC3
PC1
PC2
PC3
PC1 PC2
PC3
PC1
PC2
PCS
Layer
0
Layer
Layer
Layer1
Layer
Layer
Figure Susceptibilities decompose into interpretable loadings over data. Among the tokens with the coefficients of the largest magnitude in each principal component for the per-token susceptibility PCA, the percentage following each of the six patterns. The components are computed separately for each layer.
Spacing: a token made up of one or more characters from 0.
An
At.
Ar
NE..
Right delimiter: brackets and composite tokens, e.g. )
We randomly sample N 20000 tokens from each dataset and form the data matrix
(() C3C dE..sssE
where H(II indexes the heads in layer I 10, D all datasets, and Xa.; - is the estimated per-token susceptibility for the observable Φc, where (r, y) is the ith sampled token y in context x in dataset d. The data matrix has size N x (H(().. We perform PCA on this data matrix (including standardizing the columns) and find the top three PCs explaining resp. 95.34%, 1.83%, 0.73% the variance for layer O and 99.14%, 0.39%, 0.11% for layer 1. We examine the principal components to find the top positive and negative per-token susceptibilities and find the frequency of our six patterns among these top tokens. The resulting "data loadings" are shown in Figure 2 and in Appendix we include the background frequencies. The "component loadings" are shown in Figure 3 In the following provide interpretations for each principal component.
CC
8.8 C.8
PC1: Word segmentation The first PC is uniform across the heads in layer 0 and layer 1 and explains the majority of the variance. The interpretation is visually apparent in Figure 4 on most tokens the heads have similar susceptibilities which are broadly positive for word endings, induction patterns and right delimiters and negative for word starts and spaces. Note that, as with human infants, one of the first problems language model must solve in acquiring language is word segmentation: identifying boundaries
stream of tokens (Goldwater et al.) 2009). It is therefore notable that the strongest pattern in the per-token susceptibilities seems to be associated with the model having learned to segment the token stream into words.
PC2: The induction circuit In data, PC2 is dominated by the dichotomy between word endings (positive) and induction patterns (negative). In components, PC2 has positive loadings on all heads identified in Wang et al. (2024) as being part of the induction circuit for this model, and negative loadings on the remaining heads. The induction circuit consists of the induction heads (16). 117 composing with the previous-token heads (:1), 0:4 and current-token head 0:5.
In Wang et al. (2024) it was found that the negative effect of ablating the heads 1:0 1:55 was mainly on prediction of n-grams and skip n-grams and were thus termed layer-1 (L.1) multigram heads. The
7

---

Published as a conference paper at ICLR 2026
0.4
0.
07
0.0
0.0
-02
-02
0.0
100
K01
13h7
KITE
Khz
KMJ
Khu
Kns
KnB
KhT
DRO
Dh1
Dn2
Flid
phs
1000
2011
LinG
Layer
Layer
Mangen Comtrrecshenn
taketinn
Cun
Pevtoonn # lyckleeh
PCT: Uniform Mode PC1 shows that
(b) PC2: Induction vs. Multigram, PC2 distingushes crcatttnsscctteeec from multgram nends.
PCR Head Specialization. distingus shes corrent
Beods leam regresent word boundones.
colen heads from jrenioss tokonn herd; and Dyck reads.
Figure 3: Susceptibilities decompose into interpretable loadings over components. The loadings of the top two principal components for per-token susceptibility PCA on attention heads.
interpretation of PC2 is therefore that the induction circuit tends to express induction patterns which the remaining heads (including the layer 1 multigram heads) suppress, with the role of expression and suppression reversed for word endings.
We note that the appearance of the induction circuit as the heads with positive loadings in PC2 holds across all four independently trained seeds that considered (Appendix [ In this sense structural inference finds the induction circuit and associates it with a pattern in the data distribution.
PC3: Bracket matching In data, PC3 has more variation between the two layers than the earlier
PCs, but both layers there is significant negative loading on right delimiters. This is interesting
because among the five positively loaded components in Figure 3 are 0.7, 1:3, 1:5 which were identified using ablations in Wang et al. (2024 Appendix B.3) as Dyck heads meaning that they are involved in predicting matching closing brackets of various kinds.
The role of 1:4 here is unclear, but was noted in (Wang et al. 2024 Appendix B.4.2) that this head seems to specialize in verb-particle phrases and other patterns including :// ... which could be viewed as generalized bracketings. The presence of this head in PC3 could also be explained by the data loadings on other patterns besides delimiters. Overall this component seems somewhat related to structure in the model for predicting brackets previously identified in Wang et al. (2024), although as clearly as PC2 is related to the induction eircuit.
To check the robustness of the claim to have distinguished the layer 1 multigram heads from the induction heads by per-token susceptibility PCA, we run an independent analysis in Appendix G.] using PCA of the full susceptibilities including the values over training.
5 RELATED WORK
Ablations and mechanistic interpretability. Susceptibilities measure correlations various changes in losses that result from perturbing the weights. A somewhat analogous of techniques ablations, which involve perturbing activations. These are widely used in mechanistic interpretability test hypotheses about the computation being performed by parts of neural networks (Chan et al.)
2022; Wang et al. 2023a Rauker et al. 2023 Bereska and Gavves 2024). By performing these interventions and observing the change in losses, researchers aim to infer facts about internal structure.
In this paper we compare susceptibilities to zero ablation (Meyes et al. 2020 Hamblin et al. 2023
Nanda et 2023 Morcos al., 2018 Zhou et al.
2018
Influence functions. Classical influence functions measure the effect of up-weighting one training example on an estimator (Cook and Weisberg 1980, Koh and Liang 2017). Susceptibilities are closely related to the Bayesian form of influence functions (Giordano and Broderick 2023 Iba 2023).
Suppose we have a dataset D2 = {(14, vi } 12 We introduce a single new sample (Jnnww Vnew with negative log likelihood f(w; Inew Ynew) into the likelihood with (small) temperaure parameter h controlling how "strongly" this new sample is weighted. Concretely, we define
p(w]Dn, h) x (xp(-nI...w)) exp( h {ww; Knew, Vuew) ((x))
Next we fix a test point (Irest Mtest) and define an observable d(w) E (w; Itests Frest) Then
it is easy check that x in this case is -Cov D., I[ [w; Trest Vrest); ((w; Fnew ynow) Thus
8

---

Published as a conference paper at ICLR 2026
influence functions, in this sense, are special case of susceptibilities. However, in this paper focus on a different special case, where the observables are tied to parts of the model.
Suppression behavior in neural networks. Mechanistic interpretability has discovered many examples of suppression in language models, such as anti-induction heads that suppress induction patterns (Olsson et al. 2022 McDougall et al.) (022)) and negative name-mover heads that decrease the probability of an earlier name being copied in the Indirect Object Identification (IOI) circuit (Wang et al., 2023a McGrath et al. (2023) explore the phenomenon of self-repair, where later layers counteract the effects of ablation in earlier layers. McDougall et al. (2024) introduce the idea of copy suppression more generally, identifying a specific head in
that actively inhibits copying patterns. Further evidence of copy suppression and self-repair across models in the Pythia family (Biderman et al., (023) presented in [Tigees et al. (2024) and [Rushing and Nanda (2024). |Rshhnng and Nanda (2024) also speculate that self-repair might be an incorrect framing, since ablations are inherently evaluating the model off-distribution. These negative components are a challenge for scaling mechanistic interpretability (McDougall al.) 2022) as self-repair obscures the effect of ablations; see McGrath et al. (2023) and Rushing and Nanda (2024 $5.1).
Lad et al. (024)) suggest that predictions in neural networks with residual connections (such as transformers) result from an ensemble of many components which vote (Veit et al., 2016), and some of those votes may be "against".. Gurnee et al. (2024a) study universal neurons across training seeds of GPT2 models and find consistently that models learn neurons in later layers that suppress particular sets of tokens. Building on perspective in Geva et al. (2022) they endorse view of networks building predictions by both "promoting and suppressing concepts in the vocabulary space". In this vein Yan and Jia (2025) show that language models answer factual queries by promoting many answers and then suppressing some of them. More generally, suppression or inhibition has been a key part of artificial neural networks since McCulloch and Pitts (1990); Wang al. (2023b).
6 LIMITATIONS AND FUTURE WORK
The main limitation of our methodology is the quality of the approximate posterior sampling method. The effect of hyperparameter selection in SGLD on susceptibility estimation remains poorly un- derstood. Furthermore, SGLD generates correlated samples. Averaging over multiple chains only partially mitigates this issue. In this paper examine small 3M parameter model, but as SGLD is inherently scalable, we do not expect major obstacles in applying the methods to larger models. For example, Wang et al. (2024) use SGLD to estimate similar observables in Pythia-70M, The main limitation for scalability is the need for separate posterior samples for each component; for details on costs up to the scale of 1.4B parameters Appendix C.5
7 CONCLUSION
We have introduced susceptibilities, novel interpretability framework inspired by statistical physics. In this analogy we view neural network as complex material whose internal structure is reflected by the differential response of parts of that structure to range of "external fields" in the form of variations in the data distribution. Applying this methodology to 3M --paameterr transformer demonstrates that this analogy is fruitful: attention heads exhibit meaningfully differentiated susceptibilities, and structural inference via response matrix analysis successfully separates known functional modules.
This work builds on the literature establishing singular learning theory (Watanabe) 2009) and local learning coefficient estimation as principled tools for understanding neural networks and their development (Lau et al. 2025 Hoogland et al. 2025 Wang et al. 2024 Urdshals and Urdshals 2025) Carroll et al. 2025). The study of the balance of expression and suppression using susceptibilities fits naturally into existing literature within mechanistic interpretability (Section 5). We hope these techniques will lead to scalable and theoretically principled tools for interpretability in large neural networks, parallel to existing approaches within mechanistic interpretability such as ablations or sparse auto-encoders, but with deeper foundations in the mathematical theory of generalization.

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
Appendix outlines experimental details of the model and of hyperparameters used in sampling and Appendix B lists hyperlinks the datasets used.
ACKNOWLEDGMENTTS
This work was supported by the Survival and Flourishing Fund, Open Philanthropy (now Coefficient Giving), and the Manifund Regranting Program.
REFERENCES
A. Altland and B. D. Simons. Condensed matter field theory. Cambridge University Press, 2010.
A. Amadei, B. Linssen, and H. Berendsen, Essential dynamics proteins. Proteins: Structure, Function, and Bioinformaties, 17(4):412-425, 1993.
A. Anandkumar, R. Ge, D. Hsu, S. M. Kakade, and M. Telgarsky. A method of moments for mixture models and hidden Markov models. Conference on Learning Theory, pages 33-1, 2012.
V. Balasubramanian. Statistical inference, Occam's razor, and Statistical Mechanics the Space of Probability Distributions. Neural Computation, 92):349-3668 1997.
M. Bauer, M. Bruveris, and P. W. Michor. Uniqueness of the Fisher-Rao metric on the space of smooth densities. Bulletin of the London Mathematical Society, 48(3):499-506, 2016.
L. Bereska and S. Gavves. Mechanistic Interpretability for AI Safety A Review. Transac- tions on Machine Learning Research, 2024. URL https:://ppeneeil.eee net/forum Id=ePUVetPKu6
S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430.
PMLR, 2023.
P. G. Bissiri, C. C. Holmes, and S. G. Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5): 100-11000111 2016.
K. L. Briggman, D. Abarbanel, and W. Kristan Optical imaging of neuronal populations during decision-making. Science, 3076571) 2005.
L. Carroll, Hoogland, M. Farrugia-Roberts, and D. Murfet. Dynamics of transient structure in in-context linear regression transformers, 2025.
17745
L. Chan, A. Murria--Alonse N. Goldwosky-Dii R. Greenblatt, J. Nitishinskaya, A. Radhakrishnan, B. Shlegeris, and N. Thomas. Causal scrubbing, method for rigorously testing interpretability hypotheses. Al Alignment Forum, 2022.
Z. Chen and D. Murfet. Modes of sequence models and learning coefficients, 2025. URL https 7/arxiv. org/ 20tg//b.b/2000.110/
R. D. Cook and S. Weisberg. Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22(4):495-508, 1980.
Devin Gulliver. Monologypill-uncoyrrgheed http://hnginnnccccoo/aaeessooooo uncopyrighted, Mar. 2025.
R. Eldan and Y. Li. Tinystories: How small can language models be and still speak coherent english?,
2023. URL https://arxiv abllabb...000000 07759
10

---

Published as a conference paper at ICLR 2026
N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, D. Hernandez, A. Jones, J. Kermion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and
T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z.
C. Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
L. Gao, S. Biderman, S. Black, I.. Golding, T. Hoppe, C. Foster, J. Phang. H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: an 800GB dataset of diverse text for lan- guage modeling. Preprint arXiv:2101.0002 [es.CL], 2020.
I. M. Gelfand and G. E. Shilov. Generalized functions, Volume 1: Properties and Operations. Academic Press, new York and London, 1977.
M. Geva, A. Caciularu, K. Wang. and Y. Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30-45, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational org/
Linguistics. doi: 10.18653/v 1/2022.emmip-main.3 URL 2022. em www.p-mainnmn.mmm---mmmmmmmmmmmm momm.ann.3
R. Giordano and T. Broderick. The Bayesian infinitesimal jackknife for variance. arXiv preprint arXiv:2305.06466, 2023.
S. Goldwater, T. L. Griffiths, and M. Johnson. A bayesian framework for word segmentation: Exploring the effects of context. Cognition, 12(1)221-54, 2009.
W. Gurmee, T. Horsley, Z. C. Guo, T.R. Kheirkhah, Q. Sun, W. Hathaway, Nanda, and D. Bertsimas. Universal neurons in GPT2 language models. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856. URL htttps://openeeeeeeeeee
net wtweefooummmeee forum (ttrrmmidd77ee14400000
W. Gurnee, T. Horsley, Z. C. Guo, T. R. Kheirkhah, Q. Sun, W. Hathaway, N. Nanda, and D. Bertsimas. Universal neurons in gpt2 language models, 2024b. /tll:/ttpss//rxxx...rr/r/////// 12181
2401
C. Hamblin, T. Konkle, and G. Alvarez. Pruning for Feature-Preservinn Circuits in CNNs, Apr. 2023.
Z. S. Harris. Distributional structure. Word, 10(2-3):146-162, 1954.
Hoogland, G. Wang, M. Farrugia-Roberts, L. Carroll, S. Wei, and D. Murfet. Loss landscape degeneracy drives stagewise development in transformers. Transactions in Machine Learning Research, 2025.
Y. Iba. W-kemel and essential subspace for frequentist evaluation of bayesian estimators. arXiv preprint arXiv:2311.13017 2023.
P. W. Koh and P. Liang. Understanding black- box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
V. Lad, W. Gurnee, and M. Tegmark. The remarkaale robustness of LLMs: Stages of inference?, 2024. URL htttps::/arxi..orr////////
19384
V. Lad, J. H. Lee, W. Gurnee, and M. Tegmark. The remarkable robustness of Ilms: Stages of inference?, 2025. /tl/iittss//arxivooorr/abb//////
19384
C. H. LaMont and P. A. Wiggins. Correspondence between thermodynamics and inference. Physical Review E, 99550024, 2019.
E. Lau, Z. Furman, G. Wang, D. Murfet, and S. Wei. The local learning coefficient: A singularity- aware complexity measure. International Conference on Artificial Intelligence and Statistics, 2025.
W. S. McCulloch and W. Pitts. A Logical Calculus of the Ideas Immanent in Nervous Activity. Bulletin of Mathematical Biology, 52:99-115, 1990.
11

---

Published as a conference paper at ICLR 2026
C. McDougall, A. Conmy, C. Rushing, T. McGrath, and N. Nanda. Copy suppression: Compre- hensively understanding a motif in language model attention heads. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 337-363, 2024.
T. McGrath, M. Rahtz, J. Kramar, V. Mikulik, and S. Legg. The Hydra effect: Emergent self-repair in language model computations, 2023. URL (ttpstt:: pettps://arxiie //tp///aavv../////////eeeeeaaeva wwtp:/wwwwwgw/////////// 157771
R. Meyes, C. W. de Puiseau, A. Posada-Moreno, and T. Meisen. Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations. Preprint arxiv:2004.01 01244000..11444444 [cs, q-bio], May 2020.
A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, and M. Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018. URL http:///eeanreeee....eeeeeeeraaa
net/ Eorum?id=r11u0
N. Nanda, J. Lieberum, and B. Barak. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.052/1 2023.
C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodss, D. Hernandez, Johnston, A. Jones, Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, Clark, J. Kaplan, S. McCandlish, and
C. Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022.
M. Phuong and M. Hutter. Formal algorithms for transformers. Preprint arXiv:2207.09233 [cs.LG], 2022.
T. Rauker, A. Ho, S. Casper, and D. Hadfield-Menell. Toward Transparent Al: A Survey on Interpreting the Inner Structures of Deep Neural Networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 464 483, Los Alamitos, CA, USA, 2023. IEEE Computer Society.
T. T. Rogers and J. L. McClelland. Semantic Cognition: A Parallel Distributed Processing Approach. MIT Press, 2004.
C. Rushing and N. Nanda. Explorations of self-repair in language models. N0t.:2402.13390 2024.
arXiv preprint
A. M. Saxe, J. L. McClelland, and S. Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23):11537--1144 2019.
C. Tigges, M. Hanna, Q. Yu, and S. Biderman. LLM circuit analyses are consistent across training and scale. In The eighth- Annual Conference Neural Information Processing Systems, 2024.
5vNudIE
E. Urdshals and J. Urdshals. Structure development in list-sorting transformers. ICML SMUNN Workshop, 2025.
E. Urdshals, F. Lau, J. Hoogland, S. van Wingerden, and D. Murfet. Compressibility measures complexity: Minimum description length meets singular learning theory, 2025. URL httpss
12077
A. Veit, M. Wilber, and S. Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016.
G. Wang. J. Hoogland, S. van Wingerden, Z. Furman, and D. Murfet. Differentiation and special- ization of attention heads via the refined local learning coefficient. International Conference on Learning Representations, 2024.
K. R. Wang. A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference Learning Representations, 2023a.
12

---

Published as a conference paper at ICLR 2026
Q. Wang, M. A. Powell, A. Geisa, E. Bridgeford, C. E. Priebe, and J. T. Vogelstein. Why do networks have inhibitoryneegative connections? In Proceedings of the IEEE/CVF International Conference Computer Viston, pages 22551-22559 2023b.
Watanabe. Almost all learning machines are singular. In 2007 IEEE Symposium on Foundations of Computational Intelligence, pages 383-388. IEEE, 2007.
S. Watanabe. Algebraic geometry and statistical learning theory. Cambridge University Press, 2009.
M. Welling and Y. W. Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In Proceedings of the 28th International Conference on Machine Learning, 2011.
S. M. Xie, S. Santurkar, T. Ma, and P. Liang. Data selection for language models via importance resampling. Preprint arXiv:2302.0316) [cs.C1.]. 2023.
L. Yan and R. Jia. Promote, suppress, iterate: How language models answer one-to-many factual queries, 2025. URL httpa//x.....r.//////////// 20475
B. Zhou, Y. Sun, D. Bau, and A. Torralba. Revisiting the Importance of Individual Units in CNNs via Ablation. Preprint 1rXiv:1806.0283 806.02891
[es], June 2018.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
The appendix provides supplementary material to support and expand upon the main text. It is organized as follows:
Appendix
A Contains mathematical proofs for lemmas presented in the main text.
Appendix Describes the datasets used in our experiments.
Appendix This section details the experimental and computational methodologies, covering loss definitions (Appendix C.I susceptibility estimation (Appendices C.2 to C.4
scalability to larger models (Appendix C.5 SGLD parameters (Appendix C.6). data mixing
(Appendix C.7 and pattern definitions (Appendix C.8).
Appendix D: This section offers further theoretical context for susceptibilities, including analogies to physics (Appendix [D.I), discussions on perturbation scale (Appendix [D.2). relation to Local Learning Coefficients (LLCs) (Appendix D.3). connection data modes (Appendix D.4), and a geometric interpretation (Appendix D.6
Appendix E: This section presents additional empirical results, including metric compar- isons (Appendix [..)) context length effects (Appendix [E.2), explanations of susceptibility differences (Appendix E.3 bimodal token analysis (Appendix E.4), and further PCA details (Appendix E.5
Appendix Presents a detailed breakdown of top per-token susceptibilities across various attention heads and datasets.
Appendix G Details alternative structural inference approach using PCA on susceptibil- ity trajectories over training time.
Appendix D Analysis of the per-token susceptibility PCA for three additional training runs.
J Appendix Information on the sample variance of the principal components and loadings heads.
14

---

Published as a conference paper at ICLR 2026
A
PROOFS
Proof of Lemma 2.2 Let Z be as in Section 2.2
Then
n33 x
(.))
(16)
0h
h=
(w) (ep((-33L(ww))p((
(17)
0h
Z
h=0
0
Dh (0) n3((0
(18)
0h
7
1==0
Since
0
/
|-
nB
L [w)
exp(-nB (xp(-n8L"(u)))((uoeer
(19)
Oh Z
(Zn k2z
Ōh
we have
n1 Z
AL) 3-
(20)
Ōh Z
k==
Combining (18) and 20
we obtain
n, x =
n (T))
Dividing through by n, gives the result.
Proof of Lemma
2.3
We compute
0
AL
I
Öh
h=0
θ
4h (z. y) log plyl, w) wddddy
0h
Oh (x, y) logp(v|æ, w) wddrdy
(al (xx y) g(z, v)) log plyla, w)dady
L I. (w) I(w)
using the definition of on 9h from Section 2.2
Lemma A.1. We have
/ XxY X(ex) of (a, v) dx dy.
x
Proof. Exchanging the order of integration gives
J o (I, v)X dy = - / d'(x,)) v) Cova (o(w), (w)) L(w) dy
/ d'(x, 2) ( ((w)) ((w)) - [(w)) w-)) (d(w): w²) ), ((())))))) (w) L(w): w Jo dy
(o(w)) (I- L(w) wn)
+ (((w): w²) )8 L (ww) L L(w); w0)
= = Covp (6) AL] =x
where (1() ʃ d' (w),) dy.
15

---

Published as a conference paper at ICLR 2026
Table 1: Details of the Pile (Gao al. 2020) subset datasets used in our analysis.
Dataset
Description
Size (Rows)
PILE-GITHUS
Code and documentation from GitHub
100k
FLE-PILL-CO
Web crawl data from Common Crawl Scientific abstracts from PubMed
100k
PILE-PUBMED ABSTRACTS
100k
PILE-USPTO BACKGROUNDS
Patent background sections
100k
PILE-PUBMED CENTRAL
Full-text scientific articles
100k
PLEESTTCKKECHIANGE
Questions and answers from tech forums
100k
PLEEWIKKPEDDA_ER
English Wikipedia articles
100k
PILE-PREELAW
Legal opinions and case law
100k
PILE-ARXIV
Scientific papers from arXiv
100k
PILE-DM MATHEMATICS
Mathematics problems and solutions
100k
FIL--ENRON EMAILS
Corporate emails from Enron
100k
PILE- HACKERNEESS
Tech discussions from Hacker News
100k
PILE-NIII EXPORTER
NIH grant applications
100k
PILE SEGE_SUUSESSSMIE
Combined samples from all subsets Combined samples from all subsets
6.66k
PILEIM
1M
Note that in the main text actually consider mixture between and (1 5hhg + öh q' and so
AL 7h (w) - L(w). Substituting in the above we obtain not = q/ (xx ()))), do dy but rather
x 8h / g'(x,, x)x dy
(21)
since L (w)) = (1 (h) + 8h L'w)..
B DATA
The datasets used to evaluate model performance described in Table These generated by filtering for the first 100k rows from subsets of the uncopyrighted Pile (Devin Gulliver (022), which was generated by dropping copyrighted subsets from the Pile (Gao al. 2020)) Several subsets were omitted because there were too few rows to easily retrieve 100k samples.
We use two github datasets PILE GITHHBB and AUTHUB-ALL. The former is pile subset, while the latter is CODEPARROTT/GTTHU-CCOM and is included for consistency with (Wang et al.) (222)).
C METHODS
C.1 EMPIRICAL LOSS
Recall that denotes the set of tokens, and our sample space is X Y where X =U- g and Y = Σ. For a probability distribution P over tokens we denote by P[t] the probability of token t, and fw is the function from contexts to probability distributions of next tokens computed by the transformer then for z E * and y E we set
ply|,, w) softmax((fu()))))
See Phuong and Hutter (2022) for a formal definition of transformers. In practice for transformer training pairs (x, y) are not sampled i.i.d, but instead are generating from full contexts SK (t....,tt)) E ? We denote by Szx the sub-sequence (f1....,ta (x) of Sg. Our dataset D,, is
available on HuggingFace at httpss/muggingaaee wwt.://wwa...ccooeeceeeeoooeeeee t Cimaeus / /tttttaaseeeeeeeeee aset pasets-pples from The Pile (Gao et al. 2020) and specifically from the uncopyrighted version (Devin Gulliver 2025). with cleaning performed by removing all content from the Books3, BookCorpus2, OpenSubtitles, YTSubbitles, and OWT2 subsets.
datasets pileppeee----e---------- subsets- 5at1--66335555777222223333333330
These datasets are derived
16

---

Published as a conference paper at ICLR 2026
collection of full contexts, {5k }
The empirical loss with respect to Dn is
K-1
1 Σ K
(w((w) =
log (softtma
(22)
k=1
We can think of this as the negative log- regllikeiihood for biased samples {(S ( iskck, from q(z, y) over Y as the negative oogliiilihood for the unbiased samples {5k j==1 over full contexts with probability model
K-1
p(Sk () = II plt , w)
In this paper K = 1024 and all full contexts begin with a BOS token < </endfftext 1> (which has index 4999 in our tokenizer) so p does not model the first token. In the case where we think of p as modelling full contexts the factor K K-i is unnatural (although standard transformer training).
we set (w) = (K (2)) then
nB L, = n2 K-1 Ln
so the effective inverse temperature in tempered posteriors is R-I- Similarly if we want treat Ln (w) as the empirical negative log-likel for the biased set of n n(K 1) samples then
nB In n((K 1) In
so the effective inverse temperature is again 8 K-I-
C.2 LOCAL SUSCEPTIBILITY
Let us consider the observable & from 7 for some component and a variation in the data distribution in 3 for some q'. Substituting this observable into (12), we denote the result by
(w="CC,')) = Lo w", "²,"",,""""") + (d:w", w",
(23)
where (-:w",C, C)2 denotes an expectation defined as in Wang et al. (024)) where we fix the pa- rameters u² To explain: the generalized function 8(u
in means that the expectations involving & are computing an integral over the weight-refined posterior, where only parameters in the
circuit C (represented by the variables v) are allowed vary. However the expectation (AT; w²) )
is defined with respect to the "full" posterior, where all parameters are allowed to vary.
If we let (=r) denote approximate samples from the weight-refined posterior for C' (that is, the same kind of samples we would use to define the weight-refined but not also data-refined LLC) and let {w]]r_r denote approximate samples from the full posterior (the kind of samples we would use to define the ordinary LLC) then the susceptibility is
x(w':C,g')
= 1
(w:) - In(w")) Ann (w:)
(.n(w:))
1=1
[n(ws) - Ln( [ Σ ALn(w))
+
2
1=1
(w.) Ln(w")) {L (wr) [n(we)}
(wx) (n(
Σ (La (wi) (n((1)]]
(24)
C.3 IMPLEMENTING SUSCEPTIBILITIES
Let & denote the susceptibility estimate 24 for the probe data distribution q' and weight restric- tion C. where wy are SGLD weight samples from the weight restricted posterior, wo are SGLD
17

---

Published as a conference paper at ICLR 2026
weight samples from the full not weight restricted posterior, w² are the initial model weights, Ln is the loss function on the pretraining dataset on a random batch of size n. in this case 1erilee-mm2,,, and L is the loss function random batch of size = (1 - 6 + 5hg'. That is, a mixture between the pretraining dataset and probe dataset o' using the process specified in Appendix C.7
q timaaustaa//.......a....a /
These susceptibilities were found for each of the datasets listed in Appendix B for each checkpoint step in
= {0, 100, 200, 300, 400, 600, 700, 800, 900, 1000, 1500, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 12500, 15000, 17500, 20000, 25000, 30000, 40000, 49900)
and each attention head weight restriction, as well as no weight restriction at all. For these experiments 64, and 8h 0.1. SGLD hyperprameters can be found in Appendix C.6
we had a batch size
The hardware used for the computation consisted of 10 A100 GPUs run in parallel, each computing all weight-refined susceptibilites on all datasets for an evenly distributed subset of S. each GPU took about 20 hours to complete its task.
C.4 PER-TOKEN SUSCEPTIBILITIES
With the same notation as Appendix
we estimate per-token susceptibilities using the equation
{(x,y)
(w)) { Ln (we) - (nw (( - Ln bnx))}
(Ln(wr) Ln(w") >}
Σ (w)= - Lm Ln)
For total of approximately M 160 x 1023 163680 samples, across 160 different contexts, where each context d² (c....., cf) is list of k I 1023 tokens, with the property that for all j E [2, ki), (1) Drr-
We choose these contexts using the datasets Dataset.........a..... shuffle method with seed 0 after tokenizing the dataset under consideration, and taking the 160 contexts.
To visualize these per- per-sample susceptibilitics, let
x(c)) = Σ
Ln (we) - Ln(w")] (liors 1103) (m))
[=1
+
Σ (w) - Ln
1=1
Then this is a real number, and we can map it to a color value via either or where the only differences
Require: X(c)) E R
1: max & max,, 18(c))
2: if x(e)) > 0 then
[x(eT)] max(x)
3: color +- rgba
0,255, O,
4: else
[e(eT) max(x)
5: color + rgba
255, 0, O.
6: end if
are the shade green and the quadratic versus linear opacity scaling.
Each of can also be decoded into string of characters using our model's tokenizer. Therefore highlight each such string the corresponding color computed above. Let s(c)) be this highlighted
18

---

Published as a conference paper at ICLR 2026
Require: X(c)) E
1: max(x) , 18(())
2: if x(ee) > 0 then
[(())
3: color -- rgba
0, 12,1, 0, max(x)
4: else
[R() max(x)
5:
color rgba
255,, 0, O,
6:
end if
string. Then this produces Figure 9 usiin the first coloring algorithm while the latter is used to produce Figure and Figure
For Figure 9 we sort each per-sample susceptibility in a descending manner, giving us the list
{ 2 2 X(c) We then choose the first 100 of these susceptibilities, and visualize their surrounding context by concatenating each of their highlighted strings. With + denoting concatenation,
8 11-20 2200
+
+
((5++000)
8 sc,) 10
200
+
s(c ((
And also choose the last 100 of these susceptibilities, and visualize the surrounding context the same way
8( PM
J1 2200
+
+
8 J
(1++200)
s(c 31 112000
((C)
TMM-10
For Figure 4 and Figure 1 the same context samples from each dataset but do sorting of the data and instead search manually for representative examples.
Here all the same definitions and parameters in Appendix CS and SGLD sampling with hyperparameters as Appendix [C.6] modified to only use 100 draws. We run these per-token susceptibilities for all datasets in Appendix B and on checkpoint step 49900.
The hardware used for this computation consisted of 16 A100 GPUs, each computing a quarter of the weight restrictions for each seed. Each GPU took 50 hours to complete its task
C.5 SCALABILITY LARGER MODELS
There are two main obstacles to scaling the results in the paper:
1. The computational cost of SGLD sampling and evaluating losses to compute susceptibilities.
2. The complexity of larger models.
Regarding the first point, estimate the cost of compute follows: for given model, let 7 be the time cost of a training step, F be the cost of a forward pass, the number of probe datasets, m the number of model components being analyzed, and C some roughly constant O(100) number of SGLD samples. Then we expect that the cost of estimating susceptibilities on given checkpoint of that model scales like
mC + kF).
In practice, we take approximately log-spaced checkpoints of models, so the total cost of analysis scales logarithmically with the total number of training steps and typically ends up comparable to the cost of training. Some preliminary work suggests that we may be able to significantly reduce the computational cost in larger models without loss of signal, such as by using much less than the entire set of attention heads. Not accounting for these cost improvements, we were for example able to susceptibilities the full set of attention heads (384 in total) on Pythia 41 Om and Pythia 1.4b
19

---

Published as a conference paper at ICLR 2026
After fev iterations of the feedback locp, the measured light proflles shown in Fig. Miiggonnt (conting oux Light palterns) and Pig. (iag:disel (dpascele speli patierns) show increased accur and note clasely resenble their carqeE patterns. A sumnazy of the imprevenenta to these optical traps due to the foedback
process, along with the number of iterations required (+Motaaf and the light-usage effich be found
G
V[tabErea
08
04
Pattern
Before
After
Before
Aftes
ONIorE #hright [9[
Indented Ring
21.9
9.8
21.5
Mapt
Square lattics
22.0
15.3
3.6
14.8
Ring lattice
13.7
10.0
2.5
1.1
10
12.5
Burnazy of Lxprevementa including erior, iterations (paa)
and Effici Bocy Ganaas
[tabiresk]
[Continugui patterna eptinised by feecback.
Zing pattern
restriction (ap in Fig. Milg:flon I showing the neasured Bignal Begion intensity using the initial HRAF-C alculated phase profile (left) and after 8 iterations of feedback optimi (rightl, which improve Sepsi on on"(Vieft (10V 18 fron 39.8188 to 1M (Belus) The interwity acound the siscunfezence of the ring showing the tarqet pattezn (zed), and the neasured profile before (azeea) and after (blue) teedback (b) RMS progression for the ring pattern with restriction, recorded each of the feedback algorithm. (c) Gagaiian double-we
Bignal Region intensity before
(right) feedback.
Blo%) cut actoss the centre of the sells, showing normalised pixei Intensity
Figure 4: Visualizing per-token susceptibilities for three heads sample from ARXIV. Each token is highlighted in three segments (top, middle, buttom) which correspond to the per-token susceptibilities for three heads (0:0.12 177). The susceptibility values standardized for each head, with solid red representing a z-score of -3 and solid green representing a z-score of +3, Green means positive susceptibility and red means negative.
(Biderman et al., 2023) for around $2,000 and $5,000 respectively, for a single checkpoint each. For sampling more generally, Urdshals et al. (2255) establishes LLC estimation at scale by studying Pythia models up to 6.9B (see e.g. Figure 3).
We also note that, apart from SGLD sampling, the scaling behavior of the compute cost is similar to e.g. circuit discovery via ablations, which also requires evaluating forward passes for a set of model components being ablated.
As model size grows, these computational costs require paralellization across multiple GPUs, but this is done in the standard way and is well-supported. Overall, there theoretical or practical opstables to scaling to 10B parameters, and the costs are not prohibitive.
Regarding (2), the difficulties paper are simple, and so the patterns in the data and corresponding structures the model are remarkably visible in the susceptibility matrix. We do not expect PCA on its own to be sufficient to discover interesting structure in larger models, which "understand" more complex patterns. In these settings we also make use of non-linear dimension reduction techniques like UMAP, and more sophisticated data analysis to study the internal structure in models. The present paper should be viewed as validation that simple data analysis, simple models, reveals simple patterns; we agree that it remains open question (to be addressed in future work) whether more complex data analysis reveals complex structures larger models.
more uncertain. The small transformers studied in the present
Intuitively, our method should be thought of as somewhat comparable to EK-FAC for data attribution: while is computationally nontrivial to estimate Hessian inverses in large models and the results not straightforward to interpret, this is routinely done (see c.g. R. Grosse et al "Studying large language model generalization with influence functions") because the information gained is believed to be worth the cost.
C.6 SGLD HYPERPARAMETERS
Following Wang et al. 2024) we used SGLD with hyperparameters y = 300, n,3 = 30, E 0.001, batch size 64, 4 chains, and 200 draws for the regular susceptibilities in Appendix C.3 but only 100 draws and batch size 16 for the per-token susceptibilities as in Appendix C.4
Note that for these experiments did not use RMSProp SGLD.
20

---

Published as a conference paper at ICLR 2026
100
Word
Percentage of Tokenss
Word Part
Word End Induction
Spacing
Formatting
Right Delimiter
Left Delimiter
20
wxw
dm mathematics
EC
enron, smails
freelaw
github-all
hackemews
nift exporter
putmeed abstracts
pubmed central
stackexchange
uspta hackgrounds
wikipedia en
Figure 5: Percentages of tokens in each dataset which follow given pattern. Note that not all patterns are mutually exclusive, which is why some totals exceed 100.
C.7 DATASET MIXING
To create the mixed dataset Doh consisting of samples from qoh -8h)g öhg' we interleaved 1 iin order to ensure even mixing, the pretraining dataset with each probe dataset using Algorithm determinacy, and the property that if q' q. then D D
Let (I,, 14) be samples from and (x', vf) be samples from q', with N = 1, 000, 000 the size of the pretraining dataset, and M 100, 000 the size of the probe dataset. PILEIM
Algorithm 1 Data Mixing Algorithm
Require: D2 = {x,, n)) Dis = [(2). W0 5h e [0,
1: Dah min(NNM) +-
2: probe count <- 0
3: 1+1
4: while j < N, M do
5: target probe count [(j 1)8h]
6:
if probe count < target probe count then D Min(N,M) Ahh N.M) {( 1/))
7:
8:
probe count + probe count + 1
9:
else
DÅh end M.) + 5oh MiiuNN U {(),,,,))
10:
11:
3 -j++
13:
end while
This new dataset is then shuffled, and during SGLD minibatches of size 64 are drawn from this shuffled pre-computed data-distribution for loss estimates.
C.8 TOKEN PATTERNS
Let Σ denote the set tokens in our tokenizer.
Definition C.1. A left delimiter token is an element the set of tokens
< 5
21

---

Published as a conference paper at ICLR 2026
right delimiter token is an element of the set of tokens
1. ); 8) $
We call token delimiter token if it is either left or right delimiter token.
The asymmetry between left and right delimiters is due to the tokenizer. For our model right delimiters seem much more important than left delimiters.
Definition C.2. A formatting token is an element of the set of tokens
eenoffeett
Definition C.3. A word start single token that begins with space and followed by lower upper case letters. That is, it is token which when de-tokenized matches the regular expression [A-2a [A-2a-2] 2] +
Definition C.4. A spacing token is token which when de-tokenized is a sequence of characters from the set
At
Definition C.5. A numeric token is token which when de-tokenized and with spaces removed, consists of one or more digits.
The patterns defined above independent of the context in which token appears. By contrast, the subsequent definitions apply to a token in given context.
Definition C.6. A word end token is token which when de-tokenized is made up of upper or lower case letters and which is followed in its context by single formatting token, delimiter or space.
Definition C.7. word part token a token which not a word ending in its context and which when de-tokenized consists of upper lower case letters.
Definition C.8. An induction pattern is sequence xyUsy where U E Σ" and x, y E E, satisfying the following conditions:
The conditional probability of y following a satisfies g(y (x) S 0.05.
o'
An,
the to :, and by in be
Note that can be the empty sequence and may contain occurrences of x, y. In a given context classify a token as an induction pattern token if it is for an induction pattern ryUry within the context.
We use estimated conditional probabilities based samples from the Pile. Note that the sets of left delimiters, right delimiters, formatting tokens, word start tokens and word part tokens are pairwise disjoint. The set of induction pattern tokens and word part tokens are disjoint. The percentage of N 20000 tokens sampled from each dataset which fit each of these patterns are given in Figure
D THEORETICAL BACKGROUND
D.1 ANALOGY WITH MAGNETIC SUSCEPTIBILITY
The Bayesian susceptibility introduced above is the direct analogue of susceptibilities in physics, such as the magnetic susceptibility. We recall briefly the linear response of a spin system placed in an external magnetic field H. With energy E(or (which may also depend H) and magnetization M(a) = E, oi the field- perturbedd Boltzmann weight is
expl expl- (B[() H M()))) ZH 3n=fexxl-B(C -H00((E--)]]).) HMM) do.
pH
Zu
22

---

Published as a conference paper at ICLR 2026
The magnetic susceptibility is the first derivative of the equilibrium magnetisation,
①
(M)
(M) - M(o)
Xmag
OH H=o'
Its sign carries immediate physical meaning:
Sign of Xmag
Interpretation
Xmag
(paramagnetism) (insensitive)
spins align with H; response reinforces the field system insensitive already saturated
Xmag
Xmag
0
(diamagnetism)
induced currents oppose H; response cancels the field
The Bayesian susceptibility parallels the magnetic susceptibility, with the probe parameter replacing the magnetic field strength H and Φ(w) replacing the magnetization Mo).
D.2 THE SCALE OF
Since the scale of the parameter h is arbitrary, there is sense in which susceptibilities are only well-defined up to rescaling. This is casy to understand if we consider multiple variations of the form 3), say q >> q' and q - q" where the KI. divergence between q, q. much larger than between q. q' By default therefore we should not read too much into comparisons of magnitudes of susceptibilities across different variations of the data distribution.
then For example if we set h h for some
1 0
K
0 Th
nB3 0h (0)
B.
h-0
nz3
(0)
-
(25)
h=0
so that ALn (w) is finite difference quotient. This could be motivated by the prineiple given above that we should rescale the deformation parameter h since our interval is "really" of length 5h.
It would also be natural to include factor 1/5h in 13)
Remark D.1. Given target distribution q' (ax, y) and the mixture from Lemma 2.3 However, if the difference between of and g is large, w" may not be a local minima of (w). The likelihood of this being reasonable assumption increases as we move of closer g-
obtain AL L'ww) L [ww
Since susceptibilities are in any case infinitesimal, this suggests that we replace the original variation from q to q' with variation from g (1 5h)g öhg' for some small 8h. In this case the mixture is
h [0, 1] (1 h)g h(1 8h)q höhg'
(1 höh q + höhq'
In this case, again by Lemma 2.3 will have that AL is finite difference
AL(w) == (w) L(w).
For local susceptibilities we always choose variation which is sufficiently small, in this sense.
D.3 SUSCEPTIBILITIES AS A DERIVATIVE
Let w² local minima of L(w). Recall from Lau et al. (2025): Watanabe 2009) that the estimated local learning coefficient (LLCC) is defined by
((w) n.3
E "ww""," . (w)) Inn (ww))
(26)
where R "www is the expectation with respect the Gibbs posterior
Suppose we approximate the derivative in the local susceptibility by difference quotient
Xtinite Xfnite(w")) (w*) 1 n.9 5h (0; w") 9.86 sh (0:: u=))))))
23

---

Published as a conference paper at ICLR 2026
When the observable is associated to component C' as in Theorem 2.4 but with dependence
d(w) 8(u =²) [w) L (w))
the expectations in question up scalars, estimates of the weight- and data-refined LLCs as introduced in Wang et al. 2024) once we replace the annealed posteriors by the ordinary ones, since
E whe'
[i(u ((u-w')( "') - 1 ((((w")
1 \(w"; ;0.. C).
n.
Hence Xfinite (w*) ) related the difference quotient
1
; gah, C) Xww"; :q.
(n.3² 8h
which measures rate of change of the refined LLC as function of the shift in the data distribution.
D.4 MODES AND MATRIX FACTORIZATIONS
In the main text we have focused on a decomposition of susceptibilities x as an integral of per-token susceptibilities X(v.y) (Theorem A. Such a presentation implicitly relates choice of token sequences as basis of function space containing the conditional distributions q (z). and it is conceptually useful to consider an alternative basis of "patterns" or more precisely modes. In this appendix we explain how make this precise in the setting of (Chen and Murfet 2025).
That paper explains how to think about the data distribution over sequences as a tensor, and use tensor decomposition methods to produce natural orthonormal bases of function space. Similar techniques are commonly used in the field of natural language processing (Anandkumar et al. 2012).
We fix finite alphabet of tokens V and let
* Hb,1 (2* RR")
E where k. I > O. In the main text 1 S k 1 and / Here E denotes the vector space of formal linear combinations of sequences of tokens of length 7. The inner product is defined by
be the space of functions *
+ R
(.g) - ((()), g(x)dx.
We set x Y L with the counting measure, q(x, u) is the distribution for x E E* E E and ((x) = q (I) denotes the distribution over sequences of length k. Let C E H be the function corresponding to the true conditional probabilities
C(x) = / ((y/x)) ydy.
Vo E E
(27)
As explained in Chen and Murfet 2025 by applying tensor decomposition methods such SVD to C we obtain a natural orthonormal basis {Cays IeEA.9E+++ of H. Here A indexes right singular vectors, A indexes left singular vectors for nonzero singular values, and ++ 2 A is an extended set of indices associated an arbitrary extension of the left singular vectors an orthonormal basis of E The examples given in Chen and Murfet (2025) justify relating such modes to atomic patterns in the data distribution. In this section when we over a, 3 we always mean E A and ++ respectively. Sometimes we use y in place of 3.
Let us consider variation of the data distribution y(z, v) which is a "concept shift", that is, we vary the conditional distribution q(ylz) (z) but not (zx). Let us write g' ('(y/z) for some other conditional distribution and set gz( y) = q' (1))z))(()) Let C² denote the equivalent of (27) but for g'
Recall that ((.w)) (w) = - log plyl w). Let x be a susceptibility as defined in Section 2.3
For consistency with Chen and Murfet 2025 the inverse temperature.
continue to use 3 index, note this is distinct from
24

---

Published as a conference paper at ICLR 2026
Definition D.2. We call
C' -- cab
(28)
0
the mode decomposition of the data distribution shift.
Definition D.3. Given w E W we define ((w) E H by
(x)) / ((x.)) ydy.
Given a E A we define aas : W - R by & (w)(w) = (D(w), eas)
Lemma D.4. We have
x= Σ can Cova [0o Par]
Proof. We compute that
AL = ( ((vl=))g(>>) log plyl, w) drdy
=- =- e / (a)())(V))(z)) log p(y w) drdy
(B(w))
so the conclusion follows from Lemma 2.2 and bilinearity of the covariance.
Definition D.5. For any E A,7 E A we call
Xay = Cov [Ø) Panl
the susceptibility of é to the mode pair y.
This depends on the observable é but not on the variation in the data distribution. From Lemma D.4 obtain that the expression of the susceptibility in the mode basis is
X =
Σ
Xory
(29)
0,7
for some coefficients e
D.5 STRUCTURAL INFERENCE AND MODES
In this section provide motivation for the approach introduced in Section 3.3 by explaining how the theory of modes provides a factorization the data matrix. We use the notation introduced the main text and the previous section. We denote by X the susceptibility for da and observable Øj--
We denote by Xan.j the susceptibility for a mode pair a, 7 and the observable Φj. as introduced in Appendix [D.] Ideally would like measure Xa but in practice do not have the ability to sample from variations of the data distribution which only affect single modes. However, by Lemma D.4 (for mode-aligned concept shifts)
day Xays
(30)
aεA
where we note that ed,ay is from Definition
D.2 and does not depend on j. This is matrix equation
X
CPP
(31)
where C' is the (D) x (IAJAA++) matrix with entries cd.a7 and P is the ([A]A++1)) |H] matrix with entries Xar.j. Then C' gives the coefficients of each variation of the data distribution in each mode and P is the matrix of mode susceptibilities. More informally, coefficients between variations in the data and modes, and P as the coupling coefficients between observables and modes.
think of C as the coupling
25

---

Published as a conference paper at ICLR 2026
DM math
(D
(o(w)
Training 4
F
(12)
R
arXiv
(b))
Statistical manifold of data distributions 2
Frechét manifold of "Posteriors" Prob(W)
Figure 6: Susceptibilities as components of tangent map. We can think of susceptibilities as components of the tangent map from data distributions to the expectations of family observables with respect to the posterior formed for that data distribution, with fixed model. In this diagram we indicate q E L, the training distribution, and tangent vectors in this space correspouding to the mixtures with ARXIV and DM MATHEMATICS. The tangent map of F sends these tangent vectors to tangent vectors in the space of "posteriors".
The quantities X5 may be estimated from data, that is, we obtain empirical estimate of the data matrix X. On the other hand C, P are not directly observable because it is intractable to determine the complete set of modes of complex distribution. However 3I suggests way apply methods of data analysis to the matrix X order to infer the modes and their couplings to the observables.
Applying matrix factorization methods like SVD to the data matrix X is therefore way to attempt to recover (linear combinations of) the underlying modes of the data distributions, or at least a coarse-graining of them. In this interpretation, the right singular vectors tell about the relationship between attention heads and modes, and the principal components tell us about the coupling between modes and datasets.
D.6 SUSCEPTIBILITES AS TANGENT MAP
This is all summarized neatly by Figure 6 We can define smooth map F which sends distribution o' E 2 over sequences of tokens to (for fixed n, () the annealed and tempered posterior distribution F(4') (woq') on W (Bauer et al. (0166). Suitably well-behaved observables Φ, W - R define, by taking expectations, functions this manifold. Given a {Øj}jEH therefore obtain a map
point in the Frechét manifold Prob(W)) of positive densities of integral
(d))(-)
G=
Prob(W) - R
(0s)e-)
Composing with F gives a map D - IR whose tangent map at q is
T (G o = (w) (G) (F) T,9
Tio
(32)
where (Ø) denotes the vector of expectation values of all observables with respect to the unperturbed (annealed, tempered) posterior (w). By definition this linear map, or rather its restriction to the subspace of the tangent space spanned by the tangents to the mixtures with the q² has as its matrix the (transposed) response matrix X (up to a factor of n.3). The relationship between the matrix factorizations in (31) and 32 seems important.
26

---

Published as a conference paper at ICLR 2026
STXTA
Figure 7: Pairwise comparisons of per-token susceptibilities, ablation, and loss as density maps for attention head (0:0. The color bar legend on the right of each subplot indicates the number of points represented by each colored hex.
E SUPPORTING RESULTS
In this section we collect data and analysis that was performed to validate the basic methodology of susceptibilities, and which supports the headline results of the main text.
E. PER-TOKEN METRIC COMPARISONS
In Figure g select a pair of attention heads 0:0, 1:2 and pair of datasets GITHUB, DM_ MALMATHEMMATCS and for data sampled from each dataset we compute the pair (8, a) where is the per-token susceptibility and is the loss after ablation minus the pre-ablation loss (we call this the ablation delta). The correlation between these two metrics is very small.
specific example where the induction pattern is particularly visible in the positive susceptibilities, while there is no discernible induction pattern in the zero-ablation loss differences or original per-token loss.
In Figure
E.2 PER-TOKEN SUSCEPTIBILITY VARIATION WITH CONTEXT
In Figure 10 we plot the change in average per-token susceptibility for each atten- tion head using 160 contexts from each of the following datasets: pilelm, arxiv, dm, mathematics, enron emails, freelaw, github, nih aaho_eer,,er, philpapers, pubmed_abstraa abstracts, pubmed_central. stackexchange, uapto_backgrounde wi wikipedia_en.
27

---

Published as a conference paper at ICLR 2026
Permlllen abletton dell
= 0..03
-0.001
0,001
-0,009
Per-token atbatoo
0.5
-0.5
-3.0
0
Per-toion susceptiblity
Per-token suscepsbiltty
Figure 8: Susceptibilities ablations. Each plot shows distributions of per-token susceptibilities versus zero ablation loss differences as density maps. Each subplot corresponds to a pair of attention head and Pile subset, with datasets fixed across rows (top: GITHUB, bottom: DM MATHEMATICS) and heads fixed across columns (left: (0:0, right: (12)). Hexagon colors indicate count of tokens with susceptibility and ablation delta values in given region. Shown inset the Pearson correlation coefficients, indicating negligible correlation.
usas tureau of Land Haxagenent. 1999, Lowden Rance Preseraled Fire Roruev Fira: Raport, KiLatass Fare
USIT Sareau of Land Nanagarent. 3050, Lowden Rancs Prescr dee keview Final Report. vildland Fire Lessors Conter. (Gntine) J0ly 22, (ostes July
1502 Bureati of Last managements 1999; Lowdes narch
Freecribed thed Fire Rextie Firal Reportt Mildlast Fire
rescors rearee Center- jeetine] July 22, 1999. trer July 2013.]
Lessora: Leamnl Center. [ontinel zuly 22, 1238. ltitte
2013:]
Srssiii-ierrieeeesssss www//ee Escaec Excapet _redees.pct.
Grande PraseFibed PArn: Inveatigation Report. Nitdland rire
National Parki Services
USB2 Natsoral Park Service, 2800. Cerra Grasde
158: National Park Servace, JRl at, 2009, certe Sriece:
Prescrabet Firei Investigatian Repert, Mililand fire
Frescribed Firel Eivestigation Reports wilddaand Fire
Lessers Learne Conter. [tellne) Myy 18, 2099. (catee):
Leceste Learzed Center. (Dntice)) May In, 2009. (Coeed:
LeasBER Learned Conter (Dnlirel May 18, 2000. (cicedt
Jane 28, 1
Jusz 28, 2013.1
Junc 28g 2013.1
http: http://wwwtttieeeeeeeeeeeee setiooc smonts /erroo Grande
attp://weaart Leftire leseeee ret snents jerraa
.Aeport.pof.
_epport.por.
Apper Lppll
Prescratted:
wald fira Lensons Center. [0ntine] actibeer 2286.
[inline] October
sted: 2013.1
itod: June 2013.1
stees June 28, 2013:]
wet///cceeeereeeeeeeeeeee...eeee Por Escapss
areps//wm.. wildfi www.iieeeeeeer..ee... www.s/eeeeree//ooeeeeeeee s/Re, Eoropee,
2S9rT_118400.0 pof,
aport port__33400..pef.
spert 111.11111.0000 pdl.
Figure The same sample from freelaw is shown three times, from left to right with per- token susceptibilities, per-token ablation loss differences, and original token loss highlighted, with susceptibilities and ablations calculated for (1:4). On the left, green indicates a positive susceptibility and red negative one. In the middle, green indicates negative ablation loss difference (ablating the head improves performance) and red a positive ablation loss difference. On the right, a deeper red highlight indicates higher loss that token.
We note that in general the per-token susceptibilities increase with context length, with the strongest rate of increase being in the layer multigram heads (1:0-:5 from length 10 onwards. It clear what the full explanation for this phenomenon is, but it is consistent with the observation that these heads tend suppress induction patterns (Section 4.2) and the longer the context, the more token pairs zy are recognizable as induction patterns.
E.3 PER-TOKEN SUSCEPTIBILITIES EXPLAIN SUSCEPTIBILITY DIFFERENCES
is natural to ask why some heads are outliers for particular datasets but not others. For instance 0:1 is negative outlier (by which mean that it lies below the other heads) for GITHUB but not FREELAW or HACKERNEWS (see Appendix F). Based on Appendix D.4 model for susceptibilities is that head is sensitive to some patterns in the data distribution more than others (we say it is
28

---

Published as a conference paper at ICLR 2026
10h4, 10h5, lah6, 11h0, 11h1, 11h2, 11h3, 11h4, 11h5
0.5
11h6, 11h7
0.0
Average partoken susceplibiiity
-0.5
-1.5
10a
Context position
Figure 10: Per-token susceptibilities function of context length. The per-token susceptibilities for each attention head averaged over pairs x, where r has a given length. Layer 0 heads are shown in blue, layer 1 induction heads in orange and the other layer heads in red. We note strong dependence on context length for these heads 1::0-(1:5.
Per-Token Token Suscentibilties Average
0.50
025
0.00
-075
-0.08
-0.05
-0.04
-0.02
0.02
0.06
Susceptibilitiie
Figure 11: Comparison of average per-token susceptibilities and susceptibilities for every attention head and dataset. The dotted red line is the line of best fit.
susceptible the pattern) and we vary the data distribution this will drive different responses from different heads.
this basis our hypothesis that 0:1 negatively susceptible to a pattern that is common in GITHUB but not in the other two datasets. To explore this we turn to the per-token susceptibilities (Definition 2.5 which we use to explain differences in overall susceptibilities. For this to be sound methodology we need the averaged per-token susceptibilities to co-vary correctly with the susceptibility. In Figure 0.958 and the line of best fit has slope 9.794. Since dh 0.1 we would predict that the slope should be 10. We view this as sufficiently well-coorelated that we can use per-token susceptibilities to study differences between susceptibilities.
we plot one against the other. The Pearson's 's correlation coefficient is
29

---

Published as a conference paper at ICLR 2026
03030
02200
Abvoluut Suin of Wleggaue Suscept
72020
230.0
40020
33020
23000
23000
12020
Figure 12: Top negative susceptibility tokens. Shown is the overall sum of negative susceptibility magnitude from 163k tokens sampled from GITHUB-ALE Left: (0:0, Right: 0:1). In both cases the top contributing token is space
Let us now turn to detailed study of 0:0 versus 0:1 GITHUB (at the end of training). For 163k tokens sampled from GITHUB-ALL we form the data in Table Note that to obtain the scaled sum add up the total positive and total negative susceptibility, compute the average over tokens by dividing by the total number of tokens (163k), and then multiply by 5h 0.1.
Head
Total pos.
Total neg.
Sum
Scaled sum
Mean
Std
0:0
171,561
-14.344
26,219
0.016
-1.49
0.52
0:1
139,458
-205,255
-65,797
-0.040
-1.89
0.73
Table 2: Total negative and positive per-token susceptibility for two attention heads on GITHUB-ALL. and statisties for per-token susceptibilities X(z.y) with y the space token.
Comparing to the susceptibilities in Figure 28 we see that 0:0 lies in the range [0.01, and 0:1 in the range [-0.05, -..06.. The gap in total positive per-token susceptibility 32k gap of 60k the negative susceptibility. Since we want to understand why 00:1 is a negative outlier, we focus the latter gap. In Figure we show the total contributions of individual tokens y (over all contexts 2) to the total negative susceptibility for the two heads. In both cases we see that the bulk comes from the space token which contributes roughly an order of magnitude more than the next most important token. In fact accounts for 44% of the total negative susceptibility for (0:0) and 39% for (0:1. The space token appears 42,764 times this dataset, and the statisties of the per-token susceptibilities for (in any context) are shown in the rightmost two columsn of Table The full distributions are shown in Figure
The difference in the means is enough to explain about 17k of the total 60k gap in negative suscep- tibility, so about a third. This is evidence that primary reason that 00:1 is a negative outlier on GITHUB-ALL is that this head is substantially more susceptible to the space token than other heads (e.g. it 27% larger magnitude than the mean susceptibility of (0:0) and spaces are more frequent in GITHIUB than FREELAW ILACKERNEWS (compare 43k occurrences for GITIIUB with 32k for FREELAW and 16k for HACKERNEWS) This is not surprising since code is often structured using space tokens.
E.4 BIMODAL TOKENS
The per-token susceptibility X(x.)) is function of context and an individual token y being predicted that context. In Appendix E.3 we saw how differences in susceptibilities heads can potentially be attributed to por-token susceptibilities, but there the analysis was of token y
30

---

Published as a conference paper at ICLR 2026
14000
Requency
0000
6000
4000
4000
2000
Suscoptibility Value
Susceptibility
Figure 13: Histogram of susceptibilities. Shown is histogram of per-token susceptibility values for the space token across 163k tokens sampled from GITHUB- ALL. Left: O.O, Right: 0.1.
independent of its context Since not all occurrences of semantically the same, should expect that the distribution of susceptibilities across æ for a fixed y should sometimes reflect this.
Indeed, in this section we report interesting phenomena where this distribution can be bimodal: there are some tokens for which there are two kinds contexts æ with the susceptibilities X(x.)) in each context clustering around well-separated values.
The head 0:0 and token to On both GITHUB-AN ALL and ARXIV we observe that token to is bimodal for the attention head 0:0), shown in Figure 14 On ARXIV there is one mode with positive susceptibility and another with negative susceptibility. A typical positive instance is
app ears
to
be
relative ly
ins
ens itive
where the overset number is the susceptibility of the outlined token. A typical instance of the negative mode
2.23
to
ong.
That is, the examples in the positive mode are normal occurrences to as a word, whereas the negative mode consists of occurrences that are part of LaTeX commands. For GITHUB-ALE ALL the token also bimodal, with one positive mode and a negative mode. Typical instances of the positive mode again occurrences of to in normal English sentences and typical negative instance is
22.17 to
if
by
ip
The head 1:7 and token /. Similarly we see in Figure 14 positive and negative mode in the distribution of susceptibility values for (1:7) and the token HACKERNEWS. A typical instance of the positive mode is forward slash URL:
4.10
2.35
pr org 7 se cono
/
while typical negative instances are "either/or" constructions
-2.76
33.1
ving
ittiing has ic p
go
mult
ier ror
Interestingly the distribution of susceptibilities is also bimodal for the same head and token on LENOO_ with the positive mode still being about slashes in URLs, but the negative mode is closer to zero and occurs much more frequently (see Figure 15). Typical negative instances are
-2.73
1.12
0.33
ay
M ann
C
oo
En
ron
11
09
000
31

---

Published as a conference paper at ICLR 2026
Frequeeccy 20
125
100
75-
Susceptibility
Susceptibiiity
Figure 14: Bimodal tokens. Histograms showing frequency of certain susceptibility values for particular tokens and heads in particular datasets. Left: Attention head 0:0 and token to in ARXIV. Right: Attention head 177 and token in BIACKERNEWS.
200
150
Frequency
1:00
Susceptibility Value
Figure 15: Bimodal token. Histograms showing frequency of certain susceptibility values for 1:7 and token / in ENRON EMAILS.
32

---

Published as a conference paper at ICLR 2026
E.5 PER-TOKEN SUSCEPTIBILITY PCA
Here we collect some additional analysis of the PCs in Section 4.2
In Figure T6 and Figure 17 compare the complete distribution of susceptibilities for all heads USPTO_ BACKGROUNDS for all tokens vs just the induction pattern tokens, and we see here how the distribution for the layer multigram heads (and to significant but lesser extent 1:6) shifts the right.
33

---

Published as a conference paper at ICLR 2026
All Tokens
uspto backgrounge
Density
10h
10h2
138.3
28
Ith-1
Iih-
LBb
0.0
-15
15
20
Susceptibility
Figure 16: Distribution of per-token susceptibilities for all heads on USPTO BACKGROUNDS, mea- sured on all tokens. Curves are produced by KDE smoothing of data from 163k tokens.
Induction Tokens
uspto backgrounds
Density
R0h1
10h2
(yyer:::
1063
[h1 F0 67
l1h4
1b4
107 mv2.22
10
Susceptibility
Figure 17: Distribution of per-token susceptibilities for all heads on mea- sured on just induction tokens. Curves are produced by KDE smoothing.
34

---

Published as a conference paper at ICLR 2026
F TOP SUSCEPTIBILITIES
In this section we give additional details for each dataset (see Appendix The purpose of measuring susceptibilities is to these "responses" external perturbations in the data distribution to infer internal structure. Thus the first test for the methodology whether or not the set of perturbations that we are using (e-g. Pile subsets) elicits sufficient variety of responses. To see this it suffices to compare the plots of susceptibilities for each head across training on some datasets, for instance HACKERNEWS (Figure[2) and GITHUB-ALL (Figure 28).
Note that by definition the susceptibility with respect a variation of the Pile in the direction of dataset q' zero if is distributed identically to q. Thus for subsets that are similarly distributed to the overall Pile we should expect susceptibilities to be small; see Appendix F.9
Attention head h in layer I is denoted Eh). Tokens are denoted t and the token with high (positive or negative) susceptibility is denoted with border like t In this section ær stands for a numeric token, e.g.
13.
In each section we plot the susceptibility of each head for particular dataset, and provide the three largest positive and negative susceptibility tokens taken from the one hundred tokens with largest susceptibility magnitude found in a sample of contexts. Note that these outlier tokens not necessarily a good indicator the "function" a head.
The per-token susceptibility X(r.y) is function of the predicted token and context x. We typically only give y with at most some small segment of n, e.g.
1 W.
wa
vel
ength
where only the last two tokens of the context æ is given. In some cases we also show the largest magnitude negative per-token susceptibilities, with the same conventions. Recall from Section 3.2 that we interpret X(a.u) > for a particular head as a tendency for that head to act to suppress the prediction of in context x, so to a large extent this appendix is a "catalogue of suppression".
In the following :// always appears as part of http :// or https :// unless specified otherwise, and #### appears as part of a "line" underneath a title or separating sections of document. The token appears as part of HTML,, class or similar markup.
F.1 ARXIV
Layer
Layer 1
0.06
0.08
0.04
0.06
0.02
0.04
0.00
0.02
0.02
0.00
0.04
-0.0
10a
10a
10*
10a
10a
10*
Steps t
Steps
10h0
10h2
10h4
10h6
11h0
11h2
11h4
11h6
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 18: ARXIV susceptibilities for each head over training.
The susceptibilities for each head are shown in Figure 18
Examples of positive per-token susceptibilities:
35

---

Published as a conference paper at ICLR 2026
0:2
vel
(nggh (pecially],
differe nces
vel ength
ome
pecial
vel ength
Examples of negative per-token susceptibilities:
te
x
S
1:4
om Us
1:7:
lendoftext.
ree
F.2
DM MATHEMATTTTSS
Layer
Layer 1
0.04
0.06
0.02
0.04
0.00
0.02
-0.02
0.00
0.04
0.02
-0.06
10a
104
10a
10a
Steps t
Steps
10h2
10h4
10h6
11h0
11h2
11h4
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 19: DM_ MATHEMATICS susceptibilities for each head training.
The susceptibilities for each head are shown in Figure T9
Examples of positive per-token susceptibilities:
0:0
D
eter
mine
0:1> der
iv
ative repl ac
ement
0:4
der
iv ative D eter mine
repl
emment
Examples of negative per-token susceptibilitie:
0:0+
is
be
0:1:
O
is
be
0:4|
O
is
be
F.3
ENRON_EMA LSS
The susceptibilities for each head are shown in Figure 20
Examples of positive per-token susceptibilities:
D3
#
1-1
im
nate
36

---

Published as a conference paper at ICLR 2026
Layer
Layer
0.00
0.005
-0.1
0.000
0.005
0.02
0.010
-..03
0.015
0.04
0.020
0.025
10a
10a
10*
10a
10a
10*
Steps
Steps t
10h0
10h2
10h4
10h6
11h0
11h2
11h4
11h6
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 20: ENRON EMAILS susceptibilities for each head over training.
Examples of negative per-token susceptibilities:
<nnooftext
Cc
as
1:1
eenoftext >
For
as
endoftext >
R
ose
F.4
NIIL_EXPOTTER
Layer 0
Layer 1
0.0075
0.015
0.0050
0.0025
0.010
0.0000
0.005
-0.0025
0.000
-0.0050
-0.05
-0.0075
-0.0100
-0.010
102
10a
10a
102
10a
10a
Steps t
Steps
10h0
10h2
10h4
10h6
11h0
11h2
11h4
11h6
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 21: NII_EXPORTER susceptibilities for each head over training.
The susceptibilities for each head are shown in Figure 21 Below x stands for a token representing number.
Examples of positive per-token susceptibilities:
0:6 differe nnces)
al
wate el im
Lnaae
1:3 th y roidd
X
prot ot ype
1:5c: y reid
e
inate
full ength
Examples of negative per-token susceptibilities:
37

---

Published as a conference paper at ICLR 2026
0:6
qual
1
obvious
1:3
qual
EV
1:5:
qual
ty
Ca
2
ion
F.5 SUOMED_ABSTRACTS
Layer 0
Layer 1
0.005
0.02
0.000
0.01
0.005
-0.000
-0.015
-0.01
-0.020
-0.02
10a
10a
10a
10a
104
Steps t
Steps t
10h0
10h2
10h4
10h6
11h0
11h2
11h4
11h6
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 22: PUBMED_ABSS DOMMEDDABSSRACTSS susceptibilities for each head over training.
The susceptibilities for each head are shown in Figure 22 number.
Below stands for a token representing
Examples of positive per-token susceptibilities:
0:0
)
differe nces
0:5k
g
demon st rration) differe nices
1:5:
++
Th roid
8)
Examples of negative per-token susceptibilities:
0:0
med
um
0
pub
oper i nn
is
0:5
ili
oC
oC
ge
ous
cl id
ined
ion
es 0
1:5:
SC
ro
ful
ace
um
ev
al
u at atnn
F.6 PUBMED PUBMED_CENTRAL CENTRAL
The susceptibilities for each head shown in Figure 23
Examples of positive per-token susceptibilities:
di differe
nces
1:4
full
ength
full
engt
Examples of negative per-token susceptibilities:
pro
Lns
u1
in 0.
end
ere
nd
one
in
ase
eet
ex
m
al
un
ate
pro
38

---

Published as a conference paper at ICLR 2026
Layer
Layer
0.01
0.03
0.00
0.02
-0.01
0.01
-0.02
0.00
0001
0.03
10a
10a
104
10a
10a
10a
Steps t
Steps
10h0
10h2
10h4
10h6
11h0
11h2
11h4
11h6
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 23: PUBMED_CENTRAN susceptibilities for each head over training.
Layer
Layer 1
0.015
0.03
0.010
0.02
0.005
0.01
0.000
0.005
10a
10a
10a
10a
10a
10a
Steps
Steps
10h0
1oh2
10h4
10h6
11h0
11h2
11h4
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 24: USPTO_ BAPPO__BCCKGROUDDS susceptibilities for each head over training.
39

---

Published as a conference paper at ICLR 2026
F.7 USPTO BACKGROUNDS
The susceptibilities for each head are shown in Figure 24
Examples of positive per-token susceptibilities:
0
::2
ve
enggth
es ppec a aalllly
di ffere
nices
1:2:
wa
vel
ength od er berg
mag
a itude
1:4:
wa
Vel
ength
mag n itude
ht
m
Examples of negative per-token susceptiblitiee:
O D:2:
of
color
color
of
1:4
met er
color
F.8 WIKIPEDIA_EN
Layer 0
Layer
0.030
0.020
0.025
0.015
0.020
0.010
0.015
0.005
0.010
0.000
0.005
-0.055
0.000
0.010
102
10a
10a
102
10a
10a
Steps
Steps
10h2
10h4
11h2
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 25: WIKIPEDIA EN susceptibilities for each head over training.
The susceptibilities for each head are shown in Figure 25
Examples positive per-token susceptibilities:
0:0
e
eerences di ffere nces
1:33
R
e
ererces
al bum
Y is rae
1:4:
E
k berq
:// al bum
Examples of negative susceptibilities:
0:0:
<eendfftett
endoftext
record on
1:4:
< endoftext
of record
F.9 CC
The susceptibilities for each head shown in Figure 26 Instances of ": typically occur part of dictionaries, e.g. name ":" but the preceding tokens vary.
Examples of positive per-token susceptibilities:
40

---

Published as a conference paper at ICLR 2026
Layer
Layer 1
0.000
0.000
-0.005
-0.005
0.010
0.010
0.015
0..000
0.015
-0.025
0.020
10a
10a
10"
10a
10a
10
Steps
Steps
10h0
10h2
10h4
10h6
11h0
11h2
11h4
11h6
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 26: PILE-CC susceptibilities for each head over training.
0:1
th
is
more
from
0:7:
23
and
1:6:
ery
y
mp
ic
under
ground
Examples of negative per-token susceptibilities:
est
ival
com
0:7:
est
iiall
com
st
one
1:6:
games
st
one
We note that the susceptibilities for heads on CC are small relative to other datasets. Note that by definition the susceptibility should be zero for variation q o' where o' is distributed identically to q, so this could because CC is a significant fraction the Pile (18% according to Gao et al. (2020)). In Table we give a variant of Table 2 where compute the total positive and negative per-token susceptibility of 0:0 on the datasets CC, GITHUB-ALL.
Dataset
Total pos.
Total neg.
Sum
Scaled sum
GITHUB-ALL
171,561
-145,342
17,816
0.016
CC
128,026
-130,095
-966
-0.0013
Table 3: Total negative and positive per-tok por-token susceptibility for 0:0 on two datasets, and the divided by the total number of tokens and multipled by 5h 0.1.
We note that the total positive and negative per-token susceptibilities are of similar order of magnitude for both datasets, but their difference is an order of magnitude smaller (966 vs 17816) and this explains the order of magnitude difference in the final susceptibility. Recall that macroscopic bodies have near zero electric charge because the exact cancellation of very large amounts of positive and negative charge in physical matter at equilibrium; it seems reasonable intuition that near zero susceptibilitics are likewise caused by near-exact cancellations rather than both positive and negative token susceptibilities being small.
Also note that, given the similarity between the tokens with top positive susceptibility shown above, it is surprise that Figure 26 shows little differentiation of the heads. This particular variation in the data distribution away from the full Pile is not enough to "break the symmetry" between the heads, by exposing them distributions of patterns for which they have distinct preferences.
41

---

Published as a conference paper at ICLR 2026
Layer
Layer
0.00
0.00
0.01
0.01
0.02
0.03
0.02
0.04
-0.03
0.05
10a
10a
10*
102
10a
10*
Steps
Steps t
10h0
10h2
10h4
10h6
11h0
11h2
11h4
11h6
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 27: HACKERNEWSS susceptibilities for each head over training.
F.10
HACKERNEWS
The susceptibilities for each head are shown in Figure 27
Examples of positive per-token susceptibilities:
0:0
ve
just
1:1:
al
one
I
ve
1:5
ve
Examples of negative per-token susceptibilities:
0:0
recommend
and mon et ary
1:1
and
the
CO ff ee
and
recommend mon
et ary
F.11 GITHUB-ALL
Layer 0
Layer 1
0.06
0.00
0.04
0..02
0.04
0.00
-0.66
0000
-0.08
10a
10a
10a
10a
10a
Steps t
Steps
10h2
10h4
10h6
11h0
11h2
11h4
10h1
10h3
10h5
10h7
11h1
11h3
11h5
11h7
Figure 28: GITHUB-ALL susceptibilities for each head over training.
42

---

Published as a conference paper at ICLR 2026
The susceptibilities for each head are shown in Figure 28
Examples of positive per-token susceptibilities:
0:0:
Get
St
ring
ength
011
count
1:6:
ength
Examples of negative per-token susceptibilities:
0:0:
< </endoftext >>
int count
0:1
de f
connect
int count con st base
1::6:
con
st
base
con st st int
count
F.12
FREELAW
Layer
Layer 1
0.005
0.006
0.000
0.005
0.002
-0.010
-0.015
0.002
0.004
0.020
0.006
-0.025
10?
10*
10a
102
10
10
Steps
Steps
10h0
10h2
10h4
10h6
11h0
11h2
11h4
11h6
10h1
10h3
10h5
10h7
11h1
11h3
I1h5
11h7
Figure 29: FREELAW susceptibilities for each head over training.
The susceptibilities for each head are shown in Figure 29
Examples of positive per-token susceptibilities:
0:6
An
ysis
di differe
nces
1:4:
d ys sev
uments
1:5:
d
ys
sey
K
i Lcks
Examples of negative per-token susceptibilities:
0:6::
on
rem
edy
defense
1:4%
A
ri
in
ona
up
Ct
Rep
resent
a
vves
S up
Ct
We note here the role of the token splitting word across multiple lines, and in abbreviations. In the negative susceptibilities there are several other examples of the tokens being used other split words and abbreviations respectively.
43

---

Published as a conference paper at ICLR 2026
Layer 0
Layer
0.00
0.01
0.00
-0.02
0.01
0.04
-0.02
0.03
-..6
0.04
0.08
10?
10a
10*
10²
10a
10a
Steps
Steps
10h0
10h2
10h4
10h6
11h0
11h2
l1h4
1th6
10h1
10h3
lohs
10h7
11h3
1ths
11h7
Figure 30: STACKEXCHANGE susceptibilities for each head over training.
F.13 STACKEXCHANGE
The susceptibilities for each head are shown in Figure 30 Examples of positive per-token susceptibilities:
ou ble
differe nces
1:3:
Examples of negative per-token susceptibilities:
0
W
M
lement
0:4
T
Lmment
with
H
T
LE
[ement
where
44

---

Published as a conference paper at ICLR 2026
g-
60
-24
682
186 197
1250
101
032
12:0
12M
Figure 31: Loadings of PCs on heads. The loadings of each principal component on each head is shown, with the first principal component being a uniform mode, the second principal component loading (in layer on the multigram heads, and the third principal component loading on the induction heads and 1:55
G STRUCTURAL INFERENCE VIA TRAJECTORY PCA
In the main text (Section 4.2) we do structural inference on our small language model using PCA on data matrix of per-token susceptibilities at the end of training. To provide independent check the basic results we provide in this appendix details of an alternative analysis using PCA data matrix of susceptibilities (not per-token) of heads (Appendix G.1 and then present the results (Appendix G.2
training. First explain the methodology
G.1 JOINT TRAJECTORY PCA
We adapt a multi-trajectory variant of trajectory PCA (Amadei al. 1993 Briggman al. 2005) to study how the behaviors of attention heads in our small language model evolve with respect to different data distributions; see also Carroll et (2025 $4.2).
For each attention head E H and each dataset d E D, we measure the susceptibility x at each training checkpoint E T. The checkpoints used are the same ones given Appendix C.3 This gives us, for each dataset d, a trajectory through a space with dimensions corresponding to attention heads (t) (x² x5(),, x (t) E |ll,, We combine these trajectories into
data matrix E IR | x [H]] where each row represents the susceptibility measurements across all attention heads for specific dataset at specific checkpoint. For each d E D, we aggregate the row vectors from each checkpoint into matrix X₂ E <R[[x|H| and then stack each Xa vertically into X E TR. [||]
na(t1) (a(t2)
Xdi
Xa
for d ED, x=
Xd
(r(((r)
We standardize each column of to have zero mean and unit variance, ensuring that attention heads with larger susceptibility magnitudes do not dominate the analysis. Let Xau denote this standardized matrix. We then perform singular value decomposition (SVD)
Xud UEVT
(33)
where U e RIPIITixe contains the left singular vectors, Σ E Rexe is diagonal matrix of singular values, and V E [xx contains the right singular vectors where c is the chosen number of principal components. The principal components are given by the columns of UE and the loadings that indicate how attention heads contribute to each principal component are given by the rows of V
For each dataset we project trajectory (a(() onto the reduced space defined by the first principal components Ta(()= (na(()= 14) VA where is the mean vector used during standardization and Vi consists of the first columns V. This gives us a oow-dimensional representation TT(() e E of the trajectory for dataset at checkpoint t.
G.2 RESULTS
We perform joint trajectory PCA as a concrete realisation of the structural inference proposed in We found that three principal components explained 98% of the variance (with the top
Section 3.3
45

---

Published as a conference paper at ICLR 2026
three PCs explaining respectively 85.78% 10.76%, 1.63%). In more detail, the loadings of the PCs as presented in Figure aree:
are:
PC1 is uniform mode, loading on all heads.
PC2 loads strongly positively on 0:1 and negatively on 1:0 1:5, that is, the layer multigram heads.
PC3 loads strongly positively on (0:0, 0:5 and negatively on 0:2.0:7 in layer 0 and strongly negatively on (1:6) 1:7 layer 1.
We see therefore that PC2 separates the layer multigram heads and induction heads, as noted in Section 4.2 for for the per-token susceptibility PC2. Since the susceptibility for a head is the expectation over the probe data distribution of the per-token susceptibilities, there is broad agreement between the structure revealed by the trajectory PCA and the analysis in Section 4.2 Nonetheless the switch from per-token to overall susceptibility, and the inclusion of multiple checkpoints, provides non-trivial check on the robustness of that analysis.
not a surprise to see that
H COMPARISON TO DIRECT LOGIT EFFECTS
Previous work (Gurnee al. 2024b) (Lad et al. 2025 has defined prediction and suppression neurons with reference to Wo W where W is the output matrix for head h, and Wy the unembedding matrix for the network, following Elhage contribution of neuron in head h to token 2 in our vocabulary E. In particular, one says neuron index head predictive when the effect distribution (Wa W)) JJ has a high kurtosis and positive skew, and suppressive when the effect distribution has high kurtosis and negative skew.
al. (021). Entry (i.j) ) is interpreted as the direct
Given our own characterization of negative susceptibilities as indicating expression, one hypothesis is that a negative susceptibility for head h on a token /. means that neurons in head h typically have a positive effect on the logits of t, while a positive susceptibility indicates neurons typically have a negative effect.
Let X = ((I1,1),,... vi),,,.,..,, (In, Mn)] DN be n samples for which we have computed X(x.v.)) from our dataset DN, and U (ui,.... Um C Σ be the set of unique yt's in this subset. Then for each
U define Xy Xy=u= = [(1g) My) X X = Σ M(=s where X(2.)) the per-sample susceptibility of head sample (x, y). so that Xymu is the average per-sample susceptibility on samples with the completion u.
yj = "} = {(z), Mjj) M..),..., (T My. }}. and
For each attention head and every token u E U we plot the quantity Xy against the average value of (Wo ) ).,- The results for each head are shown in Figure 32 along with the corresponding 2 and slope of each graph.
We note that both the 2 values and the slopes are very small, with slopes having inconsistent signs, rejecting the hypothesized connection.
I ADDITIONAL SEEDS
Three additional models were trained with the same architecture and training data, but different random seeds; we refer these as seeds 2, 3, 4 (with the original being seed 1).
J VARIANCE OF PRINCIPAL COMPONENTS
In Figure 2 and Figure 3 of the main text we present the data and component loadings of the top principal components of the per-token susceptibility PCA. Recall that this is done separately for the two layers. A prerequisite for assigning meaning to these principal components that they are suitably stable with respect the tokens chosen.
In this appendix we present the top five principal components the per-token susceptibility PCA for both layers, including data for all nine patterns identified in Appendix C.8 as well information
46

---

Published as a conference paper at ICLR 2026
p² 0.018, slope=-0.00 0.0001
/² 0.034, slope 00001
0.001
p² 0.059, slope=0.002
/ 0.010, slope=-0.001
Toleen Aug Neuron Effect
0.1
0.02
-2.22
-181
0.030.
slope-
2 0.047. 0.0pe-0.001
0.207, slope-0.005
0.066,
siope--0.003
Token Avg Neuron Effect
3,02
2.02
-2.22
Number points heragram
-1A1
-2.01
0.037,
003.7./0pp-0.0000 0.002
e - 0.118, slope 0.004
1 = 0.010. slope=0.001
0.046, slope=-0.007
Tolon Aug Neuron Effect
6.073
2
6.20
6.000
-0.200
0:7
1:3
1:7
-.0199 slopee..lpppe...oe.. 0.000
/ 0.009, slope--0.002
0.000
0.071.
0lope-0.003
2 - 0.003, slope-0.002
Tokenn Avg Neuron Eftect
-202
-3.4
-3.0
Avg Susceptibilft
Token Avg Suscepaiblitt
Avg Susceptibility
Taken Avg Suscoptibllit
Figure 32: Average token susceptibility plotted against average neuron effect. Hex bin plots for each head I h plotting the points (xti With) ... (X m., (W With)amm. for each U and with (Wr Whth) )u; indicating the average value of (Wn Wrih) ).... The head, yd value, and slope are indicated above each plot, and hexagrams are coloured according to the number such points which lie inside them.
47

---

Published as a conference paper at ICLR 2026
End
Posiidee
70
25
60
20
20.1
15
Top Negattre
20.7
15
30
100
20
Neguilie
32.5
21.8
92.1
17.9
Induction
Spacing
Right Delimiter
Top 1% Positvee
15.7
6.3
42.9
5.2
4.9
0.0
11.7
6.7
22.2
10% Positive
23.3
6.1
5.6
1.9
4.9
10
30% Negattve
0.4
Nogattve
0.0
PC1
PC2
PC2
PC2
PC1
PC2
PC1
PC2
Layer
Layer
Layer
Layer
Layer
Layer
PC1
PC2
0.75
0.50
0.25
0.00
0.1
-0.25
0.0
Hil
Figure 33: Per-token susceptibility PCAs for Seed Among the tokens with the coefficients the largest magnitude in each principal component for the por-token susceptibility PCA, the percentage following each of the six patterns (Top). The loadings of the principal components heads (Bottom). In Wang et al. 2004 Appendix G) it was found that in this seed the previous-token head is 0.1,, the current-token head (0:6) and the induction head is 1:0.
48

---

Published as a conference paper at ICLR 2026
Word
End
25
Positire
20.5
20
50
Top 10% Negative
32.0
14.9
29.3
11.8
5.1
18.0
5.1
16.9
15
2.2
17.5
2.4
13.1
30
-00
10
20
Top 1% Negattve
34.5
13.2
30.8
13.8
-5
9.5
24.0
8.9
28.9
3.9
19.9
41
22.5
Induction on
Spacing
Right Delimiter
Top 1% Positive
17.7
5.3
44.7
5.3
4.5
1.1
1.0
14.1
1.7
25.7
0.2
600
Top 10% Pooitive
23.5
5.7
36.8
5.3
1.7
2.4
2.1
0.3
6.6
2.1
5.1
1.4
Top 10% Negattre
3.3
56.8
2.0
53.6
30
39.5
3.0
41.5
2.9
0.2
5.0
0.2
2.7
10
Top 1% Negattve
5.0
76.66
9.2
25.3
5.3
0.5
PC1
PC2
PC1
PC2
FC1
PC2
PC1
PC2
PC1
PC2
PC1
PC2
Layer
Layer
Layer
Layer
Layer
Layer
PC1
PC2
0.50
0.25
0.1
0.0
-0.50
Hil
1INE
1115
Figure 34: Per-token susceptibility PCAs for Seed Among the tokens with the coefficients of the largest magnitude in each principal component for the por-token susceptibility PCA, the percentage following each of the six patterns (Top). The loadings of the principal components heads (Bottom). In Wang et al. 2004 Appendix G) it was found that in this seed the previous-token head is 0:4) the current-token head
(1:2 and the induction head is 1:6..
49

---

Published as a conference paper at ICLR 2026
End
70
25
50
15
Top 10% Negative-
28.3
17..2
27.6
11.2
4.8
11.6
16.2
2.4
14.0
30
10
Thp 1% Negative
29:1
19.2
25.1
15.7
8.7
12.4
7.8
25.2
3.8
6.9
3.9
24.5
10
Induction
Spacing
Right Delimiter
Top 13 Positive
21.0
12.8
45.3
4.7
70
4.4
0.7
0.3
0.1
40
10.0
1.0
17.1
0.8
15.0
60
12.5
Top 10% Postive
23.6
9.0
36.8
4.7
50
1.8
1.5
2.1
0.2
6.4
0.9
5.7
1.2
Top 10% Negative
2
50.3
1.4
52.5
46.1
6.8
45.0
3.3
7.0
0.3
3.5
7.5
Top 1% Negative
28.9
1.3
67.2
20.2
0.3
0.5
6.3
0.5
2.2
2.5
PC1
PC2
PC1
FC2
PC1
PC2
PC1
PC2
PC1
PC2
PC1
PC2
Layer
Layer
Layer
Layer
Layer
Layer
PC1
PC2
0.5
0.4
0.2
0.0
0.0
1056
(067)
1113
M1
Figure 35: Per-token susceptibility PCAs for Seed 4. Among the tokens with the coefficients of the largest magnitude in each principal component for the per-token susceptibility PCA, the percentage following each of the six patterns (Top). The loadings of the principal components heads (Bottom). In Wang et al. 22024) Appendix G) it was found that in this seed the previous-token heads are 0:0 and 0:3), the toreent-token head is 0:6 and the induction heads are 11:1 and 1:3)
50

---

Published as a conference paper at ICLR 2026
about sample variance. As in the rest of the paper (with the exception of Appendix 1 all data is for seed 1. The stochasticity here is the 20000 sampled tokens from each dataset. We report the mean and standard deviation of the percentages and head loadings in Figure 36 Figure 37 for layer 0 and Figure 38 Figure for for layer 1.
The conclusion is that the sample variance is sufficiently small to justify the interpretations given in the main text.
Word Start
Word Part
Word End
Positive
34
26.5
23.
20
2377
203
242
22.9
23.1
23.3
210
22.3
2313
44
11.
23.4
25.0
11-2
16.3
163
26-1
17.
16.3
52.6
42.
25.8
22.2
233
30
Thp 10% Negstive
21.1
20.3
16.7
263
56
204
54
20.8
5.6
132
15.6
12.0
25.7
23.7
26.3
17.
20
1% Negative
219
-3.3
263
216
216
23.7
212
21.
215
21.5
250
20.3
234
10
Induction
Spacing
Formatting
Posttive
223
23.5
24-4
233
211
70
24
20.0
200
201
35
252
33
231
15.0
12.5
Pocktiv
26.3
40.5
31.
237
2611
50
1.8
3.3
93
202
21.6
4-3
4.3
35
74
18
Thp 10% Negstive
3.
20.1
$3.9
263
262
10
21.0
26-3
20.
16.3
23.1
1.6
25.
11.6
1.
5.0
Negative
233
224
10
221
204
215
202
230
:40
21.3
210
213
34
2.5
Right Delimiter
Left Delimiter
Numeric
17.5
Postive
226
26.3
26.2
467
234
206
80.6
20.0
200
2.1
2633
230
264
Poottiv
63
23
06
16
423
12.5
1/
1.2
98
204
1.
2.3
34
03
0.4
34
10.0 A8
15
Thp 10% Negstive
02
203
65
202
1.2
203
53
203
235
4.2
28.3
61
20.3
5.0
Negative
05
31
21
226
206
21.
242
203
28.7
21.
225
223
-5
P21
PC2
PC3
Pd4
PC5
PC1
122
PC3
PC4
PC5
PC1
PC2
PC3
PC4
PCS
Figure 36: Per-token susceptibility PCA for layer 0 heads showing mean and standard deviation of loadings of principal components on data patterns across 10 independent draws of 20000 tokens from each dataset.
PC1
PC2
PC3
0.8
0.3
0.4
0.6
0.4
0.2
0.2
0.0
0.0
-0.
0.0
10h0
10h1
10m2
10h3
10m4
10h5
10m6
10h7
10h0
10m1
10n2
10h3
10h4
1ons
10h6
10h7
10h0
10m1
10h2
10n3
10m4
lons
1one
1007
Figure 37: Loadings of the top three principal components on layer attention heads, showing
mean and standard deviation across 10 independent draws 20000 tokens from each dataset.
K STATEMENT ON USE OF LLMS
LLMs were used in the course of this research to support literature review for related work well as to generate some of the code used to implement experiments and to analyze and plot the data. All I. L.M.--gnerrted output was reviewed by human author.
51

---

Published as a conference paper at ICLR 2026
Word Start
Word Part
Word End
50
70
Pesitive
84
2-4
16.
28.7
26-8
23.2
215
23.4
2-0
14.0
13
16.5
31.8
40
Top 10% Posttive
10
26-2
21.7
21.
17.4
12.8
23.2
50
30
g
Top 10% Nogazive
206
20
40.5
88.5
50
10
+0.
42.3
60
61.
+33
+20
622
+3.3
43.
+1.
10
Induction
Spacing
Formatting
60
17.5
Posttive
24.3
A1.
22.0
21.7
50
22.0
20.0
203
407
432
43.3
48.5
23.5
202
Rosttive
3,9
26.1
16.
12
23
1.
6.5
14
24
12.5
*
10.0
Thp 10% Negatiie
30
11.
312
3.3
144
12.
12.1
47
15.2
7.5
5.0
Tpp 1% Negattie
23.6
10
230
-02
-
22.2
-25
Right Delimiter
Delimiter
Top 1% Posttive
:05
20
21.0
20.0
12
10
10% Positive
255
272
233
13
20.
234
201
204
203
233
2633
282
253
233
20.4
Negative
5-1
28
10
14
15
203
27A
120
16
20.5
44
Tp 1% Negative
6-1
3.2
11.8
24
32
6.1
644
1-1
244
4.
15
34
PC2
PCS
0
PCS
-0
Figure 38: Per-token susceptibility PCA for layer 1 heads showing mean and standard deviation of loadings of principal components on data patterns across 10 independent draws of 20000 tokens from each dataset.
PC1
PC2
PC3
0.6
0.6
0.3
0.4
0.4
0.2
0.0
0.1
-0.4
0.0
-0.4
11
11h
11h3
11h4
11h5
11h6-
11/7
1h0
111
112
11h3
11h4
11h6.
11h7
11h0
11h2
n1h
11h4
115
11h6
117
Figure 39: Loadings of the top three principal components layer attention heads, showing
mean and standard deviation across 10 independent draws of
20000 tokens from each dataset.
52


Reviews:
[{"review_id": "kYdQ4J9W1m", "reviewer": "Reviewer_Cmff", "summary": "The authors introduce a novel interpretability framework for transformer networks through susceptibilities. The authors investigate how much do variations in the data distribution influence specific components (i.e. attention heads) of the network. The authors highlight two interesting patterns of behavior identified by a standardized notion of susceptibility, resulting from a change in the parameters, namely expression (loss decreases, probability of next token increases) and suppression (loss decreases, probability of next token decreases). Through experiments on a two-layer attention-only Transformer trained on a subset of the Pile, these patterns are related to existing findings from mechanistic interpretability, rediscovering the induction head, as well as identifying how simple linguistic capabilities are conducted within the network. \n\nOverall, the paper presents a strong mathematical foundation and a novel approach for uncovering mechanistic patterns within transformer networks. The paper is well written, albeit the math is quite dense. My main concerns with the paper are the method scaling to larger models, which the authors proactively address in Appendix C.5, but I believe remains a limitation of their work. Finally, I am slightly concerned how the method should be applied in cases where there is no clear next-token point where susceptibility should be estimated (open-ended generation tasks), as the studied patterns are relatively simple to identify from data.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Strong mathematical foundation\n- Provides an empirically validated novel method for discovering mechanistic patterns within transformer networks", "weaknesses": "- Computational costs when scaling (mentioned as a weakness)\n- Applicability beyond simple tasks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J0la5pvali", "reviewer": "Reviewer_KJxa", "summary": "The authors provide a framework for approximating the effect of a small change in the training data on model components, called susceptibilities. They show that this is equivalent to the covariance between a component’s loss (in this case, attention heads) and the change in total loss under the perturbed data distribution. They then estimate, using locally sampled posteriors with SGLD, susceptibilities for loss on each pair of tokens in the vocabulary, for all attention heads in a 3M-parameter transformer, trained on subsets of Pile. The top principal components of this combined covariance matrix are sensitive to certain classes of interpretable token patterns, such as the induction.", "questions": "1. Could the authors explain why $\\delta h = 0.1$ is justified? And additionally, compute $\\chi$ for range of values of $\\delta h$ to check how different the susceptibilities are?\n2.  How to interpret the functional roles of these heads with respect to susceptibilities, given that the DLA plot doesn't show any correlation? One could conduct causal ablations on a few samples as a sanity check, but it is unclear if it would be correlated, given that DLA shows no signs of life. Does this method actually find a causal, induction circuit?\n\nIt would be great if the authors could address these two points in the paper (or point me to it, if it already exists). \n\nSome writing suggestions:\n1. A notation table (and maybe an algorithm table) would be extremely helpful as the paper is quite mathematically dense. \n2. Add some more description about the key findings. Eg: suppression and expression in the introduction", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The assumptions are clearly stated, and the theory seems well grounded. \n2. Applying susceptibilities to attention heads yields interpretable patterns, previously found in small attention-only transformers via mechanistic interpretability.", "weaknesses": "1. Although novel, this attribution method seems very hard to scale to larger models. \n2. The heads that express or suppress the reported patterns in Figure 2 are not yet mechanistically explained. The correlation with Direct Logit Attribution (Figure 32) shows no sign that they are actually responsible for the behaviour. \n3. Relaxing sampling to the local posterior makes sense for small changes, but it is unclear if this holds when 10% of the data is replaced. It would be great if this could be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oEDHO9EWJU", "reviewer": "Reviewer_B7pi", "summary": "The authors introduce a new interpretability framework grounded in statistical physics and Bayesian learning theory. The key idea is to view a neural network as a Bayesian statistical mechanical system, where the model’s parameters and interactions can be explored through small perturbations of the data distribution. Small shifts in the input data (like moving from natural text to programming code) produce first-order linear responses in specific components of the model. These responses (called susceptibilities) reveal how strongly and in what direction each component reacts to data changes, providing a principled way to quantify its sensitivity and functional role within the network.", "questions": "* as I mentioned in the weaknesses section, I think the paper would be stronger if it demonstrated how the proposed method could be applied to analyze more complex patterns or behaviors (such as bias or knowledge acquisition).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The framework is theoretically rigorous, grounded in statistical physics and Bayesian learning theory, while also being empirically validated through concrete experiments.\n\n* The proposed approach is novel:  Susceptibility analysis connects the functional behavior of model components to shifts in the training distribution and shows that heads with similar response patterns cluster into interpretable groups.", "weaknesses": "* The analysis (Sec 4) focuses on a very simple set of patterns (Word Start, Word Part, Word End, Induction Pattern, Right Delimiter). In addition, the framework is demonstrated only on a small toy language model (3M, 2 attention layers, without MLP). Although the authors note that they do not anticipate major obstacles in scaling the method to larger models, applying it to a larger model could have strengthened the work by demonstrating the ability to capture more complex behaviors.\n\n* This is only a suggestion, but I think the presentation could be made more accessible to readers who are less familiar with the theoretical background.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xoyGJot82m", "reviewer": "Reviewer_Wjna", "summary": "This paper introduces susceptibilities, a new interpretability framework grounded in Bayesian statistical mechanics that measures how specific neural network components respond to controlled changes in the data distribution. By estimating these susceptibilities locally using SGLD, the authors define a notion of expression vs. suppression for model components and construct response matrices that reveal internal structure. Applied to a 3M-parameter transformer trained on the Pile, the method automatically separates known functional modules such as the induction circuit and identifies linguistic patterns like word segmentation and bracket matching.", "questions": "1. The empirical results in Section 4.2 show that the first principal component (PC1), interpreted as “word segmentation,” accounts for an overwhelming portion of the variance (95.3% in Layer 0 and 99.1% in Layer 1). Could the authors clarify how the proposed method can reveal more complex circuits when the learned representations appear to be dominated by such a simple feature?\n2. Since susceptibility is fundamentally correlational, did you conduct any causal validation, such as steering or ablating the susceptibility-identified heads, to assess their influence on the model’s outputs?\n3. Appendix C.5 presents a cost model that scales linearly with the number of components. While this may be feasible when analyzing only attention heads, it seems impractical once MLP neurons are included. Do you plan to extend the framework to investigate MLP neurons as well?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This work introduces a principled and novel framework for understanding the internal computations of deep neural networks, drawing inspiration from Bayesian statistical mechanics and singular learning theory.\n2. The proposed methodology offers an unsupervised pipeline for discovering functional circuits within neural networks.\n3. It not only provides empirical results demonstrating the effectiveness of the framework, but also includes sanity checks to ensure its practical value in understanding transformer-based language models.\n4. The approach establishes strong connections to physics, which may encourage researchers in the field to contribute to the study of the interpretability of neural networks.", "weaknesses": "1. Although the theoretical framing is elegant, its practical usefulness remains uncertain. The empirical evaluation focuses only on a small toy model with two attention layers, which is not representative of typical language models.\n   - Training on only a subset of the Pile dataset may be too simplistic, potentially explaining why the first principal component accounts for  95–99% of the representational variance.\n2. The interpretation of principal components remains largely manual and speculative, which may lead to inconsistent or subjective interpretations of the empirical findings.\n3. The proposed susceptibility framework captures correlations in the local loss landscape, in contrast to many mechanistic interpretability approaches that emphasize causal relationships.\n4. Finally, the paper requires expertise in Bayesian statistics, which may limit accessibility and reduce its potential audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)