Equivalent Linear Mappings of Large Language Models
Authors: James Robert Golden
Venue: Accepted by TMLR ()

Abstract: Despite significant progress in transformer interpretability, an understanding of the computational mechanisms of large language models (LLMs) remains a fundamental challenge. Many approaches interpret a network's hidden representations but remain agnostic about how those representations are generated. We address this by mapping LLM inference for a given input sequence to an equivalent and interpretable linear system which reconstructs the predicted output embedding with relative error below $10^{-13}$ at double floating-point precision, requiring no additional model training. We exploit a property of transformer decoders wherein every operation (gated activations, attention, and normalization) can be expressed as $A(x) \cdot x$, where $A(x)$ represents an input-dependent linear transform and $x$ preserves the linear pathway. To expose this linear structure, we strategically detach components of the gradient computation with respect to an input sequence, freezing the $A(x)$ terms at their values computed during inference, such that the Jacobian yields an equivalent linear mapping. This ``detached’’ Jacobian of the model reconstructs the output with one linear operator per input token, which is shown for Qwen 3, Gemma 3 and Llama 3, up to Qwen 3 14B. These linear representations demonstrate that LLMs operate in extremely low-dimensional subspaces where the singular vectors can be decoded to interpretable semantic concepts. The computation for each intermediate output also has a linear equivalent, and we examine how the linear representations of individual layers and their attention and multilayer perceptron modules build predictions, and use these as steering operators to insert semantic concepts into unrelated text. Despite their expressive power and global nonlinearity, modern LLMs can be interpreted through equivalent linear representations that reveal low-dimensional semantic structures in the next-token prediction process. Code is available at https://github.com/jamesgolden1/equivalent-linear-LLMs/.

Published in Transactions on Machine Learning Research (10/2025)
Equivalent Linear Mappings of Large Language Models
James R. Golden
jumesgolden @graail com
Ookland, CA
Reviewed on OpenReview: https: // /poperevvee..... net/ forum? id= oDNdJa.uEy
Abstract
Despite significant progress in transformer interpretability, an understanding of the com- putational mechanisms of large language models (LLMs) remains a fundamental challenge. Many approaches interpret a network's hidden representations but remain agnostic about how those representations are generated. We address this by mapping LLM inference for a given input sequence to an oquivalent and interpretable lincar system which reconstructs the predicted output embedding with relative error below 10-13 at double floating-pon precision, requiring no additional model training. We exploit property of transformer de- coders wherein every operation (gated activations, attention, and normalization) can be expressed as A(x) I, where A(x) represents an input-dependent linear transform and æ preserves the linear pathway. To expose this linear structure, we strategically detach com- ponents of the gradient computation with respect to an input sequence, freezing the A(x) terms their values computed during inference, such that the Jacobian yiclds an equivalent linear mapping. This "delached" Jacobian of the model reconstructs the output with one linear operator per input token, which is shown for Qwen 3, Gemma 3 and Llama 3, up to Qwen 3 14B. These linear representations demonstrate that LLMs operate in extremely low- dimensional subspaces where the singular vectors can be decoded to interpretable semantic concepts. The computation for each intermediate output also has a linear equivalent, and we examine how the linear representations of individual layers and their attention and mul- lilayer perceptron modules build predictions, and use these as steering operators to insert semantic concepts into unrelated text. Despite their expressive power and global nonlinear- ity, modern LLMs be interpreted through equivalent lincar representations that reveal low-dimensional semantic structures in the next-token prediction process. Code is available at http:://ggiaab..oomjammooleleetp cool:/aamosgollenm/gquuvlallttte - leee-rnnneearaaa
1
Introduction
The transformer decoder is the architecture of choice for large language models (Vaswani et al.
2017 and
efforts toward conceptual understanding of its mechanisms ongoing (Sharkey et al. 2025).
Significant
Templeton et al. 2024 Lieberum et al., 2024), linear probes (Alain & Bengio 2016), "white-box" alternative architec-
insights include sparse autoencoders for conceptual activations in L.LMs Bricken et al. 2023
tures Yu et al. 2023) and analytic results on generalization (Cowsik et al.) 2024). While transformers are
complex globally nonlinear functions of their input, we demonstrate how compute an equivalent linear sys- tem that reconstructs the predicted output embodding for a given input sequence up double doating-point precision.
Our approach directly extends the framework of Elhage et al 2021 who analyzed attention-only trans- formers as interpretable lincar circuits, but were limited to small models without MLPs (duc to gated activation functions) or normalization ion layers. We show that by detaching nonlinear terms from the gradient computation, modern LLMs with gated activations (as well as softmax attention and normalization can be decomposed into an equivalent lincar system for a given input. Recently, Kadkhodaic al. (2023) showed that powerful image denoising diffusion models with ReLU activations and certain architectural constraints are piecewise linear functions which can be computed via the Jacobian and be clearly interpreted

---

Published in Transactions on Machine Learning Research (10/2025)
A
Lawear
B
Model: Gwen/Owen3 3-44
The bridge out of Matin is the [[only]] Detached Jacobian Reconstruction
RMS Norm
Liner n_own
FF SwiGLU
10.0
2.56c
Oeginal Jacobian Reconstrution Ideniiyy (inear)
Detached Jacobian Reconstruct on
Food Forward See Ii
Activssion
RMS Norn
2.5
Linor (up. proft
Lincer ppall_pratt
0.0
Grouped O.ary/Atenten
-25
Detached Gradient
-10.0
Emtwerding
-10.0
-7.5
2.5
Duiput
Figure 1: A) A schematic of the transformer decoder (Grattafiori et al. 2024 Nvidia 2024). The PyTorch gradient detach operations for components outlined in red effectively freeze the nonlinear activations for a given input sequence, creating a linear path for the gradient with respect to the input embedding vectors, but do not change the output. The output embedding prediction can be mapped to an equivalent linear system by the Jacobian autograd operation. The feedforward module with a gated linear activation function is shown in expanded form to demonstrate how the gating term can be detached from the gradient to form a linear path, achieving linearity for given input. The RMSNorm layers and softmax attention blocks also must be detached from the gradient. B) For the input sequence "The bridge out of Marin is the", the elements of the predicted output embodding vector of the model compared to the elements from the Jacobian reconstruction for both the original Jacobian (blue points) and delached Jacobian operations (red points), shown for Qwen 3 14B. Note that the detached Jacobian reconstructions match the predicted embedding, with relative error (the norm of the reconstruction error divided by the norm of the output embedding) less than 10-13 for double Noating-poinl precision. See reconstructions for Llama 3.2 3B and Gemmna 3 4B in Fig. A2
low-dimensional adaptive linear filters with comprehensible singular vectors. For many open-weight LLMs, every component operation (gated activations, attention, and normalization) can be expressed in the form A(x).2, where A(x) represents input-dependent lincar transform and preserves the linear pathway. The gradient operation with respect. to the inpul can be manipulated al inference by freezing the A(x) terms at their values during inference operation with the detach operation such that the output embedding predic- tion has a linear equivalent as in Fig. 1 This "detached" Jacobian J computation captures the complete forward operation of the model, including activation functions and attention modules, although must be recomputed for each input sequence (as it is not piecewise linear but "pointwise" linear).
This approach allows to analyze a model from imput embeddings to prodicted output embedding as equivalent linear system for a particular input sequence. By examining the singular value decomposition (SVD) of the equivalent linear system, we can measure the local dimensionality of the learned manifolds involved in next-token prediction and can decode the singular vectors into output tokens. This analysis can also be done layer by layer, or for individual attention and multilayer perceptron (MLP) modules, in order to observe how these models compose next-token predictions.
We demonstrate equivalent linearity in model families including Qwen 3, Gemma 3, Llama 3, at a range of sizes up to Qwen 3 14B. (See the appendix for additional equivalent linear demonstrations for Deepseek R1 0528 Qwen 3 8B Distill, Phi 4, Mistral Ministral and OLMo 2). This approach offers a path to interpreting LLMs for specific inputs that could serve as a complement to other powerful interpretability methods. While this is a local method that is somewhat computationally intensive, this approach does not require additional training as required for sparse autoencoders. For example, training sparse autoencoders for Gemma 2 9B (Lieberum et al., 2024) required substantial compute across multiple feature widths and layers, and must be repeated for each new model and layer. Our approach works immodiately on LLMs with gated activations and zero-bias linear layers, and produces more exact representation for interpretation than other methods.
2

---

Published in Transactions on Machine Learning Research (10/2025)
If equivalent linear mapping were applied to next-token prediction at scale, this would offer form of interpretability the difficult but tractable problem of analyzing many equivalent linear systems.
2 Method
2.1
The Jacobian of a deep ReLU Network
Mohan et al. (2019) observed that deep ReLU networks for image denoising which utilize zero-bias linear layers "adaptive linear" functions due to their homogeneity of order at given fixed input, which enables interpretation as an equivalent lincar system. Given the homogeneity at fixed input, the network's output can be reproduced by numerically computing the Jacobian matrix of the network at a particular input image xim and multiplying it by Xim
Vim ((x]m -Xmm
Due to the global nonlinearity of the network, the Jacobian must usually be computed again at every input of interest. The Jacobian may be the same for similar inputs in the same piecewise region of the response
[Balestriero & Baraniuk 2021 Black et al. 2022) (but this will be demonstrated to not be the case for transformer architectures)
2.2 The Jacobian of a transformer decoder
Many open weight LLMs also use linear layers with zero bias, as required for lincarity in the architecture of Mohan et al. (201)) A transformer decoder predicts an output. token embedding, y given a sequence of k input tokens (to, t ti.., tk) mapped to imput embedding vectors x (Xo, X1.,, Xk., where t" and x' represent a particular sequence. The output embedding prediction is a nonlincar function of the input embedding vectors Xo, Xk. as L.Mss utilize nonlinear gated activation functions for layer outputs (SwiGLU for Llama 3, GELU for Gemma. 3 and Swish for Qwen 3) as well as normalization and softmax attention blocks.
Gated activations like Swish(x) x sigmoid(B x), with a lincar term and a nonlinear term, are also an "adaptive" linear function or, more generally, an adaptive homogeneous function of order 1 (Mohan et all 2019). If the sigmoid(3 x) term that gives rise to the nonlincarity is frozen for specifie numerical input, e.g. embedding vector x5 (Elhage et al.) 2021) (or equivalently detached from the computational graph with respect to the input), then we have linear function valid only at xq where numerically compute a Jacobian matrix that carries out Swish(x0) as a linear operation.
[ holds and we can
Below we show that computing the Jacobian after effectively substituting specific values for the nonlinear terms also works for other gated activation functions, normalization layers and softmax attention blocks. We further demonstrate that for given input sequence we can apply necessary gradient detachments so that the entire transformer decoder is an adaptive homogeneous function of order and mumerically compute the equivalent linear system that reproduces the transformer output y"beddinggy".
The Jacobian (xx of a transformer is the set of matrices generated by taking the partial derivative of the decoder inference function yxx) = Xx.,, xx), with respect to each clement of each x; (where X; for Llama 3.2 3B has length 3072, for example, and therefore the Jacobian matrix for each embedding vector is a square matrix of this size). If transformer docoder were naturally a homogencous function of order 1, this Jacobian would generate an equivalent representation of the network.
However, this is not the case. In order to numerically compute equivalent linear representation, we introduce "detached" Jacobian J+, which is set of matrices that captures the full nonlincar forward computation for a particular input sequence x* as a linear system. The detached Jacobian is the numerical Jacobian of the LLM forward operation when its gradient includes a specific set of detach() operations for the noulinear terms in the normalization, activation and attention operations that force the function to be "adaptively" homogeneous of order 1. The detached Jacobian operates on its corresponding input embedding vector to provide reconstruction of the LLM forward operation (shown in Fig. and validated in Fig. 13
3

---

Published in Transactions on Machine Learning Research (10/2025)
by the PyTTrch "allclose" function for absolute and relative tolerances of 10 10-13).
k (x²") x
y
(2)
The conventional Jacobian J for particular input sequence x" (as in Mohan et al. (0099 does not gener- ate an accurate reconstruction the nonlinear LLM forward operation since the transformer function is not homogeneous of order 1. The detached Jacobian J evaluated at x" is the result an alternative gradient path through the same network which homogeneous with respect to the input x", The detached Jacobian J+ only generates an accurate reconstruction at x* and not in the local neighborhood due the strong non- linearity of the decoder inference function. The detached Jacobian matrices differ for every input sequence and must be computed numerically for every sequence.
2.3 Nonlinear layers as linear operators for given input
In order to achieve linearity, modifications must be made to the gradient computations of the RMSNorm operation, the activation function (SwiGLU in Llama 3.2) and the softmax term the attention block output.
2.3.1 Normalization
Normalization layers like LayerNorm (Xu al. 2019 or RMSNorm (Zhang & Sennrich, 2019) with zero bias nonlinear with respect to their input because they include division by the square root of the variance of the imput.
norm(x)
Vear(x
Mohan et al. (2019) devised novel bias-free batch-norm layer which detaches the variance term from the network's computational graph (see their code implementation). Their batch-norm layer returns the same values as the standard batch-norm layer, but it is linear at inference as the nonlinear operation is removed from the gradient computation. This also similar to the "freezing" of nonlinear terms attention-only transformers from Elhage et al. (2021).
We make similar change for Llama 3.2 3B by altering how the gradient with respect to the input is computed at inference for RMSNorm. This is accomplished by substituting the value for the input vector x" for only the variance term as in 4 In PyTorch, this is accomplished by cloning and detaching the x tensor within the variance operation, so its value will be treated as a constant. The gradient operation is still tracked for x in the numerator, so that term will be treated as a variable by the PyTorrh autograd function for computing the Jacobian. The gradient of the function then computed at (we assume for simplicity an input sequence of length 1).
norm(x) / (xa(x*)
(4)
We define the detached Jacobian as follows:
J norm
(5)
Ox
We can rewrite the pointwise linear RMSNorm follows:
norm(x") =J xx((x)
(6)
4

---

Published in Transactions on Machine Learning Research (10/2025)
At inference for given input, we now have linear RMSNorm whose output is mumerically identical to the one used in training. However, when we take the gradient with respect to the input vector x in eval mode, the numerical output is the detached Jacobian matrix J which we can use to reconstruct the normalization output as a lincar system.
The goal apply this same approach for other nonlinear functions the decoder such that the entire computation from the input embedding vectors the predicted output is linear for a given input, and we compute and interpret the set of detached Jacobian matrices.
2.3.2 Activation functions
While Mohan et al. (2019) relied on ReLU activation functions, which do not require any changes to achieve linearity, Llama 3.2 3B uses SwiGLU (Shazeer 2020), Gemma 3 uses approximate GELU (Hendrycks & Gimpel 2016 and Qwen uses Swish for activation functions. There is a linear x term in each of these, and the gradients can be cloned and detached from the nonlincar terms. This manipulation produces a pointwise linear Swish layer with respect the input x.
Swish (x) = x sigmoid(3 x)
(7)
Swish (x*)= sigmoid(3 xx)x=xx
(8)
0
Swish (x") = O1xxx Swish xx
(9)
Swish (x") = =J swiishh (x"iis(("..xx"" xx
(10)
Detaching the gradient from the Swish output thus allows for pointwise linear form of Swish at inference. A similar procedure may be carried out for SwiGLU with Llama 3 and GELU with Gemma 3 (see supplement,
eq. 17
2.3.3
Attention
The softmax operation at the output of the attention block can also be detached, with the linear relationship preserved through the subsequent multiplication with V, which is a linear function of x. Below, Q = WQx K W and V Wvx.
QK Vlg
Aln(Q, K, V) softmar
V
(11)
QKT
Attn(x) [softmaz
Wvx
(12)
θ Attn(x*) = (1 =([==((((()))) Attn(z))lxxx)) x'
(13)
Attn(x") = J
Attn (x') xx*
(14)
The linear term within V makes it possible for the attention block be pointwise linear at inference, the gradient for the softmax output is detached.
5

---

Published in Transactions on Machine Learning Research (10/2025)
2.3.4 The Transformer Decoder
With the the above gradient detachments for the normalization layers, activation functions and attention blocks, the transformer decoder network is linear with respect to x" when evaluated at x" (shown here for length k).
y' (x x
(15)
The output of the network incorporating the above gradient detachments is unchanged from the original architecture but has an equivalent linear representation.
3 Results
3.1
Pointwise linearity of the predicted output
In order to validate whether the detached Jacobian achieves reconstruction with linear representation, we can compare the predicted output embedding vector for given input token sequence to the reconstruction of the output. As a baseline, we can also compute the reconstruction using the conventional Jacobian as in Mohan et al. (2019) and examine its accuracy. Given the above argument that the appropriate gradient detachments are necessary to achieve output roconstruction, we expect the detached Jacobian to accomplish reconstruction, but the conventional Jacobian to fail.
Fig. 3 compares the network output to both the conventional and detached Jacobian reconstructions for Llama 3.2 3B and Qwen 3 14B. The reconstruction of the output embedding with the detached Jacobian ma- trices falls on the identity line when compared with the output embedding, showing accurate reconst while the reconstruction with the conventional Jacobian is not at all close to the output. This comparison therefore demonstrates the validity of the reconstruction with the linear system of the detached Jacobian for Qwen 14B for a particular input.
In order to examine the fidelity of the detached Jacobian reconstruction, compared the reconstruction against the network output using PoTorch function allclose with varying tolerance levels. The reconstructions achieved numerical agreement within a relative and absolute tolerance of 10 This tolerance is approxi- mately 50 times the machine epsilon 2.2 10-16 for 64-bit floating-point numbers, indicating high-fidelity reconstruction that is numerically equivalent to the reference implementation for practical purposcs. As an additional metric, the norm of the detached Jacobian reconst reconstruction error divided by the norm out of the output on the order of 10-14.
The numerical computation of the full detached Jacobian matrix takes the order of 10 seconds for input sequence of 8 tokens for Llama. 3.2 3B in float32 GPU with 24 GB VRAM. In contrast, the full Jacobian matrix for the same sequence at float64 precision with Qwen 3 14B on GPU with 40 GB VRAM takes 20 seconds. An approximate method for computing the top k singular vectors of the detached Jacobian without forming the full matrix utilizing Lanezos iteration has also been implemented in JAX for Gemma 4B, allowing for the efficient computation of the top 16 singular vectors of the detached Jacobian for up to 100 input-token input. The maximum length tested on GPU with 80 GB VRAM was over 400 tokens for only the top singular vector corresponding to each token. The Lanczos method trades reconstruction precision for scalability while preserving interpretability, and examples available in the code repository.
3.2 Single-unit feature selectivity and invariance
Since the detached Jacobian applied to the input embedding reproduces the predicted output embedding vector, and the elements of the predicted output embedding vector are the units of the last transformer layer, the rows of the detached Jacobian matrices represent the input features which the last layer units are selective and invariant for that particular input sequence (Kadkhodaie et al. 2023 Mohan et al. 2019

---

Published in Transactions on Machine Learning Research (10/2025)
A
pppt-predichhnn "The bridge out Marin is the [mmos]]"" Coss the Detached jacobian Largest Norms
Model: meta ilamot lama: 3.2. Instruct
B
input-prediction "The bridge out Mar is the [tmost])"
Model: meta laamsUaae instruct
Detachod Jacobian
Totein
Token
Rows::
12
Token 1,
Token Rows: bridge
Toten 2, Ve. bridge
Tolen Roow: planea
Tokan 3,
12
Tokan Rows: off
Tohan 4, Vu off
Rows: Marn
10an1
Taken 5, Martn
Tokan Nows:
Tokan 5, Va.
25-
Rows, _the
Sirgaatt 4300 www.:nggeetbeeee
Takeo the
Takoon 0. Cots: first
Takeen 0, Us: most
Takeen
Takrn Cols: most
100
Token 2, Us most
Cols: Marin
Taken
Taken Cols: most
Token 4, Ue most
Thkeen Cols: Marin
Token 5, Marin
Tolen Cols: most
Token 6, movi
Token Coll: most
Totan most
Serted CHAMmn index
Singubr value index
Figure 2: Given the sequence "The bridge out of Marin is the", the most likely prediction is "most" for Llama 3.2 3B. The detached Jacobian matrices for each token represent an equivalent linear system that computes the predicted output embedding. A) We show the features which drive large responses in single units in the last decoder layer, which are the rows of the detached Jacobian with the largest norm values, and decode cach of those onto the most likely input embedding loken. The block of words al the top shows the ordered decoded "feature" input tokens from the largest rows of the detached Jacobian matrix for the input tokens. A similar operation is carried out for columns of the largest norm values, which are decoded to the output token space. Note that the activation distribution of column magnitudes is fairly sparse, with only a few units driving the response. B) We take the singular value decomposition of the detached Jacobian matrix corresponding each input token, which summarizes the modes driving the response, and decode the right and left singular vectors V and U to input and output embeddings, shown in colors. The singular value spectrum is extremely low rank, and decoding the U singular vectors returns candidate output token, including "most" and "first". Decoding the V singular vectors returns variants of the input tokens like "bridge", "Marin" and "is", as well as others that are not clearly related to the input sequence.
The activation of a particular unit in the last layer is determined by the inner product of a row of the detached Jacobian and the input embedding vector. We can sort by the magnilude of row norms, then map the largest- magnitude rows of the detached Jacobian back to the input embedding space (via cosine similarity to the input embedding matrix, since input embeddings are not typically mapped back to tokens during normal model operation) to determine the tokens that cause each unit be strongly positive or negative. We can see in Fig. A that the units respond strolgly to the words of the prompt, including "bridge", "Marin" and "is". Decoding of the rows of the detached Jacobian for each token as well the distribution of activations for this sequence is shown in Fig. DA. The columns of the Jacobian can also be decoded in the conventional manner to the output token space with the unembedding layer, and these turn out to be tokens that could be predicted, which include words like "most" or "first", which could be acceptable outputs.
3.3 Singular vectors of the detached Jacobian
An alternative approach is to look at the singular value decomposition of the detached Jacobian J UEVT following Mohan et al. (2019). Since the detached Jacobian represents the forward computation, the fact that the SVD is very low rank shows the entire forward computation can be approximated with only a few singular vectors operating on the input embeddings.
Unlike image denoising models (Mohan et al. 2019 Kadkhodaic et al., 2023) where input and output spaces are similar and singular vectors U and V have a high cosine similarity, corresponding left and right singular vectors of LLMs differ substantially. This reflects the asymmetric nature of next-token prediction, right
7

---

Published in Transactions on Machine Learning Research (10/2025)
singular vectors V capture which input token features drive the computation, while left singular vectors U capture which output token directions are predicted.
In Fig. 2B, the singular vectors are decoded for Llama 3.2 3B (and for other models in supplemental Fig. A3
The right singular vectors V are decoded to input tokens in the same way the rows of the detached Jacobian were above (nearest-neighhor to input embeddings from cosine similarity), and we see similar decoding of the top tokens to the features driving the most active single units. The left singular vectors U can be decoded to output embedding tokens (with the conventional method from the unembedding matrix), and "most" is the strongest, it was in the columns of the detached Jacobian matrices.
Layee R5_.
lagger [miett
frist wingest lateet Sutteet list thind
Iridge
bridge
landac: onew
exriect first
Inseet
Suctret
hightsoy
Layer
leidge Bhrigge
Bridge
Inidge
brdgge
Bridge
elennly
Layer 25_2
Inldow bridge
Beidge
bridge parlisment
Iedt
june
INCIDENT syrchollem
Laoee 20_0
Bnd [ssse] largeni longott lnteat gatesse
july
lutigg
beidaee
pellaphue galenny cornectins
Hishnay
Lext
cace Lardert nonds
Layse 26
I
Ieidge [brilgni
meteploe
Beidgrs BBidge
Imiggs
bralge
stouekures teid (hrilge
duaceet yine Morgits
Laye palliweett etractures
hrrldwe Pasliomet
(beldg
(rste
panilied
gyuhollen 'nttaphor
Laoss
lirlage lnsagor
Iubdae
botileneck
Layes 27_
Iesdge
Beidge
Brilgs
Indge
Beidge
baii Bridge
EXIT
jooi
exits
Lasus 27__
lettla
besige
Brodge
alouctizzce
Bdil EMU
kodi
Indidaal
EXTRA Incklants
Laous 25_0
letdax
lsurrtt largest Best
busdat last oldin. (RASE
Iunda:
beidges
Batdas
Bridae)
lilawwyy
bottharck higbaraus
Hilabwey reed catte
Layer 24
Inldgo
Ibrioge
Beidge
Ltrdge
Hollway
Jigloays
estex IXTT
eich
Loch
Laoee 26_2
letdan
besige
Uridgaa
Brodge
Exit
exit
Rait
sodi
EXi Sudkaus Moril
Kavall localan iictoptor
Layer 39 0
wrddgn
only
Succh fact thine longess fithh (nost
Inidg:
Teillges
Beidge
Tridge
Inly
Teet first
Bighway
Ukidd highwwys (esi)
foorth
Laye
highisaye frewwy
Lagus 21_2
letdaa
bekigee
betdas
structuns brii structure
Bail
Higanne
Suddtan Miaul Eivia mutaaper lncilood
Layer 30 0
Iridge
Imstt
longes fouett Think Teat
janly fita
Inidge
Teeiges
Beidge
Brilge
highway
only
biidge
last first
Highoway
rond hightws
Laye 20_3
krldgn
[Bildge
cost frewwy
couctiise road moade highwaay
bridgs
Bisage
brilgw
Ihriage
Layer
Inidgn
hridgre Twidge brld
sill sreh willing Theeuuda shig
Momppie Kysoit
Jodan Sad
Toses
Layer W
Inldan
joots
laat longeet fost taild
Inldge
beidgex
Bildge
last
Hot
(oct)
Hlighwwy
Lays 1
cuss wirince Intectie sat saline internor lou Spoodway
cussl conatline cusaial Corat Coastal route
brslge
Bestae
bralac
bariage
underwaterr beid
Layer
Twidgn [hrilgee)
(wiiign Thrilge structure
ship will willing
Jeedan Menophi
Kownit
Miscesippi
longett langest
huldgn
(rrldg
(trige
only
Highara
Laos 82_1
miscke calloe stesmn callua Truil besed vucuum
ounal cocatal cassiltue pause Cuart Rosts bouch
brsige:
underrsuse coomecting dack public member
Layer 2t 2
Iwidge [hestge [bndggs] [Brssge] best [Bridge
ship soll dock seiling sebb
Kowsit Jondan Mocpplis is Blineston Missinippe Nile
Laye 31_5 0
coty Hnt last
mostt) thiid maln sscand sutject
buidan [Brldg brrdgw
(tridge
cols
oaly last tiut bey maln cane mooel
astt
Layer
phart interio oalis Fores mosts
coast coeatline cosstal Coust mone bendh Charrel
span pultic member libwaay platform footing interwreinn deck
Laovs 21_2
betiles brrige
strustuze Betdars besk
Brida:
orku sattralt and woad
Koruit Nike Edusenius Suoketccavvan Tubss
Table 1: The top eight tokens decoded from the largest three singular vectors of the detached Jacobian for the layer outputs from Qwen 3 14B for the sequence "The bridge out of Marin is the" with the prediction [[only]] Legend: "Bridge", "'nly" "highway" "exit" "most." Semantic concepts emerge clearly by layer 25. The predicted token 'only' appears prominently in later layers alongside related infrastructure and geographic concepts. Note the progression from general bridge concepts in early layers to specific architectural terms (span, deck, platform, floating), geographic terms (coast, coastline, route, beach) and locations with notable bridges in the final layer. See also supplemental Tables 4 and for the longer tables for Llama, Gemma and Qwen.
3.4 Comparative Analysis of Singular Vectors in Llama and Qwen 3
A direct comparative analysis of the singular vectors derived from the detached Jacobian matrices of Llama 3.2B and Qwen 3 4B offers lens through which to view not only the shared computational principles of modern LLMs but also their distinct data-driven approaches. While both models demonstrate a consistent hierarchical organization of their predictive computations, they diverge significantly in their semantic rich- ness, their approach to multi-lingua representations, and their tokenization strategies. These differences are made visible by the SVD of their equivalent linear mappings and reveal unique styles that likely reflect their underlying training datasets.
In terms of their singular value spectra over 100 examples, Fig. 3 shows that both Llama 3 and Qwen 3 are consistently low-rank. The first token for Qwen has a low average rank at 1.01 than Llama 3 at 1.06, but Qwen's 's next singular vectors are all higher rank than those of Llama. Llama's "beginning of text" token is surprisingly of lower rank than the first text. token.
8

---

Published in Transactions on Machine Learning Research (10/2025)
A
Owen stable rank detached jacobian, 100 phrases
B
Uame 3.2 stable rank of detached jecobien. 100 phrases
25
Thk BoT, mean-1 1,5555
Treen mean:1.00m .0077
Token maan-110230
Token mean-1.2804
Token 1. meaa- 10057
Tosen mean-133706
Token meanm 1427
Toxen mean=1.4053
Thlenn 1.150
Count
10
5
1.0
1.2
1.4
2.0
2.2
2.4
1.00
1.05
1.30
1.15
1.20
3.25
1.30
1.35
1.40
Stable rant
Stable rank
Figure 3: For 100 short input phrases, the stable rank distribution as a function of input token number. Note [BoT] > token and Qwen 4B does not.
that Llama 3.2 3B uses
In terms of the semantic content of the singular vectors, both Llama and Qwen 3 employ a similar hierarchical strategy. The first singular vector Up with largest magnitude establishes the foundational layer of prediction. This vector primarily contains high-frequency tokens that provide grammatical structure reprosent the most probable continuations. For example, in "Should have known," both models place "better" and common puncluation in their Un vectors. This shared paltern reinforces the hypothesis that the dominant computational axis in transformers is dedicated to establishing a coherent structural and scaffold upon which more nuanced semantic meaning can built. See section A.5 in appendix for more examples of each of these analyses. 21 phrases out of 100 fit this category.
Llama 3 (Abstract Semantics) Qwen 3 (Direct Semantics)
A distinction in semantic processing is pronounced in the secondary singular vectors (U1 and U2). Llama 3 consistently demonstrates rich and abstract English-centric semantic space. For the input "Will break," " its U1 vector contains a diverse set of conceptual possibilities like "confidentiality"" "independence, " "promises," " and "ground." This indicates a capacity to reason about abstract concepts that can be "broken." Qwen 3's vectors for the same phrase are more direct and action-oriented, fcaturing tokens like "tics," " "neck," and "dance," " alongside Chinese characters for "stifl" and "can't." This highlights Llama 3's deep modeling of the nuances and abstractions within the English language. 14 phrases out of 100 fit this category.
Qwen 3's Multilingual Reasoning
Perhaps the most obvious difference revealed by this analysis is Qwen 3's multilingual and cross-lingual representation capability, which is largely absent in Llama 3's vectors for the analysed Euglish prompts. In nearly every example, Qwen 3's secondary vectors are populated with non non-English tokens tokens-primarils Chinese, but also Russian and others that are conceptually related to the input phrase. For "The broken," " Qwen 3's U1 vector includes Chinese tokens for "bicycle," " "vase," " "necklace," " and "window" all concrete examples of breakable objects. This demonstrates that Qwen does not operate a constrained linguistic space; rather, it accesses unified, cross- lons-llnngual conceptual representation to generate predictions. 38 phrases out of 100 fit this category.
Examples of sub-word Fragments in Qwen 3
We also observed difference in tokenization and morphological strategy. Qwen 3's secondary vectors fre- quently contain what appear to be sub-word fragments or tokenization artifacts (c.g., "e," "cer," "ection," " "ing"). The persistent recurrence of these tokens, often in the U2 vector, suggests that part of Qwen 3's computational process involves constructing or modifying words at a morphological level. This could be an efficient mechanism for handling its multilingual vocabulary. Llama tends operate with wholc-word semantic tokens, indicating a different approach vocabulary representation. 33 phrases out of 100 fit this category.
9

---

the 10th anniversary of the Broadway show, Published in Transactions on Machine Learning Research (10/2025)
3.5 Layer output singular vectors
Table shows the top eight tokens decoded from the largest three singular vectors of the detached Jacobians of selected layer outputs for Qwen 3 14B. The words "bridge" (and its variants), "highway", "exit", "most" and "only" are highlighted to show their appearances in decoded singular vectors. Early layers are excluded as the tokens are unintelligible. The emergence of intelligible tokens later layers shown in the tables as something like phase change the representation. Qwen generates infrastructure and engineering related concepts before producing "only".
Fig. 4 shows the normalized singular value spectra of the detached Jacobian at the output of every layer. Laam 3.2 3B has 28 layers, and decoding the largest singular vectors shows that the word representation of these intermediate operations is not interpretable until later layers. From the decoding of the top singular vector by layer, "only" emerges in layer 19. From the map of the progression of the projection of the lop two singular vectors onto the top two singular vectors of the last layer in Fig. 43, we first see a shift at layer toward the prediction.
Since the layer-by-layer operations are only linear, the stable rank R Ei 5 serves measure of the effectively dimensionality of the subspace of the representation at particular layer.
When looking at Wo ky the cumulative layer transform up through layer k, the dimensionality of the detached Jacobian steadily decreases. When considering each layer as its own individual transform W (where to_ = I W; for the simplified scenario of a single input token; there are other cross-token terms not shown here for mid-layer detached Jacobians for longer input sequences), we also see a large peak in dimensionality near the end.
Model
Layer intervention
II'm going to ari- Input sequence 'Im going ari- zoma to see the'
Normal response
Steered response 'Imm going ari- zona to see the Canyon,
Llama 3.1 8B
24 36
zona to see the Grand Canyon. I've heard it's must see. I've also heard it's bit of a trek to
Grand and I'm planning to like the Bright Golden Gate Bridge (I think that's name of the trail) in the Grand Canyon?' 'Here is painting of the Golden Gate Bridge San Fran- cisco. The Golden Gate Bridge one of the most famous bridges in the world.
Qwen 8B
24 36
HHre is painting of the
'Here is paint- ing of the same scene in the pre- vious question, but now the two peo- ple are standing on the same side the building.
Gemma 3 12B
33 / 48
went to new york
II went to new york the rial and museum. It very moving and emotional ex- perience.'
II went to new york to
to see the
"The Golden Gate Bridge Bridge." was great show.
Table 2: Detached Jacobian matrices as steering operators, pilot results with Llama 3.1 8B, Qwen 3 8B and Gemma 12B.
3.6 The detached Jacobian as conceptual steering operator
Steering vectors are well-known technique for altering LLM outputs (Liu et al., 2023 where a vector with certain properties is added to mid-layer representation, and the is passed through the rest. of
10

---

Published in Transactions on Machine Learning Research (10/2025)
A
Model: lamsst ama Instrrt
B
Modal meta- Sanitaalamaaimmaaaa
Iecot-predction: "Tee bridge out of Madnn the [mmast]]"
Input sng. The oridae our of Mari the [imonti"
Ce betached jacobian up la LAYES oulput
Layer 30 Ut steam omers, 9, res, UC heck, nop...
1.-
Layer Uc erf. dances, Canadians, swins, Nathenatic Delux....
Layer Ue logen, dir, ytu D. lestly pooudo, D. uniform..
Uei Bomb, uzai. Exolos,
clape, explosives.
"Layer 11, Les legen, J. prick, mid, aver, Vk, seed, Purks..
Dreaer "waat
Layer 13, oller, legen,
accordingy, manipuisted,
Layer 15, lu zero undonw, nnow appropriate, lowest, dity, improved, betlar..
rovrr war
Layer 17, diagically cornec,
appostae, Inwesd
Layer 19, tue only, perfect, lest, correct, less. best, combined, wont... highest,
wear)]
Layer 21; only,
Layer 23, Ukt most test, least, best. nighest, strongest, smallsst longest...
Dalyy ". perkert
Layer 24,
best. Hignest least, strongest worst onl...
Layer 25, Uki most least. best lergest. only, nighest, smatlest lest...
Layer 26, Uci
least. same, very. only. final, frrt...
Cayer 27, Ue: only, most, last, final, first, same, next, new.
28, Ue:
bez. only, longesi, last. firsi..
[ptar spprr
reaat Stapdar vator 0 ist. fras
C
Modal: meta Loma 3.2 Instruct limest]f
D
Model mets toma/Liam 3.2 38 instruct
Cumolatira Derathed (ecabran Matri Dimensionalit
out
layer depth
Layarsine Detached Jarablan syer Matrio D mendinwiity an function of layer depth
[[mest]]"
kee Dugput outh resibaal
layer Cutpatin reddunt Outpl (wit 1o
170
dutes malccal
2302
892
497
Lage: BARA
Figure 4: Since the transform representing the model forward operation is linear after detachment, we can also decompose each transformer layer as linear operation as well. A) The singular value spectrum for the cumulative transform up to layer i. Note that later layers are lower rank than earlier layers. The top singular vectors of the later layers show a clear relation to the prediction of "most". B) The projection of the top two singular vectors onto the top two singular vectors of the final layer. The singular vectors of the first 10 layers are very different than those of the last layer, so the projections remain close to the origin. At layer 11, they begin to approach those of the output layer. C) A measurement of the dimensionality of the cumulative transform up to the output of each layer as the stable rank. Within each layer, the outputs of the atlention and MLP modules (prior adding the residual terms) can also be decomposed linear mappings. The dimensionality decreases deeper into the network at each of these points, except for slight increase for the attention and MLP module outputs in layer 3. D) The dimensionality of the detached Jacobian for the layer-wise wise transform at layer i for the layer output, as well as the attention module output and MLP module output.
the network to generate output token. Here we utilize the detached Jacobian as an operator instead of an additive vector, and compute it from an intermediate layer for a "steering" phrase like "The Golden Gate" (after the "Golden Gate Claude" demo (Templeton et al., 2024)). ). The model predicts "Bridge", and this detached Jacobian matrix is used to steer the continuation of a new phrase toward this concept. For a new input phrase, like "Here a painting of the", the "new" input sequence's embedding vectors xhew are multiplied by the detached Jacobian previously computed from the steering concept J (xiteer), scaled by X and added to the layer activation faa from the "new" input.
11

---

Published in Transactions on Machine Learning Research (10/2025)
(x) = A fii fxx(())w))) + (1 A (Xttee)) nne
(16)
This steered intermediate representation then put through the remaining layers the network and the next token decoded. The detached Jacobian must. only be computed once for the steering concept, and therefore this method rather efficient. Table shows how the detached Jacobian from an intermediate layer imposes the Golden Gate Bridge the semantic output coherent with the rest of the input sentence, even when it is difficult to make logical connection. Beyond demonstrating practical utility, the success of the steering operator provides validation that the detached Jacobian captures actual semantic representations.
4 Discussion
The detached Jacobian approach allows for linear representations of the transformer decoder to be found for each input sequence, without changing the output. The intermediate outputs of each layer and the attention and MLP modules are also accurately reproduced by the detached Jacobian function.
The detached Jacobian operation accurate only at the specific operating point at which the matrices were compuled by the PyTorch autograd function. A short distance away in the input embedding neighborhood, the detached Jacobian will be extremely different because the manifold is highly curved. (Although local neighborhood validity is less applicable to LLMs which map tokens to embedding voctors, as inputs will only ever discretely sample the embedding space, and there is not an obvious need for exploring the local neighborhood to embedding vectors that do not represent words from the input vocabulary). The manifold is not piecewise linear, but only has a linear equivalent at the operating point, which can be found numerically for every input sequence.
5 Conclusion
While our current analysis covers a limited range of examples, the approach suggests a path toward large- scale interpretability by computing the detached Jacobian for many token predictions in a given dataset and analyzing the resulting lincar systems understand semantic patterns across diverse contexts. Given the low-rank nature of the detached Jacobian, our Lanczos method, which efficiently computes only the top singular vectors of the Jacobian, is step toward making this practical. Future work should explore this scaling potential, moving toward comprehensive equivalent linear analysis of LLM behavior across tasks, domains, and model architectures.
12

---

Published in Transactions on Machine Learning Research (10/2025)
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXio:1610.0m 01644, 2016.
Randall Balestriero and Richard Baraniuk. Fast jacobian-vector product for deep networks. ar preprint arXiv:2104.00221 2021.
Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, et al. Interpreting neural networks through the polytope lens. arXiv preprint Xr::22221. 12312, 2022.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hadfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https///rraallooneerrriiiiaasss
Aditya Cowsik, Tamra Nebabu, Xiao-Liang Qi, and Surya Ganguli. Geometric dynamics of signal propagation predict trainability of transformers. Xix preprint or arXiv:240300277 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Hemighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-oodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathemat- ical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformon dicruutsppn///001//aaooottttttn pttppp/////aa//o///oo// Sercuits.pub/200
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishck Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint ar 2ix:2447.217/6 2024.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint orXix:1606.000 15,
2016.
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphanc Mallat. Generalization in diffusion models arises from geometry-sdaplive harmonic representation. arXiv preprint ar Xrx::22300.00 0.05557, 2023.
Tom Licberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all once on gemma 2. arXiv preprint 2rXiv:2408.05147, 2024.
Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06608 2023.
Sreyas Mohan, Zahra Kadkhodaie, Eero P Simoncelli, and Carlos Vernandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. arXiv preprint
arXiv:1906.05// 2019.
Nvidia. Accelerating hugging face laaaa 2 and llama models with transformer engine. wtweerrennegeeeeeguuudeeee/// wwwepleeeteeeleeeeeeeeeeeeeeeeee 11ama/ tutorial tutorial_acceleeeae accelerate hf 1al_llmaa__aa with html 2024.
https: //dpss://doc...o
wwwiia.iii.....m.. com/ /ow.doo.iiiiiaaaoooooo.ooooi.cid
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky- Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, el al. Open problems in mechanistic inter- pretability, Transactions on Machine Learning Research, 2025.
Noam Shazeer. Glu variants improve transformer. arxiv preprint arXi::2002.05200 2020.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas I, Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Transformer Circwils Thread, 2024. URL httpgs//trraasfforrrrrriiiisssss pub/
bondex.html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. (ncurips), 2017. arXiv preprint arXiv: 706.00066.. 10: $0140525X16001833 2017.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in neural information processing systems, 32, 2019.
Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, and Yi Ma. White-box transformers via sparse rate reduction. Advances in Newral Information Processing Systems, 36:9422 9457, 2023.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
14

---

Published in Transactions on Machine Learning Research (10/2025)
A
Appendix
A.1 Code availability
Code is provided as a zip file (and will be made available on github).
A.2 Pointwise linear GELU
Gemma 3 uses the approximate GELU activation function. Below y 0.44715. Here is the derivation of the pointwise linear version of GELU used for Gemma 3 in the preceding analysis.
GELU (x) 2x (1 + tanh 222 (x + )x²)))
(17)
GELU (x) 2x 1 tanh 22/m (x + + x=x
(18)
② (ELU(x)]Xxxx)).. GELU (xt)= (1 0x
(19)
A.3 Singular vectors across model families
shows this same analysis for Llama 3, Qwen and Gemma across two different sizes of each. Note the low-rank structure of each of the detached Jacobians, well the differing decoding of the top singular vectors from each input embedding vector. The first or "beginning of sequence" token has the highest magnitude in each spectrum reflecting how the positional encoding is entangled with semantic information in the detach Jacobian representation.
Fig A3
A.4 Additional models
Pointwise linearity for Deepseek RI 0528 Qwen 3 8B Distill, Phi 4, Mistral Ministral and OLMo 2 are shown on the following page. See Fig. A4
A.5 Examples for comparative analysis of singular vectors in Llama and Qwen 3
Shared High-Probability Tokens in Up
This pattern shows both models using their primary singular vector (U)) establish foundation of common, structurally likely next words.
For the phrase "To see," both models prioritize articles and question words.
Qwen 3 Un: the a this an all how what and
Llama Up: the a and what an if For "To complete," " both models identify determiners as the most probable continuations.
Qwen 3 Up: the this his my your
Llama Up: this the and
For "The final result," " the Uo vectors in both models are dominated by common prepositions and linking verbs that would grammatically follow the phrase.
Qwen 3 Ug: of is in for from after was
15

---

Published in Transactions on Machine Learning Research (10/2025)
Llama Up: of is in and
Both models use their primary singular vector (U)) to propose very similar sets of common, structurally-likely next words. This highlights a shared foundational strategy of prioritizing grammatical coherence.
21 phrases out of 100 fit this category.
Before they: Both suggest verbs like were, could, start.
While walking: Both suggest prepositions of movement like in, through, on, along, around.
To see: Both prioritize articles (the, a) and question words (shat, how).
Will break: Both suggest particles like down and up, and articles like the, a.
Must leave: Both list determiners (the, a, this) and prepositions (in, at).
Should take: Both include a, the, into, and care.
After reading: Both list the, this, a, about, and "
When finished: Both suggest and with.
To begin: Both prioritize, and with.
May open: Both suggest a, the, up, and in.
Could drive: Both include a, the, in, and
During lunch: Both list time, and break.
To learn: Both prioritize the, more, about, and how.
The green: Both include and, light, is.
The old man: Both list linking verbs (was, is) and conjunctions (and).
To build they: Both suggest modal verbs (have, need, must, would).
The fast car: Both include is, has, and,
The tall building: Both list is, in, has, with.
To create: Both prioritize articles a, an, the.
The response: Both include to, is, of.
The solution: Both list to, of, is, for.
Llama 3 (Abstract Semantics) Qwen 3 (Direct Semantics)
This pattern illustrates how Llama 3's secondary vectors often explore a wider and more abstract conceptual space compared to Qwen 3's more direct and action-oriented suggestions.
For the phrase "Should take," Llama 3 suggests abstract responsibilities or concepts one should "take on," while Qwen 3 suggests direct objects or actions.
Llama 3 Up:
utmost admission inspiration revision discipline quitting
responsibility guidance
Qwen 3 U;: refuge aways -away brib # (half-day) FlK (lunch) away i (this sum)
16

---

Published in Transactions on Machine Learning Research (10/2025)
For "To imagine," " Llama 3's vectors include abstract and philosophical concepts imagine, whereas Qwen 3 focuses on more concrete items like "scenarios."
Llama U1: reconstruct ethical erect owning peace embodied meanings yourself
Qwen 3 Ui: scenarios Scenario
h (scene) scenario th fi (is kind of) oha Scenario worlds
For "The discovery," Llama 3's vectors describe the impact and nature of discovery (revolutionary, baffling), while Qwen 3's vectors describe the event of a discovery (a journey, unintentional bulletin).
Llama U1: revolution shed bust of details vind catapult baff
Qwen U/: गा (shock) # (anecdote) Vin (New West/ New Zealand) T-P (a) 2 Ni (journey)
TB (unintentional) Hi (bulletin) ń| (small town)
A.5.1 Llama 3 (Abstract Semantics) vs. Qwen (Direct Semantics)
Here, Llama 3's secondary vectors explore broader, more abstract concepts, while Qwen 3's are more concrete and action-oriented.
14 phrases out of 100 show this strong contrast.
Will break: Llama confidentiality, independence; Qwen ties, neck, dance.
Must leave: Llama departing, orientation; Qwen immediately, room.
Should take: Llama admission, inspiration, discipline; Qwen refuge, advantage.
The broken: Llama. fragments, promises, torn; Qwen window, clock,
To begin: Llama brainstorm, conceptual; Qwen start, validate.
May open: Llama invitation, plea; Qwen windows, sesame.
Could drive: Llama distracted, fleets, uninsured: Qwen drunk, uphill.
The discovery: Llama revolution, catapult; Qwen journey, bulletin.
To prevent: Llama vulnerability, security; Qwen corrosion, fires.
The solution: Llama vector, lattice, eigen; Qwen set, definition.
complete: Llama projects, tasks; Qwen orders, assigument.
Were planning: Llama launching, upcoming: Qwen permission, meetings.
The evidence: Llama overwhelmingly, against; Qwen suggests, linking.
To create: Llama customized, empowering: Qwen custom, interactive.
Qwen 3's Multilingual Reasoning
This pattern showcases Qwen 3's unique ability to access a cross-lingual conceptual space, populating its secondary vectors with semantically relevant non-English tokens.
For the phrase "The fast car," " Qwen 3's U1 vector includes multiple Chinese words related to speed and motion.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Qwen 3 U1: overt A (speed) ]z]) (surpass)
(motion) Speed if (chase)
- (speed)
riages
For "The fresh bread smelled," " Qwen 3's U2 vector is a list of Chinese synonyms and related concepts for "smell" and "fragrance."
Qwen 3 U2: smell smells [ Saniff/smll) + T (fragrance/scent) smelling
(smell/hear)
f (fragrant)
क (aroma/fragrance)
For "Should help her," the Un vector remarkably contains relevant concepts from multiple languages, includ- ing Chinese (career development, alleviate), Russian (cope/handle), and Vietnamese (support/hell hhelp).
Qwen 3 U:: 3 (career development) спранитьея haandle/copo/ hō trØ (support/help) * # (alleviate) unpack H I1'
(manage work) i (transition) *ilk (studies)
38 phrases out of 100 contain clear examples of multilinguaa reasoning.
Examples of Sub-word Fragments in Qwen
For the phrase "While walking," " the second singular vector for the token "walking" is almost entirely com- posed of these fragments, including common suffixes.
Vector (Token 1, U2): ection ing eer ignKey cion ging eed
For "To begin," " the U2 vector includes the common suffixes -ments and -ly, suggesting a mode for building nouns and adverbs.
Vector (Token 1, U2): ments eel hips eed s eve ly
For "The deep water," the U2 vector for the token "water" contains fragments like -ness and -ection.
Vector (Token 2, U2): y eel
ection yth
A.5.2 Qwen 3's WWord-Building Vector
This category identifies phrases where secondary Qwen 3 vector is dominated by sub-word fragments and morphological units (c.g., -ing, -tion, -eer, -ness).
33 phrases out of 100 clearly display a dedicated morphological vector.
18

---

Published in Transactions on Machine Learning Research (10/2025)
la </brgin of toxt emboedlfe): yor The
A
2y(x) modellng) (89))
Ous-token impult
Prodicted token: The
4/>
4 detached Jacoblan matrix the 1-th token
B
<[bogin of text/s. The (ee xo) cabbedits, 1y).
75(5) model(s. x) (10(s (i76s
Two-tokon iaput: ex 1. The
Predicted
1g <jheglo text 1, Thes 12 in 1/0, OKe Kar %al enbeedt, Sp feh Yox
201
yvbx) model(sa, Nox x) 2, 000i J. (10%, tt Ja (x)%s
Thows-token impatt: Ups,
Predicted token 201
J <b
2r':
4, 3,2
Figure A1: An overview of next-token prediction in the Llama 3.2 3B transformer decoder and decomposition of the predicted embedding vector computation using the detached Jacobian. Generating three tokens with only < BoT| > as input produces "The 201". For each prediction, each input token t mapped to an embedding vector x;, and the network generates the embedding of next token. The phrase turns out be "The 2019-2020 season". The detached Jacobian J+(x) of the predicted output embedding with respect to the input embeddings is composed of matrix corresponding to each input vector. Each detached Jacobian matrix JF(x) is a function of the entire input sequence but operates only on its corresponding input embedding vector. The matrices tend to be extremely low rank, shown in the inset figures, and the matrix Jo varies across A), B) and C) above because the input sequences differ. Since the detached Jacobian captures the entirety of the model operation in a linear system (numerically, for given input sequence), tools like the SVD can be used interpret the model and its sub-components.
19

---

Published in Transactions on Machine Learning Research (10/2025)
A
Model: metalllama/Llama- ---.aam-------nnstruu The bridge out of Marin is the [[most]]
B
Model: mete: mete-liamatl lammarLamm 3 3-3-Instruct input-prodiction: "The bridge out the [Imost]]'
SVD
Jacoolan
Detached Jacobian Reconstruction Error 2.53e-15
Token Ve: now. D. few:
quickly...
10.0
Taken 1, Vir The, the, St, THF, The...
Man
Original Jacobian Reconstruction
Taken Vic bridge, Bridge, bridges, Brid, Bridges..
pacoolan_iinuus Hinpur,, eew.edddngggg.... vector
7.5
Identity llocally linear)
Detached Jacobian Reconstructian
Toben Ver out,
Out, -ouf, out...
Bian
Taken 4, Vic off, into, from, over, exit...
5.0
Taken 5, Vi: Marin, San, marin, Mission, Marie...
Takein Vir was, are, Bis, is...
Singua valoe magiture
220
Taken the, The, -the, _the...
Taken 0, Uc most, best, one, I, only, Mar... most, one, best, only, first..
Token UK
B. best...
200
Token 3, Ust most, one, Marin, best, only. Mar,
(jacoiaan,
-2.5
Token 4, Uk most, Marin, one, best, only, Mar, last...
Marin, most, Bay, only, one, Mar...
-5.0
Token Uke most. best, first, same, longest...
Token 7, Uie most, longest, first, one, finel, last, same...
-7.5
-10.0
-10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
10.0
Output Embedding
C
Model: Owen/Owen3-114
D
Input-preddiction: "The brigge out of Marin is the ([only]]
Model: Qwenn/wen2-148
The bridge out Marin is the [[only]]
SVD of the Detached jacobian
Detached Jacobian Reconstruction Error 2.56e-15
:80
loken Vi; The. Human, The, "The, IHE...
Toonn
10.0
Vic bridge. Bridge, church, prisen, bridge...
Original Jacobian Reconstruction
Vic Out, off, Dout..
Token
Tolen
Sum (Jacobian. Fimputt embedding vector I)
7.5
Identity (locally linear)
Token Vic of, Of, Uof, -of...
Detached Jacobian Reconstruction
Taken Vi: Marin, marin, Mart, Maric, margin...
Token Vic is, are, [Is, was, -Is..
5.0
Token Vic the, The, the, an. THE...
Snggitr vaue mugatude
002
2.5
Taken Uk only, most, first, main, last, one, second...
Token 1, Ul Golden, only, main, toll, longest, subject, first, one...
Token 2, Ulc only, Golden, last, final, main, first, one, most...
Tokon Uk: Golden, only, last, bridge, main, most, final...
-2.5
Token Ule Golden,
feg. San, SF, Mend, Token Uie only, bridge, main, longest, third, second, largest...
-5.0
Taken Us gateway, bridge, Golden, key, main, choke, northern, same...
-10.0
-10.0
-7.5
-5.0
225
0.0
5.0
7.5
10.0
Output Embedding
Singuler value indix
E
Model: moogle/gemma-3-4....
F
Input+prediction: "The bridge aut of Marin is the [[Golden]]' SVD the Detached jacobian
Model: googlelgemma-33-0g
The bridge out of Marin the [[Golden]] Detached Jacobian Recanstruction Error 2.87e-15
Token 0, ebrooos doesm, Don. Don...
Token Vol MITT Stochastic, Biology, [ro lingustics...
Taken 2, Vs: Bridge, bridge, Bridges, cdu, Su...
2ton
Sum (Jacobian, Jihput embedding, vector_] ()
Taken 3, Vsa 14, Zha, Jackie, Emily, Wu...
Taken Vi: deity, канториы... .. Corvette...
Shor
Token Vir Marin, Marin, Sonoma. Chev, Reussesu...
Token Vo: Dylan, Wendy, Emms, Amanda, isabelle... Token Vs: the, the. THE. this, its..
Snpplor volve mogntud:
Token Ust one, most, only, Golden, main, longest... most, Golden, best,
Token Uet
Taken 2, Uni bridge, Bridge, Richmond, truss, Puente, Golden, Tall, Gareer...
Token 3, Ui: bridge, Golden, Bridge, Fuente, Highway, Richnond,, cause, one...
Taken Un bridge, Puente, Golden, Bridge, Highway, Marin, Richmond, toll...
-10
Uv: Marin, Sonoma, Contra,
Richmend, contra, Doyle ...
laken Us: bridge, Golden, Uridge, Puente, highwwy, Hay, Brooklyn, Interstate...
Token Us: bridge. Bridge, Brooklyn, world, way. Golden, famous, infamous...
Original Jacobian Reconstruction
Identity llocally linear)
-20
Detached Jacabian Raconstruction
Output Embedding
Singuaa value Index
10
Figure A2: The detached Jacobian rectton error and SVD for Llama 3.2 3B, Qwen 3 14B and Gemma 3 4B
20

---

Published in Transactions on Machine Learning Research (10/2025)
A
Mike indaar an arie 22 Iaia
B
lepal specisiie UNB Bul
Magis Dest
ISee'' the
Bieer erdees lnitees Frest Edeget
Intent Bedheds Eegiider.
3 OUT
si de from, el.
1.10 2%, Jobtee Paityy
Iiiar Mari pe mara Moa--
1966
entor Indoon, Liiker,
3Sken
finst male
wi thai ony,
Mardd oarnnnn 10 Ae buachio New Dooking
E
Wpcrepadctaal Mode: Quen Queer The tez Spral
F
Tdan No. salweyy
Loan
Thar the TNC The
Iohn Colorr Asomond
Thar K Soleen wiy Car trite
Toar pdeen cartar pereeer Dortens (ec Thrn.
Figure A3: Singular value decomposition of the detached Jacobian for different families and sizes of language models (from 3B to 70B parameters) evaluating the input sequence "The bridge out of Marin is the", followed by predicted token. The left singular vectors decode to tokens related to bridges and local geography, particularly the Golden Gate Bridge, while singular value spectra all have extremely low rank (see below for quantification). Each row shows top tokens associated with different singular vectors, demonstrating how models encode semantic knowledge about the input sequence and the prediction. See Fig. A4 for Deepseek R1 0528 Qwen 3 SB Distill, Phi 4, Mistral Ministral and OLMo 2.
21

---

Published in Transactions on Machine Learning Research (10/2025)
Model: deepseck derpteck.a/DeeppSeckk R1 0528 Qwen3 8B
Vosel despseek iDeeppeec- A1 2528 Ceenn ap input stodiction: bragge
The bridge out of Marin is the [[only]]
RAB 5912 the Dutached Jacatian
the [[rry]]"
Detached Jacobian Reconstruction Error 1.06e-00
You.
Tiken 1. brippe, bridpes, EL brear Clanon..
Original jacobian Reconstration
Thien 3, Vi nut, Bout, in, OUT, InE...
Sum (lecbbion jindd. enteetiinggg penteetingg.eerrrr peccur
identity locaaly linar)
Tolen 3. li seun, E. for..
40
Tolur 3, Vi ane Da be..
Ialen 5, Vie ths, IHc. the
20
bra
Tolar c, Us only, first mrstt lest man, best same, longsst...
Meken i, Us bridge. br bridge structure crosunng spon longus, sity ontinnce..
kdeas 2, Us read cnly, warst, last, way, rouls, froi, bes..
bottlenek, muuee Harine, ferry Thion 5, Uk only, first bott enect, way, same, beghnning gatwaay, latt..
'oeen red
-20
TWien
-00-
"iean 6, Ui gateway, pottienecs, tharough, I., sind, beginiinn, Gatawayy cummination...
Singuler va inder
Mooci nicresofuue min ressoning
Model: microsofiyPhi- por-i----miirrrasaanini
crideitiitooon bulld [1oil
build we need [[to]]]
SYD thetee Debached pacation
Detached Jacobian Reconstruction
3.C
Original Jocobien Reconstruction
Tatan 2. ID, I3 avotitee, CCC peat.. Takoon 3. Vo [EE emegranats BD3 am,
Sum (accston wappinnnieee..... www.tttttiigggg..... vectori)
Iddr loccally lineer)
02
D
Datached Jacobian Reconstrucion
tro morges Path
A&l, beliel,
Thkm skrepee belett atllnggeee comtivit dry, remmideiion exlect,, centiment wanden Thaen 3 Untimpletty, Cs optim, Indrrectly simulta reoutly nepembely, dimilaty, repari
Output Emtbedding
Modal: mistralal/Mnistrall.e Instruct-2410
Thare was thunder and [lighhting]]
hosttonddztie: "There was thun and [ilithenneg""
Model midrail (Minkars Andruct--2110
SVD
jacab
Detached Jacobian Reconstruction Emor 2 2.04a.06
Toare 2 to (PIBBEE 1ii (P ooy pO9C
30
Original Jacobian Reconstruction
Jalion 2. Vi (SUHFO] (VIDOLE) [MG eND), (mogeod) [PG) (MIDULE) (SUFFi0) (NNOEEOI,, prassod) 1900
Tieen 1. Vi (SUFFR)) (VIDOLE) (MG END), (Pteno)) 1990)..
dacally linear)
Delached Jacobien Reconstruction
20-
(arntiin sipput enberddingggg.
2002
10
Teien 3. Ur lghoning, a. light, rain, there. the. it. thunder...
Tilion 3. Us Eghhning, light raln, thore. lightig the..
Tiken Ephoning, light
Treen Rchening, tohn,
lithtin,, stm....
-10
Treen 4, aphining, light, thunder rain, the lighing...
-20
-30
-30
1.
Outout Embedding q
Sirgilrr
Madel: a 3Jlena//OLAo 2-1244-78
inputepredicion:
Model: allonaOUMM 2 1124-78
A fluffy blue creature roamedd the [[Earth]]
SeD of the Decached 18000 on
Detached Jacobian Reconstruction Error 1.299.06
welt
meiing
Original jacabian Reconstruction
"Tiber Me mute, HI raping clerder, mec...
cum (eesbinn sireaat enbbanddinggggo pvertee
Detached Jacobian Reconstruction
denthy docally linerr)
"Iaknn Virro Ra,
"Ileen 3. We constructs, youlte, conrts, steves entiles..
100
the =jondetassts a. the
10
streets, forest,
"ihon river ground, aroo, place surface vilago ship. lorist.
oken 2. bh river wastes edge. eastem ground place, upper, ruins...
"iken 3. bi corpoe westes, corpees, piice comneed melee, wllign fet
"aken Ui foreet, planes nelds streets cau, eaith Nits strent "shen lu room, surfece, streets aby, listaa land, oity, road.
-10-
sau, grounee,
-20-
-20
Outaut Embedding
Siquiar
22 Figure A4: The detached Jacobian reconstruction crror and SVD for Deepsock R1 0528 Qwen 8B, Phi 4 Mini 4B, Mistral Ministral 8B and OLMo2 7B.

---

Published in Transactions on Machine Learning Research (10/2025)
A
Mises Pet dar Sevee Dh
B
Hockt Overr Oner1 240 Demached
This THE
Tilce Sinn thinn Bnce
2 W.: nen, Sren, /ree,
1941 you lysten, pilets hydden, Systime 111
3. syster, yiler, syslems, Myyiier Jooters
Trken eegaios,
Titio it Ind will
7.hmn 2 E. clows fre,
Tikan the of autoratically
biln thi R. of A sllwes C. for
130
Sin der volue
C
Tpslicaee "Wee df Bansir becdlan
trax: sueckikee "Wat singe Lisweil
koddaan
114
Tates
Tolan diding singer,
Trick D. the, undl thi 15 bV loke the, and Phi,
Maai 2.9 and bewby the bett
Ioon Un bodds bost bine Intten, telggins SAs compottions
Tolan Ut. and the to
SoM door (pp
F
Restrooed ogen cay 1741 the betsited
Sro of the besacied
Ichon 1o
Takan
2: Se oper, Grm, OPEAL
Token 2.30 doer, Door, 49, Doors, ddee
1din Me deer. Dank stie. ga0% Daw, parts
3 Uii. policy to, and
Talan U.b Li an, was
3 Uot poltcy to,
Thean b. billyy approucs
Twee 2: wuys kente, pallcy heaprrs tell way
G
laa prosirt (iae
H
troet 5xb af Detached betten
And the beewihed
Thlan Vi
Isve Va
hover, prevec poweree Preenzon
Tiken Vi prewent, preserta
Prover,
Iaan 2 Sactte A.. and ar thi,
Sear Lirthi and
rloping
Take 2.50 eo. over
Figure A5: Comparison of detached Jacobians for the same phrase across models.
23

---

Published in Transactions on Machine Learning Research (10/2025)
Tepat
Type bdee
Sapai Ielas 3
Lape
ou gan oray Yodl poin akea resuted
Jos Notad Sasrmook noms miss upettar boo cagu Yead uscal
Aoi tnk estino matual tiaes moch STUW spar
Tanl Rir Tim centrn Spare oer PabdBatto
daraas
Lave
of the Borie sorttredinn
rot miner oad
11.00
Ealon cisped apal almay Buy
vil tiee entean
Layw 11_1
Kit daalii aledin Lindinga Bridge polas
déle cos magleh siken
sen Basd oe
Burgir Coppect Billge Salariian Pronkran
Kely Chtcingaan Jarna Sceal Siadimm wnta
Eayan 12.0
ad. Mage ges tyal ndl dallatted Megs
Vi tlss Slyand Tillaneis wil 7os
Iudyy
Leolge
Risiy Duiggs Miad iln
Eaya
optn
copoo
Onttingias
Laye 15 0
Fen otne you wll orra 19s3
Kik (reu 5MB25 pder se apd corpect
bridge
Ieitpe
Sridgn
kaics (iidge tell
Laye 16
Britgt (estgg speeiee skm Syjtene wit line
790 et BI alakm [Rail
RAD Dekknon
Uplossde Prdklietin clacals
jone
dei
Layer 37 0
Lm yn tesk ents armil
jesty riwet
sik /ces gr (Mittg (bricge Passitattcn 'Jor
[irdgg] testom
[Briegrs a cheke ling
Laya 37
Brlge Marta boidgg
arlaywoi Bratu lrrigge shilk jomsectia
kys IDSNIVTY 799 Alletedd Bridge Local
Piutliatin ANDEL Inapiration bast stoales
11
1oslat hall
hotting Btano dctal lend Booting lccalio
Bridge Beidge)
Brige] [Bidge] Bealem
Tayer 16.
Baster [estey
(isign Marn Bralge
(ealon bosl
Bilge
Tealgn) GUNTRY IIeeTTE Quarin Chiiee
Pedillation y Woa MENR Tregieci box Rovil onl
Tayw 3
Iealshit walalstoel Barle Semmiae del Lan aore
Ir dhoelem may iou mer Tell
Sue moil Tenters Jeeam SP Trave
lestge
Layw 11_1
Mach Deslge
boldg
KY
Briige
Ddds aiina laayy bed HD Oskcnd voatas
Roors 19656 scats ANMEL Itvy
Laver 15 2
Fitcbe Loe LA Islgar Seithers lonte LA
theefem sen Usbland snokae tell prigos
SP San Uskand
(Golor)
Hitakers cheert ot
lotil:
[Begge] Benigy
linigg
Beoipi
Layer 20.1
Marh of BF toll selapest Grifand Merkeley
Usis Unitiedd coerns fling 150ke and letakee
Sotte rosle
[llpheas
Hoose eftu noste Hoy
Tasmr
uéfor PI/K0
Fateler
Eagas $1.0
Tesles uil Inilgm)
lovil Sai (Tilles TF
luiligi tull Biidgg
Teidge Tallal Biolgs moowing
bidge icll Beoggn
Thilggss Jul Tat Bhilgg
Tayer
SP
Eapas 21.3
Jok (las [ariip
Faai Suutaasa FLIRD Roseo
Muis Oalloal Fog Taye Bickeby SP
Mara Oulkcal SP Sau Stadled Dalidaa
Bridge
betdge
Irilpw
Eagss 15_1
Maris wiens Birkeey
cotia tlgg di2 Oliysihl
Buikisss Roste
Backnan
made Rosto and Alsasie
25.2
fibe slik (trlagee dualicc GMT
Bridge safe
Muls des Dedelay afBowslt
Muzo al ST Saluduce Betaky
Layer 23.0
1e0 San berdge
leshi Say (GiMee
Kith
connecting
bil brtiege (oookn) Ten Brintge tel besigge Tod
bexigns
to spen Berdgn TNIl
Marin fiatoe Rocel 1922 Chmminaars
21-2
Mili Bedaky strmalam
Madi BI Dockol
Brrislyy Bi
Baaken
tpoar
Eays: 1
Maris di (poto madi aids
ra CoeEts my and udil 291
Pwili
Soate Les
Alcesic Lee more Siglong
Lever 21
2GK 20te wenlie solk Bogg Rocearind Bil
Mail de 10794 dee Brikaey app BiS
Maric 31
Ontinn Eanell of Salalans
Laye 28
tell bestgn 14
custectigg
unl
Mill Intigg (elden 14 Vall Hindge Looden
beidge
latert teilgo Ta0l
Morn 1it Suyealer
Easis 25 3
dik odls like diala Bug Galausenat Bosan
Mode dee des Taper atle pir und
Muis SF
al Ralannal
Easm 26 0
tell wineet mose Sen (aalin) or lotigg joste
Goldinn tell (eesgg tel yoa carme Golder romiieg
1pan vall bestgi opors Caiiken riming vibal
Tayer 31
Merrs all (ooeestral remnetiiie welehyeage
Ilb meching
Swilie Vin Suste mich are snell Sarto
dGk Biy milk spate Aoonn lit Haley piec
Madl por muie 255 the lande Iiza
Layer 27 4
witjec bey) dite 57 di (o0t consecting toll
1ptn toll (oolden) brow bil Bas Toll ialy
spin 'all tol spece Veu Spen syrebll laoe
Laser
Borth muteerae Tahee Unmented
triinng post
Tiili Surra Atorre Seree mhn mite no
Taser 36
Ricussed lally ubjest ose bas sselled til
Orllie) tell Riderased open Son Toll Inly
1pen cill ssreho bil abos Reburell only
Tasm
Cosenaaa Toleets
Palle Auaeie Sunta Sume Lesking mato
Laym 25 0
203) lichocedd the, lse (soe] one sthject wibbe
Golden (so0] sell Heal Richnced the (Geden
last tial (GoMM)) watt letee tell (tse)
Laves 15
Gallea) The Greamtto tolre
sedlig leiter Ped
Vi Serto Partfo Servenl Rin Come sinling Muric SF Bizknood
Lapm 2
Nach alea dnd mile Caoke inlin anika
Maiie dli maai aon ccae any agaar D0y
Table 3: The top three singular vectors of the detached Jacobian for the layer outputs from Llama 3.1 8B for the sequence "The bridge out of Marin is the" with the prediction [[Golde]]]. Legend: "Bridge"
"only"
"highway"
"exit"
"most"
24

---

Published in Transactions on Machine Learning Research (10/2025)
Odfa Marc Sux Eatay
Parkla Dis
Layer -L.0
laat Sie
Bht
Gual pelalieeet bose ramarrad Nigit
Laps
tae undallly way
Iitte Eii Nape paaae Iasten Bidles
Tose
Lapt JU.
Uk feet how seroey
Inee Binle inne
Detenral Goste Guaaes
kitlgn loos
Tagn
Reggttng
annry oF 1ET vay
TSC Tqpeti Moatee
kegkek ageay
Tawer 29.2
Weet repari capll Tellaly seregit conceces explar
Overry
Geapee Sark Teden CM Bidlo
Maliin Ortwh
Tipdker Wanceower Piellia Sweein Baove
Tapr
Siep (unds (nea)] fist unoul very
jefe
Firt cnteellin lonk
Teitll
Tedell
(esden)
(molkeg beingin kook siltary
3
TAk Opc
Dit
Hogs Falaber Oee
beatee
Taye 21
(wway [onty (ee (exee Merooe
Feet
Teteel Teleel Teieen) (esten) Hoklgni Ore dhg che dng teileve
Layer 11_11
Dedag (Endei (eidge (eidgee piare Tel Tac
Sedgg
belaae
Detagn
Biddge
Tolgin Tel
[est (outal imt frea sacckte uns Ganee Satten
31_2
gleme 121631
reask tackhaer
Trato Crosdes (eldgs
Arcitectinl Solgiere Sualic
Mezzenn Muren office
[ete] (oakgr
[elden)
22_1
Styica Prante duccke sscgin Moldlegg Vib Whesie
Sodge Tel
Icld bode Todl Suttet
(oot) caMmL coatt
Lawer 22.2
3055 vrnut baciaance pactee gilte) srieesk grem
Sr
Britge
Geal Desigs
oete Ndwan rosts rotes seat Bertes Seovey esprommay
Bitlge Bndge
paice
Sevens Marts Sape
Expeli Colifrsse
Bistor
Calberians Calseris Coliarniee Yolk Morerry Yourche
Layer 21,0
Caliass (40)
beidgs Bennay
Packk Biilge
Cileca
bridgl
Dawmy
Bilgg ue
(iidgi
(eidge (indge
Billge bokgel
bolglng paars
1
Bidgg Langee
Srilal
cieceelie belgging
trillesd bater rosl Nightigs
Laper 25.0
Scage Seidgn
Tellgee
Cillica
Srigl
Seidgee
Calficna
beltge Bayy
Srige
beidgn
Dridge Bilgge brlge
bolgle
Layer 2
Dedai Bridge
Telige
brllgee paste
Setgpe
Bridgl
beita
Srabwin Icets tallbed Seoncy tolds read
till
Laye 20,2
Maate Mora Bumene
Seces Geans
Jr-4pm
aiige
Luskge
Leikgs)
Osakk Palki Nilveske Seeatees Heaglä
Casago
Nitaala Detaet
[eat]]
3
Brdggt belgpe
Teldan beligng
poaers
Bista trApsa
mallead Desncy pablwcy
Surve roo!
Layer 2122
Stnte Mara 31 Darrew Sununa
Brige
Bisken
Voal Auglo
Alahass Oaake
27.0
Scige
Calficca
Intast
Bilig
Vaii Mern feasuz Draeze
Street
Thil
Sche Angber Midorms Cate Birrhaaan Netrods
20.0
(Glde) Caliiciia
Callernia
Varie Mern oat Mial
Scactrpe Tinte Simarter
Supple
Callicea (S688)))
[aldan legeet
2-
Mtrt nare Sepoes
Seggr Sergttr
Table 4: The top three singular vectors of the detached Jacobian for the layer outputs from Gemma 3 4B for the sequence "The bridge out of Marin is the" with the prediction [[Golde]]]. Legend: "Bridge"
"only"
"highway"
"exit" "
"most"
25

---

Published in Transactions on Machine Learning Research (10/2025)
Loat miee
Ippal Inats
lopat taken
Layer
AUSE arw abe auttophes thet Filee stren
Liber
villa buision gotatte dopcom bine
Laver
We laterr Fil nor for and cen
Iyr loonling whith
Faiiiies compaion Beratlla FR...
Lager 2
1KY REGARD of
REGARD massage
emly slow callnng outtound lnttor tight
Lavir
Deal Eeiti one UXCEGT
Drizay raEnt lag odich batteed
Mangee mods USooso TORT Meatesa mastin Spurs
bye wyap
CONSTANTES
Layer
melopler
stet up
lesige
Iany
Leytr x
nic DK Sir Eapas Jos Th
nt3ea Sbarot Kina
lusseiscy KiDie
INCIDERT
Layar 29_0
selupo CONSTANTS gitch Uncoumeis masoage TRY
mntluphuc besige
Inrgesl ascol
unly
firs cesterptoor
finl hanlet use Sercell
Legionar
bobilunek
Layer
wrtoylor une Derno maleddnn OR30
Inidge
Thidge
bildyy
hridyy
aveus WithFrw rgi TTiiing pricete Frtering
Lacer 25
Ro laver Baller
und Jilte
rillig COMPONENT siull mirlippie
Layer
fra (lird [ssoed] lnegest fouth colmination
ontiepice first handest
bondest
frated
Layer 1
mitaglor makeskin ary BKAK
bril
heldge
bildgrs
beldge
Taidge
scmuss Lacalitong Whildemr
Lastr
Butter nellngg
pinootann antol iettpphee
Lawer
lagget
longee
beldgr
[iidge
Posiige
gateway
(esii)
hiphway
Layer 2
Iridar Iadiagr
Bididge
Bniiggs
ldld
Inldans
Ieldee
Beldge
beldge
rxit edtte steralyy
Layer
Inldgrs Ieekiig
beldge
pailiareed
wdit
Jans
INCIERNT symbolisn
Lesr 20.0
drd Issat
lacgeat bust
lougest ants gakenay
lutdan
bridase
mittapict aocenny connecting
Mialinyy Srtt
odll
lust bacolt runds
Layer 16
hildar Ioldagr
mettaphor
Brddge
Lhiday
Ioldens
beldge
strectuee brid
heloge
Marglos
Layrr 2
peclimmant stractszze
buidges
Puriiaaes
berdge
Edil
Gal
tbols
Roil
peateked
systholias mxttaber
Layer 27_0
dee largott
beldge
loopet
(nsit)
oldet lakeut
Iuldas
buidace
Balder
Boldans
Iurt Bott
eslt
Sigasoy
bottlenock best road chole
lindaae
Bosdae
Layer 27_2
Iridas landae
bridace
Boidge structuros
Ioll
Rait
(ast
calt
lucideas EXTRA
locidaaza
Layer 28. 0
bmndge Jorgest largesi first brsies last cloot (msel
lesdge
bidggs
Bestge
highney
esit
boitkneck highwwyys
Higliwyy
hot road cists
Layer 2
briday
luidar
bridgre
Bnidge bele
Iodt
Kolt
jast
rate
Sacdaan Moesi
Revalt Sucldent metappor
Layer 29_0 0
only
fourth lest
thanl longosi Silk (msel
lendg:
bindge
Bettlge
only
lest fest
Ingloway Ulre hegisnerr (soil) lour.li
Layer 2
lmndge
lnndges
Tesdge
Meretton beal steucture
Batl
Higinna
Suclaan Movtl
Elis necepler inssdent
Layer 30_0 o
hildgy (ooel
longest foorth tiird we innly rah
Ieldge bildgs
Taldge
highway
only
baritge
Duct fiee
Hightws
naul highans
Laver
bindges
Besiggss
Bridge
freewwy
Tighway
conskliee madd highways
hrrlge
Bridge
brrigge
Layrr 30
lesiae Merusture drextuzee larslaos
lanila: besé
vail seal welling Bermunda
Nleapho Buwse
Jondur
Suidum Jonns
Layer 31_1 0
bridgy Inooe]
Ioly
lark lingest fine thind lerget
bnidge buidgs [Tuligs
Toilg
only
lazt
highway
fine: Thrklgg
Insit
[lighow
Inort]
Layer
cual
Istence airlise invertor Los Specdseue
cosst cose cosatal
Cuast zotte beuch Cocatal
Braige
Irralge unxeewater
Layer
boidgge
beddgee
hest (iiddg strurture
calling dock reals
Jardan Miruphic Kuwait Miutsippp
Layer 35
bridg (noo]]
only finst lost longes thind lerget
Inidge [hrddgr bnddgrs Tevilg
Iwiddy
only lask firs
Beglowyy (nsek) main tosto Iesh
Layer 12_1
Sneekor alrine steems attlinee Well loved cacuun
coust conetal cosatiine wate Coaet Hoste beach
hridgs spaz undeewnder connecting dedk public nember
Lawe
|Pildee best [Pildge
dock salling seals
Kosalt Jondan Merephie Rlivestoon Misdoiipp
Layer 2
only Bost
theed second subjec:
Intder Larider briders
Botder
cnly
fost key
Lover
Junt
Room
Gronzal
libuary plilfom
rock
Layer
Teles
Table 5: The top three singular vectors of the detached Jacobian for the layer outputs from Qwen 3 14B for the sequence "The bridge out of Marin is the" with the prediction [[only]]. Legend: "Bridge"
"only"
"highway"
"exit"
"most"
26


Reviews:
[{"review_id": "cln55AStET", "reviewer": "Reviewer_QBRf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength: Detached Jacobian as an analysis tool\n\nThe linearization of an LLM, and the analysis of such linearization, is an interesting idea for interpretability. There is a rich literature on linear maps which can be leveraged in such cases (e.g., the authors gesture to Lanczos-based approach from numerical linear algebra for more efficient SVDs). As far as I know, the definition of the detached Jacobian and its analysis is novel.\n\n## Weakness 1: Theoretical justification of detached Jacobian\n\nIn my opinion, there are a couple of shortcomings with the definition and the detached Jacobian and its analysis:\n\n**There is insufficient discussion and justification of this method in relation to a Taylor approximation**. One can also linearize a (non-linear) function at a particular input $x$ via Taylor’s theorem, which yields the approximation $f(x’) = f(x) + J(x) (x’ - x)$. This is also exact at $x’ = x$, and much studied than the detached Jacobian (e.g., bounded error in the neighborhood of x). I think it is fine to introduce a new notion, but there should be an elaboration on why, e.g., if you *require* a linear map $f(x’) = L(x) x’$, as opposed to the Taylor approximation’s affine map, this should be justified.\n\nOne experiment that might help justify the detached Jacobian is comparing quality of the approximation of the detached Jacobian vs. the true Taylor approximation in progressively larger neighborhoods around $x$. Showing they are very similar would suggest they are both valid ways of doing an approximation of an LLM at some input.\n\n**The perfect reconstruction of the detached Jacobian might be trivial.** Figure 3 shows that the detached Jacobian “nearly exactly match the predicted embedding”. This is true, and the errors here ($10^{-3}$ for float16 and $10^{-6}$ for float32) suggest they are equivalent up to machine precision. However, as far as I understand, the detached Jacobian is *defined* to perfectly reconstruct the output for a particular input, so this result is unsurprising. Moreover, because of what is mentioned above, we know a Taylor approximation would also satisfy this property, though the function here is affine. So the fact that the (true) Jacobian is presented as having large error feels slightly misleading, unless you can justify the need for a linear map (i.e., compared the true vs. detached Jacobians without any bias).\n\nAs an extreme example, we know that the detached Jacobian at a point $x$, $D$, is a linear map that satisfies $D x= f(x)$, but there infinite linear maps satisfying such a property, so Figure 3 would remain exactly the same for all those alternative definitions of a detached Jacobian, though obviously the subsequent analysis would change. Why is it that the detached Jacobian *specifically* is the right construction? Intuitively it feels “closer” to the way the LLM operates, but hopefully there is a more grounded way to justify this.\n\n## Weakness 2: More comprehensive and clearer empirical evaluation\n\nThe results of the empirical evaluation are interesting, but they feel limited in scope. As far as I can tell, sections 3.2, 3.3, 3.4 and 3.5 consider a single sentence “The bridge out of Marin is the”. While delving deep into one example is good, the paper would benefit from a more comprehensive quantitative evaluation. For example, for comparisons across model families, you could take 100-1000 sentences and see how similar the predictions from the singular vectors/large magnitude rows/etc. are across model families.\n\nI found Section 3.6 but a bit hard to follow. I think there should be more explanation of the method and empirical setup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper defines the notion of the detached Jacobian (for a particular input sequence) as a tool for analyzing the behavior of LLMs. Given a particular input sequence, the detached Jacobian is constructed by “detaching” (i.e., telling the automatic differentiation library that certain variables be treated as constants for the purposes of differentiation) nodes of the computation graph such that the Jacobian of modified graph perfectly reconstructs the output of the original function on that input sequence. The high level idea is that given a non-linear operation in the graph, e.g., $g(x) = \\sigma(x) \\cdot x$, is can be replaced with a linear variant $f(x’) = \\sigma(x) \\cdot x’$ where $\\sigma(x)$ is treated as a constant and thus the Jacobian at any $x’$ is $\\sigma(x)$. When considering the original input, $x = x’$, $f(x)$ is a perfect reconstruction of the original output $g(x)$. The authors use this to (i) correlate high-magnitude rows in the detached Jacobian to particular tokens that follow logically from the original sentence, (ii) correlate singular vectors of the detached Jacobian with tokens, (iii) show the low (stable) rank of the detached Jacobian, and (iv) explore the detached Jacobian as a tool for output steering.\n\nOverall, I think the detached Jacobian is a potentially promising tool for interpretability, but the paper needs to further strengthen the theoretical justification of this object and its empirical analysis.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Further theoretical justification for the detached Jacobian in relation to the Taylor approximation**. The section above has some ideas on possible experiments/theoretical arguments that could be made. Addressing this would be critical in my opinion.\n2. **A more comprehensive, quantitative experiment that considers other prefixes** (see section above for one possibility). I think this might not be critical in the presence of a very compelling theoretical justification for the detached Jacobian, but strong empirical insights could be used to argue for the usefulness of the detached Jacobian in lieu of a rigorous theoretical argument.\n3. **Clarification on Section 3.6**. This would strengthen the paper, but it might not be critical (I would still appreciate clarification on the setup during the rebuttal).", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "otJ5BAaoz7", "reviewer": "Reviewer_4yX4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper presents a new approach to interpretability of LLMs, that does not rely on training a separate layer (e.g. a sparse autoencoder) and then using an LLM to interpret the resulting features. Thus the features extracted correspond directly already existing embedding features, rather than automated labelling by an LLM. It proposes approaches to analyse and steer features. \n- Good explanation of method and how to detach particular parts of the inference computation to obtain a linear approximation of the LLM inference function for a given input.\n- Implementation provided that shows the approach works for a wide selection of open-weights LLMs.\n\n## Weaknesses\n- The paper uses the term local linearity, which commonly refers to a function being well approximated by a tangent line in the neighbourhood of a certain point. In the paper, however, the terms seems to mean a linear approximation of a non-linear function that is valid for a particular point. Thus while the main point paper, that LLM can be approximated linearly at each point holds, referring to this as a locally linear mapping seems misleading.\nThe authors themselves point out that: *This numerical Jacobian computation captures the complete forward operation of the model, including activation functions and attention, although it is only valid for that particular input sequence (more “pointwise” linear than \npiecewise linear)*.\n- The figures are cluttered and it is hard to understand them given the captions provided. Figure 3 attempts to show that the detached Jacobian is a good linear approximation of inference, but has 6 small subplots for different models, rather than 1 or 2 which would be a sufficient example.\n- Overall it seems like the paper is really trying to fit within a 12 page limit, which is harming legibility and clarity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission:\n- Shows that inference in an LLM can be approximated linearly at any given point using a detached Jacobian.\n- Presents how to use the linear approximation to extract potentially interpretable features from layers of the LLM.\n- Shows how steering can be performed using the extracted features.\nTo summarise, an alternative approach to interpreting LLMs other than autoencoders is proposed and implemented.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Content\n- Please change *local linearity* to *linear approximation at fixed point* or some other appropriate term as the use of local linearity in this context is non-standard and confusing.\n- Would be useful to add why this is an improvement or what advantages it may offer over existing interpretability methods.\n- I don't see the relevance of figure 2, as far as I understand, is are the equations above the plot which could be typeset in the document, savings space. \n- Please focus on one or two examples in Figure 3 and limit it to one or two subplots with a more informative keys which demonstrate the main finding (LLM inference can be linearly approximated for a given point).\n- Figures 4 and 5 are hard to read with the lists of words overlain on the plot. Please separate them out. Maybe combine figures 4 and 5 into one (they are very similar) and this would give the space to present the lists of words in a more legible format. The shared figure could have 2 or at most 4 subplots. \n- Axis label of figure 6B is not visible. Please fix.\n- Tables 1,2, and 3 are too small to be legible it would be better if only a subset of the input tokens, as well as a subset of the layers were presented which would greatly improve legibility. \n- Typo on page 11 line 2 (main text, not caption) - should be rest not rets.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4uUuLTcsvP", "reviewer": "Reviewer_P4MZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Interesting method\n- The conceptual steering experiment is potentially useful as a mechanism for fine-grained control over the model behaviours.\n\nWeaknesses:\n- There is not enough discussion of why the detached Jacobian specifically provides a better linearization than the true Jacobian\n- The graphs and especially the tables contain too much text which is way too small to read, even at high zoom. This isn't just a font issue, these graphs and tables should be restructured to be more concise.\n- All analyses seem to have been done on a single sentence in the main text of the paper, with a few more sentences in the appendices. A more quantitative study on datasets of non-trivial size (e.g. 1000 examples), both in-distribution and out-of-distribution for the models, should have been performed in order to make the results more robust.\n- The conclusion mention possible safety applications w.r.t. singular vectors for bias, misinformation or toxic content, but these are not identified in the paper, and it's unclear if they can be identified. This seems like an overclaim.\n\nEDIT:\n\nThe authors have largely addressed my concerns in the last revision of the manuscript.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper present a method of representation analysis for Transformer LLMs based on linearization.\n\nSpecifically, the method apply to Transformer models without bias parameters and where all non-linear functions (activations, normalization and attention) are in the form of f(x, ...) = x * g(x, ...). The authors observe that in the forward pass the non-linear gating function g can be replaced by a \"detached\" Jacobian of f, obtained by autodiff with cloning the inputs of g and not propagating gradient backwards (corresponding to using the Pytorch \"detach()\" method on the cloned tensors). This is in general not the true Jacobian of f, but according to the paper it provides a better approximation at the operating point compared to the true Jacobian, which only provides a first-order Taylor approximation.\n\nThe authors apply this method to multiple mechanistic interpretability analyses:\n- Analyse how the input and output embeddings linearly relate to each other at a specific input sequence, including a token relevance analysis\n- Analyse the approximate rank (computed by SVD) of the mapping, finding that it is very low.\n- They perform a \"conceptual steering\" experiment, where they mix the linearization of the model at one sequence with the computation of another sequence, observing semantic transfer between the two sequences.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Quantitative analyses over in-distribution and out-of-distribution datasets.\n- More concise and readable tables and graphs\n- Better theoretical justification for the detached Jacobian method\n- Remove unsupported claims about safety applications, or support these claims by experimental evidence", "broader_impact_concerns": "Broader Impact Statement is not present, and in my opinion not necessary for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)