Large Language Models are Locally Linear Mappings
Authors: name: James R. Golden
Published: 2025-05-30 07:08:33

Abstract: We demonstrate that the inference operations of several open-weight large
language models (LLMs) can be mapped to an exactly equivalent linear system for
an input sequence without modifying the model weights or altering output
predictions. Extending techniques from image diffusion models that exhibit
local or piecewise linearity, we strategically alter the gradient computation
with respect to a given input sequence for a next-token prediction such that
the Jacobian of the model nearly exactly reproduces the forward prediction with
a linear system. We demonstrate this approach across models (Llama 3, Gemma 3,
Qwen 3, Phi 4, Mistral Ministral and OLMo 2, up to Llama 3.3 70B Q4) and show
through the singular value decomposition of the detached Jacobian that these
LLMs operate in extremely low-dimensional subspaces where many of the largest
singular vectors decode to concepts related to the most-likely output token.
This approach also allows us to examine the operation of each successive layer
(and its attention and MLP components) as nearly-exact linear systems and
observe the emergence of semantic concepts. Despite their expressive power and
global nonlinearity, modern LLMs can be interpreted through nearly-exact
locally linear decompositions that provide insights into their internal
representations and reveal interpretable semantic structures in the next-token
prediction process.