I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?
Authors: Yuhang Liu
Dong Gong
Yichao Cai
Erdun Gao
Zhen Zhang
Biwei Huang
Mingming Gong
Anton van den Hengel
Javen Qinfeng Shi
Venue: ICLR 2026 Poster (2026)
Keywords: large language model representations
human-interpretable concept
latent variable model
Primary area: interpretability and explainable AI

Abstract: Recent empirical evidence shows that LLM representations encode human-interpretable concepts. Nevertheless, the mechanisms by which these representations emerge remain largely unexplored. To shed further light on this, we introduce a novel generative model that generates tokens on the basis of such concepts formulated as latent discrete variables. Under mild conditions, even when the mapping from the latent space to the observed space is non-invertible, we establish rigorous identifiability result: the representations learned by LLMs through next-token prediction can be approximately modeled as the logarithm of the posterior probabilities of these latent discrete concepts given input context, up to an invertible linear transformation. This theoretical finding: 1) provides evidence that LLMs capture essential underlying generative factors, 2) offers a unified and principled perspective for understanding the linear representation hypothesis, and 3) motivates a theoretically grounded approach for evaluating sparse autoencoders. Empirically, we validate our theoretical results through evaluations on both simulation data and the Pythia, Llama, and DeepSeek model families.

Published as a conference paper at ICLR 2026
I PREDICT THEREFORE I AM: Is NEXT TOKEN PRE- DICTION ENOUGH TO LEARN HUMAN-INTERPRETABLE CONCEPTS FROM DATA?
Yuhang Liu"2, Dong Gong', Yichao Cai², Erdun Gao Zhen Zhang Biwei Huang",
Mingming Gong Anton van den Hengel Javen Qinfeng Shi
'Responsible AI Research Centre, Australia
2 Australlaa Institute for Machine Learning, Adelaide University
3 Schooo of Computer Science and Engineering, The University New South Wales
*Halıcogluu Data Science Institute, University of California San Diego
School of Mathematics and Statisties, The University of Melbourne yuhang liu01@adelaide edu.au
Project: google http:///ewe////aaaaaaa/////////a
ABSTRACT
Recent empirical evidence shows that LLM representations encode human- interpretable concepts. Nevertheless, the mechanisms by which these representa- tions emerge remain largely unexplored. To shed further light on this, introduce a novel generative model that generates tokens on the basis of such concepts formu- lated latent discrete variables. Under mild conditions, even when the mapping from the latent space to the observed space is non-invertible, we establish rigorous identifiability result: the representations learned by LLMSs through next-token pre- diction can be approximately modeled as the logarithm of the posterior probabilities these latent discrete concepts given input context, up to an linear transforma- tion. This theoretical finding: 1) provides evidence that LLMs capture essential underlying generative factors, 2) offers a unified and principled perspective for understanding the linear representation hypothesis, and 3) motivates theoretically grounded approach for evaluating sparse autoencoders. Empirically, we validate theoretical results through evaluations both simulation data and the Pythia, Ilama, and DeepSeek model families.
INTRODUCTION
Large language models (LLMs) are trained on extensive datasets, primarily sourced from the Internet, enabling them excel in wide range of downstream tasks, such language translation, text summarization, and question answering (Zhao et al., 2024; Bommasani et al., 2021; ; Olah et al., 2020). Despite this success, their internal representations remain largely opaque, fact that has sparked series of efforts toward theoretical understanding and practical interpretability. A promising direction arises from recent empirical evidence showing that LLM representations (often called activations in the natural language processing community) encode latent concepts, such as sentiment (Turner et al., 2023) or writing style (Lyu al., 2023), that align with human- cittrrrnterpeet abstractions (Acerbi Stubbersfield, 2023; Manning et al., 2020; Sajjad et al., 2022; Sharkey et al., 2025). Uncovering the underlying reasons for this phenomenon, i.e., developing a principled framework that formally links learned representations to latent concepts, holds promise for advancing
understanding of LLMs.
Some recent works attempt to formulate the problem of linking LLM representations to latent concepts through latent variable models (Park et al., 2023; 2024; Rajendran et al., 2024) in which human- caman-interpretable concepts are formulated as latent variables. For example, Park et al. (2023) propose latent variable model that focuses solely binary latent concepts. The work of Park et al. (2024) extends this formalization to categorical concepts, but its focus on hierarchical relationships between concepts may not capture other possible structures in textual data. Rajendran et al. (2024)

---

Published as a conference paper at ICLR 2026
instead model both latent concepts and, in particular, observed text data, continuous variables which may deviate from the discrete sense of language. See Appendix A for additional related work.
p(x,)) y) = Σ ply (c)p((//cpppc
Bayes'
p X
Σ
Diveraity Condition, Condition,
rula
Coverage Condition.
discrete latent variables
Lodiupp Table
ciscrate variables
Traneformer
y discrate observed variable
identifibblity Result: Linear traneformetion
1)==)
= Allog p c ((c)
Interence Model, e.g. Liama 1(x)
A depends diversity condition.
expe ((,09)
quantthes the lineertty.
Define approximately invertbilit by: pCc= x,
ply
Σ, exp((,(x)(x,6)))
Concepts encoded matrix A.
Where e arg max yx))
Sporse autoencoders root the matrix A.
Neke-Token Prediction
The proposed Leten: Variable Modial
ddntifabillyy Analyas
An overview of the main contributions of this work. On the left, we illustrate the proposed latent variable model that represents concepts as latent variables c, which generates both the input x and output y within next-token prediction framework. Leveraging Bayes' rule and the certain conditions, we establish an identifiability result: LLM representations are related to linear transfor- mation of the logarithm of the posterior distribution of latent variables conditioned on input context, i.e. (x) A log p(c c- c,|x)]; + b o(e), where b is constant, and o(E) represents term that grows asymptotically smaller than e as e -
Figure 1:
We investigate here how L.MMs, despite relying solely on next-token prediction, can grasp human- interpretable concepts, through a new latent variable model without the limitations above.
We begin by introducing latent variable model, illustrated on the left of Figure 1. In this model, human-interpretable concepts are formulated as latent variables connected by arbitrary causal relation- ships, which in turn generate observed text data through an underlying generative process. Building on this model we develop theoretical framework to assess whether next-token prediction enables LLMs trained on such text data to recover the underlying latent variables. Through identifiability analysis, we show that, under mild conditions, L.M representations approximately correspond to linear transformation of the logarithm of the posterior distribution of latent variables conditioned on input context. We refer to this as the linear property of LLM representations.
Taking step further, we demonstrate that this linear property provides both theoretical and practical insights for linear phenomena observed in empirical studies. On the theoretical side, it offers unified perspective for understanding the various forms from the linear representation hypothesis, e.g., steering vector (Turner et al., 2023) and linear probing (Park al., 2023; Marks & Tegmark, 2023). On the practical side, the linear property motivates a principled evaluation strategy for sparse autoencoders (SAFs), which aim extract concepts in unsupervised manner (Huben et al., 2024; Bricken et al., 2023). Specifically, supervised concept extraction methods, e.g., linear probing, can serve as an upper bound evaluate how well SAE features recover the underlying concepts.
Contributions. To investigate the relationship between LLM representations and human- numan-interpretable concepts, this paper makes the following contributions. We introduce novel latent variable model in Sec. 2, in which numan--nterprettble concepts are formulated as latent variables. Building on this setup, we establish the existence of linear transformation between LLM representations and -umaaninterpreetable concepts in Sec. 3. In Sec. 4, we show how the linear property above provides theoretical support for the empirical linear representation hypothesis in L.MMs and establishes a unified perspective. In Sec. 5, we show how the linear property motivate principled approach for evaluating SAEs. Additionally, we offer an early exploration of a promising method: structured SAEs, a variant that incorporates structured sparsity, to improve performance under this evaluation. In Sec. 6, we present experiments on simulation data, and real data across the Pythia (Biderman al., 2023), Llama (Touvron et al., 2023a; Dubey al., 2024), and DeepSeck-RR R1 (Guo al., 2025) models families, with results consistent with our findings. We further present empirical results on the proposed evaluation approach, and the structured SAES.
2

---

Published as a conference paper at ICLR 2026
2 SETUP: LATENT VARIABLE MODEL FOR TEXT DATA GENERATION
We begin by a novel latent variable model designed to capture text data generation process, as depicted on the left in Figure 1. The proposed generative model can be expressed probabilistically as:
p(x. = Σ. p(xxe))((lc)p((c,
where we formulate human-interpretable concepts as latent variables c, and observed variables x and y represent text generated by the mapping g on c. We here distinguish observed variables into x and y, to align the input, i.e., context, and output token in next-token prediction framework, respectively.
Our goal is to consider realistic assumptions on the proposed latent variable model, ensuring the model's flexibility to closely approximate real text data. To this end, we do not impose any specific graph structure over latent variables. For example, over latent variables from the perspective of causal representation learning. Most importantly, we highlight the following two factors that distinguish our model from existing latent variable models (Rajendran et al., 2024; Yan et al., 2024; Jin et al., 2020; Jiang et al., 2024), underscoring its novelty.
allow for any directed acyclic graph structure
Discrete Modeling for Text Data Some existing latent variable models (Rajendran al., 2024; Yan et al., 2024) for text data predominantly assume that both latent and observed variables are continuous, primarily to simplify identifiability analysis. However, this assumption may overlook the largely discrete characteristics of text data. In contrast, we assume throughout this work that all variables, both latent variables and observed x and y, are discrete. This assumption can be justified for observed variables x and y, considering the inherently discrete structure of text data. Likewise, latent variables, which represent underlying concepts, can well-suited to discrete modeling, as they often correspond to categorical distinctions or finite sets of semantic properties commonly observed in text data. For instance, in a topic modeling scenario, latent variables can represent distinct topics, such "sports", "politics", or "technology", each of which forms a discrete category. This discrete structure aligns with the way humans typically interpret and classify information in text, making the assumption not only intuitive but also practical for real-world applications, as also assumed in prior work (Blei et al., 2003; Jin et al., 2020; Jiang et al., 2024).
No Invertibility Requirement Unlike existing approaches (Jiang et al., 2024; Rajendran et al., 2024), we do not require the mapping g from latent c to observed variables (x, y) to be invertible. Allowing non-invertibility a deliberate design choice driven by fact that the mapping from latent to observed space is often complex and unknown. Imposing constraints on it may unnecessarily limit its flexibility. Moreover, in the context of text data, there are two key considerations. First, in natural language processing, the mapping from latent observed space often involves many-to- one relationship. For example, different combinations of emotional concepts can lead to the same sentiment label. In sentiment analysis, the combination of "positive sentiment" and "excitement" might result in an observed outcome such as "This is amazing!" (Miller, 1995). Second, some latent concepts may not be explicitly manifested in the observed sentence. For instance, speaker's intent, tone, or implicit connotations may influence how sentence is constructed, yet remain unobservable in surface-level text. This phenomenon particularly prominent in pragmatics and discourse analysis, where unspoken contextual factors significantly shape meaning (Levinson, 1983). To formulate approximate invertibility, we define the mapping g from latent to observed space as follows:
Definition 2.1.
We define the degree of approximate invertibility of the mapping g from to (x, by introducing an error term as follows: 1 - p(c = c* xx y) = e, where 0 I e < 1, and = arg maxe p(c|x, y) represents the dominant mode of the posterior.
Remark 2.2. Such relaxed condition naturally implies that may not achieve exact identifiability results, as established in previous works, due to inevitable information loss in the context of non- invertible mappings. Nevertheless, this -on-irvertibility still allows for considering identifiability in an approximate sense, which will be discussed in Sec. 3.
3 THEORY: IDENTIFIABILITY OF THE PROPOSED LATENT VARIABLE MODEL
We now turn to the identifiability analysis of the proposed latent variable model, i.c., whether the latent variables can be uniquely recovered (up an equivalence class) from observed data and y
3

---

Published as a conference paper at ICLR 2026
under certain assumptions. We conduct this analysis within the next-token prediction framework, widely adopted and empirically significant paradigm for training LLMs (Brown et al., 2020; Touvron et al., 2023b; Bi et al., 2024; Zhao et al., 2023).
We begin by introducing the general form of next-token prediction, which serves as the foundation for analysis. The goal of next-token prediction is to learn model that predicts the conditional distribution over the next token p(y)x), which can be achieved by applying the softmax function over the logits produced by the model's final layer. This process mirrors multinomial logistic regression, where the model approximates the true conditional distribution p(y) x) by minimizing the cross- entropy loss. In theory, assuming sufficient training data, a sufficiently expressive architecture and proper optimization, the model's ' predictions will converge to the true conditional distribution. We emphasize that these assumptions are standard in the literature on identifiability analyses.
exp (fx(())T,
p(ulx) = E,, exp fx
(2)
where yj denotes specific value of the observed variable y. The function fx() maps input to representation space, and f, (v) corresponds to the model's final-layer weights, ie., look-up table.
On the other hand, the true ply x) can be derived from Eq. using Bayes' rule:
plyxx) Σ. plyle)p(c(xx)
By the expression for p(y x) both Eq. 2 and Eq. 3, we can align the right-hand sides of these two equations. Taking the logarithm of both sides yields:
fx) (v) log
exp(fx fx,))) = log E. p(ylc)p(c(x).
(4)
We now, as shown in Eq. 4, establish an initial connection between the LLM representations fx in the inference model (left-hand side) and the latent variables in the generative model (right-hand side). To further study this connection, we introduce the following diversity condition.
Notation Regarding Let = [ct .c.,,.... cm'] denote the latent variables, where ct Va with [V&| Ti, k = ,,...,7,, m, so that c can take = II denote all possible values of c.
possible values. Let C; i = I,...,f,
Diversity Condition We assume there exist e' + 1 values of where El denotes the representation dimension of LLMs, such that the following matrix invertible:
L [fy(w)) (y0), (y(2)) (y(),
(v() [10)] E "
This assumption was originally developed in nonlinear independent component analysis (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Khemakhem et al., 2020), and has been adopted in identifiability analysis for L.LMs (Marconato et al., 2024; Roeder al., 2021; Rajendran al., 2024). The invertibility of É indicates that there exists a sufficiently diverse set values such that the corresponding difference vectors (25) (yn) form a linearly independent set spanning the image of fy. This assumption generally mild: as noted by Roeder al. (2021), the probability that randomly initialized and stochastically updated parameters of f, produce linearly dependent difference vectors is effectively zero.
TV Condition we assume:
For any two tokens in you V1,...ye ye in the Diversity Condition above, e.g., Vs. yo in,
TV (plc v.c bo)) = 2 Σ p(c v.) p(c b0) =
log
where c is defined in Definition 2.1, TV denotes the total variation distance. In the context of the latent variable model in Eq. 1, where y, denotes a single token and c represents the full latent space, this condition essentially requires that the posterior distribution p(c y) changes slowly with respect to some token y. This seems intuitive. For example, a given token is typically generated by only a small subset of latent concepts, while each latent concept may correspond to many different tokens. Under such a common one-to-man structure, the posterior distributions for different tokens may not fluctuate dramatically.
4

---

Published as a conference paper at ICLR 2026
Coverage Condition. While the TV condition controls how much the posterior p(c ()) moves across anchor tokens, bounding the remainder additionally requires a mild regularity assumption on the conditional posteriors p(c y4) and p(c x. yo). Specifically, we assumee that there exists constant such that
sup p(c x, y:) pog x, x, S 8.
This condition states that, on the latent configurations that are relevant for generating (x, y:) and (x, vo), the conditional posteriors do not collapse and do not differ by arbitrarily large log-factors. Such a regularity assumption is intuitive: a given token Yi is typically generated by only small subset of latent factors, and the observed variable usually retains sufficient information about these factors. Hence the conditional posteriors p(c x, y/ and p(c x, yo) remain well-behaved and their invertible.
log-difference shrinks the generative map becomes
Theorem 3.1. Under the diversity condition above, the true latent variables related to LLM representations (x),, which are learned through the next-token prediction framework, by the following relationship:
(xx)= Allogp(c = c/(x);; + ( 'hy.
(5)
where
hy [bb
Nyy0 ... Ayee Nyy with hy; 1p(c c:ly [log cily yj x)]:
( byy 1/1) bly = y0), ...], with bly = yj) = P Plgg(y = ys |c)]
= ( LI with L E TR' Lig,- = p(c = ci l.1v yy= -リi) p(c = c; yo)
Moreover,
when 0, we have: (x) Allogp(c=c.ppp b,"
when e - and 5 - 0, under the TV condition, we have: fx(x) = +.
=
Remark 3.2. This theorem establishes precise relationship between the LLM representations fx( and the underlying latent concepts c. This not only provides theoretical grounding for understanding LLM representations, but also offers a unified prospective for the linear representation hypothesis (Sec. 4). Beyond this, it suggests a principled approach for evaluating SAES (Sec. 5).
THEORY + INSIGHTS: UNIFYING THE LINEAR HYPOTHESIS
In this section, we demonstrate how the identifiability result presented in Theorem 3.1 supports the linear representation hypothesis in LLMs. To this end, we first briefly introduce the linear representation hypothesis and then explain how it can be understood and unified through the linear matrix A from our identifiability result.
4.1 THE LINEAR REPRESENTATION HYPOTHESIS
The linear representation hypothesis suggests that human-interpretabbe concepts in LLMs are repre- sented linearly. This idea is supported by empirical evidence various forms, including:
Each concept is represented (i.e., vector offset) in representations of pairs that vary only in one latent concept of interest, e.g.. gender. For instance, Rep("men")-Rer ReeeReeppoooeee = Rep("king" "eeiin""""Re(("uueeeeeeeee (Mikolov al., 2013; Pennington et al., 2014; Turner et al., 2023).
Concepts Directions:
direction determined by the differences
Concept Manipulability: Previous studies have demonstrated that the value of a concept can be altered independently from others by introducing a corresponding steering vector (Li et al., 2024; Wang al., 2023; Turner al., 2023). For example, transitioning output from false truthful
Here, [] denotes a vector indexed by all latent configurations c,, and later we will also use + to denote a vector indexed all possible values of single concept
5

---

Published as a conference paper at ICLR 2026
answer can be accomplished by adding vector offset derived from counterfactual pairs that differ solely in the false/truthful concept.
Linear Probing: The value of a concept is often measured using linear probe. For instance, the probability that the output language is French is logit- in the representation of the input. this context, the linear weights can be interpreted as representing the concept of English/French (Park et al., 2023; Marks & Tegmark, 2023).
4.2 UNDERSTANDING THE LINEAR REPRESENTATION HYPOTHESIS
The linear representation hypothesis has received growing empirical support in recent years. While recent work has aimed to develop unified frameworks for a deeper understanding of this phenomenon (Park et al., 2023; Marconato et al., 2024; Jiang al., 2024), our approach seeks to explain it through the lens of identifiability. Before proceeding, we first provide the following definition:
Definition 4.1. We define the degree of approximate invertibility of the mapping from the latent variables с to the observed variables x by introducing an error term éx, as follows: 1 - p(c = <% x) = Ex. where 0 < Ex 1, and x arg max. p(c x) represents the dominant mode of x).
Next, we present two key corollaries derived from our identifiability results.
Corollary 4.2 (Concepts Are Encoded in the Matrix A). Suppose that Theorem 3.1 holds, i.e., (x( x [log p(c = C; x)], + b. Let Xy and x1 be a pair that differ only in the i-th concept c². Then as <x 0, (x1) - fx()) ~ A ([ogg(c* x1) log p(c xx)],.), where A = AB* where B (0, 1 is binary lifting matrix that broadcasts each entry of [log p(c x)]<+ to x)}:
the corresponding index in [log p(p
Understanding Concepts Directions The corollary explains why the representation difference Rep("man") (ep"""oman") can be closely approximated by Rep("king") - Repp""quent In both the ("man",, "wommn") and ("king". "queen") pairs, the primary distinguishing factor is the latent concept of gender, while other concepts such as royalty remain largely unchanged. As result, both Rep("man") Rep( Rep("woman") and Rep("king" - Rep("queen") are driven by the same expression A ([logp(( x1) - log p(c xxo) ).) the case a binary concept c², the expression is further reduced to vector direction corresponding the concept of interest. See Appendix G.1 for details.
Understanding Concept Manipulability The corollary also supports the notion that a con- cept's value can be adjusted by adding corresponding steering vector, such as Rep("man"). Rep("womann). Adding the steering vector effectively modifies the original representation to produce a new representation, i.e., fx = (x) + a
p(c |xi) - log xx)][²)
A [log p(c cx + aB* [log p(c x1) log p(ct x (n)) b. where represents an in- troduced weight (Wang et al., 2023; Turner et al., 2023). Thus, manipulating a concept via a steering vector is, in essence, equivalent to modifying the posterior distribution of the concept of interest given
A
x, which directly impacts the model's output.
To understand Linear Probing, we first introduce the following corollary:
Corollary 4.3 (Linear Classifiability of Representations). Suppose that Theorem 3.1 holds, i.e. fx()) = A plo C; x)); + b. Let xu and x1 be pair data that differ only in the i-th concept variable dt. Then, as ex - 0, the corresponding representations (fx (xn), fo (x(x)) )) are linearly separable along the variation of cx In particular, there exists linear classifier with weight matrix W such that W z I, and the corresponding logit [p(ck x)]ct.
This corollary supports that latent concepts, e.g., English vs. French, can be reliably classified using linear probing on the model's representations. This linear separability enables alignment berween the model's 's predictive distribution and the true class distribution, achieved via cross-entropy minimization. A specific analysis for the binary case of concept ct is provided in Appendix G.2.
4.3 UNIFYING THE LINEAR REPRESENTATION HYPOTHESIS
Taken together, Corollary 4.2 shows that the linear transformation A underlies both concepts as directions and concept manipulability, while Corollary 4.3 demonstrates that linear probing is

---

Published as a conference paper at ICLR 2026
supported by classifier W satisfying W ~ I. This establishes a unified view understanding the linear representation hypothesis: the various forms are all connected through the same underlying linear matrix A, providing unified theoretical perspective on how LLM representations linearly encode concepts. Importantly, according to Theorem 3.1, the matrix is defined as A (f.")-',, which, in essence, arises from the data diversity condition outlined in Diversity Condition.
5 THEORY + PRACTICE: EVALUATING SAES AND STRUCTURED SAES
Broadly speaking, SAEs are designed with two primary objectives. First, they aim to learn set of latent features z such that sparse linear combinations fz can accurately recon- struct LLM representations, i.e., (x) R Bz. Second, they seek to ensure that each learned feature 2, corresponds to monosemantic, cuuman-interpredd
Evaluating SAES
concept, thereby enabling mechanistic un- derstanding of LLMs. While reconstruction loss is commonly used to assess how well representations reconstructed (Rajamanoharan et al., 2024a;b; Gao et al., 2025; Braun et al., 2024), it is limited proxy for the second objective. A key challenge lies in the absence of ground truth for the underlying concepts (Kantamneni et al., 2025), making the evaluation of feature disentanglement challenging.
To address this issue, we propose a new evaluation method for SAES grounded in our theoretical insights. Specifically, based on Theorem 3.1, the LLM representations (x) can be approximated as (x) 2 A [logp(c = C, x)]; 2. Meanwhile, SAEs are trained to reconstruct f (x) by ,z.. Combining the two, we arrive at: Bz
plogpic c x)]; This suggests that SAE features z are linearly related to [log p(c = C; x)];- Therefore, if we expect each latent dimension to encode only single concept c²t, it should depend only on the posterior of that concept log p(c* x). Consequently, we can evaluate whether each 2; has successfully learned a monosemantic concept by measuring its linear correlation with p(c x). The question is: how can we obtain p(ck x)?
Based on Corollary 4.3, this can be achieved using paired data that differ only in the k-th concept variable et Specifically, we can construct paired data (Xo, that differ in only single binary concept ct, with labels ct = for X and c² = for X1. We then train a lincar classifier in a supervised manner the corresponding LLM representations (xo) and f& (x()), with the goal of predicting their labels. Once the classifier trained, the resulting logit provides a estimate of the posterior probability p(c = x). See Appendix I.2 for more rigorous details.
Structured SAFS Building on the evaluation method described above, our experiments (See Figure 4) suggest that sparsity regularization alone may may not be sufficient fully disentangle the underlying concepts in L.LM representations. Revisiting Theorem 1, the identifiability result depends on the proposed latent variable model. In this model, the complex dependencies in text data are encoded by the latent variables, whose interdependencies capture the underlying structure of the data. That is, latent variables are likely to exhibit strong interdependencies. Motivated by this, we propose exploring structured SAEs that incorporate additional regularization beyond sparsity to encourage learned features to model potential relationships among concepts.
In particular, we experiment with low-rank structures to complement sparsity, while noting that other forms of structured regularization could also be considered. Formally, structured SAEs minimize the following objective:
C= C [| - (x) A (ISIIP: +m|])]
(6)
where (x) denotes reconstruction, X is the dynamically adjusted sparsity coefficient at step i, following the p-annealing SAE strategy (Karvonen et al., 2024), 7 a hyperparameter that balances the sparsity penalty and the low-rank regularization, the learned features = S + R.
|S| pL E. PS is the adaptive Lp, norm promoting sparsity, ||R]|noe is the nuclear norm, used to enforce low-rank structure on R. See Appendix I.I for more implementation details.
6 EMPIRICAL EVALUATION ON SIMULATED AND REAL DATA WITH LLMS
We begin by conducting experiments on synthetic data, which is generated through the following process: First, we create random directed acyclic graphs (DAGs) with latent variables.
Simulation
For simplicity, we omit the constant b in this section, which does not affect the subsequent analysis.
7

---

Published as a conference paper at ICLR 2026
representing concepts. For each random DAG, the conditional probabilities of each variable given its parents are modeled using Bernoulli distributions, where the parameters are sampled uniformly from [0.2, 0.8]. To simulate nonlinear mixture process, we then convert the latent variable samples into one-hot format and randomly apply a permutation matrix to the one-hot encoding, generating one-hot observed samples. These are then transformed into binary observed samples. To simulate next-token prediction, we randomly mask part of the binary observed data, e.g.,. Tú, and predict it by use the remaining portion X Refer to Appendix H.I for more details.
Evaluation In Theorem 3.1, we demonstrate that the representations learned by next-token pre- diction approximate a linear transformation of log p(c]x), and this approximation becomes tighter when the mapping from to x is approximately invertible. Building on this, Corollary 4.3 establishes that for a data pair (c, x) differing only in the concept of interest, the representations fx fxx) is lin- early separable. Therefore, validate Theorem 3.1, we can assess the degree to which the learned representations can be classified linearly for data pairs (c.
Performance Across the Degree Invertibility
Performance Across Graphs and Size of Variables
X
Classification Accuracy
0.23
Classification Accurocy
0:599
0.90
0:25
0.43
0.80
6435
Dimension of Observed Varlables
Number of Latent Variables
Figure 2: Classification accuracy vs. number of observed variables (left) and graph structures (right).
Our initial experiments investigate the relationship between the degree of invertibility of the mapping from с to x and the approximation of the identifiability result in Theorem 3.1. This exploration provides empirical insights into how invertibility influences the recovery of latent variables. To this end, we fix the size the latent variables and gradually increase the size of the observed variables, thereby enhancing the degree of invertibility of the mapping from demonstrates that classification accuracy improves as the size of the observed variables x increases, aligning with results in Theorem 3.1.
The left of Figure
We then examine the impact of latent graph structures on our identifiability results. To this end, we randomly generate DAG structures imposed on c. Specifically, random Erdös-Rényi (ER) graphs (ERDAS & R&wi, 1959) are generated with varying numbers of expected edges. For instance, ERA denotes graphs with d nodes and kd expected edges. The right panel of Figure illustrates the relationship between classification accuracy and the size of the latent variables e under different settings, including ERI, ER2, and ER3. The results demonstrate that our identifiability findings hold consistently across various graph structures and latent variable sizes, as evidenced by the linear classification accuracy.
Experiments with LLMs We now present experiments on pre-trained LLMs. While is challeng- ing to collect all latent variables from real-world data to directly evaluate our identifiability result in Thorem 3.1, we can instead assess our corollaries indirectly validate it. For Corollary 4.2, prior studies have already demonstrated the linear representation properties of LLM embeddings using counterfactual pairs that differ only in single concept of interest (Mikolov et al., 2013; Pennington et al., 20:4; Turner et al., 2023; Li et al., 2024; Wang al., 2023; Park et al., 2023). Therefore, shift our focus to a new property highlighted in Corollary 4.3, specifically the relationship between W and Ä k which has not been explored in prior work. To investigate this, we require counterfactual pairs that differ only in single concept. Constructing such counterfactual sentences is highly non-trivial. even for human annotators, due to the complexities of semantics and the need for precise control contextual variations, as noted in prior studies (Park et al,, 2023; Jiang al., 2024).
8

---

Published as a conference paper at ICLR 2026
A. Pythia
W. on Pythia-1.4
x W, on Pythip 2.8p
A. x W. on Pythis 6
X. W. on Pythin-12b
A W. on Llama-2-70
Ax on LLama-2-13b
Ax LLama-2-700 2-70b
Figure 3: Results of the product A, x W across the LLaMA-2 2 and Pythia model families. Here, A, represents matrix derived from the feature differences of 27 counterfactual pairs, while W, is weight matrix obtained from a linear classifier trained on these features. The product approximates the identity matrix, supporting the theoretical findings in Corollary 4.3.
For our experiments, utilize 27 counterfactual pairs from Park et al. (2023) that differ only in binary concept, which are constructed based on the Big Analogy Test dataset (Gladkova et al., 2016). More details can be found in Sec. H.2 in Appendix.
Guided by Corollary 4.2, we use these 27 counterfactual pairs obtain A&. Specifically, for each pair differing in a binary concept, we compute the representation difference, and stacking all such vectors yields the matrix A, E 22 where each row corresponds to concept direction, and dim denotes the representation dimension of the pre-trained LLM used. See a binary special case of Corollary 4.2 in Appendix G.1 for further details. To obtain W, motivated by Corollary 4.3, we train a linear classifier for each binary concept using the representations of the counterfactual pairs. The resulting weight vectors stacked to form matrix W, Rdimx27 where each column corresponds to the decision boundary for one concept. See binary special case of Corollary 4.3 in Appendix G.2 for further details. Finally, we first normalize both A, and W. to remove the effect of scaling, which does not affect the semantics of direction or decision boundary, and then examine the product A, W., According to Corollary 4.3, the f-th entry of this product corresponds to the inner product between the representation difference vector for the i-th concept and the classifier weight vector for the j-th concept. When = j. this inner product should be close to 1, indicating that the classifier is aligned with the concept direction. When + j. the inner product should be less than 1, since the classifier for one concept should not be aligned with the concept direction of a different concept. Therefore, the matrix product A, W, should approximate the identity matrix I. Figure displays the results of this product across the LLaMA-2 and Pythia model families. Refer to Appendix K for more results LLaMA-3 and DeepSeek-R1. The results show that the product approximates the identity matrix, which is consistent with the theoretical finding in Corollary 4.3.
Experiments on SAES We finally conduct experiment SAES, training four sparse variants, including top-k SAE (Gao et al., 2025), batch-top-k SAE (Bussmann et al., 2024), p-annealing SAE (Karvonen al., 2024), and the proposed structured SAE. Following Theorem 3.1, each SAE trained representations from the final hidden layer across Pythia-70m, Pythia-1 1.4b and Pythia-2.8b (Biderman al., 2023), with The Pile corpus (Gao et al., 2020). For evaluation, as mentioned in Sec. 5, use 27 counterfactual pairs from (Park et al., 2023) again to train linear classifier using LogisticcRegressioon implementation from the scikit-learn library, which provides logits, i.c., unnormalized p(c* 1|x), for each of the 27 pairs. These counterfactual pairs also passed through the trained SAEs to extract features z. We search for the best- matching feature z; for each p(c*) x) according to Pearson correlation between exp(21) and p(c*\xx) to measure linear correlation.
Figure 4 reports the average Pearson correlations across the 27 counterfactual concepts (left, detailed results in Appendix !) and the reconstruction measured by mean squared emror (MSE, right). We

---

Published as a conference paper at ICLR 2026
Correlation across
Methods
Squared across
0,80
25
2320
0.75
Poarson Correlation
20
15
Stivitum
0.65
kiph
0.60
2320
4500
0.50
7yt
Pythio11440 1.40-.1.4.
Pythia
Pythia70mm
Pythib-1.44 4.
Pytria.2.80
Model
Model
Figure 4: Comparison of SAE models. Pearson correlation scores (left) and MSE on the used counterfactual pairs (right) across different scales of Pythia.
draw two key conclusions. First, the proposed evaluation framework shows that: it differentiates SAE variants with sensitivity and provides interpretable evidence of how the learned features align with binary concepts. The trends are consistent with conventional reconstruction metrics such as MSE, reinforcing the reliability of our approach. Second, the results highlight the advantage of the proposed structured SAEs, which benefits from incorporating structured regularization. This advantage is consistently observed under both our evaluation framework and standard reconstruction metrics.
We emphasize again that obtaining counterfactual pairs is challenging. Even this compact, high- precision benchmark, all four SAEs achieve Pearson correlations below 0.8 (1.0 indicates perfect recovery). Consequently, this benehmark is effeective in differentiating the performance of the four SAEs. We hope motivates the creation of more such high-quality counterfactual pairs.
CONCLUSION
In this work, we propose a latent variable model to capture the generative process of text, representing high-level, human- interpretable concepts latent variables. Under mild assumptions, our analysis provides a key insight into LLMs: linear property, whereby LLM representations approximate a linear transformation of the posterior latent variables. This establishes foundational framework for understanding next-token prediction. Furthermore, we show that this linear property offers unified theoretical perspective on various forms of the linear representation hypothesis, and motivates principled evaluation strategy for sparse autoencoders. These findings open avenues for deeper exploration of how LLMS learn and represent complex patterns. Building on our results, we suggest the following future direction: develop methods to linearly unmix LLM representations to directly extract the probabilities individual high-level concepts from the latent posterior. Further details are provided in Appendix L.
ACKNOWLEDGMENT
This project was partially funded by the Responsible AI Research Centre (Yuhang Liu, Zhen Zhang, Erdun Gao, Anton van den Hengel, and Javen Qinfeng Shi). Dong Gong was partially supported by the Australian Government through the Australian Research Council Discovery Early Career Researcher Award Australian Government through the Australian Research Council Discovery Projects (DP240102088). The authors also thank the anonymous reviewers for their constructive feedback.
Mingming Gong was partially supported by the
Ethics Statement. This study complies with the ethical standards of ICLR. relies exclusively on public datasets and does not pose foreseeable negative impacts.
Reproducibility Statement. We provide thorough descriptions of the methodology and experi- ments. The code and instructions will be openly available once the paper is published.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Alberto Acerbi and Joseph M Stubbersfield. Large language models show human-like content biases in transmission chain experiments. Proceedings of the National Academy of Sciences, 120(44): 1231379110,, 2023.
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing over- complete dictionaries for sparse representation. IEEE Transactions on signal processing. 54(11): 4311-4322, 2006.
Kartik Abuja,
Divyat Mahajan, Yixin Wang. and Yoshua Bengio. Interventional causal representation learning. In /CML, pp. 372-407. PMLR, 2023.
Sanjeev Arora, Rong Ge. Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. Conference on learning theory, pp. 113-149. PMLR, 2015.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang. Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma. and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495 2018.
Francis Bach, Rodolphe Jenatton, Julien Mairal, and Guillaume Obozinski. Structured sparsity through convex optimization. Statistical Science, 27(4):450 2012.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.022 0.25444 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
David M Blei, Andrew Y Ng. and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):933-1022, 2003.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jcannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint 10Xi::2008.07255: 2021.
Dan Braun, Jordan Taylor, Nicholas Goldowsky- Diil, and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning. Advances in Neural Information Processing Systems, 37:107286-107325 2024.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal representa- tion learning. arXiv preprint arXiv:2203. 16437, 2022.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume. Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/transffrmer-ciie wttp:cist.subb///2/////aaa/a wwtppp/02//ooooommmiiici features/index.hmm
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf. and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. arXiv preprint arXiv:2306.02233. 2023.

---

Published as a conference paper at ICLR 2026
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 202.
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410, 2024.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1337, 2011.
Venkat Chandrasekaran, Sujay Sanghavi, Pablo A Parrilo, and Alan S Willsky. Sparse and low-rank matrix decompositions. IFAC Proceedings Volumes, (4310):4433-498 2009.
Venkat Chandrasckaran, Sujay Sanghavi, Pablo A Parrilo, and Alan S Willsky, Rank-sparsity incoherence for matrix decomposition. SIAM Journal on Optimization, 21(2):572-500 2011.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788. 2024.
Bogdan Dumitrescu and Paul Irofti. Dictionary learning algorithms and applications. Springer, 2018.
Julian Eggert and Edgar Korner. Sparse coding and nmf. In 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No. 04CH37541), volume 4, pp. 2529-2533, IEEE, 2004.
Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing. Springer Science & Business Media, 2010.
Michael Elad and Alfred M Bruckstein. generalized uncertainty principle and sparse representation in pairs of bases. IEEE Transactions Information Theory, (000::555822661 2002.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, 7ac Hadfield-Dodss, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209. /0652, 2022.
P ERDdS and A R&wi. On random graphs i. Publ. math. debrecen, 6(290-297):18, 1959.
B Everett. An introduction to latent variable models. Springer Science & Business Media, 2013.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406. 04099, 2024.
Leo Gao, Tom Dupre Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2025. URL bttps::/openreviii net forum? d=tcs2t htt 9ZNKD.
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't. t. In Proceedings of the NAACL Student Research Workshop. pp. 8-15, 2016.
Yuqi Gu and David B Dunson. Bayesian pyramids: Identifiable multilayer discrete latent structure models for discrete data. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(2):399-426 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 2rXiv:2501. /2948 2025.
12

---

Published as a conference paper at ICLR 2026
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.0207,, 2023.
Wes Gumee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical learning with sparsity. Mono- graphs on statistics and applied probability, 2015.
Junzhou Huang, Tong Zhang, and Dimitris Metaxas. Learning with structured sparsity. In Proceedings of the 26th Annual International Conference Machine Learning, 2009.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Tweifth International Conference on Learning Representations, 2024.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. NeurlPS, 29, 2016.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859 868. PMLR, 2019.
Rodolphe Jenatton, Julien Mairal, Francis R Bach, and Guillaume R Obozinski. Proximal methods for sparse hicrarchical dictionary learning. In Proceedings of the 27th international conference on machine learning, 2010.
Yibo Jiang, Goutham Rajendran, Pradcep Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. arXiv preprint arXiv:2403.03867, 2024.
Shuning Jin, Sam Wiseman, Karl Stratos, and Karen Livescu. Discrete latent variable representations for low-resource text classification. arXiv preprint arXiv:2006.06226, 2020.
Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? case study in sparse probing. arXiv preprint arXiv:2502. 16681, 2025.
Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Riggs Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. Measuring progress in dictionary learning for language model interpretability with board game models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In AISTAS, pp. 2207-2211 PMLR, 2020.
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning latent causal graphs via mixture oracles. Advances in Neural Information Processing Systems, 34:18081 18181 2021.
Lingjing Kong, Guangyi Chen, Biwei Huang, Eric P Xing, Yuejie Chi, and Kun Zhang. Learning discrete concepts in latent hierarchical models. arXiv preprint arXiv:2406.005: 2024.
Stephen C Levinson. Pragmatics. Cambridge UP, 1983.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
intervention:
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifying weight- variant latent causal models. arXiv preprint ar08 14153, 2022.
13

---

Published as a conference paper at ICLR 2026
Yuhang Liu, Zhen Zhang, Dong Gong, Erdun Gao, Biwei Huang. Mingming Gong, Anton den Hengel, Kun Zhang, and Javen Qinfeng Shi. Towards identifiable latent additive noise models. arXiv preprint ar 4rXiv:2403.15711 2024a.
Yuhang Liu, Zhen Zhang. Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifiable latent polynomial causal models through the lens of change. In The Twelfth International Conference on Learning Representations, 2024b.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Latent covariate shift: Unlocking partial identifiability for multi-source domain adaptation. Transactions on Machine Learning Research, 2025.
Yuhang Liu, Zhen Zhang, Dong Gong, Erdun Gao, Biwei Huang, Mingming Gong, Anton den Hengel, Kun Zhang, and Javen Qinfeng Shi. Beyond DAGS: A latent partial causal model for multimodal learning. In The Fourteenth International Conference on Learning Representations, 2026.
Qing Lyu, Marianna Apidianaki, and Chris Callison-Burchh Representation of lexical stylistic features in language models' embedding space. arXiv preprint arXiv:2305.18657 2023.
Julien Mairal, Rodolphe Jenatton, Francis Bach, and Guillaume R Obozinski. Network flow algo- rithms for structured sparsity. In Advances in Neural Information Processing Systems, 2010.
Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):300463000 -0054, 2020.
Emanuele Marconato, Sébastien Lachapelle, Sebastian Weichwald, and Luigi Gresele. All none: Identifiable linear properties of next-token predictors in language modeling. arXiv preprint arXiv:2410.23501, 2024.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
Riccardo Massidda, Atticus Geiger, Thomas Ieard, and Davide Bacciu. Causal abstraction with soft interventions. In Conference on Causal Learning and Reasoning, pp. 68 877 PMLR, 2023.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013.
George A Miller. Wordnet: a Texical database for english. Communications of the ACM, 38(11): 39 41, 1995.
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.155:4 2022.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-super sequence models. arXiv preprint (xXiv:2309000441 2023.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 503):e00024-001 2020.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:231 1.03110.0088... 2023.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. arXiv preprint UrXiv:2406.01506 2024.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
14

---

Published as a conference paper at ICLR 2026
Senthooran Rajamanoharan, Arthur Commy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. arXiv preprint 2.Xtv:2404.161114 2024a.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 2401:24077.4435,, 2024b.
Goutham Rajendran, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradeep Kumar Ravikumar. From causal to concept-based representation learning. In NeuriPS, 2024.
Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning, pp. 9030-9039, PMLR, 2021.
Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Firoj Alam, Abdul Rafae Khan, and Jia Xu. Analyzing encoded concepts transformer language models. arXiv preprint arXiv:2206. 13289, 2022.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-634 2021.
Anna Seigal, Chandler Squires, and Caroline Uhler. Linear causal disentanglement via interventions. arXiv preprint arXiv:2211.16400 2022.
Lee Sharkey, Bital Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky- Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretabilityy arXiv preprint arXiv:2501. 2025.
Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Weakly supervised disentangled generative causal representation learning. The Journal of Machine Learning Research, 23(1):10994-110 1104, 2022.
Pablo Sprechmann, Alexander M Bronstein, and Guillermo Sapiro. Learning efficient sparse and low rank models. IEEE transactions pattern analysis and machine intelligence. 37(9):1821-188 2015.
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. arXiv preprini 23. 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922. 2023b.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization, arXiv e-prints, pp. Xixvv22308, 2023.
Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.082230 2023.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In NeurIPS, 2021.
Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Barein- boim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. Advances Neural Information Processing Systems, 36, 2023.
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for based conditional model. In /CML 2023 Workshop on Structured Probabilistic Inference ({&} Generative Modeling, 2023.
15

---

Published as a conference paper at ICLR 2026
Hanqi Yan, Lingjing Kong, Lin Gui, Yuejie Chi, Eric Xing, Yulan He, and Kun Zhang. Counterfactual generation with identifiability guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables, Journal of the Royal Statistical Society Series B: Statistical Methodology. 2006.
Tianren Zhang, Guanyu Chen, and Feng Chen. When do neural networks learn world models? In ICML, 2025.
Haiyan Zhao, Hanjic Chen, Fan Yang. Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang. Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2):1-38, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
Jieyu Zheng and Markus Meister. The unbearable slowness of being: Why do we live at 10 bits/s? Neuron, 2024.
Zihan Zhou, Xiaodong Li, John Wright, Emmanuel Candes, and Yi Ma. Stable principal component pursuit. In 2010 IEEE international sympostum information theory, pp. 1518-1522. IEEE, 2010.
16

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
A Related Work
18
B Limitations
19
C Further Justification for the Assumptions Used
19
D Proof Theorem 3.1
20
E Proof Corollary 4.2
23
F Proof of Corollary 4.3
25
G Extension of Corollaries 4.2 and 4.3 for a Binary Concept
26
G.1 Extension of Corollary 4.2 for Binary Concept
26
G.2 Extension of Corollary 4.3 for Binary Concept
27
Experimental Details Supporting Theoretical Results
28
H.1 Simulation Details
28
H.2 Experimental Details on LLMs
28
Experiment on Sparse Autoencoders
29
1.1
Implementation the Proposed Structured SAE
................................
29
1.2 Details of Evaluation Metric
................................
30
I.3
Experiments and Results..
31
J Further Discussion: Observations LLMs and World Models
33
J.1 LLMs Mimic the Human World Model, Not the World Itself
33
J.2 LLMs Versus Pure Vision Models: A Fundamental Difference
34
K More Results on Llama-3 and DeepSeek-RI
35
L Future Directions
36
M Acknowledgment of LLMS Usage
36
N and Null Baseline Analysis
36
Assessing Robustness the Linear Result in Theorem 3.1
42
Sensitivity SAE evaluation linear probe quality
Empirical Verification of the TV and Approximate Invertibility
44
R Additional Counterfactual Pair Analysis on Pythia Models
44
17

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Linearity of Representation in LLMs Recent studies have established the empirical finding that concepts in LLMs are often linearly encoded, a phenomenon known as the linear representation hypothesis (Mikolov et al., 2013: Pennington et al., 2014; Arora al., 2016; Elhage et al.. 2022; Burns et al., 2022; Tigges et al., 2023; Nanda et al., 2023; Moschella et al., 2022; Park et al., 2023; Li et al., 2024; Gurnee et al., 2023; Rajendran et al., 2024). Building on this observation, recent works (Park et al., 2023; Marconato et al., 2024) attempt to unity these findings into cohesive framework, aiming to deepen our understanding and potentially inspire new insights. However, these works do not address why and how such linear properties emerge. Some previous works (Marconato et al., 2024; Roeder et al., 2021) have demonstrated identifiability results within the inference space, such as establishing connections between features derived from different inference models. However, these results are confined to the inference space and do not connect to the true latent variables in latent variable models. A recent work (Jiang al., 2024) seeks explain the origins of these linear properties but employs latent variable model that differs from ours. From a technical perspective, their explanation is rooted in the implicit bias of gradient descent. In contrast, our work provides an explanation grounded in identifiability theory. This distinction highlights our focus on connecting the observed linear properties directly to the identifiability of true latent variables in latent variable models, offering more comprehensive and theoretically robust understanding of these phenomena. In addition, recent work (Rajendran et al., 2024) assumes continuous latent and observed variables, while we assume both latent variables and observed variables to be discrete, aligning more closely with modeling natural language.
Causal Representation Learning This work is closely related to causal representation learning (Schölkopf et al., 2021), which aims to identify high-level latent causal variables from low-level observational data. Many prior studies (Brehmer al., 2022; Von Kügelgen et al., 2021; Massidda et al., 2023; von Kügelgen et al., 2023; Ahuja et al., 2023; Seigal et al., 2022; Shen et al., 2022; Liu et al., 2022; Buchholz et al., 2023; Varici et al., 2023; Liu et al., 2024b; 2025; 2024a; 2026; Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Khemakhem et al., 2020) have developed theoretical frameworks supporting the recovery of true latent variables up to simple transformations. However, these works primarily focus on continuous spaces and do not address the next-token prediction framework employed by LLMs. A subset of works (Gu & Dunson, 2023; Kong et al., 2024; Kivva et al., 202 has explored causal representation learning in discrete spaces for both latent and observed variables, often imposing specific graph structures and assuming invertible mappings from latent to observed spaces. Despite their focus discrete settings, none of these studies consider the next-token prediction framework in LLMs. In contrast, our work overcomes these limitations. We analyze approximate identifiability without relying on strict invertibility assumptions, which better aligns with the complex and often non- invertible relationships observed in real-world data. While a recent study has examined non non-invertible mappings from latent to observed spaces, they rely on additional historical information to effectively restore invertibility. In our approach, we focus on achieving approximate identifiability without requiring such additional information. A very recent work (Rajendran et al., 2024) explores identifiability analysis for LLMs; however, they model both observed text data and latent variables as continuous and still assume invertibility in their analysis. In contrast, our framework explicitly considers discrete latent and observed variables, relaxed invertibility assumptions, and directly aligns with the next-token prediction paradigm, offering more realistic and generalizable approach to causal representation learning in LLMs. Our work also differs from Zhang et al. (2025), who study identifiability through multi-task learning for the latent variables in the latent variable model of Everett (2013), where the mapping from latent to observation is assumed to be invertible. In contrast, we focus specifically on the next-token prediction framework used in LLMs, and analyze a different latent variable model that explicitly allows the mapping from latent to observed space to be non-invertible.
Sparse Autoencoders Polysemanticity, a phenomenon observed in recent studies, roughly speaking, refers to cases where single representation encodes multiple distinct, human-interpretate concepts (Arora et al., 2018). Early investigations suggested that neural networks represent features by linear superposition and motivated efforts to disentangle human- iammaninterpretablle concepts from such linear mixing (Elhage et al., 2022). This can be achieved by using sparse autoencoders (Huben et al., 2024; Rajamanoharan et al., 2024a;b; Gao et al., 2025; Braun et al., 2024; Bricken et al., 2023), technique
18

---

Published as a conference paper at ICLR 2026
closely related to the well-known framework of dictionary learning (Dumitrescu & Irofti, 2018; Eggert & Korner, 2004; Elad, 2010; Elad & Bruckstein, 2002; Aharon et al., 2006; Arora et al., 2015). In contrast to these works, propose structured SAEs to model the dependencies among latent concepts. Furthermore, motivated by our theoretical findings, we introduce new evaluation method for SAEs, grounded in our justified theoretical results.
Structured Sparsity Structured sparsity extends the classical sparse modeling paradigm by enforc- ing correlations or constraints among groups of features, rather than treating all entries independently. Early formulations include Group lasso (Yuan & Lin, 2006; Bach et al., 2012; Hastie al., 2015), graph-structured sparsity (Huang et al., 2009), and hierarchical or tree-structured sparsity (Jenatton al., 2010; Mairal et al., 2010), which allow prior knowledge about the data structure to guide feature selection. Moreover, low-rank plus structured-spant dimensional subspaces with structured sparse deviations, capturing both global and concept-speee components (Chandrasekaran et al., 2009; Sprechmann et al., 2015). Classical optimization- based approaches use nuclear-norm regularization for the low-rank component and structured norms for the sparse component (Candès et al., 2011; Chandrasckaran al., 201 I; Zhou et al., 2010). The pro- posed Structured SAEs bring srructuredspaassee ideas into LLMs by integrating structured-sparye constraints directly into autoencoder architectures. In doing so, they build bridge between classical structured-pparitt methods and modern neural representations in large language models. We do not attempt enumerate all existing work on structured sparsity, but expected that exploring possible grructured-ppassin techniques can further benefit LLMs.
decompositions further combine shared low-
B LIMITATIONS
One of the main contributions of this work is establishing an identifiability result for the next-token prediction framework, a widely used approach for training LLMs. This result hinges on a key assumption:Intuitivlyy this requires that the posterior distribution p(c y) changes slowly across tokens. While given token is typically generated by only small subset of latent concepts, and each latent concept may correspond to multiple tokens, rigorously verifying this condition in practice is challenging. On the other hand, this TV condition primarily serves supplementary requirement to control the influence of non-dominant points arising from the non-invertibility of the mapping from latent to observed space. It is possible that more general conditions could be formulated to replace or relax this assumption, which remains an interesting direction for future work.
Similar to existing works (Park et al., 2023; Jiang et al., 2024; Marconato et al., 2024; Rajendran et al., 2024), our analysis is limited to the last layer in LLMs and does not provide a justification for intermediate layers. Formalizing the relationship between representations in intermediate layers and identifiability analysis presents additional challenges. This complexity arises from the intricate nature of intermediate layer representations. Therefore, extending identifiability results to intermediate layers remains an open and more challenging problem.
Moreover, prior empirical studies have shown that linearity and concept representations often emerge in intermediate layers (Nanda et al., 2023; Gurnee & Tegmark, 2023). In some cases, these intermediate-layer linear representations may even diminish or transform by the time they reach the output layer (Marks & Tegmark, 2023). Consequently, while our theoretical results rigorously address identifiability the last layer, they do not capture potential linear structures present in intermediate layers. This limitation is particularly relevant for applications such as Structured SAES, which are sometimes trained on intermediate-layer activations rather than the final output (Gao et al., 2024). Addressing identifiability and linear representation formation across all layers remains an important direction for future work.
C FURTHER JUSTIFICATION FOR THE ASSUMPTIONS USED
We emphasize that, while the diversity condition is primarily theoretical assumption ensuring identifiability, its plausibility can be assessed empirically using pre-trained LLMs on real data. Directly observing all latent variables c is infeasible, but the corollaries derived from our theory, including Corollary 4.2 and Corollary 4.3, provide predictions that can be indirectly validated. Prior work has shown that LLM embeddings exhibit linear representation properties when comparing
19

---

Published as a conference paper at ICLR 2026
counterfactual pairs differing in single concept (Mikolov et al., 2013; Pennington et al., 2014; Turne et al., 2023; Li et al., 2024; Wang et al., 2023; Park et al., 2023), supporting the theoretical prediction of sufficient diversity in latent directions (Corollary 4.2). In addition, experiments show that the alignment between representation differences and classifier weights across multiple LLM families (Figures 3 and closely matches the predictions of Corollary 4.3, suggesting that the diversity condition might plausibly hold in practice.
D PROOF OF THEOREM 3.1
Proof. Next-token prediction mirrors multinomial logistic regression, where the model approximates the true conditional distribution, i.e., p(y(x), /x). by minimizing the cross-entropy loss. In this framework, given sufficient training data, a sufficiently expressive architecture, and effective optimization, the model's predictions expected converge the true conditional distribution, as follows:
f p(uxx) Ey (x
On the other hand, for the proposed latent variable model described in Eq. 1), the true conditional distribution p(y x) can be derived using Bayes' rule:
plyx)) Σ. p(yle)p(c(x))
(8)
By comparing Eq. and Eq. 7, we arrive at the following relationship:
(x)(fx(x"',,
E. p(ule)(((xx)
(9)
Eg of expl (x) fyy()))
Taking the logarithm on both sides of Eq. we obtain:
Px (x() T f. (v) - log (xx) log P(Pe))))cch)
(10)
where -(x)= Ey, exp (x) (3;)) represents the normalization constant. This transformation
provides a direct link between the representations f(x) learned by next-token prediction (Eq. 7) and the true posterior distribution p(c]x).
Now, let focus on the right of Eq. 10. we can obtain that:
log Σ plulc)p(c(x) = log p(y]x),
(11)
Ep(clu) [log((])))]
(12)
Ep(clv) llog (píclu. x) plyl x)
p(cly, x) )]
(13)
Ep(el)) logg(us c|x)] Eprciv) (logo(llu.veee
(14)
Ep(cls) [log)(c())] + Ep(cl)) log ((yle, Epcele) log p(cpycy
x)]
p(vic)
(15)
where p(yl c, x) = p(ylc), due to the conditional independence of and x given e, as defined in the proposed latent variable model.
Together with Eq. 15 and Eq. 10, we have: fx ff log ((x) = Epte [log((c/x]] Ep(ely) [logr(wlos Ep(ely) [p (ogg x)].
(16)
Define Eptely) Pogp(y/c)] by. and define vector [p(c c,(x)], as the vector constructed by the probabilities of all possible values of c, conditional As result, Ep(cly) log p(c)x)
20

---

Published as a conference paper at ICLR 2026
Ee ((cly) Ip(c c,ly)]? [log cx(x)];, and similarly Bptcls) log logp(ccy,,0
(p(c c,lu)l; pog c,(y,x)]; Then can re-write Eq. 16 follows:
fx(x)" (v()) log (xx =lpce c [log c,(x)],
(17)
- [[p(c c,ly)l [log [lg c, (.gg,)): + by
(18)
Let yo....., ye be the points provided by the diversity condition outlined in Section 3, for 0, we have:
fxx) - log (xx =lp(c - cily - yo)l? logp(c - c,|x)];
wp(c c.,l mo)l? [logp(c caly yo, x)]: tbya
(19)
Avo
where we define hwo obtain:
1p(c c;ly [logH c:ly yo, yo. x)]i For y 1, we similarly
(x() - log (xx =lplc= -:ily - [ogplc/c=p) - -hy + by.
(20)
Subtracting Eq. 19 from Eq. 20, we get the following expression:
(fr( (v0)
(xx))) (x() -(p(c= = c:ly -11) - p(c cily = vo)l? [lgr(c ci/x)];
- (bby hyy)) + by: byo
(21)
According the diversity condition, where y take l' values, obtain total of e equations similar to Eq. 21. Collecting all of these equations, we have:
(v)( (yO). (y(e) - (vo)) T fx()
(22)
ir
([v(c c:ly = v1) p(c = cily vo)li, ..., [p(c = cily = yr) - p(c coly = 3n)]aa)
[log
c; cx(x)];
[hy
Nyo
hgo
+
(byr
byo .... by
byol
(23)
by
by
According to the diversity condition, the matrix L of size E' x l' is invertible, as result, we arrive:
(x) (L 'L log p(c c/(x)]; ) - -
(24)
Now, we focus on the term by on the right-hand side of Eq. 24. Note that by. Epte is constant with respect to c, as the expectation integrates over all possible values c. As a result, the entire term - denoted as b, is also constant.
We next examine the term hy in Eq. 24, considering the cases where the mapping from the latent space to the observed space is exactly invertible and where it only approximately invertible.
Invertible According definition 2.1, when the mapping g from latent space to observed space is invertible, meaning that for
- p(c c* v) e, we have = Then, for by, we analyze each component, i.c., fy, hy. where
(25)
hy. LE Ep(c/==g [logp(c x. v:)l, hyo (p((el==oo) [logp(c x, x=y= vo)]
(26)
21

---

Published as a conference paper at ICLR 2026
When 0, the posterior distribution p(c x, y) becomes delta distribution centered at c², i.e.
p(c x, y) 5(c c²), which implies that the posterior is concentrated at a single point c², satisfying logp(c" xx y) = 0. In this case, we have
(27)
hyg
Ago 0.
(28)
Approximately invertible. Assume the generative map g c c (x,y) is approximately invertible so that, for observed pair (x,y). y). the posterior concentrates on a single mode c* arg maxe p(c v) with
:: p(c"
xx
O.
(29)
Let
(o(c) = ppc yo),
ro(c): = p(c x, yo);
(30)
and consider the remainder
Ah :h hy,
hyo = Σ [icc - Σ logroc)
(31)
Using the decomposition
Ah Σ (9.(c) 40 (c)) logr.(c)+ + E (o(()) log (c))),
(32)
we bound each term under the approximate-invertiinne assumption.
For the first term, split the sum at c*:
Σ (x(()) qu(c)) log (c) =(q((c)) 9o(c')) logri(c")
(33)
+
Σ (q((c) (o(c))))((c))))))) log (c))
(34)
cxee
By approximate concentration, ri(c*) x 1 - and for & c* we have r:(c) S so rog")(c") and log rc(c) loge e for e # ct. Hence
+ 0
Σ (a:(c) (o(c)) log (cc) - Σ
(c((c) (o(c)] + (a(cc")-o00) 90(c')l) llogl.
(35)
oxte
The prefactor is exactly 2TV p() 80)). so
Σ (q((c) - qu(c)) logri(c) $ 2TV(p(c p(c. 3o)) llg
(36)
For the second term,
Σ (log(()) - logro(c)),
(37)
By the Coverage Condition, we have:
(logr;(c) logra(c)) s 8,
(38)
thus,
Σ -oo))()og(() 8
(39)
Combining the two parts, we obtain
A S (p(c v.).p(c vo)) loog cl + 8.
(40)
Thus, if the TV condition
TV (p(c (iNell),pe u.),((e 30)) of 1
(41)
logge
holds and + 0, then
A + 0 as e -
Therefore, under approximate invertibility together with the TV and mild coverage conditions above, we have hy, - hyo - 0, completing the proof for the approximately-inveriiin case.
22

---

Published as a conference paper at ICLR 2026
E PROOF OF COROLLARY 4.2
Proof. We first prove that: when 6x + 0, i.c., p(c x) becomes sharply peaked at cx we can approximate:
p(c) se p(c xx) p(c k x))
(42)
where e denotes all concepts exeept ott To this end, we analysis their KI. (Kullback-Leibler) residual term:
Residual DRL. (p(c x)
p(c* x)p((c |x))
(43)
Since when 6x - 0, p(c x) is sharply peaked at particular configuration c; = (cht , cocke) ie..: if c otherwise,
plc x) = [1 -1 6x (xx r(c),
(44)
where Eote; r(c) = 1.
Main Term in KI.
The KI. divergence is given by:
plc x) p(ct x)p((=kk] |x)
DRL (p(c x)]lp(c* x))p(c= xx) Σ p(c x)
(45)
The main contribution comes from c c
Ex (x) log p(cko x)) plc kn xx)
(46)
Note that:
p(cko x) =
xx)= = p(ct+ c x) 2 p(c c ko x) = 6x:
(47)
e-k
and similarly:
plc ko x)) 2 1 - 6x.
(48)
Hence:
x)) :ppppce p(cc-he
xx) 2 (1 - Ex)? (x=ex)?
(49)
1 -
(50)
plcke
x) ppcc
kn x) 1 Cx
Thus, the main term becomes:
1- Ex
(1 (x-6x) log p(ck=
51 - (x) (x)]lo log
- (1 ex) log(1 ex). (51)
x) p(c k+
x)
1- 6x
Tail Term
For c + or the contribution to the KL divergence is:
(x((c)
1 r(c) )
log
(((()))
(xr(c)log
6x x(ct)r(c-x))
(52)
Summing over all c # ct we obtain:
) -xx log Σ r(c)+ +ex Σ ((c))log
r(c)
r(c)
Σ r(c) log cyte
((ct)r(c-b))
(((t)r((--))
cfe
(53)
Since the term r(c) is bounded, we conclude:
Tail Term o(Ex log (x)
(54)
which vanishes faster than the main term as 6x 0.
23

---

Published as a conference paper at ICLR 2026
Final Bound Combining both contributions in Main Term and Tail Term, get:
DKL (p(c x) p(c x) p(c xx) s -(1 (x) log(1 6x) + o(Ex 6x). (55)
Here when 6x - DKL (p(c x) p(c xx) p(c x)) - O.
Now that we have shown that Eq. 42 holds, we can rewrite the term [log p(c; x)], as:
[logp(c; x)], ~ B* p(e x)] + +B [logp(c**
xx)_-*x.
(56)
Here, B* and B-k are binary broadcasting matrices that expand the marginal log-probability vectors [ogg(ec (cf x)] and [logr(e=* x)]c== to the full configuration space llgg(es x)]i, respec- tively.
As result, for pair xo and xı that differ only in the i-th concept variable c², their difference in posterior distribution is:
[log p(g x1) - logp(e; xo)];
B [lg x1) logp(c" ( xx)] ct + B +B*
[logp(c x1) logp(c [xx0] xxo]e-t- (57)
We now show that:
p(c=k k x1) p(c xo) +, as 6x +
(58)
so the term B- [logp(c
p(c - x1) - lngg-oop(ee logp(cc-
(x0)le= in Eq. 57 vanishes.
Recall Eq. 44, we have
p(c x) = 1 -1- 11-xx Exi
if (c , c
(59)
6x r((),
otherwise,
Then, for c have:
plc x)= = Σ c Ix).
(60)
Combining this with Eq. 59, have:
p(c xx) = - -xx 6x
Σ ot clippate
= cx + o((x))
(61)
p(c k x)) =x)=xx =
Ex -() o(Ex), for c cht
(62)
Both Eq. 61 and Eq. 61 only depends on e. As result, if xu and x1 have the same values on the components relevant to -k the difference between p(c x1 and p(c-k xo) is of order o(ex), as:
p(c i 1
x1) = =pc | xo) - o(Ex),
(63)
which implies:
logp(c x1) - -logp(c xo) = log (1+ olex)
= =(((x)-
(64)
p(c |xo)
Consequently,
B (lo x1) - logp(c-* xx0) = o(Ex) + 6x + 0.
(65)
Together with Eq. 57 and the result from Theorem 3.1, we get:
(x1) f (xn) x AB* (logg(c" [x1]]a= [logp(c* xxo)]a).
(66)
24

---

Published as a conference paper at ICLR 2026
F PROOF OF COROLLARY 4.3
Proof. Again, when 6x - 0, i.e., when p(c x) becomes sharply peaked at cx. we can approximate:
p(c x) ss p(c"
x) p(cc" -k |x);
(67)
Given the above, neglecting the constant term in the result in Theorem 3.1,
& (x) Ri A [logp(c = c, x)];
(68)
which can be rewritten
(x) A |logp(c x)], s A(B [logp(c [x)] + +B* [logp(c-* |x)]
(69)
In this case, for data pair (Xn, X1) that differ only in the latent variable c², the representations (x( passed to linear classifier with weights W.
The classifier produces the logits:
logits x W (A [logp(c* x)] + +B --k [logp(c>* x))c
(70)
For correct classification under cross entropy loss, the logits must match the true probabilities:
logits = [p(c* x)],ar
(71)
where we omit constant scaling factors for simplicity, corresponding to the normalization applied prior to the softmax operation in the cross-entropy loss.
Combining Eq. and Eq. 71, the weight matrix W must satisfy the condition:
W(AB*) ~ I,
(72)
which ensures that the classifier produces the correct logits. Here denotes the identify matrix.
25

---

Published as a conference paper at ICLR 2026
G EXTENSION OF COROLLARIES 4.2 AND 4.3 FOR A BINARY CONCEPT
When considering pair that differ only in concept of interest, i.e., c² and assuming that the concept binary, both Corollaries 4.2 and 4.3 can be further refined, yielding the following results.
G.1 EXTENSION OF COROLLARY 4.2 FOR A BINARY CONCEPT
Corollary G.1 (Binary Concept Direction). Suppose that Theorem 3.1 holds, and let e be binary concept variable, i.e., ch {0, 13. Xe and x1 be a pair of inputs that differ only in the i-th binary concept ct, with ch = for Xa and et for X1. Then, as Ex - 0, the representation difference simplifies as:
(x1) - (xx) ~ A ([logp(c* x11) - logp(c xo)] = log =0 x1) - -
[-1
(73)
1
or, ~ logp(c = xo) A
[11
(74)
where A = AB*, B is a binary lifting matrix that broadcasts each entry of [logp(c* |x)]et to the corresponlingg index [logpic = c; x)]; This shows that changes in a binary concept are encoded in a specific direction in the representation space defined by A#.
Proof. Recall Eq. 44, the joint concept distribution conditioned on input follows peaked structure:
Gx.
cf
p(c x)
(75)
6x r(c), otherwise,
where cx = (c kk)) is the dominant concept configuration for x, and r(c) is a normalized residual distribution over all non-dominant
Let Xo and x1 differ only in the i-th binary concept variable et, with:
(0,c ex = kt,c
(76)
Then the marginal probabilities for c are:
For X1:
plc =1 x1) = -xxi) +xx
Σ
r(1,cT)) = - 6xi .01
(77)
c-
where 01 := Eo- (
plc* = 01xx)) = =1 -pp(c = x1) Cxi -a1
(78)
For Xy:
p(c =0 =0 xo) (1 (1xx + txo
Σ
r(0,c Ex, 000
(79)
where 0g 1= Ee kefe to r(0, ck)).
plc =1 =1
xo) = 1 -p(ct =1 xx) = 6xo 000
(80)
Taking logarithmic, we have:
log p(c x1 ) = log(1 - fxi 01) Rd 0,
as 6x +
(81)
log p(c* = x1 ) = log((x; 01).
(82)
log p(c* 0 xx) = llog(1 - (x 000) ~ 0, as 6x +0 0
(83)
log p(c xo) log((xo av) ~ log(Gx) a1) log p =0 - 6x 0.
(84)
26

---

Published as a conference paper at ICLR 2026
Then the vector difference:
logplc k 0 | x1) logp(ck = 0 x0)
[log p(c x1) log p(c²) xu)l
(85)
logple* = 1 X1) - log x()
log p(c = x1)
(86)
log plct = 1 xxo
1
log loe =0 x1)
(87)
Finally, the representation difference becomes:
(x)) - (xo) = A ([logp(e* x1) logp(c xx)] ar) = log p(c =0
-0|x, x/)
A
--1
(88)
Here, note that: log p(c* = 0 x1) logp(c 1 Xo) as shown in Eq. 84.
G.2 EXTENSION OF COROLLARY 4.3 FOR A BINARY CONCEPT
Corollary G.2 (Binary Concept Classification). Suppose that Theorem 3.1 holds, i.e., fx(x) ss [logp(c- - c; x)]; + b. Let Xn and X1 be pair data that differ only in the i-th binary concept variable ct, with labels ctt where ch 0 for Xo and e 1 for Xt. Then when 6x + 0, the corresponding representations (f(xo), f(x1)) are linearly separable with weight vector satisfying
w A
= 1. The corresponding logit is the (unnormalized) p(c² =1|x)
Proof. When éx + O, the posterior p(c x) becomes sharply peaked at a unique mode cx. This implies iadd-eddeeeneece of c and given x, allowing us write:
p(c x) Rs p(c*
x) ppcc x).
(89)
Taking logs and substituting into Theorem 3.1 (neglecting the bias term b), we obtain:
fx (x)
[logp(c; x)), A [loglo x)]e+ +B B "loglo x)]e x),
(90)
where B and B denote the lifting operators that map the marginal log-probabilities of ct and c into the joint log-probability vector space.
Define A = AB* Then:
(x() A llog((e x)]c + (terms involving only e *))
(91)
Consider linear classifier with weight vector w applied fx(x):
p(c* x
logit = w. fx) ~ w
1 - p(pcc =0 x)
+ const.
(92)
Let w A = [so, $1]. Then:
logit 8o loc 0 x) 81 log = x) const (81 -8)) log p(c 1|
|x) const.
(93)
For the classifier to correctly separate Xo and x under cross-entropy loss, we require:
logit log |
(94)
where we omit the normalization constant prior to the softmax, since does not affect the analysis. This is equivalent to:
w A *
R 1,
(95)
This completes the proof.
27

---

Published as a conference paper at ICLR 2026
H EXPERIMENTAL DETAILS SUPPORTING THEORETICAL RESULTS
H.1 SIMULATION DETAILS
For the left side of Figure 2, which investigates the relationship between the degree of invertibility in the mapping from exclude other uncertain factors that might affect the result. To achieve this, we keep the number of latent variables constant (i.e., 3) and ensure that the graph structure follows chain structure. Based on this structure, we model the conditional probabílities of each variable, given its parents, using Bernoulli distributions. The parameters of these distributions are uniformly sampled from the interval [0.2, 0.8], which are then used to generate the latent variables. Subsequently, we apply a one-hot encoding to these samples to obtain one-hot formal representations. These one-hot samples are then randomly permuted. For 3 latent variables, there are 2 possible permutations, each corresponding to 3 observed binary variables, resulting randomly sample from these observed variables, varying the sample size. For example, as shown on the left in Figure 2, we can select different variables as observed variables. Clearly, as the number of observed variables increases, the mutual information between observed and latent variables also increases. As a result, the degree of invertibility from latent to observed variables increases.
and the approximation of the identifiability result in Theorem 3.1, we aim to
total of 2 3 different observed binary variables. We then
For the right side of Figure 2, explore the robustness of our identifiability result in Theorem 3.1 with respect to both the graph structure and the size of the latent variables. To this end, we randomly generate DAG structures in the latent space using where ERA denotes graphs with d nodes and kd expected edges. For each ERK configuration, we also vary the size of the latent variables from 4 to 8, allowing us examine how the size of latent variables influences the identifiability results. In terms of the observed variables, we adapt the experimental setup from the left side of Figure latent variable sizes. This ensures that the degree of invertibility from the latent space to the observed space remains sufficiently high, a crucial factor for the accuracy of our identifiability analysis.
(ER) graphs (ERDAS & R&wi, 1959),
determine the appropriate observed variable size for different
Throughout the simulation, we use the following: In each experiment, we randomly mask one observed variable Tü, and use the remaining observed variables to predict it. Specifically, the remaining variables and the corresponding mask matrix are used as inputs to an embedding layer. This embedding layer transforms the input into a high-dimensional feature representation. The generated embeddings are then passed through Multi-Layer Perceptron (MLP)-based architecture to extract meaningful features, e.g.. fx (x). The MLP model consists of three layers, each with 256 hidden units. After ach layer, we apply Batch Normalization unit to stabilize training and ReLU nonlinear activation function to introduce nonlinearity. The final output of the MLP is used to predict the masked variable through linear classification layer. This allows model can predict missing or masked values based on the remaining observed variables. We employ the Adam optimizer with a learning rate of 1e 4. To ensure robustness and account for potential variability in the results, we conduct each experimental setting with five different runs, each initialized with different random seed. This procedure helps mitigate the effects of random initialization and provides a more reliable evaluation of the model's 's performance.
assess how well the
For evaluation, we use the LogisticRegression classifier from the seikit-learn library, which operates on the features extracted from the output of the MI.P-base architecture described above.
H.2 EXPERIMENTAL DETAILS ON LLMS
Unlike in simulation studies, where we have access to the complete set of latent variables, in real- world scenarios, their true values remain inherently unknown. This limitation arises from the nature of latent variables, they are unobserved and must be inferred indirectly from the data. As consequence, we cannot directly validate the linear identifiability results established in Theorem 3.1, since such validation would require explicit knowledge of these latent variables.
However, we can instead verify Corollary 4.3, which is direct consequence of Theorem 3.1. By doing so, we provide indirect empirical evidence supporting the theoretical identifiability results. To achieve this, need collect counterfactual pairs of data instances that differ in controlled and specific ways. These counterfactual pairs are essential for testing the implications of our theory in the context of real- world data.
28

---

Published as a conference paper at ICLR 2026
Generating such counterfactual pairs, however, presents significant challenges. First, the inherent complexity and nuances of natural language make it difficult to create pairs that differ in precisely the intended contexts while leaving other aspects unchanged. Second, as highlighted in prior works (Park et al., 2023; Jiang et al., 2024), constructing such counterfactual sentences is highly non-trivial task, for human annotators, due to the intricacies of semantics and the need for precise control over contextual variations.
We utilize the 27 counterfactual pairs introduced in (Park al., 2023), which provide structured and well-curated set of counterfactual pairs. These concepts encompass a wide range of semantic and morphological transformations, as detailed in Table By leveraging this established dataset, we ensure consistency with previous research while facilitating a robust and meaningful evaluation of Corollary 4.3.
We first use these 27 counterfactual pairs to construct a A, with size 27 x dam by using the differences in the representations these 27 counterfactual pairs, where dim corresponds to the feature dimension of the used LLM, such as 4096 for the Llama-27B model. To construct the corresponding matrix W., we train linear classifier using these 27 counterfactual pairs. Specifically, we use the representations of the counterfactual pairs as input and the corresponding values of the latent variables as output. As result, the corresponding linear weights for the 27 counterfactual pairs be used to create W In our experiments, we employ the LogisticRegression classifier from the scikit-learn library.
Note that, since the 27 counterfactual pairs involve only binary concepts, each of A, corresponds to the direction a concept associated with counterfactual pair, as stated in Corollary G.I. Similarly, discussed in Corollary G.2 each column of W. denotes the classifier direction for specific pair and aligns with the corresponding row of A, for that concept. As result, after normalizing A, and W, remove the arbitrary logit scaling (which is irrelevant before the softmax in cross-entropy loss), we expect to observe that A, W zs I. Therefore, we apply normalization to both A, and W prior to computing their product.
EXPERIMENT ON SPARSE AUTOENCODERS
I.1 IMPLEMENTATION OF THE PROPOSED STRUCTURED SAE
The proposed structured SAE employs regularization terms: structured regularization model the dependence among latent concepts, and a sparsity regularization based on the assumption that latent concepts may be sparsely activated. We implement follows:
S ReLU(W, (fx(x)) - ba) + ba),
(96)
R RcLU((::(fe) ba) + bi),
(97)
z S R,
(98)
X = w + b
(99)
Here:
S denotes the sparse representations from the sparse encoder (with parameters w.. b.);
R denotes the structured representation from the structured encoder (with parameters
Wi, by);
Z the combined representations used for reconstruction;
Fx (x) denote the reconstruction of (x).
The loss function for training is:
C = E Deiin [[(xxx) (x) (ISI) R|/
(100)
where:
29

---

Published as a conference paper at ICLR 2026
Table Concept names, one example the counterfactual pairs, and the number of used pairs, taken from (Park et al., 2023).
Concept
Example
Word Pair Counts
verb + 3pSg
(accept, accepts)
50
verb = Ving
(add, adding)
50
verb + Ved
(accept, accepted)
50
Ving = 3pSg
(adding, adds)
50
Ving Ved
(adding, added)
50
3pSg + Ved
(adds, added)
50
verb 3 able
(accept, acceptable)
50
verb > V
(begin, beginner)
50
verb + V tion
(compile, compilation)
50
verb - ment
(agree, agreement)
50
11
adj
adj
(able, unable)
50
12
adj + adj ly
(according, accordingly)
50
small + big
(brief, long)
25
14
thing color
(ant, black)
50
15
thing = part
(bus, seats)
50
country = capital
(Austria, Vienna)
158
17
pronoun = possessive
(he, his)
4
18
male + female
(actor, actress)
52
lower = upper
(always, Always)
73
20
plural
(album, albums)
100
21
adj + comparative
(bad, worse)
87
adj + superlative
(bad, worst)
87
23
frequent = infrequent
(bad, terrible)
86
24
English = French
(April, avril)
116
25
French + German
(ami, Freund)
128
26
French Spanish
(annee, año)
180
27
German + Spanish
(Arbeit, trabajo)
228
X₂ is the dynamically adjusted sparsity coefficient step t, following p-annealing SAE (Karvonen et al., 2024);
is hyperparameter that balances the sparsity penalty and the low-rank regularization;
|||p: = E, |s is the adaptive Lp, promoting sparsity, following p-annealing SAE (Karvonen et al., 2024);
|R| nuc is the nuclear norm, used to encourage low-rank structure.
We estimate the nuclear norm using the top-ksvd singular values:
kand
R. nuc
dis
(101)
where {ar are the largest keed singular values, obtained via low-rank SVD (c.g., PyTorch's avd lowaok...... The value of ksnd is tunable parameter (e.g., kond 64) that trades off approxi- mation accuracy and computational cost.
1.2 DETAILS EVALUATION METRIC
Obtaining f 1|x) from Supervised Linear Classification. For evaluation, we first use 27 counterfactual pairs from (Park et al., 2023), also see Table 1, to train a linear classification using Logistic
classifier from the scikit-leam library, to obtain logits, i... unnormalized p(c 1[x.. for each of the 27 pairs. As a result, for each concept in these 27 concepts, we can obtain the corresponding logit. Figure 5 shows classification accuracy of Regeesinno classifier.
30

---

Published as a conference paper at ICLR 2026
Stacking these 27 logits yields the logit vector
u = (W1 U2. U27).
(102)
use the representations (x() the same 27 counterfactual pairs from (Park et al., 2023) as input to trained SAE, extracting the corresponding latent features Let denote the dlement-wise exponentiation of z, ie.. exp(z). This yiclds a feature matrix of size 27 D, where D is the dimensionality of z:
Extracting 2; from trained SAEs.
[1 Z2
227 )²
(103)
Correlation Matrix and Assignment. the Pearson correlation
For the logit vector u, and the feature matrix 2, we compute
R = corru,, a.,,,
d d=I,..., D
(104)
where ž.d denotes the d-th column of z.
Note that the estimated features z from the SAE are subject to permutation indeterminacy. To address this, we apply the Hungarian algorithm to solve the assignment problem R₃, This yields the optimal assignment for each concept, allowing us compute the assigned Pearson correlation. We report the mean Pearson correlation across the 27 concepts.
1.3 EXPERIMENTS AND RESULTS
We train four SAE variants- top-k SAE (Gao et al., 2025), batch-top-k SAE (Bussmann al., 2024), p-annealing SAE (Karvonen et al., 2024), and our proposed structured SAE. Each SAE is trained three times activations from the final hidden layer of the pretrained Pythia 70m, Pythia 1,4b, and Pythia 2.8b (Biderman et al., 2023) (download from https:/ : / h/tuggingfacc.e.emc/ooooo.a using training data from the first 200 million tokens of the Pile corpus (Gao et al., 2020) (download from https:/ /hugg i www.acca.....oo co soo/dataeees El GleutherrA/tiin pile wwduplicatees
Experimental setup. We set the feature dimension of all SAE variants to be the same (D = -2,22677 768) and are trained for 000 optimization steps with batch size of 10000. optimizer with an initial learning rate of x 10-1 and linearly warm up the learning rate during the first 200 steps. For the top-k and batch-top-k SAEs, we set & = 32. For p-annealing SAEs, apply sparsity warm-up of 400 steps and an initial sparsity penalty coefficient X 0.1. The p-anncaling-Lotk SAE uses the same p-anncaling settings and additionally applics a low-rank scaling factor y 0.1.
employ the Adam
Compute resources. GPUs (40 GB each).
All experiments are conducted on server equipped with four NVIDIA A100
Pearson Correlation Coefficient. We report Pearson Correlation Coefficient (PCC) at the 20 000-step checkpoint. As summarized in Tables 3-5, the two p-annealing variants markedly outperform the fixed-k baselines. Across the 27 concepts, the proposed structured SAE achieves bet- ter PCC across different scales Pythia family, confirming that sparsity and the low-rank adaptation yield features that align more cleanly with human- inttrpp-interreeaabll concepts.
Table 2: Ablation results for the low-rank term scaling factor y. PCC is reported mean + standard deviation over multiple runs.
PCC (mean + std)
10-3
0.691 + 0.023
10
0.695 + 0.019
10
0.704 = 0.013
0.685 + 0.021
31

---

Published as a conference paper at ICLR 2026
Classification Performance per Concept
1.1
1.0
1,00
1.00
1.00
1.00 1.00
1.00 1.00 1.00 1.00
1.00
1.00 1.00
Accuracy
0.6
0.2
0.1
spog, vnd
all superratiee
country capital
empiit Frettsy
trevch, german
(meeh spaeraa
freourt, intrepient
germas sqaniah
lowsr sppeer
male fomale
Amo popieaiie
thing coler
tntrg, sart
verk tog
aed
vert ving
abile
viep Spig
wing and
umall
connt
vertt
Figure 5: Classification accuracy of logistic probes across various concepts. Each bar represents the performance for a given concept.
Ablation on the low-rank coefficient. We investigate the effect of the low-rank term scaling factor
[10-3,10-3 10- 10-1.11 while keeping all other hyperparameters fixed. As shown in Table 2, the PCC peaks at y 10 10-1, indicating that a moderate weighting of the low-rank term is beneficial. Smaller or larger values of lead to slightly worse performance, suggesting that both under- and over-emphasis on the low-rank term can hurt the model's 's ability to capture the underlying structure.
Table PCC results at training step 20,000 on Pythia- Bold values denote the best.
PCC across different SAEs (1)
Concepts
top-k
batch-top-k
p-annealing
Our structured SAE
3psg_ved
0.533 + 0.069
0.166 + 0.012
0.660
0.011
0.684 + 0.008
adj_adi_t_ly
0.828 + 0.054
0.918 0.005
0.848 0.004
0.884 0.005
acj_comparative
0.640 :::00:000 0.108
0.646 0.087
0.536
0.077
0.695 0.070
acj_superlative
0.743 : 0.017
0.757 = 0.028
0.731 = 0.021
0.792 0 0.033
acj_un_t_adj
0.362 + 0.041
0.353 + 0.030
0.510 0.031
0.771 1: 0.033
countzy_capital
0.237 + 0.008
0.247 + 0.021
0.356 0.005
0.494 1: 0.003
english_french
0.501 0.076
0.448 0.006
0.629 0.008
0.639 0.005
french_german
0.615 0.003
0.617 0.023
0.608 0.020
0.685 0.050
french_spanish
0.627 = 0.069
0.520 + 0.054
0.643 0.033
0.613 0.086
frequent__iffreqeen
0.279 + 0.014
0.292 0.017
0.490 0.011
0.509 1 0.018
german_spanish
0.583 + 0.077
0.730 T 0.010
0.688 0.021
0.713 1: 0.010
lower_upper
0.474 + 0.080
0.442 0.016
0.631 0.010
0.614 0.008
malc_fenale
0.378 = 0.040
0.326 = 0.008
0.585 = 0.030
0.625 0.006
noun_plural
0,616 = 0.000
0.501 + 0.038
0.662 0.024
0,640 - 0.011
proncun_possesssive
0.897 = 0.035
0.885 0.037
0.967 0.022
0.974 0.020
small_big
0.403 + 0.005
0.412 0.040
0.603 0.023
0.668 0.010
thing_ccler
0.950 + 0.008
0.899 + 0.042
0.918 0.020
0.915 0.016
thing_part
0.421 + 0.013
0.345 + 0.016
0.494 0.031
0.506 + 0.008
verb_3ps0
0,658 + 0.035
0.588 + 0.061
0.673 0.033
0.752 0.022
verb_v_t_abl
0.723 + 0.068
0.706 1 0.033
0.787 0.031
0.825 1 0.018
0.581 0.039
0.548 0.021
0.772 0.025
0.748 0.011
verb_v_+_ment
0.750 = 0.013
0.778 - 0.017
0.622 0.019
0.696 0.017
verb_v_t_tion
0.764 = 0.054
0.681 + 0.079
0.722 = 0.048
0.794
verb_ved
0.522 = 0.064
0.583 - 0.087
0.537 0.061
0.668 0.052
verb_ving
0.689 + 0.066
0.717 - 0.035
0.638 0.026
0.737 1 0.034
ving_3pag
0.467 + 0.057
0.403 0.029
0.704 0.029
0.704 0.010
ving_ved
0.557 + 0.043
0.538 + 0.067
0.691 0.053
0.688 4 0.022
32

---

Published as a conference paper at ICLR 2026
Table PCC results at training step 20,000 on Pythia-I.4b. Bold values denote the best.
PCC across different SAEs (1)
Concepts
top-k
batch-top-k
p-annealing
Our structured SAE
3psg_ved
0.577 + 0.015
0.536 1 0.013
0.629 0.012
0.727 0.009
adi_adi_t_ly
0.870 + 0.013
0.841 + 0.009
0.897 = 0.020
0.940 1 0.011
adi_comparative
0.760 0.018
0.738 = 0.013
0.640 0.011
0.660 + 0.010
adj_superlative
0,849 = 0.022
0.871 0.017
0.767 0.013
0.790 0.009
adj_un_t_adj
0.563 = 0.076
0.556 = 0.107
0.654 = 0.021
0.749 0.012
average
0.697 + 0.004
0.686 + 0.002
0.742 = 0.019
0.752 0.010
country_capital
0.802 + 0.032
0.795 + 0.006
0.788 0.007
0.823 0.012
english_french
0.836 + 0.014
0.810 0.066
0.813 0.008
0.814 0.011
trench_german
0.701 = 0.030
0.676 0.081
0.790 0.016
0.799 0.012
french_spanise
0.737 + 0.006
0.706 0.041
0.823 0.010
0.762 0.013
Etequent_iifrequuee
0,522 + 0.029
0.569 + 0.003
0.588 0.012
0.646 0.011
german_spanish
0.687 = 0.023
0.673 + 0.038
0.796 0.009
0.783
lower_upper
0.765 = 0.007
0.746 0.007
0.716 0.010
0.726
male_female
0.584 = 0.047
0.617 + 0.011
0.593 0.006
0.546
noun_plural
0.688 + 0.006
0.703 + 0.029
0.851 0.014
0.010
prencun_possessive
0.937 + 0.018
0.869 + 0.031
0.980 0.012
0.989 1: 0.008
small_big
0.287 + 0.007
0.300 0.008
0.536 0.013
0.533 0.010
thing_celer
0.913 0.019
0.937 = 0.009
0.928 0.012
0.904 0.011
thing_part
0.397 = 0.005
= 0 0.013
0.440 = 0.010
0.531 0.014
verb_3psg
0.675 = 0.090
0.619 0.011
0.709 0.008
0.716 0.012
verb_v_t_able
0.785 + 0.048
0.743 0.026
0.899 0.015
0.798 + 0.013
verb_v_t_er
0.707 + 0.012
0.745 0.019
0.752 0.011
0.781 0.012
verb_v_t_mad +_ment
0.769 + 0.014
0.603 + 0.033
0.831 0.010
0.741 0.009
verb_v_+_E v___tionn
0.857 + 0.025
1 0.013
0.841 0.012
0,818 0.010
verb_ved
0.599 + 0.027
0.674 1 0.032
0.682 0.009
0.710 0.008
verb_ving
0.730 + 0.004
0.721 0.053
0.800 0.013
0.794 0.011
ving_3psg
0.612 0.064
0.625 0.036
0.642 0.010
0.745 0.014
ving_ved
0.605 = 0.019
0.558 + 0.008
0.645 0.011
0.660 0.012
J FURTHER DISCUSSION: OBSERVATIONS ON LLMs AND WORLD MODELS
J.1 LLMS MIMIC THE HUMAN WORLD MODEL,, NOT THE WORLD ITSELF
As we explore the implications our linear identifiability result, is important to situate it within the broader context of human cetttiin--eecffcalllly how humans develop and interact with internal world model. In this subsection, we introduce the concept that LLMs Mimic the Human World Model, Not the World Itself, emphasizing the distinction between the vast physical world and the compressed abstraction humans use for reasoning and maaiing Our analysis shows that LLMs replicate human-like abstractions through latent variable models. We further argue that LLMs aim emulate this internal, compressed world model-rathe than the physical world itself-by learning from couaa--generate text. This distinction provides critical insight into the success of LLMs tasks aligned with human conceptualization and deepens our understanding of the relationship between language models and human cognition.
Humans gradually develop their understanding of the environment through learning from others and interacting with the world. This internal representation of our external environment is known as a world model. A key observation is that this model is not a direct reflection of the physical world but rather highly compressed abstraction of it. For instance, numbers, such as 1, 2, and 3, are abstract tools created by the human mind for reasoning and problem-solving. They do not exist as tangible entities in the natural world. Similarly, many aspects of reality that escape human senses or even the most sophisticated scientific instruments absent from mental representations.
Interestingly, our texts reflect our mental activities and emotions, providing a window into this compressed world model. A recent study reveals striking disparity between the limited information throughput human behavior (approximately 10 bits/s) and the vast sensory input available (around 10 bits/s) (Zheng & Meister, 2024). This suggests that the human world model is efficient,
33

---

Published as a conference paper at ICLR 2026
Table PCC results at training step 20,000 on Pythia-2.8b. Bold values denote the best.
PCC across different SAEs (1)
Concepts
top-k
batch-top-k
p-annealing
Our structured SAE
3psg_ved
0.589 1 0.025
0.513 1 0.024
0.645 0.012
0.734 0.027
adi_adi_t_iy
0.869 + 0.036
0.797 1 0.050
0.919 0.020
0.941 1 0.013
adi_comparative
0,847 = 0.020
0.859 = 0.010
0.817 0.015
0.757 + 0.028
adj_superlative
0,885 = 0.011
0.897 0.010
0.901 0.017
0.787 0.009
adj_un_t_adj
0,631 = 0.086
0.690 + 0.033
0.590 = 0.012
0.693 0.021
average
0.707 + 0.008
0.704 + 0.002
0.762 0.011
0.771 + 0.022
country_capital
0.810 + 0.040
0.791 + 0.034
0.892 0.014
0.832 + 0.020
english_french
0.854 + 0.010
0.834 0.010
0.842 0.015
0.847 0.027
#rench_gernan
0,659 = 0.060
0.710 + 0.035
0.743 0.016
0.799 0.022
french_spanise
0.741 = 0.027
0.711 0.012
0.828 0.011
0.794 0.019
[tequent_infrequeee
0,523 + 0.066
0.576 0.036
0.572 0.018
0.670 1: 0.028
german_spanish
0.643 = 0.040
0.646 0.063
0.816 0.021
0.805 + 0.030
lower_upper
0.772 = 0.027
0.758 + 0.010
0.704 0.012
0.739 + 0.025
male_female
0.640 0.040
=
0.721 = 0.018
0.640 + 0.020
noun_plural
0.761 + 0.024
0.737 1 0.014
0.879 0.023
0.830 1 0.027
grenoun_possessive
0.918 + 0.049
0.914 0.011
0.980 0.015
0.995 + 0.021
small_big
0.317 + 0.007
0.400 0.043
0.481 0.014
0.602 + 0.032
thing_celer
0.918 0.019
0.932 + 0.012
0.923 0.007
0.901 0.015
thing_part
0.373 = 0.033
0 0.021
0.457 0.010
0.506 0. 025
verb_3psg
0,523 = 0.071
0.578 + 0.025
0.643 0.012
0.739 + O 0.021
verb_v_t_able
0.751 + 0.022
0.736 + 0.023
0.832 0.018
0.828 + 0.026
verb_v_t_er
0.715 + 0.052
0.790 0.011
0.832 0.025
0.806 + 0.022
verb_v_t_ment +_nntt
0.784 + 0.036
4.762-0.033 0.032
0.860 0.015
0.854 + 0.028
verb_v_+_E t_tiion
0,805 + 0.053
0.845 1 0.085
0.864 = 0.023
0,861 0.030
verb_ved
0.720 + 0.041
0.728 1 0.039
0.664 0.012
0.752 1 0.018
verb_ving
0.732 + 0.060
0.755 1 0.035
0.845 0.020
0.819 1 0.027
ving_3pag
0.728 0.082
0.559 0.117
0.690 0.015
0.675 0.022
ving_ved
0.562 = 0.022
0.589 + 0.009
0.624 0.017
0.620 0.019
compressed abstraction, enabling us to reason, predict, and make decisions effectively. Despite this compression, humans have flourished as the dominant species on Earth, demonstrating the power of such a streamlined model.
LLMs aim to mimic not the vast and unbounded world but this froman--omppesse world model, which is significantly smaller and more manageable. By learning from human text, which encodes this abstraction, LLMs effectively replicate the patterns, reasoning, and abstractions that have proven successful for humans. This explains the impressive performance of LLMs in tasks that align with human understanding. Furthermore, the overlap between the latent space of LLMs (representing human concepts) and the observed space (human text) provides powerful mechanism for aligning human-like abstractions with model predictions. For instance, modifying words in the observed space often corresponds predictable changes in latent concepts.
J.2 LLMS VERSUS PURE VISION MODELS: A FUNDAMENTAL DIFFERENCE
While LLMs model human language, which has already undergone significant compression through human cognition, vision models face fundamentally different challenge. In the previous subsection, we discussed how L.MSs mimic the human world model, leveraging compressed abstractions derived from numan-generate text. In contrast, vision models operate on raw, high-dimensional data from visual inputs. Moreover, the training data for vision models represents only a tiny fraction of the universe, constrained by the limitations of capturing devices and datasets. This vastness and lack of compression make the task of building generalizable representations in vision models inherently more complex than in NLP-based LLMs.
This difference in the nature of their training data and observed space might explain the behavioral differences between LLMs and pure vision models. While LLMs benefit from the inherent abstraction and compression of human language, vision models must contend with raw, unprocessed inputs that
34

---

Published as a conference paper at ICLR 2026
require far greater generalization capabilities. This underscores the importance of understanding the nuances of each modality when designing and evaluating Al systems.
K MORE RESULTS ON LLAMA-3 AND DEEPSEEK-RI
We conduct additional experiments on recent LLMs, including Llama-3 and DeepSeek-RL, to further evaluate findings. The experimental setup strictly adheres to the settings described in Section H.2. This enables a comprehensive investigation of our findings, ensuring that the results are thoroughly evaluated and validated across diverse LLM architectures and experimental conditions. Overall, can see, the product A, x W, approximates the identity matrix, supporting the theoretical findings outlined in Corollary G.2.
A, Llama-3-8b
x W, Llama-3-70b
-AATA-ttt RAtWARE St
W. DeepSeek-R1-11
W. DeepSeek-R1.1
x W.on on BeepSeek-R1-14b
W. on DeepSeek-R1- R1-33b
wwt.:r/wwwSaaaeea......... 32.4
Figure 6: Results of the product A, x W, across the LLaMA-3 and DeepSeek-R1 model families.
Here, A, represents matrix derived from the feature differences of 27 counterfactual pairs, while W, is a weight matrix obtained from a linear classifier trained on these features.
35

---

Published as a conference paper at ICLR 2026
L FUTURE DIRECTIONS
Rethinking Invertibility Assumptions in Causal Representation Learning Our identifiability analysis is closely related the concept of identifiability in causal representation learning. Notably, to the best of our knowledge, this is the first work to explore approximate identifiability in the context of non-in non-invertible mappings from latent space to observed space- departure from the commonly upheld invertibility assumption in the causal representation learning community (Brehmer et al., 2022; Von Kügelgen et al., 2021; Massidda al., 2023; von Kügelgen et al., 2023; Ahuja al., 2023; Seigal et al., 2022; Shen et al., 2022; Liu et al., 2022). We hope that our work will inspire future research aimed at overcoming the limitations imposed by invertibility assumptions in causal representation learning.
Embedding Causal Reasoning in LLMs Through Linear Unmixing Our linear identifiability result lays foundation for uncovering latent causal relationships among concepts, especially when these variables exhibit causal dependencies within the proposed latent variable model. By showing that the representations in LLMs are linear mixtures of latent causal variables, our analysis shows that linear unmixing of these representations may allow for the identification of underlying latent causal structures. This approach not only opens up the possibility of understanding causal dynamics within LLMs but also suggests that causal reasoning, particularly in latent spaces, could be achievable through the exploration of these linear unmixing techniques. We believe this work marks a pivotal step toward embedding robust causal reasoning capabilities into LLMs.
M ACKNOWLEDGMENT OF LLMs USAGE
We disclose that large language models (LLMs) were used only to correct typos, improve grammar, and refine phrasing. All scientific contributions, including problem formulation, methodology design, theoretical proofs, experiments, and analysis of results, are exclusively the work of the authors.
N CROSS- VASISAAALNA AND NULL BASELINE ANALYSIS
Motivation. In previous experimental results in Figure 3, we demonstrate the results of the product A, x W, across the LLaMA-2 and Pythia model families. However, the concept directions A, and probing weights W, were computed using the same set of word pairs, which could artificially inflate diagonal alignment due to repeated supervision. To rigorously test the theoretical finding in Corollary 4.3, we introduce two complementary safeguards: 50%/50% % split cross-validation, and (ii) null baseline with randomly permuted labels.
Regarding 50 %50%% Cross- Validation, split each word pair into two halves. The first half is used to compute the concept direction (mean difference vector), and the second half is used exclusively for training the linear probing classifier, to ensures that the evaluation is performed on held-out data.
For Null Baseline (Permutation Labels), We use randomly permuted labels of the word pairs to train the linear probing classifier. The resulting distribution of diagonal alignments serves as a null model, reflecting the aligmment expected by chance. The results are shown in Figures 7 and 8. These findings confirm that the high diagonal aligmment observed experimental results in Figure is not trivial consequence of reusing the same supervision, but rather reflects meaningful association between the concept directions and the learned probe weights, aligning with the theoretical finding in Corollary 4.3, We also report Frobenius norm, which measures how far the obtained operator deviates from the ideal projection its overall action, with smaller values indicating closer alignment to the identity the concept subspace.
We further evaluate the setup under multiple random seeds that control the 50%/50% vross-validation split. This allows to verify that the observed performance is not an artifact of particular partition and that the conclusions hold consistently across different random splits. The results are shown in Figure 9-13, which demonstrate that the finding is robust across different random seeds.
36

---

Published as a conference paper at ICLR 2026
A. x.W. Pythia-lb
W. on -yth
W. on Pythip 2
A. W. Pythio.6.0 6.96
1A Pria-
W. Pythia-1
W. Pythia 4b
W. Pythis 2 8b
W. Pythis 6 96
[A.
[A.
JA. -
Figure 7: Results the product A, x W. under Cross-Validann Validation (top) and Null Baseline (bottom). The vross-validation ensures that the observed effect generalizes to held-out embeddings, while the null bascline provides quantitative reference to rule out spurious alignment. Both results are consistent with the theoretical finding in Corollary 4.3.
A. W. on Pythia 12b
A, x W, on Liame 2-76
A. W. Lera Sama-2-130
A. W. on Uamee 2-706
1A-W. P
x W. on Pythia-12b
A. X W. on Llama-2-7p
A. W. Lama-2-130 13b
A. W. Lama--2-70
[A, W. -
|9 AUMAL
Figure 8: Results of the product A, x W. under Validaaino (top) and Null Baseline (bottom). The cross-validation ensures that the observed effect generalizes to held-out embeddings, while the null baseline provides quantitative reference to rule out spurious alignment. Both results are consistent with the theoretical finding in Corollary 4.3.
37

---

Published as a conference paper at ICLR 2026
pythia-1b: Mean of A,W, over 10 seeds
59
12
14
12
12
555
AM
211
30
500
34
54
21
AA W. psl2 6..822
Figure 9: Results of the product A, W. & on Pythia- lb across different seeds.
Aiciia-144b Mean of A,W; over 10 seeds
85 24
LA
20
AM
20
20
21
A W. Poll 6.4450
Figure 10: Results of the product A, x W, Pythia-1.49 1.4b across different seeds.
38

---

Published as a conference paper at ICLR 2026
pythia-2.88 Rihia-2.8bb: Mean of A,W, over 10 seeds
22
20
AMA
201
25
321
30
X2
54
20
27
22
AA W. psl2 6.3901
Figure 11: Results of the product A, x W, Pythia-2.8b across different seeds.
pythia-6.8m 6.9b: Mean of A,W. over 10 seeds
221
25
11
344
515
201
20
20
201
211
21
A W. P3ll2 6.9484
Figure 12: Results of the product A, x W, Pythia-6.96 across different seeds.
39

---

Published as a conference paper at ICLR 2026
pythia-1 2b: Mean A,W, over 10 seeds
24
35
204
25
13
115
AM
20
25
321
12
30
20
20
AA W. Psl2 7.2469
Figure 13: Results of the product A, W. & on Pythia- 12b across different seeds.
Jama-2-7b Mean of A,W, over 10 seeds
385
2%
355
AM
12
20
24
25
A W. Poll2 5.4854
Figure 14: Results of the product A, x W, Llama-2-77 across different seeds.
40

---

Published as a conference paper at ICLR 2026
Llama-2-13bb Mean A,W, over seeds
AM
3V
AA W. ps12 5.5662
Figure 15: Results of the product A, W, on Llama-2- 2-1bb across different seeds.
Llama-2-706 2-70b: Mean A.W; over seeds
LA
2%
29
25
20
A W. pol2 5.8355
Figure 16: Results of the product A, x W, on Llama-2-700 across different seeds.
41

---

Published as a conference paper at ICLR 2026
O ASSESSING ROBUSTNESS THE LINEAR RESULT IN THEOREM 3.1
To assess the practical impact of the theoretical correction term h, in Theorem 3.1, which appears as the residual perturbation the ideal linear result in Theorem 3.1. We design an experiment that uses linear-probe performance as an empirical proxy for the magnitude of this term. Intuitively, if hy is negligible for a given pretrained model, the representation should closely follow the ideal linear form and línear probe trained on counterfactual concept pairs will achieve high test accuracy: conversely, a large hy should degrade probe performance. Concretely, we use the 27 concept counterfactual pairs to train a logistic linear probe to discriminate the pair. To quantify variability, we repeat the train/test (70%/30%) split three times with different random seeds and report the mean and standard deviation of test accuracy. The majority of concepts show high mean accuracy and small standard deviation, indicating that the hy correction is empirically small and that the linear approximation in Theorem 3.1 holds well in practice. The results are shown in Figures 17-21. We note that some concepts, such as the "pronoun possessive" category, have very limited sample sizes, which lead to poor and unstable test performance; consequently, the results for these concepts should be interpreted with caution. Overall, the repeated-run analysis demonstrates that our main empirical findings are robust and not artifact of particular random split, and it provides quantitative evidence that the hy term has limited influence for most concepts under study.
Linear Probe Test Accuracy for Fach Concept lquantifying hy effect)
1.0
0.6
Test
0.4
0.0
Cuben nooro
Figure 17: Testing performance of linear probing on Pythia 1b model.
Linear Probe Test Accuracy for Each Concopt Pair (quantifying by offect)
1.0
0.8
Accuracy
0.6
0.2
0.0
Figure 18: Testing performance of linear probing on Pythia- 1.4b model.
P SENSITIVITY OF SAE EVALUATION TO LINEAR PROBE QUALITY
In principle, training linear probe to approximate the ideal posterior p(clx) for evaluating SAE features requires ideal counterfactual samples. In practice, however, we often face several challenges, such the choice of regularization parameters in logistic regression, imbalanced classes, and label
42

---

Published as a conference paper at ICLR 2026
Linear Probe Test Accuracy for Fach Concept Pair lquantifying by effect)
1.0
Accuracy
0.6
0.2
Figure 19: Testing performance of linear probing Pythia-2.8h model.
Linear Probe Test Accuracy lquantifying h,
0.8
Test
0.4
0.0
Figure 20: Testing performance of linear probing on Pythia-6.9b model.
Linear Probe Test Accuracy for Each Concopt Pair Iquantifying by offect)
1.0
0.6
0.2
Figure 21: Testing performance linear probing on Pythia- 12b model.
noise. assess the sensitivity of SAE evaluation to these factors, we conducted experiments varying the probe's ss regularization strength (LogisticRegression in SKleam,, C=0.1 1=0.1,0.01. 1,000,,,,,0001,, respectively. baseline C=1), class imbalance (proportion of positive labels y = 1) set to 0.7, 0.8, 0.9, implemented by "clsss "esigtt'''' parameter in LegisticRegression in scikit-learn, baseline balanced at 1), and label noise (5%, 10%, 15% random label flips; baseline 0%)). The results are shown in Figure 22. We observe that the Pearson correlation remains stable across different regularization strengths, which is expected since a linear classifier with l regularization typically exhibits robustness to variations in strength. In contrast, for varying class imbalance and label noise, there is slight decrease in Pearson
43

---

Published as a conference paper at ICLR 2026
correlation, This behavior may be reasonable, as both class imbalance and label noise may introduce additional variability in the learned logits, affecting the evaluation.
Partse corvladion antrrtio Stwegh
irbalace
Planeee servirdiin
Figure 22: Pearson correlation across regularization strength (left), class imbalance (middle), and
Noise Levels (right).
Q EMPIRICAL VERIFICATION OF THE TV AND APPROXIMATE INVERTIBILITY
Our thcoretical analysis in Section shows that the key remainder term in the decomposition Ah= = fly. yo is governed primarily by two quantities: (i) the total-variation distance between the posteriors p(c yi) and p(c yo)). and (ii) the concentration of the conditional posterior p(c x, y). captured by the parameter in the metroximatt-ivvetibiii assumption. To assess the strength of these assumptions in practice, we conduct an empirical study on real LLM representations.
Empirical proxy for the posterior. We train 27 linear probes on the counterfactual concept pairs introduced earlier, and stack their prediction probabilities to obtain a normalized vector over the 27 concepts. This provides a psendo-pesterior over the probed concept subspace, which we treat purely as practical proxy for p(c x). Because the next-token y depends solely the input x in the autoregressive setting, the variation of this pseudo-posterior across input contexts serves as an empirical surrogate for the variation of p(c y). Across all 27 concepts, the resulting TV distances are small (Table 6), supporting the required TV condition in our remainder bound.
Posterior concentration proxy. For each instance x, we measure how concentrated the pseudo- posterior is around its dominant concept via
{xx = 1 max p x).
This quantity is not intended to estimate the true global invertibility parameter, but rather to provide conservative subspace- based proxy. Following the reviewer's suggestion, we report the 5% quantile
Eg Quantile,.oo (?(x)),
which captures concentration the high-mass portion of the distribution. As shown in Table 6, this proxy remains small for all concepts.
Remainder control. The theoretical remainder term takes the form
TV (p(c u:),p(e 30)) [loge].
Using the empirical TV values and the conservative estimate log Gool. we compute this product for each of the 27 concepts. Across the board (Table 6), the product remains small, indicating that both factors in the theoretical remainder are simultaneously well-behaved. This suggests that the remainder term is negligible for real LLM representations, reinforcing the practical relevance of our identifiability analysis.
R ADDITIONAL COUNTERFACTUAL PAIR ANALYSIS ON PYTHIA MODELS
To further strengthen the generality of our findings, we extend our evaluation, beyond morphology- and translation- -based contrasts in the used 27 pairs before, by constructing 8 new counterfactual pairs.
44

---

Published as a conference paper at ICLR 2026
Table Empirical estimates of the TV distance the anchor concept and the combined remainder proxy TV log Equs for all 27 concepts. The global proxy is 40o 0.7473 with |log Egos 0.2913. Note: Concept 21 is selected as the anchor concept (corresponding to yo).
Concepts 0-13
Concepts 14-26
ID
TV
TV. log
ID
TV
TV- log
0.1336
0.03893
14
0.1989
0.05794
0.1721
0.05014
15
0.2073
0.06040
0.2259
0.06581
16
0.1778
0.05181
0.2398
0.06985
17
0.1430
0.04165
0.2389
0.06958
18
0.1821
0.05305
0.1206
0.03513
19
0.1386
0.04037
0.1005
0.02928
20
0.1427
0.04157
0.1721
0.05013
0,0000
0.00000
0.1358
0.03957
22
0.1158
0.03374
0.1774
0.05168
23
0.1034
0.03011
10
0.1455
0.04238
24
0.1124
0.03273
0.2940
0.08565
25
0.0857
0.02497
0.0933
0.02717
26
0.0754
0.02196
13
0.1548
0.04510
These include positíve negative (e.g., good vs bad), truth vs falsity (authentic, fake), and toxic vs neutral (e.g.. dangerous, harmless). Constructing such counterfactual pairs is, however, nontrivial and inherently challenging. This constructing process may often introduce additional variability and noise. As a result, the pairs may deviate from the idealized assumption of the case that single latent concept changing while all other factors remain fixed.
Importantly, the presence of such noise does not undermine our analysis, rather, it provides an opportunity assess the robustness of our results under realistic, noisy conditions.
We therefore run the evaluation pipeline, including the cross-validation procedure and the null baseline protocol (similar Section N), these newly constructed pairs. Overall, the results, shown in Figure 23, exhibit the same qualitative patterns as the original experiments, indicating that our findings remain robust even when additional noise may be introduced. This further supports the broader claim that our framework provides a unified perspective on concept directions, probing, and representation steering.
W. on bythiecth
A W.W.W on Pythiesl Ab
A X. an Pythie2.ft
A. A.W.W on Pythie-d..h
x W. Pythie112
P.-W--PP---22mmm Aux Wi Pythip
M.-W.-Pil--..m A W. Pythia 12b
A Wi on Pythia 1.40
A. W.WW W. on Pythia 2.88
A H.W on Pythia 6
2030
L -11--11--1102
20002
Figure 23: Results of the product A, W, under Cross- Validatio (top) and Null Baseline (bottom) on new pair data. The cross-validation ensures that the observed effect generalizes to held-out embeddings, while the null baseline provides quantitative reference rule spurious alignment.
45

---

Published as a conference paper at ICLR 2026
Pyto
pycb
0:25
0.24
0.26
0.20
0.27
0.10
+0.00
-0.03
0.004
07
0.35
0.77
5:33
0.21
0.09
0-4
4.21
2.17
0.22
9:23
0.30
6,25
0.01
0.53
0.35
0.24
20.05
20,02
20.05
20.03
20.03
25.07
22.04
20.0
20.04
23.00
10.05
-0.00
10.03
60.04
10.00
10.03
60.03
10.00
$5.05
A0
60.03
10.03
60.05
40.04
10.00
60.34
0.24
0:23
0.33
0.57
0.08
0.30
0.16
0.32
0.30
0.23
0.34
0.59
0.08
0:31
0.21
0.35
0.17
0.37
0.1
0,75
4.12
2:11
0:58
2:14
0.17
0,32
0.21
0.19
0.07
0.18
10.05
20.03
80.33
10.05
$0.03
25.02
85.00
80.01
20.02
20.34
+0.03
=9.05
10.03
+0.04
10.09
s0.03
=6.06
10.02
+6.03
10.07
+9.05
10.05
20.06
+0.02
10.09
29.02
0.04
0.24
2.37
0.11
0.32
0.32
0.20
3,27
0.23
3,00
0.11
0.24
0.02
4.13
0.75
0.11
0.25
0.17
0.23
0.01
0.27
0.30
5:73
0.22
0.23
0.03
0.27
0.05
0.10
0.73
10.04
+0.02
10.04
60
10.00
10.04
02
10.04
=0.00
40
60.04
10.07
00.07
20.01
10.04
20.00
[A, W. Pala 130040
3.0040
JA_
W. P.U. - 3,0392
pythin-2 Bb. Mean of over steds
pyth 9b Newn A.W. over seeds
0.65
0.12
3.10
0.21
0.18
0.20
-0.03
0.31
0.04
D:31
0.27
0.08
0:15
0.26
0.05
0.15
0.76
0.30 20.04
0.31
0:25
0.32
20.05
-0.03
20.05
20.03
20.04
20.04
10.04
20.06
20
20.04
20.04
20.03
20.03
05
20.00
0.30
0.33
0.88
20.04
10.03
10.04
25.03
25.04
10.04
20.07
20.04
20.25
0.22
0.33
3.53
0.63
0.31
0.28
0 20
0.33
0.18
0:25
0.30
0.67
0.
0.25
0.28
0.28
20.3
#0.03
=0.05
20.05
20,02
20.02
20.55
20.04
20.05
20.02
20.03
20.03
20
50.02
20.04
20.04
20.02
20
=0.00
0.21
0.35
0.
+0 72
9.28
0:33
0.35
0.71
.0.09
0:38
1003
10.12
0.08
0.25
3.37
0.35
0.38
0.17
0.22
5,20
0.00
5.18
0.13
0.34
0:13
9.38
0.76
0.15
25.03
20.05
20.04
25.06
20.33
10.03
60.23
40.03
60.02
10.05
10.02
80.03
10.04
a0.03
00.0
60.00
10.00
10.05
80.02
s0.04
00.00
IA
2.8646
2.9439
P/7 12th
Mean of A.W. over 1 seeds
0.50
B:12
0.28 10.00
8,21
0.22
-0.02
0,29
10.00
10.04
-0.12
10.03
10.05
10.05
20.05
20.02
20.00
20.05
20.09
20.04
02.02
20.04
0.30
0.2
0.65
0,29
0.07
0.51
0.11
0,50
0:23
0.1
0.30
0.64
0.09
0.24
-.22
0.29
+0.08
+0.05
+0.04
10.07
29.05
+0.05
+0 05
+0.00
20.04
20.55
20.04
20
20.0
20.04
0:34
028
0.29
8.23
0.55
0.70
0.15
0.31
0.67
0.16
0:33
0.19
3:14
0.80
0.18
+0.05
+0.03
20.02
20.02
20.00
20.12
20.05
20.00
0:27
0.23
20.05
20.03
20.04
$0.00
[A,
P.A:
Figure 24: Results of the product A, x W, on new pair data across di different seeds.
46


Reviews:
[{"review_id": "OCebSBAQae", "reviewer": "Reviewer_TJva", "summary": "The paper models text generation via discrete, human-interpretable concepts and shows, under a diversity/invertibility condition, that accurate next-token predictors' internal features are (up to an invertible linear map and bias) equal to the log posterior over those concepts. This yields a unified explanation for linear probing, steering, and \"concept directions,\" and motivates a practical test (alignment between probe weights and concept directions). The authors add small-scale simulations, apply the alignment test to several LLMs with hand-crafted counterfactual pairs, and propose a structured sparse-autoencoder (SAE) plus a probe-based evaluation that outperform common SAE baselines on correlation/MSE.", "questions": "For the \"Structured SAE,\" the authors chose nuclear norm regularization to model concept dependencies. What was the intuition behind this specific choice (implying a low-rank structure on concept activations) versus other forms of structured regularization, such as imposing a graphical model prior on the features $\\mathbf{z}$ or using a group-lasso penalty? Just a curiosity rather than a weakness of the paper.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Unifying lens on ``linear representations.”:** The same matrix $A$ explains concept directions, steering, and probe linearity, which clarifies why these tricks often work together.\n\n**Simple checks in evals:** The $A_s W_s \\approx I$ is a neat diagnostic. I haven’t seen exactly this identity‑product visualization before. \n\n**Practical tie‑in to SAEs:** The theory suggests a supervised reference for evaluating SAE features and motivates a structured (sparse+low‑rank) variant that shows consistent gains\n\n**Cross‑model qualitative replication of concept–probe alignment**", "weaknesses": "**The diversity condition seems very strong and under-discussed**\nFrom what I understand, in Appendix D, the paper implicitly assumes $f_x(x), f_y(y) \\in \\mathbb{R}^\\ell$ so that $\\hat{L} \\in \\mathbb{R}^{\\ell \\times \\ell}$ exists with columns  $f_y(y_j) - f_y(y_0).$  \n\nThat means the representation dimension must be at least  $\\ell = \\prod_k n_k$  (the number of joint latent configurations). This seems like a very strong and unrealistic assumption and the manuscript never states this dimensionality constraint explicitly.  \n\nTo illustrate, even for a toy model with just 20 binary concepts ($k=20, n_k=2$), $\\ell = 2^{20} \\approx 1 \\text{ million}$. The representation dimension $d$ of a real LLM is large, but fixed (e.g., $d=4096$ or $d=8192$). In any realistic scenario, the representation dimension $d$ is *far smaller* than the number of possible joint concepts $\\ell$ ($d \\ll \\ell$).\n\nIf $d = \\mathrm{dim}\\, f_x \\neq \\ell,$ you need to formulate everything on the span of  $\\{ f_y(y_j) - f_y(y_0) \\}$ and use for example the Moore–Penrose pseudoinverse on a $d \\times \\ell$ matrix; otherwise the shapes don’t line up (you would have a non-square matrix). As written, the proof seems to treat $\\hat{L}$ as square without justifying $d = \\ell.$ (Same comment for $L$ as well.) The authors state in Appendix C that this is a ``weak assumption\" so please correct me if I'm not understanding something here. I think the proof should go through with a pseudo-inverse but I haven't worked out the details. \n\n**Again in appendix D, on the approximately invertible part:**\nI am not sure if the argument that the remainder term $\\mathbf{h}_y$ vanishes (Pg 20) is correct.\n\nThe term is $h_{y_j} = \\mathbb{E}_{p(\\mathbf{c} \\mid y_j)} [\\log p(\\mathbf{c} \\mid \\mathbf{x}, y_j)]$. The proof argues this vanishes because $p(\\mathbf{c} \\mid \\mathbf{x}, y_j)$ concentrates at a single $\\mathbf{c}^*$ (the $\\epsilon$-invertibility assumption).\n\nHowever, the expectation is taken with respect to $p(\\mathbf{c} \\mid y_j)$, not $p(\\mathbf{c} \\mid \\mathbf{x}, y_j)$. If I split the expectation:\n$$\nh_{y_j} = p(\\mathbf{c}^* \\mid y_j) \\log p(\\mathbf{c}^* \\mid \\mathbf{x}, y_j) +  \\sum^{\\mathbf{c} \\neq \\mathbf{c}^*} p(\\mathbf{c} \\mid y_j) \\log p(\\mathbf{c} \\mid \\mathbf{x}, y_j)\n$$\n\nAs $\\epsilon \\to 0$, we have $\\log p(\\mathbf{c}^* \\mid \\mathbf{x}, y_j) \\approx \\log(1-\\epsilon) \\approx 0$, but $\\log p(\\mathbf{c} \\neq \\mathbf{c}^* \\mid \\mathbf{x}, y_j) \\approx \\log \\epsilon$.\n\nThis seems to imply that the expression for $h_{y_j}$ becomes:\n$$\nh_{y_j} \\approx \\sum^{\\mathbf{c} \\neq \\mathbf{c}^*} p(\\mathbf{c} \\mid y_j) (\\log \\epsilon) \n$$\n\n$$ = (1 - p(\\mathbf{c}^* \\mid y_j)) \\log \\epsilon $$\n\nThis term clearly depends on $y_j$ (via $p(\\mathbf{c}^* \\mid y_j)$) and diverges to $-\\infty$ as $\\epsilon \\to 0$.\n\nTherefore, the difference $h_{y_j} - h_{y_0}$ seems to be:\n$$\nh_{y_j} - h_{y_0} \\approx \\big( p(\\mathbf{c}^* \\mid y_0) - p(\\mathbf{c}^* \\mid y_j) \\big) \\cdot \\log \\epsilon\n$$\n\nInstead of vanishing, this term seems to diverge, given that the diversity condition requires $p(\\mathbf{c} \\mid y_0) \\neq p(\\mathbf{c} \\mid y_j)$. The argument on Pg 20 that $\\mathbb{E}[\\log p(\\mathbf{c} \\mid \\mathbf{x}, y)] \\approx \\log \\epsilon$ \"not depending on $y$\" appears to miss the $(1 - p(\\mathbf{c}^* \\mid y_j))$ factor.\n\nThis feels like a significant error in the proof, so I wanted to check if I'm missing a different assumption. To make this term vanish, it seems you would need to assume $p(\\mathbf{c}^* \\mid y_j)$ is constant for all $j=0...\\ell$, which seems like a very strong constraint that might conflict with the diversity condition.\n\n\n**Some concerns about experimental evals**\nUsing the same pairs to build both $A_s$ (directions) and $W_s$ (probes) can inflate diagonal alignment. There should at least be some reporting over held-out concepts or perform cross-validation, and include null baselines (say, randomly permuted labels) to show that the effect isn’t a trivial consequence of using the same supervision twice.  \n\nAlso, quantifying ``approximate identity” by reporting, layer-wise and across seeds, something like diagonal/off-diagonal gap might help (just an idea).\n\n**Contextualize Structured SAEs**\nI like the Structured SAEs idea but low‑rank‑plus‑sparse decompositions with a nuclear‑norm regularizer have a deep literature (e.g., Candes et al., ``Robust PCA”; many variants since). Bringing L+S to SAEs is reasonable but should be contextualized and credited; please cite this prior line of work explicitly (no relation to me) and clarify what is new here (architecture? training recipe? results on LM activations?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SSIZkvfZd6", "reviewer": "Reviewer_wuUB", "summary": "This paper theoretically shows that next-token prediction leads to linear identifiability of data-generating latents under a fairly general data model. In particular, such identifiability could be obtained when the generation function is non-linear and non-invertible, and when the latents are discrete and non-Boolean. Motivated by this result, the paper proposes improved techniques for evaluating and training sparse autoencoders and empirically evaluates the proposed method both in synthetic settings and on practical LLMs.", "questions": "- While discrete concepts are beneficial for modeling text semantics such as topics, there may also be continuous concepts in the text, such as those relevant to numbers. Can the identifiability result be extended to modeling both discrete and continuous latents?\n- What is the difference between the proposed SAE evaluation approach and existing probing approaches, e.g., the one used in [5]?\n- Why does low-rank regularization help model the interdependence between latent variables/concepts in structured SAE?\n\n---\n\n[5] Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Compared to existing analysis, the identifiability result proved by this paper is more general; the introduced diversity condition makes sense to me and is likely to hold in practical pre-training of LLMs. Showing approximated identifiability in the non-invertible case is also a decent complement to the results.\n- The paper translates the theoretical results to practical algorithms on SAEs.\n- The paper is well-written and easy to follow.", "weaknesses": "- A major drawback of the results, like existing ones, is that linearity is defined only for the output activations (i.e., in the unembedding space) of LLMs. Yet, prior empirical work on the linear representation hypothesis often shows that linearity is usually formed in the _intermediate_ layers [1, 2]. For some concepts, such intermediate-layer linearity could even fade away at the output layer [3], which cannot be captured by the proved results. Although I agree with the authors that rigorously analyzing linearity for intermediate layers is (much) more difficult, I feel it is still necessary to mention it here since it may be of more interest for both theorists and practitioners (e.g., SAEs are often trained on intermediate-layer outputs rather than last-layer outputs), in my opinion.\n- The relation between theoretical results and the proposed structured SAE is rather loose (see Questions for more details).\n- (Minor) Missing related work: [4] also proves linear identifiability of latent variables under a general setting and discusses the implications for the linear representation hypothesis.\n\n---\n\n[1] Language models represent space and time. ICLR, 2024.\n\n[2] Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023.\n\n[3] The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. COLM, 2024.\n\n[4] When do neural networks learn world models? ICML, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZmqFAFHyG", "reviewer": "Reviewer_2RWt", "summary": "This paper proposes a theoretical framework connecting next-token prediction with latent variable identifiability. It argues that the representation in large language models approximates a linear transformation of the log posterior of latent concepts. The authors derive this relationship analytically and validate it through small-scale experiments showing linear alignment between counterfactual representation differences and classifier weights.", "questions": "* Could the authors please clarify how the theory (theorem 3.1) can hold in the realistic setting where $d \\ll l$?\n* The theory requires $l+1$ distinct $y$ tokens. How can this condition possibly be satisfied in practice, where $l$ is combinatorially large and $l \\gg |V|$?\n* The latent variable model assumes $x$ and $y$ are conditionally independent given $c$. How does this simplification, which breaks the autoregressive nature of language, where $x$ (past tokens) is the direct causal parent of $y$ (the next token), affect the main claim? Would the linear identifiability result still hold under a more standard model?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper provides an interesting conceptual link between next-token prediction, linear probing, and identifiability theory.\n* It provides a single, unified mathematical explanation for a diverse set of empirical phenomena (linear probing, concept steering, and vector arithmetic).\n* They test their theory on synthetic data where the ground truth $c$ is known, and then successfully test its corollaries (e.g., $A_s W_s \\approx I$) across multiple LLM families and sizes.", "weaknesses": "The paper presents an interesting finding that LLM representations could be linear latent concept posteriors. However, some assumptions may be too strong and make the analysis flawed.\n*  The main theorem (Theorem 3.1) critically depends on the invertibility of the matrix $\\hat{L}=[f_{y}(y_{1})-f_{y}(y_{0}), ..., f_{y}(y_{l})-f_{y}(y_{0})]$, which implies that the representation dimension d must equal the total number of latent configurations $l$. In practice, $l$ grows combinatorially explosive (e.g., $2^m$ for m binary concepts), while d (e.g., 4096) is a fixed hyperparameter. This d<<l seems to invalidate the proof. \n* Also, the diversity condition, requires the existence of $l+1$ distinct values of y for the matrices L and $\\hat{L}$ to be constructed.  $l$ is combinatorially explosive (e.g., $2^m$ for m binary concepts) and will be much larger than the vocabulary size |V| for any non-trivial concept space. And it is impossible to find $l+1$ if $l$ > |V|.\n* The latent variable model is  $p(x,y)=\\sum_{c}p(x|c)p(y|c)p(c)$. This assumes that the context $x$ and the next token $y$ are conditionally independent given the latent concept $c$. This is a simplification that breaks the autoregressive nature of language, where $x$ (past tokens) is the direct causal parent of $y$ (the next token).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cx3b1bDmJJ", "reviewer": "Reviewer_YUaf", "summary": "The authors study why next-token prediction (NTP) models appear to learn human‑interpretable concepts. The authors posit a discrete latent-variable generative model in which concepts c (discrete, possibly with arbitrary causal structure) generate input x and target y. Under a diversity condition and without requiring invertibility of the mapping from c to observations, they prove an identifiability result: the model’s representation f_x(x) is (approximately) an invertible linear transform of \\log p(c \\mid x) up to additive terms. This provides a unified account of “linear representation” phenomena (steering vectors, linear probing) and motivates a principled evaluation of SAEs by comparing their features to linear-probe estimates of concept posteriors. The empirical section includes synthetic validations and tests across Pythia, Llama, and DeepSeek families.", "questions": "* Can you provide sufficient conditions (in practice) under which the diversity condition holds for token distributions in large corpora?  \n* How large is the h_y correction term in typical NTP models? Can you report empirical magnitudes and the impact on linearity fits?\n* In the SAE evaluation recipe, how sensitive are conclusions to the quality of linear probes (regularization, class imbalance, definition of concept labels)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Clear theoretical link from NTP to concept posteriors via a linear transform (Theorem 3.1), with corollaries explaining steering/probing phenomena.\n* Actionable implication for evaluating SAEs by benchmarking against linear-probe posterior estimates.\n* Good validation practices: synthetic and multiple model families (Pythia, Llama, DeepSeek).", "weaknesses": "* Assumptions strength: The diversity condition is nontrivial and may not hold; more intuition or empirical checks would help.\n* Residual terms: The correction term in Theorem 3.1 (involving h_y) complicates the neat linear story; Can you quantify when is it negligible?\n* Empirical depth: SAE evaluation is promising but would benefit from a broader battery (multiple SAE variants, structured sparsity, stability across seeds).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)