LLM Interpretability with Identifiable Temporal-Instantaneous Representation
Authors: Xiangchen Song
Jiaqi Sun
Zijian Li
Yujia Zheng
Kun Zhang
Venue: NeurIPS 2025 poster ()
Keywords: identifiability
causal representation learning
large language models
interpretability
Primary area: deep_learning

Abstract: Despite Large Language Models' remarkable capabilities, understanding their internal representations remains challenging. Mechanistic interpretability tools such as sparse autoencoders (SAEs) were developed to extract interpretable features from LLMs but lack temporal dependency modeling, instantaneous relation representation, and more importantly theoretical guarantees—undermining both the theoretical foundations and the practical confidence necessary for subsequent analyses. While causal representation learning (CRL) offers theoretically-grounded approaches for uncovering latent concepts, existing methods cannot scale to LLMs' rich conceptual space due to inefficient computation. To bridge the gap, we introduce an identifiable temporal causal representation learning framework specifically designed for LLMs' high-dimensional concept space, capturing both time-delayed and instantaneous causal relations. Our approach provides theoretical guarantees and demonstrates efficacy on synthetic datasets scaled to match real-world complexity. By extending SAE techniques with our temporal causal framework, we successfully discover meaningful concept relationships in LLM activations. Our findings show that modeling both temporal and instantaneous conceptual relationships advances the interpretability of LLMs.

LLM Interpretability with Identifiable Temporal-Insannaneous Representation
Xiangchen Song t.l Jiaqi Sun
Zijian Li
Yujia Zheng'
Kun Zhang
CCarnegie Mellon University
2 Mohamed bin Zayed University of Artificial Intelligence
(xiangchensong, jaaponggiiiiuuunn
siaqisun, kunzi (aanzi)/emuu.... @cmu.ede
Abstract
Despite Large Language Models' remarkable capabilities, understanding their internal representations remains challenging. Mechanistic interpretability tools such as sparse autoencoders (SAEs) were developed to extract interpretable fea- tures from LLMs but lack temporal dependency modeling, instantuneous relation representation, and more importantly theoretical guarantees unneemminnnnniinnnnngge both the theoretical foundations and the practical confidence necessary for subsequent analyses. While causal representation learning (CRL) offers theoretically-gree groundddd approaches for uncovering latent concepts, existing methods cannot scale to LLMs' rich conceptual space due inefficient computation. To bridge the gap. we intro- duce an identifiable temporal causal representation learning framework specifically designed for LLMs' high-dimensional concept space, capturing both time-delayed and instantaneous causal relations. Our approach provides theoretical guaran- tees and demonstrates efficacy on synthetic datasets sealed to match real-world complexity. By extending SAE techniques with our temporal causal framework, we successfully discover mcaningful concept relationships in LLM activations. Our findings show that modeling both temporal and instantancous conceptual relationships advances the interpretability of LLMs.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language tasks, from question answering content generation. Despite these achievements, fundamental understanding of their internal representations remains underexplored. This gap between performance and interpretability poses significant challenges for ensuring the reliability, safety, and appropriate deployment of these increasingly powerful systems 47]
Mechanistic interpretability (MI) aims to bridge this gap by reverse-engineering neural networks understand how they process and represent information [14). Among all MI tools, sparse autoencoders (SAEs) have emerged as a promising approach for extracting interpretable features from LLMs
By decomposing the high-dimensional activations of LLMs into sparse, monosemantic features, SAES help identify the basic units of computation within these complex systems. However, SAES present several limitations that restrict their utility for comprehensive model understanding:
14 152.
First, SAEs treat each feature one another. This omission disregards semantic connections and transitions within a sequence, which are known as temporal or time-delayed relationships between feature Second, SAEs lack mechanisms to represent instantaneous or logical relationships between features, such as mutual
isolated representation, failing to capture how features influence
Equal contribution.
Altemattve approaches, such
30
attention scores from the LLM infer time- iine-delayed influence.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

exclusivity or co-oceurrence constraints [32]. These relationships complement the temporal dynamics by encoding structural dependencies within the same time step. Third, and most critically, SAEs offer no theoretical guarantees of the uniqueness of the recovered features. This absence undermines confidence that the extracted features reflect meaningful and stable latent variables, rather than arbitrary or unstable transformations [54]
Fortunately, to address these limitations, the causal representation learning (CRL) community has proposed a range of promising frameworks with theoretical guarantees [47]. For instance, [27] and 31 use sparse causal influence and interventions uncover temporal and instantaneous relationships among latent variables. However, these methods face significant scalability challenges due to the computational inefficiency of estimating Jacobians. As result, they typically scale to only dozens or hundreds of concepts [56], while interpretability in LLMs demands efficient modeling of thousands or even tens of thousands of concept features [52]. summary, although CRL offers strong theoretical guarantees for recovering meaningful features and their causal relationships, its limited scalability in high-dimennional settings remains a major obstacle to practical deployment in LLM analysis.
To bridge this gap, in this paper, we introduce computationally efficient temporal causal repre- sentation learning framework specifically designed for the high-dimensional activation space in LLMs. Our approach builds upon recent advances in both sparse autoencoders for LLMs and causal representation learning for sequential data. The key contributions of our work are:
(1) We propose a simple yet effective framework that jointly models eeee-eelayed causal relations between concepts and instantaneous constraints, providing more comprehensive understanding of how information flows through LLMs.
(2) Leveraging sparsity principle, we establish theoretical guarantees for approach, making the representations learned reliable and explainable.
(3) Grounded in the theoretical result, we design scalable and efficient algorithms tailored to the high-dimensional concept space of LLMs, significantly extending prior work in CRL.
(4) We validate our approach on synthetic datasets scaled match real-world complexity and demonstrate its effectiveness when applied to activations from real LLMMs.
2 Problem Setting
We begin by characterizing the generation process of LLM activations to establish interpretability guarantees. These activa- tions-sisn signals produced during inference- widely assumed to be linearly generated from hidden concepts, consistent with sparse autoencoder (SAE) literature 3 [8] However, existing formulations typically treat these concepts as independent, over- looking dependencies between them. In reality, earlier-token semantics often influence later tokens, and token generation de- pends jointly on the activation of multiple concepts. To account for these interactions, we introduce a data generation process with both temporal and instantaneous relations, adopting CRL terminology. Given a token sequence s (Ui...., Uk), let be the re-dimensional activation vector token for specific layer. Following the linear representation hypothesis 43 and SAE formulation
Latent Variables
Mixing Function Time-delayed
Observed Variables
Instantaneous
x, (Ft.i....,, ,
Figure 1: Graphical illustration of the data generation process.
18, we assume:
X g(:)),
where g R. R the linear mixing function, and x, and z, are observed and latent variables. respectively. Besides, each latent variable 21,; is governed by structural equation model (SEM) capturing both time-delayed and instantaneous dependencies:
Σ Σ Bij.T
Σ M 61,1 JEK.
21,
+
JEJL.
where B.G.r represents the coefficient for the time-delayed effect from zt-Ty to 24,4 Ji,r is the set of indices of latent variables that have time-delayed effect 24, with lag T; Mij represents the

---

coefficient for the instantaneous effect from žt,j to 21,6 KG is the set of indices of latent variables that have an instantaneous effect on 2 and e,, denotes the temporally and spatially independent noise extracted from a distribution Pe.. The graphical model for this process is illustrated in Figure !
To better understand this data generation process in the context of LLM activations, x; represents activations in a specific layer 1 for token vt, and the latent variables Ze can be considered as the underlying causal factors that generate these activations. In this case, the instantaneous effects (coefficients M,,) reflect semantic or syntactic relationships between different latent factors within the same token, while time-delayed effects (coefficients B,,j.r) represent dependencies on previous tokens. Putting them together, the underlying data generating process can be written as
X Az
2 Σ Σ
B B.jr2e-r + Σ M.jar Ca,i, I...., m. JEK.
Linear mixing
T>OJEJt.
Linear latent temporal SEM
The linear latent temporal SEM in Eq. ( induces two types of causal relationships: time-delayed causal graph Ga. in which an edge zt- - 21, exists if and only if B * 0, and an instantaneous causal graph Ge. in which edge Zt.j + Ze,i exists if and only if M.j * O. We assume that Ge is acyclic, i.e., a directed acyclic graph (DAG), which implies that M can be permuted to a strictly lower- triangular form. Under this assumption, the conditional distribution of 2, given its past satisfies the Markov property with respect to Ge [5 namely p(z, k21) = II= p(zz, Pa.((e,)); Pag (2,,))
Remark on the Linearity of the Model We acknowledge that the internal mechanisms of LLMs are inherently nonlinear due to activation functions and attention mechanisms. However, our linear approach is justified by several considerations. First, many successful mechanistic interpretability
30 rely on linear representation hypotheses as approximations of localized network behavior. Second, linear models provide an interpretable bridge between the complexity of neural networks and human understanding- -tey serve as simplified yet informative projections of the underlying causal mechanisms. Third, empirical evidence suggests that linear approximations can capture significant portions of variance in activations within specific contexts 37 19, particularly when examining ceature-to-faature relationships within layer.
techniques [15] 42 TO
More importantly, existing causal representation learning (CRL) methods cannot efficiently handle hundreds of latent variables, often encountering out-of-memory issues and prohibitively long compu- tation times. detailed discussion of these limitations is presented in Section 5.1 While nonlinear interactions certainly exist, our linear framework offers a tractable foundation for identifying causal relationships that can later be extended to incorporate more complex dependencies. This approach follows the scientific principle of starting with simpler models that capture essential phenomena before introducing additional complexity.
3 Theoretical Guarantees
Recent work in causal representation learning, particularly for time-series data, has made substantial progress in handling both time-delayed and instantaneous causal relations. Under general assumptions on the data-generating process, strong identifiability results can be established, including recovery of latent variables up to component-wiss transformations and estimation of the Markov network up to isomorphic equivalence. However, in our linear setting, the identifiability result of (27) is not directly applicable, since one key assumption (sufficient changes) cannot be satisfied; see Appendix [. for a detailed discussion. Therefore, in this section we establish identifiability by exploiting the autocovariance structure induced by linearity, following ideas inspired by (58).
Theorem 1 (Latent Indeterminacy). Suppose the estimated model (A, (Br M. P2) and the true model (A, (B,},-, 1,-,,,,, M. Pe) both generate x, according to Eq. 3and are observationally equivalent on the autocovariance matrices Rx (k)) for k: 0, 1...., L. If conditions A1-A4 hold, then the model parameters are identifiable up to the following indeterminacies:
é PPE, À A= AS, (I-M) = P P(I B, = -P,SS
where S E RR is invertible and P is a signed permutation matrix.
Al (Temporally white noise). Both and e, are temporally white, and has i.i.d. mutually independent components. To remove scaling indeterminacy, assume De, I and zero-mean data.
3

---

A2 (Rank sufficiency). A E Rm (m 2 n) has full column rank, and BL is full rank.
A3 (Process stability). Eq. _2 defines a stable vector autoregression, i.e, all roots of det(I - det(I-Ei=)B B) 0 lie strietly inside the unit circle.
A4 (Non-Gaussiantty) At most one component of Et is Gaussian.
Proof Sketch The proof consists of four steps. First, using the VAR representation of and the linear mixing X Az, we derive Sill-Walle--type recursions for the autocovariances R (k),). yielding a linear system whose unknowns are the transformed coefficients C, = A(I-M)--B,A Second, by stacking lagged observations, we show that the system' coefficient matrix equals the covariance of finite stacked vector of {xt ). which is positive definite-and thus nonsingular-under the assumption no nontrivial deterministic finite linear relations. This ensures identifiability of C, and of HH A(I M) -I M)- A Third, exploiting column-space arguments and invertibility of I M, show that any observationally equivalent models must satisfy Å = AS, (I- M) U2 (I M)S, and B, U B, where is invertible and U is orthogonal. Finally, non Gaussiaisyy of the innovations implies that U must be a signed permutation matrix, yielding precise characterization of the remaining indeterminacies.
Discussion This result extends [58] to temporal processes with instantancous relations. When M * 0, the indeterminacy of A increases due to the additional mixing introduced by M, changing the ambiguity from an orthogonal transformation to a general invertible one. Consequently, both I and B, inherit sighttluulipliaativve indeterminacies. Without further structural assumptions on M Br component-wise identifiability is impossible. However, since the indeterninacies are precisely characterized, additional structural constraints be imposed further improve identifiability, as shown in the following corollaries.
Corollary (Component-wise Identifiability). Suppose the estimated model (A, (Br 1M, and the true model (A, (Br) M. pe) ) both generale x, according Eq. and are observationally equivalent on the autocovariance matrices Rx(k) for k = 0, 1...... L. If conditions Al-A4 of Theorem
7 hold and, in addition, the following assumption on the instantaneous relations M is satisfied, then the latent variables z, are identifiable up to permutation and scaling, i.e., Å APD. where P a permutation matrix and D is diagonal matrix with nonzero entries.
A5 (Empty or unique column supports of M). Each column of M either has empty support contains at least one index that does not appear in the support of any other column. The model is estimated under a sparsity constraint on M.
M) = P(I M)S. Since only permutes rows, it does not affect column-suppor arguments. Under Assumption A5, any column of M has unique nonzero support element, which prevents cancellation across columns. If column S., contained more than one nonzero entry, then P(I - would necessarily have strictly larger support than the corresponding column of M, violating the assumed sparsity. Since M has zero diagonal (by the SEM assumption). the same argument applies to M. Therefore, sparsity-enforee estimation excludes such S, implying that S must be product of permutation and diagonal scaling matrix. Hence, 2, is component-wisg identifiable.
Proof. From Theorem we have A AS and (I-M)
Corollary 2 (Subspace Identifiability). Suppose the estimated model (A, [B M.pe) PE) and the true model (A, (B, M. Pe) both generate x, according Eq. 3 and are observationally equivalent on the autocovariance matrices Rx(k) for k 0, 0. L. If conditions Al-A4 of Theorem 7 hold and A6 is satisfied, then the latent variables 2, are subspace identifiable, i.e. À AS, where each column of S has nonzero entries only within a single subspace.
A6 (Subspace structure of M). The instantaneous relation matrix M admits a partition of [n] into K disjoint subsets such that M,, + 0 only if and j belong to the same subset. The model is estimated with a sparsity constraint on M.
Proof. From Theorem we have À AS and (I - M) P(I M)S. Assumption A6 implies that columns of M corresponding to different subspaces have disjoint supports. Since M has zero

---

diagonal, the same property holds for I - M. If any column S!, mixed components from different subspaces. then the corresponding column of (I MJS would necessarily contain nonzero entries from multiple subspaces, contradicting Assumption A6. Therefore, each column of S can only combine latent variables within the same subspace, implying subspace identifiability 21.
Discussion Corollary D strengthens Theorem Ty imposing a strong structural assumption (A5) on M, yielding component- wise identifiability. Corollary 2 relaxes this requirement by allowing instantaneous relations within latent subspaces, leading to subspace identifiability. Both results rely on sparsity constraints imposed during estimation. Moreover, each corollary admits a natural analogue for the lagged matrices B,. In particular, the counterpart of Corollary requires nonzero diagonal entries B,, while the analogue of Corollary2aassumeee a matching subspace structure. Furthermore, if M is forced to be strictly triangular, the right permutation indeterminacy must match the left one, i.e., P = P. otherwise the strictly triangular structure will be destroyed. In practice, we therefore enforce sparsity on both instantaneous and lagged relations and strictly lower triangular structure the instantaneous adjacency. Empirically, the recovered structures from real data approximately satisfy the assumed conditions, while synthetic experiments explicitly enforce them to validate identifiability.
Implementation
Based the data generation process in Eq. (3) together with the identifiability result presented in the previous section, we derive the following estimation process based on the standard sparse autoencoder. Illustrated in Figure 2 the whole estimation process be partitioned into three parts, namely (1) observation reconstruction, (2) independent noise estimation, and (3) sparsity regularization.
4.1 Observation Reconstruction
Time-delayed i, Instantaneous. M
Independent Estimation
First, we use a linear autoencoder to enforce the invertible linear transformation between observations x; and latent variables z,, and the reconstruction loss L, is defined as
En
(21, 22 22..... 2T)
C = Exi.T
Σ
T
xx - x
(4)
Encoder
Decoder
1=1
(x1, X2. XT
(81,
&r)
where the reconstructed observation is calculated via linear encoder and decoder:
Figure 2: Illustration of estimation pro- cess. B, represents the learned time-
delayed causal relation and M the in- stantaneous causal relation.
& Decoder(2.) and 2, Excoder(xx).
4.2 Independent Noise Estimation
In prior works [66 49 27 this terms refers the independent prior estimation, in which they essentially utilize the independence of noise to enforce the independence of latent variables 2,,5 conditioning on parent Pa(2,,). our case, since the whole process linear, we can directly estimate and enforce the independent noise condition by learning residual network by reversing the data generation process described Eq. 3
é = 2, M ΣB, 2.-
(6)
where the estimated latent variables are given by Eq. (5) Following the prior works, to enforce the independence of noise terms, model the noise distribution p(ε.,) with isomorphic Laplaciar distribution, and we minimize its KL-divergence with the estimated noise term.
C, Ez, [1])
"In prior works, the distribution is Gaussian, however, we can see that in linear case as is well discussed in linear ICA literature, the density function of an isomorphic Gaussian distribution is rotation invariant, hence we utilize the Laplacian distribution in our estimation.
5

---

4.3 Sparsity Regularization
Without any further constraint, the noise estimation module may bring redundant causal edges from ž--1. fm to s, leading to the incorrect estimation. As mentioned in Sec. [4.2] {B,} and M intuitively denote the time-delayed and instantaneous causal structures, since they describe how the 1, Re, [m]m contribute to E,, which motivate to remove these redundant causal edges with
sparsity regularization term L, by using the Ll penalty on {B,} and M. Formally, we have
C. Σ
||M]]
8
where || [1 denotes the LI Norm of a matrix. And we restrict the M to be strictly lower triangular to match the permutation indeterminacy on both sides of M and B, Finally, the total loss of the model can be formalized as:
Liotal for + oL + BL.;
where ar denote the hyper-parameters.
Experiments
Our experimental evaluation addresses five key claims regarding our proposed method: (1) our estimation approach aligns with identifiability theory, accurately recovering latent structures; (2) existing CRL methods fail to handle high-ding sigh-dimensional data at scale; (3) our method is able to recover target relations between concepts from semi-synthetic data; (4) compared with common SAEs, our proposal achieves satisfactory results on quantitative evaluation metrics (SAEBench [24]): and (5) our method effectively learns both time-delayed and instantaneous causal relations among concepts elicited from LLM activations.
5.1 Synthetic Data Experiments
First, using synthetic data, we demonstrate that our method can recover both the latent variables Zz and the causal structure including time-delayed relations B, and instantaneous relations M.
Identifiability Verification To establish the effectiveness of our approach, we generate simulated time series data with latent causal process introduced in Eq. B. We apply our method single time lag synthetic data generated with a randomly initialized matrix A and fixed transition matrices B and M visualized in Figure 3a and 3c Further details can be found in Appendix A.3
We visualize the estimated parameters by plotting the recovered matrices B and M alongside the correlation coefficient matrix used for calculating the mean correlation coefficient (MCC) score.
As shown in Figure comparing with the ground truth transition matrices B and M, we observe that both time-delayed and instantaneous causal relations have been precisely recovered. Furthermore, Figure Bel demonstrates that the latent variables 2, are also accurately recovered, confirming the identifiability properties of our method.
Ground
Estimated
Gronnd
Estimated XY
Correlation Coeffleieut Malris
4, 0.40
0.60
0.00
0.05
0.00
0,00
0.00
1.00
0.39
0.27
2, 0.00
1.00
0.00
0.20
0.00
0.80
3e.5
0.39
1.00
0.36
4,3
63
Feg
0.25
0.34
1.00
Lc-1i 21-1.2
46-15
Rt-11 21-12 21-133
(a)
(b)
(c)
(d)
(e)
Figure 3: Visualization of recovered causal graphs of latent variables. (a) and (b) show the ground truth and estimated time-delayed matrix, respectively. (c) and (d) show the ground truth and the estimated instantaneous causal relations, respectively. (e) displays the correlation between the ground truth and recovered latent variables.
Second, we scale the synthetic experiments to dimensions matching L.LM activations, illustrating why existing CRL methods fail in these high-dimensional settings.

---

Challenges on Scaling to Large Language Model Activation Dimensions Before presenting results on expanded synthetic data, we investigate the computation bottleneck: Jacobian calculation and explain why existing CRL methods do not extend efficiently to high-dimensional settings, thereby further motivating our use of a linear dynamical model.
120
0.30
13.0
Time (ho urs)
Time (9)
7.5
Memory
5.0
0.50
200
600
600
800
1000
200
6.00
600
800
3000
0.50
255
500
750
1000
Dimension
Dimension
Dimension
Figure 5: Computation time and memory usage for a single-step Jacobian function of input dimensionality. Both metries grow superlinearly and exceed the capacity of modern GPUs when the input dimension is greater than 1000.
Figure 6: MCC and total compute time in hours required train the linear model as function of input dimension.
Computation cost of Jacobian evaluation. We take IDOL 27 as representative method and measure both the wall-clock time and memory requirements for computing the Jacobian in prior network. Figure 5 demonstrates that both time and memory complexity grow polynomial with dimensionality. At dimensions of several moousnnd-wiih are common for Large Language Model activations single Jacobian evaluation will require approximately ten seconds modern GPU, such computation cannot fit into current generation hardware infrastructure. Since CRL training invokes this operation millions of times during the training process, the cumulative computational cost becomes prohibitive. As other CRL algorithms involve comparable Jacobian computations or more complex algorithms, this fundamental limitation applies broadly across the field.
Advantages of linear models. When linear model provides adequate approximation of the transition dynamics of the hidden concepts, the Jacobian calculation can be derived directly from model parameters such B and M, which significantly reduces the computational burden. Furthermore, such a linear model can scale efficiently with current-generaticn compute resources. To support this claim, we conducted scaling experiment using the linear model on synthetic data with dimensionalities ranging from 128 1024. In each setting, the model was trained on 50 million samples, simulating the typical training load in real LLM SAEs with 50 million tokens. As shown in Figure 6 the proposed method scales to substantially higher dimensions while maintaining a high MCC of approximately 0.9. Additionally, remains computationally efficient, with total computation time scaling linearly. In contrast, IDOL (27 exhausts memory when the dimensionality exceeds 200, and iCITRIS [1 fails scale beyond 16 dimensions.
5.2 Semi-synthetic Experiments
Given the previous experiments on synthetic data, our proposal has been shown to recover ground- truth relationships even when the hidden dimensionality reaches one thousand, which would be challenging for existing non non-paraman con-paraettric CRL approaches. We now proceed to evaluate real-world LLM activations, beginning with investigation (3). The experimental settings are briefly introduced below. Full details can be found in Appendix A.4.1
Table Relation recovery scores (1) for concept-relation extraction on semi-synthetic data.
Method
Legal
XML
Email
SAE+regression
0.54
0.94
0.74
Ours
19.95
8.63
2.66
Data preparation: We first examine three types of text, each exhibiting an obvious syntactic pattern. For example, in legal text, sequences often begin with "APPEALS" and end with "AFFIRMED". For illustration, we focus on the legal text contrastive corpus group. We constructed two contrastive subsets from the Pile dataset [17]: one containing legal documents with highly structured syntax and
7

---

Time-delayed Relation
Instantaneous Relation
Enc
APPEALS
AFFIRMED
Mich.
ID: 2579
ID: 1894
ID: 2592
ID: 2623
legal process where a party who not salisfied asks a nigher court to review the
The higher court has reviewed the lower court's decision and agrees with it. The original decision
The geographical location associated with the party handling case.
The location of the court along with partial information from the case identifier.
decision
or
Example Text:
United States Court of APPEALS E. D.
Mich.
AFFIRMED
Figure 7: Case study illustrates two relation types identified a United States legal text. The blue elements show time-delayed relation: the term "appeals" is typically followed by "affirmed" when a higher court confirms the lower court decision. The red elements show an instantaneous relation: two geographical location concepts (2592, 2623), activated together in the same passage.
stable temporal patterns, and the other containing unstructured non-legal text. We hypothesized that only texts containing these structured relations would yield meaningful temporal concept patterns and tested whether the model could recover them. Baseline: Since no directly applicable baseline exists, we used the standard SAEs trained above. As SAEs cannot capture concept-to -ecpcptp---cccpttttcce relations,
fitted regression model to estimate temporal relation matrices B via z, Σx B, Z Evaluation: We compute the concept recovery score by first identifying the top concept pair (i, in legal contexts (ensuring that the corresponding concepts do not fire in the non-legal text), then taking the corresponding coefficient B.d and normalizing by the standard deviation (B)) The ratio B. o(B) serves as a relation recovery score, indicating how strongly the relation stands out from noise. As shown in Table [ our method achieves significantly higher score, demonstrating successful recovery of the relation. Finally, as concept recovery is already achievable by standard SAEs, we additionally conducted steering vector semi-synthetic experiments to verify that our proposal can also recover concepts, following the approach of [23]. Further details are provided in Appendix A.4.2
5.3 Real LLM Activation Analysis
Experiment Setup train our linear model on activations from the pretrained LLM (rohia-160m-deduppe [5]. using SAELens [6] and dictionary--
(35) for activation
extraction. The model is trained on 50 million tokens from the Pile dataset [17]. To capture time-delayed influences, we < 20 in Eq, 3 and aggregate the B, matrices using max- max-pooling, preserving any causal link that appears at any time step. We evaluate three feature dimensionalities: 768 (matching the LLM's hidden size and aligned with Section 33. 3072, and 6144 the latter two following common SAE training setups. Unless specified, main text examples use 3072-dimensional features with 20. Full training details and additional results (sensitivity and ablation studies included) are in Appendix A.5
Table 2: Comparison of method against ReLU and TopK SAES SAEBench metrics.
Model
Recon. Loss 4
Sparse Prob. t Absorp. +
Autointerp
ReLU SAE
0.01
0.6555
0.0141
0.6791
TopK SAE
0.0097
0.7141
0.0280
0.6822
Ours
0.0108
0.6736
0.0139
0.6883
Quantitative Evaluation on SAEBench Before we dive into the details of concept relation recovery, we first present quantitative comparison between our method and existing SAE approaches. Since our main contribution lies in recovering temporal and instantaneous concept-to-concen relations, which not reflected in current SAE benchmarks, established SAEs on SAEBench tasks. This expectation is confirmed by the results in Table 2 Additional experiment results on larger latent size and model size can be found in Appendix A.5.5
expect model to perform par with
Case Studies We start with an illustrative case in Figure 7 demonstrating how our model uncovers interpretable concept features with both time-delayed and instantaneous causal relationships from real-world LLM activations. This example provides integrated view of how concepts are structured
8

---

Table Representative time-delayed and instantaneous concept relations discovered.
ID
From
ID
To
Coeff.
Time-delayed relations
Keywords for formal and official content (e.g., senate, state, military)
Verbs for official/formal usage (e.g-- deny, press, order, sign)
1657
1664
0.99
Adjectives of nationality (e.g. Japanese, Italians)
Nouns that follow nationality (e.g. brands, name)
2641
2674
0.81
Keywords for formal and official content (e.g., senate, state, military)
Objective adjectives in formal usage fe-g.ffatt, continuous, incomplete)
1657
1124
0.74
Instantaneous relations
Partial appellate citation with volume number
Partial appellate citation with volume number and series index
2208
227
0.23
Coding- format signals (e.g. localization tags, HTML tags)
Coding
Coding-format format content (c.g.
1714
80
0.16
key-value pairs, HTML. elements)
1582
Month (e.g. March)
363
Full date (e.g. March 23, 2000)
0.02
over time and interact within single time step. Note that feature interpretations may vary beyond this case; additional examples and discussions appear in Appendix A.5
Figure 7 [highlights two key observations: First, a time-delayed causal link between concepts related to "appeals" and "affirmed" in legal texts (features 2579 and 1594), capturing how the model reflects the procedural flow of legal judgments. Second, an instantaneous relation between two geographical location concepts (features 2592 and 2623) that are activated together in legal passages, suggesting that the model represents related spatial information simultaneously rather than sequentially. This example effectively demonstrates that both time-delayed and instantaneous relations exist among concept features, and that these are interpretable alongside the semantic meanings of the features-botl of which are essential for LLM interpretability. To further demonstrate our model's 's capacity uncover both types of causal relationships, we present a broader set of examples in Table 3 which showcases representative cases of both time-delayed and instantaneous interactions among concept features.
Time-delayed causal relations. We first observe a strong causal relation from nationality adjectives (feature 2641, "Japanese." "Italians") to the nouns they commonly modify (feature 2674, "brands," "literature"), with a coefficient of 0.81. Moreover, the coefficients across the 20-token temporal window (i.e., different B,) contribute consistently to the aggregated score. This suggests that such temporal relations can occur across a broad and uncertain time span, aligning with the semantic dynamics of real-world text generation. In formal contexts, official content words (feature 1657, "senate," "judge"") influence both formal verbs (feature 1664, "deny." "order") with a coefficient of 0.99, and objective adjectives (feature 1124, "fast," " "continuous") with coefficient of 0.74. These relationships reflect how formal language constrains both action and descriptive style over time.
Instantaneous causal relations. Table 3 three distinct categories of instantancous relations. First, we observe relationship between two partially overlapping appellate citation -eatures-fcature 2208 (volume numbers only) and feature 227 (volume number and series index) with normalized coefficient of 0.23. This illustrates how the model captures structured elements that commonly co-occur in legal documents, forming cohesive representational unit. Second, we find that coding- format signals (feature 1714, e.g., localization tags, HTML tags) have an instantancous causal relationship with coding-format content (feature 80, e.g., key-value pairs, HTML elements), with a coefficient of 0.16. This reveals how the model processes structured syntax and its associated content as co-occurring elements. Finally, our method identifies clear relationship between two features that both represent dates: feature 1582 (month only) and feature 363 (full date), suggesting complementary representations within the model's ' internal structure.
These findings demonstrate our method's ability to uncover both temporal and instantaneous causal structures in the concept space of LLM activations, offering insights into how models organize and process information. The identified relationships align with expected patterns natural language across domains such legal texts, temporal expressions, and structured formats, validating the effectiveness of our approach for analyzing information flow in large language models.

---

Related Work
LLM Interpretability Understanding the internal representations of LLMs remains challenging despite significant progress [26]. Interpretability research on LLMSs has explored multiple directions including: probing for linguistic knowledge [19], evaluating interpretability methods through con- trolled experiments (22), benchmarking SAEs' capacity to disentangle factual knowledge [12]. and developing ground-truth evaluation frameworks (53 25]. Recent work suggests that LLM repre- sentations may follow a linear organization 433 though this hypothesis has been challenged [16]. Our approach extends these efforts by focusing specifically on causal interpretability of temporal relationships in LLMs, providing a principled framework for understanding how information flows through model representations during sequential text generation. Additionally sparse autoencoders (SAEs) decompose neural activations into interpretable features [14 7 52 Initial work demon-
strated that SAEs can recover meaningful features from language model activations 41 leading
to numerous architectural innovations including alternative activation functions [50, 46 training
optimizations [8] 20], and efficient dictionary allocation mechanisms [4] 39]. Recent work has
successfully scaled SAEs to larger models 18 22 2], enabling automated interpretation of millions
of features 44.. Despite these advances, most SAE approaches treat features as isolated units without
modeling temporal relationships [I 19. lack explicit causal structure [34), and offer no identifiability limitations our work directly addresses.
guarantees [4 33
Feature-based Causal Circuits Recent methods like Sparse Feature Circuits [3 and attribution graphs [ 30 identify causal subnetworks explaining model behavior. These build on earlier
circuit analysis methods exploring component functionalities in vision and language models [2]
[0] Targeted interventional studies have revealed specific functional circuits, such as indirect object identification [55] and factual associations 377 While these methods enable mechanistic understanding of model computations 19 40,, they primarily rely on correlational measures rather than structured causal inference [16 43 But they focus stationary relationships [21] instead of
modeling evolving token-to- to-token dependencies critical for understanding sequential reasoning.
Causal Representation Learning Causal representation learning provides identifiability guarantees for latent variables [56] 27 47]. Temporal extensions model dynamics in sequential data [57
31 49]. with recent advances addressing non-stationarity 48 [ and instantaneous effects [32].
Multiple distribution methods [59 (88 can recover causal structure under specific interventions
or group structures. These approaches provide theoretical foundations for disentangling latent variables and identifying causal graphs (54). However, existing CRL algorithms cannot scale to LLM dimensions due to computational bottlenecks in calculating Jacobians. Our linearized formulation maintains identifiability guarantees while enabling application to high-dimm high-dimensionall LLM representations bridggg theoretical CRL advances with practical LLM interpretability challenges.
7 Conclusion
We introduced a causal representation learning framework for LLMs that jointly models time- delayed relationships and instantaneous constraints between latent concepts. Our approach provides theoretical identifiability guarantees while solving the scalability limitations of existing CRL methods through a computationally efficient linear formulation. Synthetic experiments validated our method's ability to recover latent causal structures from toy scale to real LLM scales. When applied to real LLM activations, our approach uncovered interpretable semantic patterns, revealing information flow pathways during text generation. Future work could leverage these causal structures for targeted alignment interventions, explore cross-layer concept transformations, and integrate with mechanistic interpretability techniques.
Acknowledgment
The authors would like to thank the anonymous reviewers for helpful comments and suggestions during the reviewing process. The authors would also like to acknowledge the support from NSF Award No. 2229881, Al Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract 1 and grants from Quris AI, Florin Court Capital, and MBZUAI-WIS WIS Joint Program, and the Al Deira Causal Education project.
10

---

References
[1] Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025.
[2] Anthropic Interpretability Team. Circuits updates - august 2024. Transformer Circuits Thread, 2024.
[3] Anthropic Interpretability Team.
https:// 1a222//pprl-updttte het///nidxxx.../ htteeex.wtmmttrrainiii saes,
Training sparse autoencoders.
transformer- circuits wwwpppppppppppeeepppp 2024. [Accessed January 20, 2025].
[4] Kola Ayonrinde, Adaptive sparse allocation with mutual choice & feature choice sparse autoencoders, 2024.
[5] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages 2397-2438 PMLR, 2023.
[6] Joseph Bloom, Curt Tigges, Anthony Duong, and David Chanin. Saelens. https: /github com/ /bloooos//ooossese..... 2024.
[7] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-oodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke. Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/transformmrr .ttml.
circuits. 2aal:/2022//ooneeeaatiteeaaaaa
[8] Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk: A simple improvement for topk-saes, 2024.
[9] Bart Bussmann, Michael Pearce, Patrick Leask, Joseph Isaac Bloom, Lee Sharkey, and Neel Nanda. Showing sae latents are not atomic using meta-saes, 2024.
[10] Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 6(1):e00024-066 2021.
[11] David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom, A is for absorption: Studying feature splitting and absorption in sparse autoencoders, 2024.
[12] Maheep Chaudhary and Atticus Geiger. Evaluating open-source sparse autoencoders disen- tangling factual knowledge in gpt-2 small, 2024.
[13) Guangyi Chen, Yifan Shen, Zhenhao Chen, Xiangchen Song, Yuewen Sun, Weiran Yao, Xiao Liu, and Kun Zhang. Caring: Learning temporal causal representation under non-invertible generation process. arXiv preprint arXiv:2401.14533 2024.
[14] Hoagy Cunningham, Aidan Ewart, Logan Riggs. Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023.
[15] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
[16] Joshua Engels, Eric Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are linear, 2024.

---

[17] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The Pile: 800GB dataset of diverse text for language modeling. arXiv preprini 1:Xiv:211100002 2020.
[18] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406. 04093, 2024.
[19] Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer seed-forward layers are key-value memories. In Marie-Francinn Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[20] Davide Ghilardi, Federico Belotti, and Marco Molinari. Efficient training of sparse autoencoders for large language models via layer groups. arXiv preprint arXiv:2410.21508 2024.
[21] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a baystack: Case studies with sparse probing, 2023.
[22] Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. Ravel: Evaluat- ing interpretabillity methods on disentangling language model representations, 2024.
[23] Shruti Joshi, Andrea Dittadi, Sébastien Lachapelle, and Dhanya Sridhar. Identifiable steering via sparse autoencoding of multi-concept shifts. arXiv preprint arXiv:2502. 12/79, 2025.
[24] Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Isaac Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum Stuart McDougall, Kola Ayonrinde, et al. Sacbench: A comprehensive benchmark for sparse autoencoders in language model interpretability. In Forty-second International Conference on Machine Learning. 2025.
[25] Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. Mcasuring progress in dictionary learning for language model interpretability with board game models, 2024.
[26] Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. Transactions Machine Learning Research, 2023.
[27] Zijian Li, Yifan Shen, Kaitao Zheng, Ruichu Cai, Xiangchen Song, Mingming Gong, Guangyi Chen, and Kun Zhang. On the identification of temporal causal representation with instantaneous dependence. In The Thirteenth International Conference on Learning Representations, 2025.
[28] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma arXiv preprint 2rXiv:2408.05147, 2024.
[29] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024.
[30] Jack Lindsey. Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persie, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025.
[31] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Causal representation learning for instantaneous and temporal effects interactive systems. In The Eleventh International Conference on Learning Representations, 2023.
12

---

[32] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557-13603. PMLR, 2022.
[33] Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control, 2024.
[34] Luke Marks, Alasdair Paren, David Krueger, and Fazl Barez. Enhancing neural network interpretability with feature-aligned sparse autoencoders, 2024.
[35] Samuel Marks, Adam Karvonen, and Aaron Mueller. dictionary learning. https://githu com/ http://pmarri//iiiiioooarrraaaaa
2024.
[36] Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2024.
[37] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359-17331 2022.
[38] Hiroshi Morioka and Aapo Hyvärinen. Causal representation learning made identifiable by grouping of observational variables. arXiv preprint arXiv:2310.15709 2023.
[39] Anish Mudide, Joshua Engels, Eric Michaud, Max Tegmark, and Christian Schroeder de Witt. Efficient dictionary learning with switch sparse autoencoders. arXiv preprint arXiv:2410.08201, 2024.
[40] Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, et al. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv preprint arXiv:2408.011 014416, 2024.
[41] Neel Nanda. Open Source Replication & Commentary Anthropic's Dictionary Learning Paper, Oct 2023.
[42] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3)ee00044000, 2020.
[43] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023.
[44] Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models, 2024.
[45] J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, 2000.
[46] Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 14435, 2024.
[47] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-63) 634, 2021.
[48] Xiangehen Song, Zijian Li, Guangyi Chen, Yujia Zheng, Yewen Fan, Xinshuai Dong, and Kun Zhang. Causal temporal representation learning with nonstationary sparse transition. Advances in Neural Information Processing Systems, 37:77098-771311 2024.
[49] Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. Temporally disentangled representation learning under unknown nonstationarity. Advances in Neural Information Processing Systems, 06:8092-8113 2023,
13

---

[50] Glen M. 2024.
Taggart.
Prolu: A nonlinearity for sparse autoencoders,
https://www www.aaaengggaaa.oooomaa orum o
non hrgpaa/eooooootttttooooorrrraaar
[51] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.00116 2024.
[52] Andrew Templeton, Timothy Conerly, Jacob Marcus, John Lindsey, Tamera Bricken, Bowen Chen, et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Technical report, Anthropic, 2024. Transformer Circuits Thread Technical Report.
[53] Constantin Venhoff, Anisoara Calinescu, Philip Torr, and Christian Schroeder de Witt. Sage: Scalable ground truth evaluations for large sparse autoencoders, 2024.
[54] Julius von Kügelgen. Identifiable causal representation learning: Unsupervised, multi-view, and mulli-enviromment. arXiv preprint arXin:2406.13377,, 13371,
[55] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022.
[56] Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning. arXiv preprini arXiv:2210. 3347,, 2022.
[57] Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:2110.05428, 2021.
(58) Kun Zhang and Aapo Hyvärinen. A general linear non- non-Gaussian state-space model: Identifi- ability, identification, and applications. In Proceedings of the Asian Conference on Machine Learming, volume 20 of Proceedings of Machine Learning Research, pages 113-128. PMLR, 14-15 Nov 2011.
[59] Kun Zhang, Shaoan Xie, Ignavier Ng. and Yujia Zheng. Causal representation learning from multiple distributions: general setting. In Proceedings of the 41st International Conference Machine Learning, ICML'22. 24. JMLR.org, 2024.
14

---

A
Technical Appendices and Supplementary Material
Contents
A.1 Discussion of Non- parametric Proof in Linear Case
15
A.2 Proof for Theorem
18
A.3
Synthetic Experiments
21
A.3.1
Fixed Structure Experiment
21
A.3.2
Scalability Experiment
22
A.4
Semi-synthetic Experiments
22
A.4.1
Target Concept Relation Recovery
................................
22
A.4.2
Steering Vector Recovery
A.5
LLM Activation Experiments
23
A.5.1
Details the Real-world Experiments Settings
23
A.5.2
Visualizations of Training Loss and Sparsity Metrics
25
A.3.3
Sensitivity and Ablation Studies
26
A.5.4
More Showcases on the Recovered Concepts and Relations from LLM Acti- vations
26
A.5.5
Additional SAEBench Results Larger Latent Sizes and Models
29
A.5.6
Statistical Testing and Absorption Analysis
29
A.5.7
Preliminary Investigation with Time Lag up 100)
29
A.5.8
Addition Experiments with Pretrained SAE
30
A.6
Compute Resources and Code
31
A. 7
Limitations
31
Societal Impacts
A.1
Discussion of Non-parametric Proof in Linear Case
In this discussion, we will replay the proof in (27 with assuming linearity to show the proof does not work in linear case. The proof sketch of the original non-parametric proof adapted linear case is shown in below. And we will show it is impossible to have full-rank coefficient matrix Cz., which leads to the failure the this derivation.
Non- non-parametric Proof Sketch in Linear Case If the learned model (Å, B, M, P.) is observa- tionally equivalent to the true model, then there exists an invertible linear map H AÅ such that p(2, (4-1) (( 14- H.. For nonadjacent (2,,s, 2,,) in Ma,, conditional independence implies log (222212211) O. Expanding this derivative via the chain rule yields linear system 822,
21, I5., S3c, and S0s, da., If the corresponding coefficient matrix Cz, is full rank, terms 0E,, 05, 05,,1" 05,, as,.
then all these terms vanish. Hence each 2₁, depends
most one Ê, and nonadjacent estimated
variables cannot involve adjacent true ones.
Suppose that the estimated model (A, B, M, P2) is observationally equivalent to the true model (A, B. M. Pe), then have Pλλ. (x:) PAZ, (x:). Applying the change of variable, we have
PAa. (xx) = A(2:))Á p(z))AA PAz, (x.) + p(2.) p(z.) |A|
+ p(2.) p(z:) AA -1 1 AÄ
A
given the invertibility of the linear mixing function Å. For the sake of simplicity, and p(ž,) HH]. Similarly, we have p(2,+1) pl H.
15

---

Let Me, be the Markov network of 2. Then, for any pair of variables & and 2t, that are not connected in Ma,, i.e., (2 2,,1) Ma, we have conditional independence on the graph due to the temporal relations between 2 and and the property of Markove networks:
2,, 1 , 2 U (2, { b).
We then make use of the fact that if two variables are independent conditioning on all the rest of the variables, then the cross second-ordes derivative of the logarithmic of the joint density w.r.t. the two variables. In our case, it is shown as:
8 log p(z,, 21-1) = 0.
Since log p(z, 2;-1) log p(2:)2c-1) (2-1) + log p(z)-1), where 2, has no functional dependence 21,8 or 2t,1, and therefore.
8 log p( 2c-1) O.
(10)
Because 2 by ((2 we have:
H
Zt
p( 21-1) = p(Ze, 11 Then, divide both sides
0
H
21-1)
pl Zo
H
p(2o(2(-1)
p(zz|z(-1))H)
(11)
p(Z1-1) H
Its partial derivative w.r.t. 2G, is
i Ologp(2, p(2, 21-1)
① log p(z, (2-)) 1) +
Ölog [H]
(12)
021,
02,, i log p(2, (2-1)
02
021,
a Olog H
+
(13)
izg,
02e,k
02,,
(E(M)
Then, its cross second order derivative w.r.t. &e and &,,1 is
8 log (0, 2
8 looo logg(zz 11-1)
8 log H
+
(14)
021,0001
02,
Ölog
p(zz 12:-1) Ozr.
82 82
(15)
EE(m]
82 logg(zz 14-1)
Paa 8z..
8 log H D
+
(16)
O2.
DEgK 04,,
vE[n
JE(n
Olog p(zz 11-1))
8 8zzd +
02 log p(zz (2-
Ōzi,i θzi, +
Σ
02,
Oz7,
Dāg, 02
iE(m]
iE(m)
82 lolo (t-
Ozy,i Ozi.j
82 log H
Σ J.M,
(17)
02,
D8 02,,1
8 log p( (2-1) 1) 022,5 O2g,5 02,, 0&g,1
Σ
8² log plzz p(ze(zi-1)
02i, Oz..j.
227
0ēg 02e,5
(et...
(18)
where 8 log [
is removed from the derivation since Eq. 17 because the Jacobian matrix
25, 1e
H
M) is constant according to Eq. 10 Besides, we assume linear mixing function in
Eq. the cross second order derivative 0 a²zes derivation since Eq. 17 as well.
O, the terms including this are removed from the
Re,
16

---

From Eq. 18 for each value of Z,. if we have R = + |M2,l different values of Zę-1. i... (re)) to make the following matrix Cz, full rank, for each non-adjacent pair of &t and & in the estimated Markov network Ma,, then we can conclude the relations between and 2t. Cz. is,
0 logg(zi Oz
Sº logp(z, logp(zile ) 0
+
Cz.
(19)
88 log 1 Ozz,
log pap (R)
iE|n]
①
From the notation above, we can write down each column of Cz. as function of 2 taking different values to form each row, According to Eq.
3 we have Zt Mz.
+ -Bz
+ 6s. which
forms the joint distribution of p(2,, 22-1) by:
Z:
M B
(I M
Z
Z
0
where the determinant of the Jacobian matrix of the tramsformation (in linear case, the Jacobian is the M)
(1
B
(I- (I-M)-1 is 10-III-- M) M)-1| By change of variable, we
transformation itself)
I
have p(z,, Z1-1) p 2t
EI M) And due the independence between ž;-1 and et have p(z, 24, 11-1) = p 2, (p(e) M) M/ Therefore, the conditional density is
o(2e/2z11 = p(ee))(TT -
( 11
II p(Gr,a)(I (MM)) MM
aE(n]
II p((I..: -a,,)2 - Ba 1) -M) aE[n]
where Ma,: denotes the a-th row of the matrix.
Since the conditional density p(zz 11- is second-smooth according to AI, the partial derivative of log p(z, 21 11-1) w.r.t. 21,4 is
i log ((/z:/z11)) θz,,
Olog Olog Ma.)Ze Bt. log (I - 1-M)-41
Σ
Oze,
Oz,
aE/n]
p'(ee,a)
Σ aE[n]
(Ia, - Ma.i)
p(ct,a)
Σ
Kaiipa (2t-1, Zt))
aE(n]
in which we use the abbreviation p(Er,a) = p((La, Ma,)Zr B., and therefore we let de (Z;, 2(-1) = p((er.a) Besides, we also let K = - M for the sake of simplicity. p(e..a)
Its partial derivative w.r.t. zt,j is:
82 log p(zz (2-1) Oz,,
(p((eaa (Er,))) pleta)
Σ KajiUas - Mas))
p(er,a)2
ac(n)
K Caa.iaaaall--l aE(n]
Σ
[/((e),))
F" x"(ce,)) given p(ct,a) p((Ia, Ma:) BB..Zi-).. p(cr,))))
where v/a (z(1111,)) Zt)
p(c),)
When j =
8 loggppur (119)(2111) 021
Σ Ka..K..Wa(aa-11222))
aE[n]

---

Therefore, each row of Cz. can be represented as vector of functions taking different values of 1t-1:
Cz. (2--1)
(r)
Ma, Ma.
(20)
aE(n]
+
Ma, Ma, Ma, 2)) (ESSANDMMM.
(21)
aEIn]
And if we have at least R Mai values of {z, (r)
1 [rejn] make the matrix
Cz. (C2, (z)-1). C2. (z-1)))
(R)
of shape R x R full-rank, then the relationships between the estimated é, and the true 2, can be solved by having nontrivial zero solutions in Eq. 18 However, impossible for Cz, to be R x R full-rank, because the concatenated two parts, (Ene(s) Mo, (a. 2:)))
are both constant linear combinations of (via (2,22) 2x (... ) Jasjn for M does not change across timestamps. It constrains the rank of Cz. be at most n, which implies any non empty instantaneous Markov network leads to the unidentifiability of 2
(r) and (ERe(H) Ma,iMa,j (a (.)) (e
A.2 Proof for Theorem
Theorem 1 (Latent Indeterminacy). Suppose the estimated model (Å, (Br),- (...)) and the true model (A, (Br}i-1. M. Pe) both generate Xę according to Eq. and are observationally equivalent on the autocovariance matrices (x(k) for k 0.1...., L. If conditions Al-A4 hold, then the model parameters are identifiale up to the following indeterminacies:
E = PEt.
Á = AS,
(I (I- = P
B
where S E = invertible and P a signed permutation matrix.
Al (Temporally white noise). Both and temporally white, and has i.i.d. mutually independent components. To remove scaling indeterminacy, assume Ee, and zero-mean data.
A2 (Rank sufficiency). A E R (m 2 n) has full column rank, and B2 is full rank.
A3 (Process stability). Eq. 22 defines stable vector autoregression, Le., all roots of
det(I - Et B.y 0 lie strietly inside the unit circle.
A4 (Non-Gaussianity). At most component of is Gaussian.
Proof. A more condensed formulation is used, which is the same as Eq. for the sake of matrix transformation presentation in the proof.
Mz, Σ B, Ct.
(22)
Azi,
(23)
Step Construct Autocovariance Matrices its autocovariance at lag k as:
According to the formulation for 2,, we can derive
I. aa- M) B, + (1- (I-M) 11- scck
T
(k) = Zt
I.
E 22, +
B
M)
Elz,efi 1I -M)
18

---

Hence,
Σ R2(k - T) M)
k # 0,
R₂(h)
R2(k - T)) M) M) + MI- M)- T Σ
k ==0.
Because X, Azt,
Rx (R)
Thus,
-R) -A"
*
(k(
A -ackk-Taa ------AAAA--B- M) - TAT +A[T A(I-M)-AA -MT ==
Simplifying,
L
Σ Rx - AA--T)AABB (I- -
k *
(k)
Rx - -)-- BI AAT MM M) TAT, =
Let
C, = A(T A B,A
Then,
Σ - -
h * 0,
(k(k)=
Σ
A(I A(I- -AT, =
Step 2: Solving the Linear System Formed by Autocovariance construct the following linear system.
Use the derived R (k), we can
Rx(0) HHT
(x(1)
Rx
(L RL - 1)T R 2) 2)
(1)
(0)
Rx(1)T
C
(2)
(1)
(x)
(24)
:
C
(L)
(x
Rx(L-2 2)
Rx
where H = A(I-Mp M)
Let's 's look at the lower mL rows of the coefficient matrix of the linear system:
(x
Rx(1)T
Rx (L 1)"
(x(1)
(x
Rx - 2)
Q
Rx - 1)
R 2)
(0)
19

---

It is clear that Q = E[x x (x]
x Because the process {x+} admits no nontrivial deterministic linear relation among x,, x I++1+1 due to c, Q x '1 is always positive definite and therefore nonsingular.
Thus, we solve the lower mL linear system obtain Ck 1,... L and then use the solved C₁ obtain HH Therefore, suppose we have models, (A, M, 1. -.) and (Å, M, 1, Σe). and they are observationally equivalent on (x h 0, 1, L, then have the relations between there parameters as follows:
AI - M) (I -I- MM- T AT
A(I - M) (I M) AT
(25)
Á(I M) B, A M)
B, Vr 1,
(26)
Step 3: Further Investigate B, and M
From the RHS of Eq. 25
we have:
HH = A M) M) A
(27)
which gives CoI(HH?) C (A)(A) Thank to the fact that A has full column rank, entailing its hasing left inverse A A I, and I M being invertible, have
A HH A - M) (I-M) M),
(28)
which further gives Col(A) S Col(HHH).
Therefore, Col(A) Col(HHH"), As we can derive Col(A) CoI(HH²) from the LHS from Eq. 25 we have: Col(A) = (A). Since both A and Å are column wise full rank, we have
Å = AS
(29)
for some invertible S E IRdxd
From HH Hi," with H A(I M) showing that H has left inverse, have
H TH 'H) II
(30)
Therefore, we can derive that H = U, where U is an orthogonal matrix. However, since A only has left inverse (the same as H), we cannot directly conclude how U plays the role in the relationship between and fl but need the following derivation.
Given the invertibility of M and Col(A) Col(A), know Col(H) Col(fr) = Á Co)(H). Since H has right inverse only in the column space Col(H), we have
H
H(H HU = (HH- HU
-U H HU.
(31)
Therefore,
Á - M) A A(I-M)-
(32)
AS(I - M) - A(I M) -U
(33)
+ M) (2
(34)
Finally, substituting Eq. 26 with Eq.
and Eq. 29
for all T 1, 1.... L we have,
Å M) B, = A(I M
- UB.S-'A" A1 -1 A(I M) BBA
(35)
+ UB B,
(36)
= B, = U.S.
(37)
So far, all the relations derived between the true model (A, BB M, Ex) and the estimated model (Å, {B, Y4 M, Ee) are underlined.
20

---

Step 4: Utilizing Non-Gaussianity Suppose the true model (A, (B,),1,,MM M, Ee) and the esti-
mated model (A, (B, 1 M,, Mis) are observationally equivalent:
I.
X A
(1 - M)
Σ
B
Et
(38)
T=1
I.
-1
A
(I-M) M) Σ B
-
(39)
AS
UT [I
M)S Σ
U'B
&
(40)
A
(I- M) Σ B,
U2.
(commute B, with =
(41)
I.
(I M)-M)
Σ B
(E: U2.) =
(42)
(a: UE:) 0.
(43)
Therefore, we can derive that é U εt. Because there is at most one Gaussian component e, and E. I, we know that U must be a signed permutation, i.e., U" where is a permutation
matrix and taking values from (+1, -1} for its nonzero entries. To conclude, we derive À = AS, (I M) (' - M)S, and B, = U,B, where B is invertible matrix and P is signed permutation matrix.
A.3 Synthetic Experiments
We conduct two synthetic verification experiments to validate our linear temporal instantaneous ICA method. Instruction provided in the synthetic/README. md file in our code repository.
A.3.1 Fixed Structure Experiment
For the first synthetic verification experiment, we generate data using fixed time-delayed influence functions and instantaneous relations with the following ground truth matrices:
T0.4
0.6
0
0
07
B
1
0
M
0.2
0
0
(44)
0
1
0
0.2
0
The data generation process follows a structured temporal model. We initialize the first hidden state 2o by sampling from uniform distribution 2( 1). For subsequent time steps, we compute the historical influence as Zuist = B - and then construct iteratively: the first dimension receives only historical influence plus noise, while remaining dimensions > 2 incorporate both historical and instantaneous dependencies:
2
(1)
hist + E
(1)
(1)
(45)
4iss
+ Winst
(f-1) +
+
i i22 2
(46)
where e is Laplace noise with scale 1.0, and Wisst 0.2. The observations are generated as X, Az; where A is 3 x 3 randomly initialized mixing matrix.
We train the model for 50,000 steps with batch size 1024 (approximately 51 million total samples)
using the Adam optimizer with learning rate 8 10-3 and weight decay 10-4, The loss function includes reconstruction error, KI. divergence term, and L. regularization penalties: 1 x 10-3 for matrix M and 10-8 for matrix B. We enforce the lower-triangular constraint on M to ensure identifiability.
21

---

A.3.2 Scalability Experiment
For the second synthetic experiment, we evaluate scalability across different dimensions ranging from 64 to 1024. We randomly sample a sparse time-delayed transition matrix B where only 10% of the entries are non-zero, generated using randomly initialized matrix with 10% masking.
For the instantaneous mixing matrix M, we use a chain structure where M M 0.5 for i 2 2 and all other entries are zero:
0
0.5
0
0.5
M
(47)
0.5
0.
The training hyperparameters are modified from the first experiment: learning rate increased to I x 10-3 and the sparsity coefficient for B increased to x 10-5 to account for the higher dimensional setting, while maintaining 10-3
Both experiments use identical noise characteristics (Laplace distribution with unit scale). sequence length of (two time steps total), and Mean Correlation Coefficient (MCC) as the primary evaluation metric to measure the quality of source recovery while accounting for permutation ambiguity inherent ICA methods.
A.4
Semi-synthetie Experiments
A.4.1 Target Concept Relation Recovery
Before attempting to recover concept relationships from real-world LLM activations, and based on the proven and verified identifiability of our model, we first present semi-synthetic setting to verify that our proposal can reveal obvious concept relations from contrastive corpus pairs.
Data Preparation We constructed two contrastive collections of texts drawn from the Pile dataset [77]. We considered three types of text: legal documents, emails, and XML files. For each type, we constructed two contrastive corpora: one containing the relation of interest, and the other lacking it. Specifically, for legal text, the relation defined by sequences beginning with "APPEALS" and ending with "AFFIRMED" for emails, sequences start with forwarding or reply markers (e.g., dashes) and end with common words like "Subject" or "Thanks"; and for XMI., sequences start with a UTF encoding label followed by tags such as "UTF-8" "!DOCTYPE". We hypothesized that only texts containing these structured relations would yield meaningful temporal concept patterns, and we directly tested whether the model can successfully recover such patterns.
Baseline Construction Although there is no directly applicable baseline, we leveraged standard SAES we had trained above to serve as our baseline method. Since SAEs themselves cannot capture the concept-to-concept relations, we train regression model to find temporal relation coefficient matrices Bs by solving the following regression task: Σ, B,
We calculate the concept recovery score by first obtaining the top fired feature index pair (i, related the legal context (restricted positions where the concepts of interest ought to fire but do not fire in the contrastive non-legal text), and then taking the corresponding entry B,, in the aggregated temporal relation coefficient matrices. We then calculate the relation B. where (B
Evaluation Metric
recovery score, similar to a signal-to-noise ratio, by: relation recovery score
(BB denotes the standard deviation of matrix B. Such ratio indicates the extent that the target concept relation entry in the matrix more significant than a random noise; the larger the score is the more significant the relation recovery.
Results All results are shown in Table which verifies that our proposal can identify the concept- relation of interest from contrastive corpus pairs. For the demonstrated results, we used the same trained model in the experiments recovering relations from real-world LLM activations.
22

---

A.4.2 Steering Vector Recovery
Except for the relationships between concepts, our model is also able to recover the concepts as current SAEs. To verify this, semi-synthetic benchmarks like SSAE [23] can offer valuable insights into concept identifiability. Following this setting, we tested whether our model can recover steering vectors from paired text. Specifically, constructed five categories of word pairs where only single interpretable concept changes, including gender, plurality, comparative, tense, and negation. While these changes intuitive, ensuring the word pairs capture clear ground-truth concept is non-trivial. Despite this challenge, our model demonstrated strong performance in identifying the underlying concept differences. Specifically, (1) the average correlation of concept differences within each category exceeded 0.86; (2) assuming one ground-trum truth pair, the correlation rose above 0.93; and (3) the maximum correlation within category reached over 0.94, These results support our claim that our model can indeed recover meaningful steering vectors. The word lists for the five categories is summarized in Table 4
A.5 LLM Activation Experiments
In addition to the experimental results presented in Section [53 of the main text, we provide here: (1) detailed settings for training and inference; (2) visualizations of training losses and sparsity values; (3)
comparisons across different hyper-paramedt settings, and (4) extended experiment on SAEBench with larger latent size and base language models.
A.5.1 Details on the Real-world Experiments Settings
Training We train our linear model on activations from the pretrained LLM pyrhia--60m-deduppes [5]. using SAELens [6] and dictionary-leamingg
33 for activation extraction. Importantly, in the original implementation of dietionary- loaded using an object named ActivationBuffer, which is refreshed with new activations once predefined consumption threshold is reached. During each refresh, random shuffling is applied. However, this randomization disrupts the temporal structure of the L.LM activations. To preserve temporal information, we modify the corresponding refresh function to disable the random shuffling. Details of this modification can be found in the examples/ DREADDEE md file in our code repository.
35 activations are
The model is trained on total of 50 million tokens from the Pile dataset [7]. To capture time- delayed influences, we consider two values of r, namely {5, 20). as described in Eq. 3 While our main results focus on the setting with 7 = 20, which offers better guarantees for capturing rich temporal semantics, this choice will be further justified in a later section of the supplementary materials. To address the distributed and uncertain nature of time-delayed dependencies- some relations manifest over longer time spans and others over shorter ones we aggregate the B, matrices using max-pooling. This operation preserves any causal link that appears at any time step. We refer to the resulting aggregated matrix as agg Unless otherwise specified, the weight of the independence constraint on the noise term is set to a 0.1 in Eq. 9
To better enforce sparsity in the hidden feature activations, we apply TopK filtering g in addition to the L sparsity term included in the final loss function. Given the importance of feature dimensionality in Sparse Autoencoders (SAEs), we evaluate three configurations: 768 (which directly matches the
LLM's hidden size and aligns with the identifiability discussion in Section 3}. 3072, and 6144 the latter follow the considerations of SAE literature. We optimize the loss function defined in Eq. D using the Adam optimizer with a learning rate of 0.01 and weight decay of 0.0001. Unless otherwise specified, we use a random seed of 123; additional experiments were conducted with seeds 456 and 789 for robustness.
Inference During inference, our primary goal is to interpret the hidden pattuurs--ptticllarryy those activated by significant entries in the time-delayed B) or instantaneous (M) relation matrices. This selection process differs from conventional SAE interpretation, which typically examines feature importance across the entire feature space by measuring activation strength for a given prompt. In contrast, our method emphasizes the relational structure of Teatures-hov they connect to form semantic transitions. We aim to understand the meaning of each feature by analyzing how both types of relations (instantaneous and time-delayed) link features together.
23

---

Table 4: Summary of word pairs in the five categories
Categories
Pairs
Gendered Pairs
(male, female), (actor, actress), (prince, princess), (king, queen), (god, goddess), (wizard, witch), (boy, girl). (man, woman), (father, mother), (son, daughter), (brother, sister), (husband, wife), (nephew, niece). (uncle, aunt), (gentleman, lady), (monk, nun), (grandfather, grandmother), (lord, lady), (spokesman, spokeswoman)
Plurality Pairs
(cat, cats), (dog, dogs). (apple, apples), (box, boxes), (child, children), (book, books), (car, cars), (tree, trees), (house, houses), (bird, birds), (chair, chairs), (table, tables), (shoe, shoes), (shirt, shirts), (sock, socks), (cup, cups), (plate, plates), (pen, pens). (bag, bags), (door, doors), (window, windows), (lamp, lamps), (phone, phones), (laptop, laptops), (flower, flowers), (cloud, clouds), (mountain, mountains), (river, rivers), (lake, lakes), (egg, eggs), (grape, grapes), (potato, potatoes), (tomato, tomatoes). (bus, buses). (kiss, kisses), (wish, wishes), (match, matches), (dish, dishes), (baby, babies), (lady, ladies). (city, cities), (party, parties), (family, families), (knife, knives), (leaf, leaves), (wolf, wolves)
Comparative Pairs
(fast, faster), (tall, taller), (small, smaller), (old, older), (young, younger), (short, shorter), (long, longer), (high, higher), (low, lower), (strong, stronger), (weak, weaker), (rich, richer), (poor, poorer), (hard, harder), (soft, softer), (loud, louder), (bright, brighter), (dark, darker), (clean, cleaner), (easy, easier), (happy, happier), (cool, cooler), (deep, deeper), (wide, wider), (narrow, narrower), (thick, thicker), (thin, thinner), (heavy, heavier), (light, lighter), (safe, safer), (cheap, cheaper)
Tense Change Pairs
(walk, walked), (run, ran), (eat, ate), (go, went), (write, wrote), (speak, spoke), (drink, drank), (drive, drove), (read, read), (sleep, slept), (sit, sat), (stand, stood), (fly, flew), (begin, began), (buy, bought), (bring, brought), (build, built), (catch, caught), (choose, chose). (come, came), (cut, cut), (dig, dug). (do, did), (draw, drew), (fall, fell), (feel, felt), (find, found), (get, got), (give, gave), (have, had), (hear, heard), (hide, hid), (hold, held), (keep, kept), (know, knew), (leave, left), (lose, lost), (make, made), (meet, met), (pay, paid), (ride, rode). (say, said), (see, saw), (sell, sold), (send, sent), (sing, sang), (sit, sat), (teach, taught), (think, thought)
Negative Prefix Pairs
(possible, impossible), (legal, illegal). (visible, invisible), (complete, incomplete), (fair, unfair), (known, unknown), (fortunate, unfortunate), (able, unable), (happy, unhappy), (certain, uncertain), (clear, unclear), (real, unreal), (necessary, unnecessary), (likely, unlikely), (available, unavailable), (comfortable, uncomfortable), (pleasant, unpleasant), (reli- able, unreliable), (acceptable, unacceptable), (usual, unusual), (wanted, unwanted), (expected, unexpected), (connected, disconnected), (under- stood, misunderstood), (placed, misplaced)
Our feature selection process involves the following steps: First, we select the top 100 coordinates (we also tried 200, though 100 proved sufficient) from either aggB or M, and extract the corresponding feature dimensions. Next, we generate 10,000 prompts from the EleutherAI/pile dataset, convert them into token streams, and feed them into the trained model to observe how each token responds to each selected concept feature. Finally, for each selected feature, we collect the tokens whose activations exceed a threshold (set to 3.0), along with their corresponding prompts. These tokens are viewed as consequences of the activation of the given feature, while the associated prompts serve as contexts that reveal the token and therefore, feature's 's meaning.
24

---

A.5.2 Visualizations of Training Loss and Sparsity Metries
Here, we compare the training dynamics across different settings by examining the reconstruction loss (Eq. [. the independence of the estimated noise term (Eq. (7) and the sparsity of both time-delayed and instantaneous relations (Eq, 8). The comparisons are made with respect to variations in hidden feature dimensionality, the sparsity weight on learned relations (i.e., 3 in Eq(). the temporal coverage of delayed relations, as determined by E {5, 20}. and the parameter of the TopK filtering of the hidden features.
We begin by examining the training dynamics with 7 = 5, comparing different settings of the sparsity constraint (3 ). TopK values ((0, 25, 100}, where indicates that TopK is disabled), and hidden dimensions (2_dim E (768, 3072, 6144)). The corresponding results are presented in Figure
X It is worth noting that certain unstable training batches occasionally impact the overall stability during training. However, since most of the configurations eventually converge and our primary interest lies in the behavior at convergence, we cap the y-axis at 5.0 to improve the clarity of the visualizations. Our key findings are summarized below.
341
2,00 10E
40077
40000
5.0000
Figure 8: Dynamics of reconstruction loss, noise independence, and time-delayed and instantaneous relations sparsity with setting to 5. The x-axis starts at 5M tokens, and the y-axis values are capped at 5 to enhance visualization clarity.
Insights on the Number of Training Tokens and the Impact of Hidden Feature Dimensionality From Figure 8. we observe that 50M training tokens are sufficient for convergence across all settings when the hidden feature dimensionality is greater than 768- specifccifl at 3072 and 6144. From the subplots in the first column, it is evident that higher-dimensional hidden features provide greater stability during training. This increased robustness likely helps mitigate the effects of noisy or unstable batches within the token stream, leading to more consistent optimization of the objective. Consequently, in the subsequent case studies, including Section 5.3 of the main content, we focus on the settings with hidden dimensionalities of 3072 and 6144.
Impact of TopK Filtering The training process is in general more stable after applying TopK filtering. More specifically, comparing the aub-diagrams from the first row in Figure g to the second
25

---

and the third rows, we can see that the decrease of the reconstruction error is significantly less effected by some of the token batches, especially, for the setting when feature dimension is set to 3072 or 6144.
Impact of Sparsity Strength In general, when 3 sets to 0.01 (pay attention to the round marker in Figure gas as oppose to star marker), both the time-delayed and the instantaneous relations show lower sparsity compared with a stronger sparsity weight. This might be due to a weaker constrain that results better optimization results, while the stronger one might increase the sharpness of the potential solution space. This also indicates that 0.01 is sufficient for achieving our goal of sparse causal relations in our model.
A.5.3 Sensitivity and Ablation Studies
and 3 We conducted additional comparisons with 3 0, 0.001, 0.005, 0.05, 1.0 and a = 0, 0.001, 0.01 to cover a broader hyperparameter range, using T = and feature dimension 3072. The results are shown in the two tables below, with our selected setting in bold text. The tables highlight that (1) concept relationships are inherently sparse, while a large ,3 disrupts optimization, and (2) has stronger effect, with 0.1 being well-balanced choice.
Sensitivity Study
Table 5: Performance comparison under different values of a
Ox
0.0
0.001
0.01
0.1
1.0
Reconstruction Loss 4
0.0227
0.0191
0.0118
0.0128
0.1121
Independence Loss |
4.3849
2.4572
0.3910
0.1448
0.5252
B. Sparsity (LI) 4
0.0012
0.0007
0.0018
0.0007
0.0058
M Sparsity (L1) 4
0.0002
0.0001
0.0002
0.0001
0.0009
Table 6: Performance comparison under different values of /
B
0.0
0.001
0.005
0.01
0.05
0.1
1.0
Reconstruction Loss 4
0.0126
0.0126
0.0126
0.0128
0.0126
0.0128
0.8950
Independence Loss
0.1522
0.1496
0.1504
0.1448
0.1550
0.1582
3.7682
B, Sparsity (L1)
0.0003
0.0005
0.0005
0.0007
0.0007
0.0013
0.0053
M Sparsity (L1) $
0.0001
0,0001
0.0001
0.0001
0.0001
0.0002
0.0007
Ablation Studies on Bias Terms Finally, we explore whether there will be potential performance improvement when additional bias terms added to our linear encoder and decoder functions in equation [5] to give more complete justification of our implementation. We also compared these two settings in the real LLM activations (feature dimension=3072, a = 0.1, 8 = 0.01, 7 = 5). The results shown in the table helow indicate that the flexibility gain of the bias terms is not significant in our model.
A.5.4 More Showcases on the Recovered Concepts and Relations from LLM Activations
In addition to the examples presented in Section 5.3 of the main text, we provide additional cases here further illustrate the diversity and interpretability of the recovered concepts and relations, highlighting how they manifest across different domains and contexts.
Time-delayed Causal Relations Table 8 showcases further examples of time-delayed causal relations extracted from LLM activations by using our model, with the same setting that we have shown in the main content Table Many of these reflect the structured nature of legal, technical, and encyclopedic language. For instance. feature 2341 (e.g., "Orders/mandate in appellate judgment") is linked to feature 2592 (e.g., "decision" and "observance"), revealing how commands or mandates precede judicial conclusions in legal discourse. Similarly, technical logs such as feature 1856 (error messages) anticipate subsequent failure indicators (feature 1833, "FAILURE"), reflecting typical diagnostic progressions in computing contexts.
26

---

Table 7: Ablation comparisons on the bias terms for the encoder and decoder
Metric (real-world)
Without Bias
With Bias
Reconstruction Loss
0.0129
0.0129
Independence Loss
0. 0.1452
0.1457
Sparsity (L1)
0.0007
0.0007
M Sparsity (LI)
0.0001
0.0001
Table 8: More examples of the discovered time-delayed relations with contextual explanations.
From ID
From Explanation
To ID
To Explanation
Context
2341
Orders/mandate in appel-
2592
"decision" and "obser- vance"
Legal
judgment
late judgment
labels
1856
Technical error message
1833
"FAILURE"
Describes the fail- ure reason
2579
"APPEALS"
2592
Court/party geographical
Appeals in legal documents
location or case handler
1833
Ajax request header: 'application', "function" (type, URL, status)
2390
Syntax and functions like "each"
Ajax request func- tion labels
1856
Volume number in case citation
2341
"mandamus" from "writ of mandamus"
Summary of case docket
2100
Page number where case starts
2579
"APPEALS"
Case
citation
structure
790
Wikipedia ship owner name
2730
"ship"
Wikipedia entity tagging
1825
Email forward/reply dashes
1641
Common words "subject", "thanks"
like Email metadata and signals
1551
Name "Wynne" (e.g., "John Wynne")
2311
"sat" (in Parliament)
Wikipedia bios for people named Wynne
1124
UTF encoding label
1657
Tags like "UTF-8", XML document "IDOCTYPE"
structure
1675
HTML starting signal
2583
Common HTML tags like "a", "pre"
HTML document recognition
1303
"default" keyword
2623 Follows "default" (e.g.. "context", " "_")
Generic technical documentation
1895
"Q", "Re", "forward"
1203
"thanks"
Email or Q&A style messages
2708
Personal pronouns (TI,, "you")
2584
Tense indicators like "will", "have"
Human language facts
Notably, semantic connections span heterogeneous domains. Wikipedia entity labeling (e.g., ship names and their categories) and web document structures (e.g., UTF labels leading to encoding declarations) both reveal meaningful temporal dependencies that LLMs internalize. The relation between personal pronouns (feature 2708, "I", "you") and tense markers (feature 2584, "will", "have"" further illustrates how human language patterns are temporally structured, even over several tokens. These cases reinforce the model's s capacity to track and anticipate semantic developments over time in content- and domain- aware aware manner.
27

---

Table 9: More examples of the discovered instantaneous relations with contextual explanations.
From ID
From Explanation
To ID
To Explanation
Context
2341
Labels 'license' in com- ment of "license control pre server"
1856
Labels 'license" in both comment and command line
Bash script con- text
Labels 'research' 2592
227 Labels 'research' with nearby nouns like "pro- gram"
Academic texts
2592
Labels 'magazine"
80 Labels 'magazine" and common related nouns like "teenage", "blogs""
Academic texts
2592
Labels 'module'
2208
Labels both 'module' and JavaScript code "exports' as in "mod- uleéxports"
2623
Labels 'https'
227 Labels both 'htips' and URLs ':/l"
Instantaneous Causal Relations Table 9 provides more instances of instantaneous relationships, highlighting features that are co-activated within the same context window. In the domain of Bash scripting, we observe co-occurrence between licensing-related comments (feature 2341) and execution commands (feature 1856), showing how LLMs jointly encode comment semantics and imperative script logic.
In academic and technical domains, common conceptual pairs such as "research" and "program", or "magazine" and related digital terms like "blogs" "websites", represented together (e.g.. features 2592 and 227 or 80). These examples suggest that the model forms composite concepts out of frequently co-occurring terms, such as publication metadata content descriptions.
In programming contexts, the instantancous link between "module" (feature 2592) and the JavaScript construct "module.ewy "modulc.exports" (feature 2208) demonstrates that the model learns the tight coupling between programming keywords. Likewise, the relation between "https" (feature 2623) and full syntactic pattern "https://" (feature 227) reflects how structured URL formats are stored as unified units in the model's activation space. Together, these examples demonstrate the model's 's ability to encode concise,
composite structures through simultaneous feature activation.
Notes on the Results Following presentation of the causal relations recovered from LLM activations, we clarify several key points regarding the interpretation of these results. First, due to variations in tokenization strategies across different corpora, many identified tokens in given sentence may correspond only to partial words. This issue can be exacerbated by noise introduced during data collection processes such as OCR web crawling. To address this, we rely on human judgment and linguistic intuition to infer and annotate the complete underlying word, ensuring that the labeling remains accurate and avoids overextending unrelated tokens. Second, the recovered time-delayed relations we present may be somewhat semantically constrained, as the clearest relations tend align with explicit syntactic structures. Many of our examples-such as those from code snippets or legal cecumennts-covee semantic information through formal syntax. While these cases are illustrative, we view the discovery of more abstract, syntactically diffuse relations in general language text as an important direction for future work. It is also important to note that the examples we present were not cherry picked; rather, they are representative cases that naturally appear throughout the dataset and were surfaced by our method. These relational patterns would not be easily discoverable using sparse autoencoders (SAEs), as SAEs do not consider interactions between features. Finally, we observe that feature pairs exhibiting strong causal relations tend to be activated under highly similar prompt conditions, indicating that these features are contextually aligned and often co-occur within the same linguistic environments.
28

---

Table 10: Gemma-2-2B instantaneeou--rrlationnoo model on SAEBench with different latent sizes.
Latent Size
Recon. Loss + Sparse Probing Acc. Absorption +
Autointerp
6k
0.0108
0.6736
0.0139
0.6883
16k
0.0059
0.6918
0.0167
0.7117
Table 11: Absorption statistics with extended training budgets.
Model
Full A Absorption Fraction
Absorption Fraction
Split Features
Pythia-160M-11
6.471 x 10-2
9.185 x 10-3 3.794 x 10-4
1.043
Gemma-2-2B-16k
1.289 x 10-2
1.269
A.5.5
Additional SAEBench Results on Larger Latent Sizes and Models
Following the same dataset and training protocol as in the main experiments, we trained the simplified instantaneous--rlaatonnnnt variant with 16k latents Gemma-2-2B and compared our 6k-latent configuration. As shown in Table TO the 16k model reduces reconstruction loss (0.0059 vs. 0.0108) and slightly improves sparse probing top-1 accuracy (0.6918 vs. 0.6736). Its absorption score is modestly higher (0.0167 vs. 0.0139) but remains small, and the Autointerp score increases (0.7117 vs. 0.6883). Overall, performance on SAEBench metrics remains at similar level across latent sizes.
We also extended the training scale to 500M tokens for Prthia-160M, and to 300M tokens for Gemma- 2-2B, both with 16k latents. In both cases we observed very small absorption fractions, and the mean number of split features remained close to one, indicating minimal feature splitting. Summary statisties are reported in Table 11
A.5.6 Statistical Testing and Absorption Analysis
To strengthen the empirical findings, we additionally performed statistical testing to assess the equivalence of reconstruction losses and the robustness of absorption scores. Using 100 samples per method (N = 300), any shift 0.00127 across groups would be detected with power 2 0.8. Pairwise Welch-TOST and Hodges-Lehmant tests with A = 0.001 confirmed equivalence: all 90% confidence intervals lay within -A, A], demonstrating statistical equivalence at a = 0.05 among the three methods.
For absorption, although rigorous hypothesis testing is challenging due to the very small magnitudes observed, we collected sufficiently large number of samples (2 200) to establish confidence intervals. The mean and 95% confidence intervals were 0.0135 + 0.0002 for the 6k model and 0.0136 0.0002 for the 16k model, which are more than sufficient to demonstrate negligible absorption in practice.
Additionally the signal-to-noise ratio (20.02 for method vs. 2.39 for the SAE baseline) already indicates strong margin. Such a large difference is unlikely to arise from random noise.
A.5.7 Preliminary Investigation with Time Lag up to 100
To address the potential limitation that a fixed value of r may be overly restrictive in capturing the rich and diverse semanties of real-world contexts, we explore a more flexible approach. Specifi- cally, different types of concept-relations may require varying numbers of steps to be successfully recovered. Furthermore, even for single concept-relation, stable recovery across different contexts may necessitate range of steps rather than single fixed value. In light of these considerations, in addition to the recovered relations shown in Table 3 Table g and Table 9 we present the relations captured within 100 steps, grouping them into bins of sizes 10, 20, and 50. This binning naturally categorizes the relations of interest, facilitating further analysis and discussion.
The increased flexibility provided by larger time lag allows us to recover greater number of concept-relations. For example, we can recover relations such as "monument" - "from" and "seek" - "opportunity".. Interestingly, increasing the time lag not only allows longer-range relations to be captured but also enables previously overlooked relations be discovered, as this flexibility improves identification of concepts entangled in the relation. To better illustrate the relations recovered with
29

---

larger time lag, we are preparing a web demo, which will soon be included in the code repository once it is ready.
To better illustrate the relations recovered with a larger time lag, we are preparing a web demo, which it is ready. However, our primary contribution is
will soon be included in the code repository to demonstrate that our model can recover relation-concepts more effectively than existing SAEs, addressing gap that is currently missing but crucial for advancing LLM interpretability. A broader and more systematic study of this phenomenon is left to future work.
A.5.8 Addition Experiments with Pretrained SAE
As ablation study we additionally construct our linear model using the pretrained Sparse Autoencoder (SAE) from Gemma Scope (28] on the Gemma 2 2B model [51]. To enable feasible qualitative evaluation, we selected the top 2, 034 most frequently activated features from the commonly used SAE gemma-2-255 2-e--22--------- 20b/20-gemmacopp pereccpeeeee-eeeeeee www--eeseeeee res-16k using the SAELens package 66. We trained our linear model on million tokens from the Pile dataset.
Since time- delayd influences may occur with variable time lags, we set a sufficiently large value for in Eq. In practice, use 20 and aggregate the time-delayed matrices B, using max-pooling that is, if a causal link exists in any of the time-lagged matrices B,, we consider that link be present in the aggregated causal structure.
Case Studies
Our analysis reveals rich causal structures among programming--elated concepts in LLM activations. We examine both time-delayed and instantaneous causal relationships, providing insights into how the model processes and generates code-related content.
Time-De Deee-Delayed Causal Relations We identified several meaningful time-delayed causal relation- ships in programming contexts. A prominent example is the causal link from a concept representing "function definitions and related code structure in programming languages" to concept representing "variable definitions and data types in programming contexts." This relationship aligns with the natural structure of programming, where global function definitions often precede and influence local variable declarations or data structures. When the model processes or generates function definitions, it subsequently activates concepts related the variables and data types that would appear within those functions.
Additional time-delayed relationships include causal links from "programming language syntax specifications" to "code implementation details" and from "algorithmic problem statements" to "solution implementation structures." These relationships demonstrate how the model captures the sequential dependencies inherent in programming tasks, where understanding of requirements or specifications precedes implementation details.
Instantaneous Causal Relations Our method also reveals interesting instantancous causal re- lationships that occur within the same time step. We observe a strong instantaneous causal link between concept representing "specific formatting and notation elements commonly used in math- ematical expressions or programming syntax" and a concept representing "mathematical symbols and expressions in technical content." " This relationship indicates that the model simultaneously processes formatting rules and the mathematical content they structure, reflecting how these aspects intrinsically connected in code representation.
We also identified instantancous causal relationships between "programming language keywords" and "syntax highlighting patterns," " as well as between "code indentation patterns" and "block structure delineation." These instantancous relationships capture the syntactic constraints that operate simultaneously within programming languages, where certain elements must co-occur for the code to be worl-formed.
These case studies demonstrate that our method can extract meaningful causal relationships from real LLM activations, providing insights into how these models process and generate structured content like code. The identified causal structures align with the logical and syntactic relationships one would expect in programming contexts, validating the effectiveness of our approach for interpretability research.
30

---

A.6 Compute Resources and Code
All experiments were conducted on a computing cluster equipped with NVIDIA L40 GPÜs. The synthetic verification experiments were run using 16 CPU cores, 32 GB of memory, and a single GPU, The Jacobian complexity experiment was executed on CPU only, as the computation did not fit within GPU VRAM; to avoid out-of-memory (OOM) errors, 32 CPU cores and 400 GB of memory were allocated. The scaled-up synthetic experiment with the linear model used CPU cores, 64 GB of memory, and one GPU. The large language model (LLM) activation experiment was performed using 16 CPU cores, GB of memory, and single GPU.
The code that can replicate the main experiments presented in our paper can be accessed via https //github.com/xiaagg
ttemptmmmppppeee inst-sae
A.7 Linitations
We acknowledge certain limitations of our work. The linear approximation, while computationally efficient and theoretically grounded, may capture all nonlinear interactions present in LLM acti- vations. Future work should explore extending our framework to incorporate bounded nonlinearities while maintaining computational tractability. Additionally, developing methods automatically interpret the discovered causal structures in terms of human-understandable concepts remains challenge. Our method also assumes specific form of temporal dependency that might fully capture the long-range dependencies that LLMs can handle. The current formulation is limited to first-order temporal dependencies, and extending this higher-order dependencies would increase computational complexity. Lastly, tokenization has been shown to critically affect LLM identifiability during our evaluations, even though it is not inherently part of LLM interpretation methods. We emphasize the importance of choosing a tokenization strategy that preserves semantic information and maximizes the effectiveness of LLM interpretation approaches.
A.8 Societal Impacts
Our interpretability approach can improve transparency, support alignment interventions, facilitate debugging and bias detection, advance scientific understanding of causal representations, and inform educational tools that raise AI literacy. At the same time, deeper insight into model internals may enable malicious manipulation, create misplaced confidence in safety tools, widen resource disparities, expose private information from training data, and distract attention from broader social and governance measures. Future work should include collaboration with ethicists, social scientists, and policy experts to guide responsible use.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Major claims are described in abstract and emphasized in introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed in Appendix A.7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification: All theorems are supported with complete and correct proof in both main text and Appendix with assumptions clearly presented in main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: detailed description of the experimental setup is provided in Appendix A..3 for the synthetic experiments and in Appendix A.5 for the LLM activation experiments. The codebase required to reproduce the experiments included the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
33

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The model data used in our experiments are either from publicly available datasets or can be generated using the codebase provided in the supplementary materials. The main experimental results can reproduced using this submitted codebase.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Pollliiiciiiiiooiiiiiioooooiiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed settings are provided in the Appendix A.3 and A.5
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The mean value of multiple and with std plotted with error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Yes the computation resources use in the experiment is provided in Ap-
pendix A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps:/neurrps.ccc http://wwwwrppaccooooo....aactta
Answer: [Yes]
Justification: authors have reviewed and conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed in seperate section in Appendix A.8
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The assets including baseline codes and the dataset and models explicitly mentioned and credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Detailed instructions have been provided along with the codebase. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
38


Reviews:
[{"review_id": "2pryT49Y7E", "reviewer": "Reviewer_Uu1b", "summary": "The authors claim four contributions:\n1. A framework to model time-delayed causal relations between concepts and instantaneous constraints, with theoretical guarantees in the direction of reliability and explainability.\n2. Algorithms tailored to causal representation learning for the high-dimensional concept space of LLMs.\n3. Synthetic experiments supporting the results, scaled to match real-world complexity and applied on real LLMs.", "questions": "1. Usually, LLM activations are viewed as attention computations, and there is no sequential time dependence. Why was the modeling choice of sequential time dependence made in this work? Is there evidence support the claim that this is superior to an attention style of activation modeling? Is there a benefit of having an SEM, which induces interventional distributions, rather than a probabilistic model?\n2. Are the latent variables $\\mathbf z_t$ usually confirmed to be identifiable, given that there are many more latent than observed variables? What is the value of showing that identifiability holds in the synthetic setting in Experiment 5.1?\n3. Is there evidence to support the claim that the proposed SEM in equations (1), (2), and (3) is consistent with observational activations generated by real LLMs? For example, how well does the proposed model fit the actual distribution of activations? Is there any baseline for how well a model can fit this distribution, and does the proposed model improve upon it?\n4. Given that the model proposed is an SEM rather than a purely observational model - is there evidence to support the claim that there are the instantaneous and temporal mechanisms can be modeled linearly in the specific way this work models them?\n5. Is there evidence to indicate whether or not the preconditions of Thm 1-3 are met, sparsity in particular?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Originality: the authors introduce a linear SEM to model LLM activations. I am unfamiliar with the related mechanistic interpretability literature, but this concept is novel to my knowledge.\n\nQuality and significance: It is shown that the underlying latent variables in the SEM can be identified under some assumptions, including sparsity of latent components. It does not seem to be shown that the SEM can model the distribution of activations; case studies are performed (selecting <=6 examples of specific types of successes in total), but I did not see an objective or large-scale analysis supporting this claim. In other words, evaluation appears to be limited.\n\nClarity: contributions, methodology, theory, and results are stated clearly.", "was_revised": true, "final_justification": "Authors claim three new results: competitiveness w.r.t. the baseline of SAE in activation reconstruction, and potential improvements in absorption and concept recovery. The results are not claimed with statistical soundness.", "rebuttal": "We thank the reviewer for the thoughtful review and the opportunity to clarify key aspects of our work. We address each concern below.\n\n> **W1. It does not seem to be shown that the SEM can model the distribution of activations; case studies are performed (selecting <=6 examples of specific types of successes in total), but I did not see an objective or large-scale analysis supporting this claim. In other words, evaluation appears to be limited.**\n\n**AW1:** Thank you for this thoughtful comment. Our primary contribution is to identify latent concepts and their temporal or instantaneous relations under sparse structural assumptions with theoretical guarantees. While modeling the full distribution of LLM activations is not our focus, our method captures the structure of the activation space well—evidenced by near-zero reconstruction loss (the first columns of Figures 8 and 9).\n\nThough only a few case studies appear in the main text due to space limits, we include over 20 additional examples in Tables 2 and 3 (appendix), along with longer-range temporal analyses demonstrating richer semantic dynamics. Additional visualizations and discussions will be included in the updated manuscript.\n\nTo address evaluation rigor, we added SAEBench experiments using standard metrics (e.g., reconstruction loss, sparse probing), showing our model performs comparably to strong SAE baselines (see our responses AQ3 to Reviewer WPzT). We also introduce a semi-synthetic legal-vs-non-legal task to evaluate our model’s ability to extract interpretable relations (see our responses AW2,W4 to Reviewer gsa4).\n\nThese tasks are aligned with our central goal: interpretable representation and relational discovery, rather than generative modeling. Finally, we emphasize that meaningful, human-aligned examples are critical for LLM interpretability. We will clarify this distinction and further strengthen both empirical and illustrative support in the revision.\n\n> **Q1a. Usually, LLM activations are viewed as attention computations, and there is no sequential time dependence. Why was the modeling choice of sequential time dependence made in this work? Is there evidence support the claim that this is superior to an attention style of activation modeling?**\n\n**AQ1a:** We first clarify that there is indeed sequential time dependence in attention computation, as the current position needs to attend to previous positions. Additionally, there is natural sequential dependence in the semantic concept space during text generation, i.e., what you think or write now definitely depends on what you thought or wrote in the past. Therefore, when using learned activations to interpret LLMs' internal processes, sequential modeling is a very natural choice.\nRegarding attention-style modeling, it is known to be highly inexplicable, which is precisely the motivation behind our proposal.\n\n> **Q1b. Is there a benefit of having an SEM, which induces interventional distributions, rather than a probabilistic model?**\n\n**AQ1b:** Actually SEM is a probabilistic model; the distinction is that an SEM specifies how variables respond to explicit interventions, not just how they co‑vary observationally. Our objective is to explain how one internal concept can influence another within the LLM. This question is inherently interventional. The benefit of an SEM over any associational probabilistic model is that it explicitly models causal mechanisms, allowing us to understand not just correlations but directional influences between concepts.\n\n> **Q2a. Are the latent variables $\\mathbf{z}_t$ usually confirmed to be identifiable, given that there are many more latent than observed variables?**\n\n**AQ2a:** Yes, this is a standard setting in CRL and sparse dictionary learning literature which usually referred with term \"over-complete\" setting. A number of approaches [3,A,B] including our work leverage the structural sparsity to establish the identifiability.\n\n>**Q2b. What is the value of showing that identifiability holds in the synthetic setting in Experiment 5.1?**\n\n**AQ2b:** For sanity check! Theoretical identifiability proofs do not automatically guarantee that practical algorithms can recover latent variables under realistic constraints (finite data, noise, optimization challenges). And experiment 5.1 bridges this theory-practice gap by using synthetic data with known ground truth to confirm our implementation converges to the theoretically identified solution, and establish performance baselines before applying to real data where ground truth is unavailable. Such synthetic-to-real validation is a standard methodology in CRL.\n\n>**Q3. Is there evidence to support the claim that the proposed SEM in equations (1), (2), and (3) is consistent with observational activations generated by real LLMs? how well does the proposed model fit the actual distribution of activations? Is there any baseline for how well a model can fit this distribution, and does the proposed model improve upon it?**\n\n**AQ3:** Yes! We first demonstrate our modeling consistency with observational activations via the convergence of reconstruction loss (Figures 8-9 in Appendix) to sufficiently small values. We also compare it with estabilished SAE baselines:\n|Model|Reconstruction Loss $\\downarrow$|\n|-|-|\n|ReLU SAE|0.0110|\n|TopK SAE|0.0097|\n|Ours|0.0108|\n\nOur method also achieves comparable reconstruction loss (~0.01), which further confirms the consistency is at least on-par with those SAEs. \n\nWe emphasize that distribution matching, while necessary, **is not our primary contribution**. Any universal approximator can achieve near-perfect reconstruction. The critical challenge is **uniquely identifying the latent causal structure** underlying these activations. Our method not only fits the data distribution but also provides theoretically grounded guarantees for recovering the true causal relationships--Section 5.2 demonstrates that recovered causal dependencies align with known semantic patterns in language--which existing methods like SAEs cannot provide.\n\n>**Q4. Given that the model proposed is an SEM rather than a purely observational model - is there evidence to support the claim that there are the instantaneous and temporal mechanisms can be modeled linearly in the specific way this work models them?**\n\n**A4:** Yes! And we address this question through three key points:\n\n**SEM v.s. purely observational model** Please allow us to clarify that SEM could model both observational and latent variables; so they are not mutually exclusive. \n\n\nWe employ latent variables in SEM to model LLM activation dynamics (rather than observationally) because LLM activations are inherently polysemantic [D]: each dimension encodes multiple entangled concepts, making direct concept-to-concept analysis infeasible in the raw activation space. Such latent variable approach is well-established in literatures.\n\n**Evidence for instantaneous and temporal mechanisms** The existence of such causal relationships has been extensively studied in the mechanistic interpretability community, with recent empirical studies [1,28] confirming these relations  (line 32 and footnote 1).\n\n**Justification for our linear modeling** Existing work [41] showed semantic features occupy linear subspaces, and SAEs [28,50,D] successfully use sparse linear model for finding concepts. Those established the fundation for linear modeling. More importantly the evidence comes from our own successful recovery of semantically meaningful causal structures (Section 5.2). This establishes linearity as an empirical reality at the relevant abstraction level rather than a simplifying assumption.\n\n> **Q5. Is there evidence to indicate whether or not the preconditions of Thm 1-3 are met, sparsity in particular?**\n\n**A5:** This is a very good question! Verifying the **latent assumptions** about hidden variables is a fundamental challenge to all research on latent variable identification [E] since latent variables are not directly measurable. We approach to their reasonableness through **indirect validation**: if the model successfully fits the observed data distribution and consistently yields to meaningful solutions, this provides evidence that the underlying assumptions are moderate and feasible. This approach provides a practical solution for this challenge and are widely acknowledged in the community [1,54].\n\n**Regarding sparsity in particular**, evidence partially comes from empirical success from the SAE and CRL literatures [25,50,A,B,C,D] to use sparsity constraints to extract concepts recover non-trival causal relations. More importantly, evidence also comes from our own successful recovery of semantically meaningful causal structures (Section 5.2) together with the convergence behavior of our loss curve during training.\n\nWe acknowledge that these assumptions, like any in this field, may not hold universally, our empirical results suggest they seem effective and feasible for the complex and real data we study.\n\n[A] Sun, Yuchen, and Kejun Huang. \"Global identifiability of overcomplete dictionary learning via l1 and volume minimization.\" The Thirteenth International Conference on Learning Representations. 2024.\n\n[B] Chen, Siyu, et al. \"Taming Polysemanticity in LLMs: Provable Feature Recovery via Sparse Autoencoders.\" arXiv preprint arXiv:2506.14002 (2025).\n\n[C] Wang, Kexin, and Anna Seigal. \"Identifiability of overcomplete independent component analysis.\" arXiv preprint arXiv:2401.14709 (2024).\n\n[D] Elhage, et al., \"Toy Models of Superposition\", Transformer Circuits Thread, 2022.\n\n[E] Peters, Jonas, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT press, 2017."}, {"review_id": "PVgBoCQXoq", "reviewer": "Reviewer_gsa4", "summary": "In this work, the authors address the problem of interpretability in large language models (LLMs) through the framework of sparse autoencoders (SAEs). A known limitation of SAE is its lack of shared mechanisms across tokens, which hampers the discovery of interpretable concepts that capture temporal dependencies in LLM activations. Moreover, SAEs typically fail to model correlated concepts, limiting their ability to recover instantaneous relationships between them. The authors propose to overcome these limitations by modeling concepts using linear structural causal models (SCMs), which allows for both temporal and instantaneous causal relationships among concepts. Building on prior results from the causal representation learning (CRL) literature, they establish identifiability guarantees for the learned concepts and causal relationships, and empirically validate these findings on a synthetic dataset. Additionally, the authors present a case study on a real-world LLM dataset, demonstrating that their approach uncovers a range of meaningful temporal and instantaneous relationships.", "questions": "Please refer to the *weaknesses* section above for major questions. In addition, I have the following comments and clarifying questions for the authors:\n\n- There seems to be a typo on line 77, I guess the matrix $A$ should have dimensions $m \\times n$? Since in the theorems ahead authors represent the latent dimension of $z$ with the symbol $n$.\n\n- There seems to be a typo on line 146, authors write $z_{it}$ while it should be $z_{ti}$.\n\n- In Theorem 1, we defined $c_t = ( z_{t-1}, z_t, z_{t+1} )$ while in Theorem 2 we defined $c_t= (z_{t-1}, z_t)$. Why do we change the definition of $c_t$ across theorems?\n\n- On line 131, authors should specify that the diagonal matrix $D$ is also invertible, as not all diagonal matrices are invertible. \n\n- Could the authors explain the intimate neighbor set used in Theorem 1?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- While the proposed approach and identifiability analysis builds largely on prior work and are not particularly novel on their own, the application of this temporal CRL framework to LLM interpretability is novel. Temporal relationships are important for LLM interpretability, and to the best of my knowledge, this is the first work to recover them using SAEs.\n\n- The authors have done a good job with the experimental design. Scaling the synthetic dataset to include more concepts is an important step in demonstrating that the proposed approach can operate at the scale required for LLM interpretability tasks. Furthermore, the results in Appendix A.3.2 effectively justify the selection of key hyperparameters, such as the time window ($\\tau$), sparsity penalty ($\\beta$), and the dimensionality of the latent concept space.\n \n\n**Weaknesses**\n\n- The motivation for extending prior temporal CRL approaches to high-dimensional settings feels somewhat unclear. The proposed method relies on the assumption of a linear mixing function between concepts $z$ and observations $x$. Therefore, comparisons to existing CRL methods such as IDOL which are designed to handle the more general nonlinear case do not seem entirely appropriate. I suggest the authors tone down this aspect of their contributions. As currently framed (e.g., lines 57–58), the claim that this work significantly advances prior CRL methods appears overstated.\n\n- The synthetic data results in Section 5.1 serve as a useful sanity check; however, a more comprehensive analysis of concept identifiability using (semi-)synthetic language benchmarks with ground-truth concepts would strengthen the evaluation. For instance, the authors could adapt synthetic benchmarks from SSAE [1] to the temporal domain. This would provide clearer insights into whether LLMs can recover true concepts from language-based synthetic datasets.\n\n- The observation that a sparsity penalty of $\\beta=0.01$ outperforms $\\beta=0.1$ for learning sparse causal mechanisms is somewhat surprising. To better understand this behavior, I recommend the authors conduct experiments with a broader range of $\\beta$ value, such as randomly sampling from the interval $(0.001, 0.1)$  to analyze the trend. Additionally, an important sanity check would be to evaluate the case $\\beta=0$  which removes the sparsity regularizer entirely. Moreover, including ablation studies on the hyperparameter $\\alpha$ would strengthen the paper; it would be valuable to see how variations in $\\alpha$ affect training dynamics and the recovered mechanisms.\n\n- While the case study results are interesting, the authors could further evaluate their approach on related tasks such as finding steering vectors using identifiable concept learning methods with LLMs [1, 2]. It would be valuable to know if the proposed approach can infer accurate steering vectors, and if not, what limitations the authors anticipate.\n\n- The paper overall is well written, however, the writing of the theoretical aspects needs improvement.\n     - The authors should avoid labeling their main theoretical result (Theorem 3) as a “theorem,” since it largely follows from Theorems 1 and 2 in prior work. Referring to it as a proposition or corollary would be more accurate.\n     -  When presenting theoretical results, even from prior works, the authors should clearly define all assumptions and key terms. For example, the notion of component-wise identifiability for latent variable recovery is never formally defined. Similarly, the term “isomorphism” used in Theorem 1 (regarding the estimated Markov network being isomorphic to the ground truth) is not properly explained.\n    - In Theorem 1, the authors never defined what is $\\varepsilon(\\mathcal{M} _{c_t})$?\n\nReferences:\n\n- [1] Joshi, Shruti, Andrea Dittadi, Sébastien Lachapelle, and Dhanya Sridhar. \"Identifiable steering via sparse autoencoding of multi-concept shifts.\" arXiv preprint arXiv:2502.12179 (2025).\n\n- [2] Rajendran, Goutham, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. \"Learning interpretable concepts: Unifying causal representation learning and foundation models.\" arXiv preprint arXiv:2402.09236 (2024).", "was_revised": true, "final_justification": "The authors addressed nearly all of my concerns during the rebuttal and I am have raised my score for leaning towards acceptance. Given my background being primarily causal representation learning rather than SAE, my concerns and suggestions were more centered around the theoretical results of identifiability and synthetic proof-of-concept experiments. I appreciate that the authors added new experiments for identifying steering vectors and concept relations. My concerns regarding the optimization objective for enforcing sparsity are not completely resolved and perhaps authors could improve more on that aspect. However, given that I am not an expert in mechanistic interpretability and what frameworks are typically used in the SAE literature, I am keeping my confidence 3.", "rebuttal": "We sincerely appreciate the reviewer's detailed feedback and constructive suggestions. We address each concern below and will incorporate these improvements into our revised manuscript.\n>**W1. The motivation for extending prior temporal CRL approaches to high-dimensional settings feels somewhat unclear. The proposed method relies on the assumption of a linear mixing function between concepts $z$ and observations $x$. Therefore, comparisons to existing CRL methods such as IDOL which are designed to handle the more general nonlinear case do not seem entirely appropriate. I suggest the authors tone down this aspect of their contributions. As currently framed (e.g., lines 57–58), the claim that this work significantly advances prior CRL methods appears overstated.**\n\n**AW1:** We acknowledge this concern and here revised our framing to better reflect our contribution: \"this work provides first scalable CRL framework capable of LLM interpretability and representation learning.\"\n\nMeanwhile, please also allow us to clarify our motivation. Our ultimate goal is to reliably discover the concept-to-concept relations behind the LLM activations. Temporal CRL methods such as IDOL are very good fit for this task, since they provide both empirical success in other domains and, more importantly, identifiability guarantees. But almost all existing CRL methods heavily suffer from scalability issues (as shown in Figure 5, they cannot scale beyond 200 dimensions). This naturally guided us to find a proper simplification for computational feasibility, in which linear approximation is one of the options.\n\nLuckily, existing work including the Linear Representation Hypothesis together with Sparse Autoencoder studies demonstrated an empirical result that LLM activations can be modeled via linear model. Such evidence makes the linear simplification / assumption less problematic in this specific LLM activation setting.\n\n> **W2,W4. The synthetic data results in Section 5.1 serve as a useful sanity check; however, a more comprehensive analysis of concept identifiability using (semi-)synthetic language benchmarks with ground-truth concepts would strengthen the evaluation. For instance, the authors could adapt synthetic benchmarks from SSAE [1] to the temporal domain. This would provide clearer insights into whether LLMs can recover true concepts from language-based synthetic datasets. While the case study results are interesting, the authors could further evaluate their approach on related tasks such as finding steering vectors using identifiable concept learning methods with LLMs [1, 2]. It would be valuable to know if the proposed approach can infer accurate steering vectors, and if not, what limitations the authors anticipate.**\n\n**AW2,W4:** Thank you for this insightful suggestion. We agree that synthetic benchmarks like SSAE [1] can offer valuable insights into concept identifiability. Our primary focus is on identifying concept relations, though we acknowledge its close ties to concept extraction and steering vector interpretation. \n\nTo address this, we tested whether our model can recover steering vectors from paired text. Specifically, we constructed five categories of word pairs where only a single interpretable concept changes, including gender, plurality, comparative, tense, and negation. While these changes are intuitive, ensuring the word pairs capture a clear ground-truth concept is non-trivial. Despite this challenge, our model demonstrated strong performance in identifying the underlying concept differences. Specifically, (1) the average correlation of concept differences within each category exceeded 0.86; (2) assuming one ground-truth pair, the correlation rose above 0.93; and (3) the maximum correlation within a category reached over 0.94. These results support our claim that our model can indeed recover meaningful steering vectors.\n\nMoreover, building on your suggestion, we adapted SSAE [1] to test whether our model captures meaningful text relations. We constructed two contrastive collections of texts drawn from the Pile dataset. One collection consists of legal documents, which often follow structured syntactic formats from start to end, leading to stable temporal concept patterns. The other contains non-legal texts without such structure. We hypothesized that only legal texts would yield meaningful relational patterns. Indeed, we found that in the legal texts, two consistently activated concepts demonstrated a strong temporal relation, with a normalized weight of 0.70 in our recovered $B$. These concepts did not fire in non-legal texts, offering a clear contrast.\n\nThese additional experiments reinforce our model’s ability to recover accurate, interpretable concept representations and relations. We have updated our manuscript to incorporate these experiments and expand on the related discussion. Thank you for your suggestion again. \n\n\n>**W3. The observation that a sparsity penalty of $\\beta=0.01$ outperforms $\\beta=0.1$ for learning sparse causal mechanisms is somewhat surprising. To better understand this behavior, I recommend the authors conduct experiments with a broader range of $\\beta$ value, such as randomly sampling from the interval $(0.001, 0.1)$ to analyze the trend. Additionally, an important sanity check would be to evaluate the case $\\beta=0$ which removes the sparsity regularizer entirely. Moreover, including ablation studies on the hyperparameter $\\alpha$ would strengthen the paper; it would be valuable to see how variations in $\\alpha$ affect training dynamics and the recovered mechanisms.**\n\n**AW3:** Thank you for the constructive suggestion. We conducted additional comparisons with $\\beta = 0$, $0.001$, $0.005$, $0.05$, $1.0$ and $\\alpha = 0$, $0.001$, $0.01$ to cover a broader hyperparameter range, using $\\tau = 5$ and feature dimension 3072. The results are shown in the two tables below, with our selected setting *in italics*. The tables highlight that (1) concept relationships are inherently sparse, while a large $\\beta$ disrupts optimization, and (2) $\\alpha$ has a stronger effect, with 0.1 being a well-balanced choice. We also expanded our manuscript to include a broader discussion of model dynamics with respect to $\\beta$ and $\\alpha$.\n\n|$\\beta$|0.0|0.001|0.005|*0.01*|0.05|0.1|1.0|\n|-|-|-|-|-|-|-|-|\n|Reconstruction Loss $\\downarrow$|0.0126|0.0126|0.0126|*0.0128*|0.0126|0.0128|0.8950|\n|Independence Loss $\\downarrow$|0.1522|0.1496|0.1504|*0.1448*|0.1550|0.1582|3.7682|\n|Bs Sparsity (L1) $\\downarrow$|0.0003|0.0005|0.0005|*0.0007*|0.0007|0.0013|0.0053|\n|M Sparsity (L1) $\\downarrow$|0.0001|0.0001|0.0001|*0.0001*|0.0001|0.0002|0.0007|\n\n|$\\alpha$|0.0|0.001|0.01|*0.1*|1.0|\n|-|-|-|-|-|-|\n|Reconstruction Loss $\\downarrow$|0.0227|0.0191|0.0118|*0.0128*|0.1121|\n|Independence Loss $\\downarrow$|4.3849|2.4572|0.3910|*0.1448*|0.5252|\n|Bs Sparsity (L1) $\\downarrow$|0.0012|0.0007|0.0018|*0.0007*|0.0058|\n|M Sparsity (L1) $\\downarrow$|0.0002|0.0001|0.0002|*0.0001*|0.0009|\n \n>**W5. The paper overall is well written, however, the writing of the theoretical aspects needs improvement.**\n\n**AW5:** Thank you for this detailed feedback on improving the theoretical presentation. We have addressed all the suggested modifications and have updated to the paper:\n\n1. We have relabeled our main result as \"Proposition 3\" instead of \"Theorem 3\" to more accurately reflect that it builds upon the foundational Theorems 1 and 2 from prior work.\n2. A component‑wise transformation on a vector $\\mathbf{x} = (x_1, \\dots, x_n) \\in \\mathbb{R}^n$ is a map $T : \\mathbb{R}^n \\to \\mathbb{R}^n$ that permutes the coordinates using a permutation $\\sigma$ and applies possibly different scalar functions $f_i : \\mathbb{R} \\to \\mathbb{R},\\; i \\in [n]$, resulting in $T(\\mathbf{x}) = (f_1(x_{\\sigma(1)}), \\dots, f_n(x_{\\sigma(n)}))$.\n3. The $\\mathcal{M}_{c_t}$ denotes the Markov network defined on the variable set of consequtive timestamps.\n4. Isomorphism of Markov networks is formaly defined as follows. Let $V(\\cdot)$ be the vertex set of any graph. An isomorphism of Markov networks $M$ and $\\hat{M}$ is a bijection between the vertex sets of $M$ and $\\hat{M}$, $f : V(M) \\to V(\\hat{M})$ such that any two vertices $u$ and $v$ of $M$ are adjacent in $M$ if and only if $f(u)$ and $f(v)$ are adjacent in $\\hat{M}$.\n\n> **Q1-4. Comments and clarifying questions**\n\n**AQ1-4:** Thank you for pointing these out. We have corrected the typo on line 77: the matrix $A$ should indeed have dimensions $m \\times n$, as $\\mathbf{x} \\in \\mathbb{R}^m$ and $\\mathbf{z} \\in \\mathbb{R}^n$. The expression on line 146 has also been corrected to $z_{t,i}$.\n\nRegarding the symbol $c_t$, we acknowledge the inconsistency in its usage across Theorems 1–3. In Theorem 2, $c_t$ refers to a context of two consecutive time steps, while in Theorems 1 and 3, it involves three time steps. To clarify, we will explicitly label the context length as $c_t^{(2)}$ and $c_t^{(3)}$ respectively.\n\nWe also now specify that the diagonal matrix $D$ on line 131 is required to be invertible.\n\n> **Q5. Could the authors explain the intimate neighbor set used in Theorem 1?**\n\n**AQ5:** Certainly. The *Intimate Neighbor Set* is defined as follows: consider a Markov network $M_{Z}$ over variable set $Z$. The intimate neighbor set of variable $Z_i$ is given by \n\n$\\Psi_{M_Z}(Z_{i}) \\triangleq \\{ Z_{j} \\;|\\; Z_{j} \\text{ is adjacent to } Z_{i} \\text{ and is also adjacent to all other neighbors of } Z_{i}, Z_{j} \\in Z \\setminus \\{Z_{i}\\} \\}$\n\nThis definition ensures that $Z_{j}$ shares maximal context with $Z_{i}$ in the conditional dependency structure, hence quantify the sparsity of the latent process. We have added this definition to the footnote on page 4 for clarity."}, {"review_id": "h71N1KXOYR", "reviewer": "Reviewer_WPzT", "summary": "This paper applies ideas from causality to improve SAEs by explicitly modeling relations between latents in the SAE within a single token, and relations between tokens in a series. The work provides theoretical guarantees about the soundness of this method when applied to data generated following the linear representation hypothesis (LRH). This method improves the scalability of casuality methods to work with high-dimensional LLM activations and features. The method is validated experimentally on synthetic examples, and qualitatively on data generated from a small LLM.", "questions": "- Can this method model hierarchical relationships between latents? E.g. if z1 can only fire if z2 is firing, but z1 can on its own, can this be captured by this model? This would correspond to something like \"z1 = noun\" and \"z2 = dog\", since \"dog\" is a \"noun\". Hierarchical relationships seem to be break standard SAEs (causing feature absorption), so if this method can handle that it would be huge.\n- Can this method only handle relations between 2 latents at a time? e.g. if a latent is and AND of 3 latents, can this method capture that?\n- In Figure 3, is (e) the correlation between A and $\\hat{A}$ not shown? Also, why is estimated $\\hat{B}$ not identical to B? It seems like the method should be able to perfectly recover B in this synthetic setting?\n- In Equation 7, why is there no nonlinearity (presumably ReLU) in the encoder? Should there also be encoder and decoder bias terms?\n- It seems like the $\\hat{A}$ matrix is equivalent to an SAE decoder dictionary - can this matrix be directly used as an SAE? Would you expect this method to learn a more correct dictionary than standard SAE training methods?\n- In the paper the trained LLM models are still very small - is it feasible to train a more standard size version of this method to compare with SAEs, e.g. 32k+ latents?\n- Can this method be directly compared to LLM SAEs using standard SAE benchmarks like sparse probing, or other metrics from SAEBench?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\nThe method provided can give insights into relations between SAE latents both within and across tokens, which is valuable for understanding how latents relate to each other. The method provided has better scalablility characteristics than existing causality based methods, allowing this method to scale to LLMs. Explicitly borrowing ideas from causality is valuable to the interpretability community, given the parallels between LLM interpretability and causality.\n\n### Weaknesses\n\n- While more scalable than existing causality methods, the method seems to still have poor scalability compared to standard SAEs. The paper only evaluates dictionaries up to size 6144, which is miniscule by LLM SAE standards. \n- The paper also does not compare this technique directly with LLM SAEs (e.g. on interpretability benchmarks like SAEBench), so it's hard to gauge if / how much this improves on standard SAEs in practice.\n- The temporal relations are modeled by fixed token distances, which seems inflexible and likely cannot capture real syntac relationships between tokens well.", "was_revised": true, "final_justification": "Applying insights from causality to SAEs is an important contribution to the field. If true that this approach also has the side-effect of improved handling of hierarchical features in SAEs without degradation in performance, as their results seem to imply, I think this is an extremely important work that will rapidly get adopted in the field.", "rebuttal": "We sincerely thanks for your insightful feedback. We're glad you appreciate the strengths in our novelty for being the first to scale causal representation learning to LLM activations, and we appreciate your constructive suggestions for strengthening our contribution. We address each point below.\n\n>**W1,Q6 While more scalable than existing causality methods, the method seems to still have poor scalability compared to standard SAEs. The paper only evaluates dictionaries up to size 6144, which is miniscule by LLM SAE standards. In the paper the trained LLM models are still very small - is it feasible to train a more standard size version of this method to compare with SAEs, e.g. 32k+ latents?**\n\n**AW1,Q6:** Thank you for this important question about scalability. While the reported dictionary size of 6,144 was indeed limited by GPU memory constraints, as you also mentioned in your question, this dimension already represents a significant advancement in terms of dimensionality for identifiable representation learning methods. Our experimental results demonstrate that even with this dictionary size, we can extract meaningful interpretations of concept-to-concept relations.\n\nFor larger dictionary sizes, our approach can be parallelized using tensor parallelism or data parallelism to accommodate greater scale. However, this would require substantial refactoring of existing open-source infrastructure tools like `TransformerLens` and `dictionary_learning` to support multi-GPU training, which falls well outside the scope of our current work. We acknowledge this limitation and consider scaling to 32k+ latents an important direction for future research.\n\n>**W2,Q7 The paper also does not compare this technique directly with LLM SAEs (e.g. on interpretability benchmarks like SAEBench), so it's hard to gauge if / how much this improves on standard SAEs in practice. Can this method be directly compared to LLM SAEs using standard SAE benchmarks like sparse probing, or other metrics from SAEBench?**\n\n**AW2,Q7:** Thanks for the suggestion. Our main contribution is the recovery of temporal and instantaneous concept‑to‑concept relations, which is not reflected in SAE benchmarks. But we do agree with the reviewer that making a comparison on established benchmarks for SAEs will strengthen our position. For a fair apples-to-apples comparison, we trained two baseline SAEs (ReLU and TopK) under similar hyperparameters and evaluated all three methods on a range of SAEBench metrics, including sparse probing. Since we did not modify the SAE objective, we expect our model to perform on par with existing SAEs on SAEBench tasks, which is confirmed by the table below.\n\n|Model|Reconstruction Loss $\\downarrow$|Sparse Probing Top 1 Accuracy $\\uparrow$|Absorption Score $\\downarrow$|Autointerp Score $\\uparrow$|\n|-|-|-|-|-|\n|ReLU SAE|0.0110|0.6555|0.0141|0.6791|\n|TopK SAE|*0.0097*|*0.7141*|0.0280|0.6822|\n|Ours|0.0108|0.6736|*0.0139*|*0.6883*|\n\n\n>**W3. The temporal relations are modeled by fixed token distances, which seems inflexible and likely cannot capture real syntac relationships between tokens well.**\n\n**AW3:** We totally agree that fixed limited token distances might not be the most ideal way to capture the rich semantic and syntactic relationships between tokens. However, it is worth pointing out that even though the distance is limited, it still successfullly reveals some meaningful temporal relations (together with instantaneous relations) among the concepts as we have shown in the real-world experiments.\nWe further conducted the analysis for longer temporal relations ($\\tau=100$). Due to the rebuttal policy, we may not show the visualization results. We will add those results to our appendix to more comprehensively showcase the interpretability our proposal may achieve.\n\n>**Q1. Can this method model hierarchical relationships between latents? E.g. if z1 can only fire if z2 is firing, but z2 can on its own, can this be captured by this model? This would correspond to something like \"z1 = noun\" and \"z2 = dog\", since \"dog\" is a \"noun\". Hierarchical relationships seem to be break standard SAEs (causing feature absorption), so if this method can handle that it would be huge.**\n\n**AQ1:** Yes, it can! The graph for instantaneous relations can naturally model the hierarchical relationships among concepts. Our case study for instantaneous relations `Month (e.g. March)` to `Full date (e.g. March 23, 2000)` is exactly such a case, `Month` can be fired on its own, but if `Full Date` is fired, then `Month` will also be fired. Such an advantage is also captured quantitatively by the better feature absorption score in the table above.\n\n>**Q2. Can this method only handle relations between 2 latents at a time? e.g. if a latent is and AND of 3 latents, can this method capture that?**\n\n**AQ2:** Thank you for this great question. Indeed, the relations recovered by our method are represented in matrices where each entry reflects pairwise relations. But they can still support the inference of higher-order logical relations, even though it's not immediately apparent. For example, if a concept activates only when multiple other concepts are active (e.g., an AND of three latents), this can be inferred by analyzing joint activation patterns over many observations. By thresholding and binarizing concept activations, one can construct truth tables and recover logical relations such as AND, OR, or XOR. While not explicitly modeled in the relation matrices, such higher-order dependencies are reflected in the data and can be recovered through post-hoc analysis.\n\n>**Q3. In Figure 3, is (e) the correlation between A and $\\hat{A}$ not shown? Also, why is estimated $\\hat{B}$ not identical to B? It seems like the method should be able to perfectly recover B in this synthetic setting?**\n\n**AQ3:** Thanks for pointing this out, our identifiability was established for latent variables $\\mathbf{z}_t$ (up to permutation and scaling) and their relations $B$ and $M$, hence the exact value matching for $A$ and $\\hat{A}$ is not required and we allow some indeterminacy for $A$. Below is an example of the ground truth $A$ and our estimated $\\hat{A}$, and $\\mathbf{z}_t$ for this setting is already identifiable (MCC > 0.99).\n\n$A$ = [[ 0.38,  0.92, -0.02],[-0.67,  0.29,  0.68], [ 0.64, -0.25,  0.73]]\n\n$\\hat{A}$ = [[-3.38, -7.56, 0.22],[ 5.34, -2.63, -5.63],[-5.22,  1.85, -6.01]]\n\nDespite the theoretical guarantee for $B$, in practice, the perfect recovery requires optimization using an L0 sparsity penalty, which is known to be non-differentiable. In the literature, the L1 alternative is widely adopted. That explains the small difference in the estimated $B$ while it can still effectively preserve most of the structure information.\n\n>**Q4. In Equation 7, why is there no nonlinearity (presumably ReLU) in the encoder? Should there also be encoder and decoder bias terms?**\n\n**AQ4: Nonlinearity**: Thanks for pointing this out! Yes, in real-world LLM experiments, we use **top-k** activation in the encoder, following mainstream SAE architectures, which is non-linear. We use simple affine transformations in synthetic experiments for theoretical tractability.\n\n**Bias terms**: Yes, there should be bias terms for more modeling flexibility. We modified the equation to include them. We also compared these two settings in the real LLM activations (feature dimension=3072, $\\alpha=0.1$, $\\beta=0.01$, $\\tau=5$). The results shown in the table below indicate that the flexibility gain of the bias terms is not significant in our model. \n\n|Metric (real-world)|Without Bias|With Bias|\n|-|-|-|\n|Reconstruction Loss|0.0129|0.0129|\n|Independence Loss|0.1452|0.1457|\n|$B$ Sparsity (L1)|0.0007|0.0007|\n|$M$ Sparsity (L1)|0.0001|0.0001|\n>**Q5. It seems like the $\\hat{A}$ matrix is equivalent to an SAE decoder dictionary - can this matrix be directly used as an SAE? Would you expect this method to learn a more correct dictionary than standard SAE training methods?**\n\n**AQ5:** Yes, this can be used as an SAE. As shown in **AW2,Q7**, the performance on SAE tasks for our method is at least on-par with existing SAEs. We acknowledge that there may not be a universally more correct dictionary; we could say our learned dictionary is relation-aware, which can be more reasonable and capable than standard SAEs."}, {"review_id": "gRx6xNCt5c", "reviewer": "Reviewer_KYay", "summary": "This paper introduces a framework that integrates sparse autoencoders with causal representation learning to interpret LLMs. It captures temporal and instantaneous causal relationships in model activations, provides theoretical guarantees, and scales to LLMs' high-dimensional concept space, enabling the discovery of meaningful and interpretable features.", "questions": "1. Will the performance drop after the method is applied?", "limitations": "This paper does not mention the limitation. As far as I know, the Interpretability will harm the performance.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper introduces the use of the linear representation hypothesis to provide theoretical guarantees for sparse autoencoders, bridging a gap between empirical interpretability and formal causal representation learning.\n2. It effectively combines sparse autoencoders with temporal causal modeling, enabling the discovery of both time-delayed and instantaneous conceptual relationships in LLM activations—something prior methods could not scale to or capture.\n\nWeakness:\n1. The paper's objective lacks clarity—while it aims to enhance LLM interpretability, the connection between the linear representation hypothesis and LLMs is not well established. In particular, it remains unclear how variables like $x_t$ and $z_t$ are concretely defined within the LLM context.\n2. The use of synthetic time series data undermines the relevance of the evaluation, as it does not accurately reflect the structural or statistical properties of LLM activations, which the method ultimately targets.\n3. The paper lacks comparison with strong baseline methods and omits quantitative evaluation metrics, making it difficult to assess the practical effectiveness or improvements of the proposed approach.", "was_revised": true, "final_justification": "Thanks for the rebuttal. The concerns are from the time series for the synthetic experiments and human-readable interpretability. First, the authors use the time series to demonstrate a language problem. It is not reasonable. Second, the authors do not provide the conditions under which the assumption (Equation 2) holds. Third, the goal of interpretability is to understand the LLM. Since the authors say, \"From the LLM perspective, concept-relations embedded in LLM activations are intrinsically tied to real natural language semantics, which are highly complex. Exactly recovering the complete set of concepts and relations remains an open goal for the entire mechanistic interpretability community. Hence, it is challenging to construct synthetic data that covers all concept-relations in natural languages. We interpret your suggestion as asking whether we can use controlled natural language data with some known concept-relations to demonstrate that such phenomena in LLMs can be recovered by our model.\" It can be evidence that the LLM as a judge is a better explainer to understand how the LLM works.", "rebuttal": "Thank you for acknowledging our contribution between empirical interpretability and formal causal representation learning and, more importantly, the scalability issue we address. We now address your concerns one by one.\n\n>**W1. The paper's objective lacks clarity—while it aims to enhance LLM interpretability, the connection between the linear representation hypothesis and LLMs is not well established. In particular, it remains unclear how variables like $x_t$ and $z_t$ are concretely defined within the LLM context.**\n\n**AW1:** The Linear Representation Hypothesis is well-established for LLMs through prior work [A], which demonstrated that LLM activations can be effectively modeled using linear methods.\n\nTo clarify our notation: $\\mathbf{x}_t \\in \\mathbb{R}^m$ represents the activation vector at layer $L$ for token position $t$, where $m$ is the model's hidden dimension. The latent variables $\\mathbf{z}_t \\in \\mathbb{R}^n$ capture $n$ interpretable concepts that generate these activations through the linear decoder $A$.\n\n**Specifically within the LLM context (line 73-90)**: Given input tokens $(v_1, v_2, \\ldots, v_T)$, we extract activations $\\mathbf{x}_t$ from a target transformer layer (e.g., layer 12 of 32) at each position $t$. Our model decomposes these activations into interpretable concepts $\\mathbf{z}_t$ (Eq. 1, 2) that exhibit temporal dependencies $B$ (how concepts evolve across tokens) and instantaneous relationships $M$ (how concepts interact within each token). We will add this clarification to Sec. 2 with specific examples to demonstrate this mapping concretely.\n\n[A] Park, Kiho, Yo Joong Choe, and Victor Veitch. \"The linear representation hypothesis and the geometry of large language models.\" ICML (2024).\n\n>**W2. The use of synthetic time series data undermines the relevance of the evaluation, as it does not accurately reflect the structural or statistical properties of LLM activations, which the method ultimately targets.**\n\n**AW2:** This synthetic-to-real validation paradigm is a widely accepted methodology in causal representation learning and related fields. We establish theoretical identifiability guarantees for learning concepts and relations. Since ground truth is unavailable in real-world scenarios, this identifiability cannot be directly testable on real LLM data. Therefore, controlled synthetic experiments with known ground truth are essential for verifying both our theoretical analysis and algorithm implementation.\n\nOnce we establish confidence that our theoretical guarantees translate into practical performance, subsequent experiments on real LLM activations (Sections 5.2 and 5.3) demonstrate the method's relevance to authentic data. We have revised the manuscript to clarify this rationale and emphasize that synthetic studies complement and validate, rather than replace, our evaluations on real LLM data.\n\n\n>**W3. The paper lacks comparison with strong baseline methods and omits quantitative evaluation metrics, making it difficult to assess the practical effectiveness or improvements of the proposed approach.**\n\n**AW3:** As acknowledged by other reviewers and mentioned on pages 6–7, our work is the first to address temporal and instantaneous relations at the LLM activation scale. We extensively searched for baselines from the causal representation learning (CRL) literature, but to our knowledge, no existing method can efficiently handle the high-dimensional activation data produced by LLMs (Figure 5). We would appreciate clarification on any specific missing baselines you have in mind.\n\nWe agree that quantitative analysis can strengthen our approach. While no existing benchmark captures causal representation learning for concept relations in LLM activations, we demonstrate our method's effectiveness through two evaluations: (1) testing on SAEBench [A] to show our model matches mainstream SAEs' performance on standard SAE tasks, and (2) presenting quantitative results on a new task tailored to our unique contribution—recovering concept-to-concept relations.\n\n**(1) SAEBench:** For fair comparison, we trained TopK and ReLU SAEs with hyperparameters (feature dimension, training cost, etc.) similar to our model:\n| Model | Reconstruction Loss $\\downarrow$ |Sparse Probing Acc. $\\uparrow$ |Absorption Score $\\downarrow$|Autointerp Score $\\uparrow$|\n|-|-|-|-|-|\n|ReLU SAE| 0.0110|0.6555|0.0141|0.6791|\n|TopK SAE| 0.0097| 0.7141|0.0280|0.6822|\n| Ours  | 0.0108   |0.6736|0.0139|0.6883|\n\nThese results demonstrate that while not specifically designed for SAE tasks, our model achieves comparable performance on these tasks. \n\n**(2) Concept Relation Recovery:** Furthermore, building on the experiments in [B], we **quantitatively** evaluate our model on concept-to-concept relation extraction. \n\n*Data preparation:* We constructed two contrastive collections of texts drawn from the Pile dataset. One collection consists of legal documents, which often follow structured syntactic formats from start to end, leading to stable temporal concept patterns. The other contains non-legal texts without such structure. We hypothesized that only legal texts would yield meaningful relational patterns, and we directly tested whether the model can successfully recover such patterns.\n\n*Baseline:* Although there is no directly applicable baseline, we leveraged standard SAEs we had trained above to serve as our baseline method. Since SAEs themselves cannot capture the concept-to-concept relations, we train a regression model to find temporal relation coefficient matrices $\\tilde{B}$s by solving the following regression task: \n\n$$\\text{min}\\_{\\tilde{B}\\_{\\tau}} ||\\mathbf{z}\\_t - \\sum_{\\tau} \\tilde{B}\\_{\\tau} \\mathbf{z}_{t-\\tau}||\\_{2}.$$\n\n*Evaluation Pipeline:* We calculate the concept recovery score by first getting the top fired feature index pair $(i,j)$ related to legal context and the corresponding entry $B_{i,j}$ in the aggregated temporal relation coefficient matrices. We then calculate the relation recovery score, similar to a signal-to-noise ratio, by: $\\texttt{relation recovery score} = \\frac{B_{i,j}}{\\sigma(B)}$, where $\\sigma(B)$ denotes the standard deviation of matrix $B$. Such ratio indicates the extent that the target concept relation entry in the matrix is more significant than a random noise; the larger the score is the more significant the relation recovery. \n\nFrom the table below, we notice a significant difference, showing that our method can successfully recover the relation.\n\n|Method|Relation Recovery Score $\\uparrow$|\n|-|-|\n|SAE+regression|2.39|\n|Ours|20.02|\n\nThese two experiments together outline our improvement over existing SAE methods by first showing our matching performance on SAE tasks and then demonstrating that our method can successfully extract meaningful concept-to-concept relations, which cannot be handled by SAEs.\n\n[A] Karvonen, Adam, et al. \"SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability.\" Forty-second International Conference on Machine Learning, 2025\n\n[B] Joshi, Shruti, Andrea Dittadi, Sébastien Lachapelle, and Dhanya Sridhar. \"Identifiable steering via sparse autoencoding of multi-concept shifts.\" arXiv preprint arXiv:2502.12179 (2025).\n\n>**Q1 Will the performance drop after the method is applied?**\n\n**AQ1:** We assume the performance you mentioned here refers to the language model's original metrics, such as downstream task accuracy and perplexity. Our interpretability method only reads transformer layer activations; it does not modify any weights, architecture, or forward computations. The model’s task performance therefore remains unchanged. While some interpretability approaches do fine‑tune model weights and can reduce performance, our analysis is not in this category. If the reviewer is considering a different notion of performance, we would appreciate further clarification.\n\n\n>**L1. This paper does not mention the limitation. As far as I know, the Interpretability will harm the performance.**\n\n**AL1:** We kindly refer the reviewer to Appendix **Section A.5** for the limitations discussion. The performance issue is discussed in the previous question."}]

Author rebuttal:

Meta-review: This paper presents a framework for the interpretability of large language models by combining notions of sparse autoencoders with a temporal causal representation learning (CRL) approach. The method introduces (linear) identifiable structural causal models to capture both temporal and instantaneous relations among concepts, and provide theoretical guarantees under the linear representation hypothesis. The authors present synthetic and semi-synthetic experiments, as well as case studies on LLM activations, and benchmark against standard SAEs.

**Strengths**:
* While the CRL approach is not novel, their applications to mechanistic interpretability is (as noted by reviewers), with the ability to recover temporal as well as instantaneous relations among concepts.
* Theoretical results providing identifiability, extending beyond what standard SAEs provide.
* Empirical results show competitive reconstruction performance to SAE baselines whenever comparisons are possible, and new experiments during rebuttal (e.g., SAEBench metrics, steering vectors, relation recovery tasks) strengthened the contribution.
* Reviewer discussions highlighted the potential for addressing feature absorption, an important challenge in SAE interpretability.

**Weaknesses**:
* Evaluation remains somewhat limited: while semi-synthetic and case study results are convincing, large-scale benchmarks and broader statistical testing are still lacking (mostly addressed during rebuttals).
* Some framing in the initial submission overstated the contribution relative to prior CRL methods (e.g., IDOL); authors agreed to soften this claim.
* Certain assumptions (linear mixing, sparsity) may restrict the generality of the approach, and their justification relies heavily on the linear representation hypothesis.

**Outcomes of the discussions**:
The back-and-forth discussion was productive. The authors added statistical testing against SAE baselines, new semi-synthetic experiments (e.g., steering vectors, legal vs. non-legal texts), and extended SAEBench evaluations. Reviewers gsa4 and WPzT increased their confidence in the work, with WPzT emphasizing its significance for the interpretability community. Reviewer Uu1b remained more skeptical, noting unresolved concerns around usefulness and statistical soundness, though they did raise their score after rebuttal. Reviewer KYay’s review remained critical, largely focusing on the use of synthetic time series; the authors responded with clarifications and further semi-synthetic results. Overall, I regard this latter review as quite superficial. 

**Summary**:
Overall, this paper presents an interesting and theoretically grounded step toward interpretable and identifiable causal structure discovery in LLMs. The novel theoretical framing, reported results, and strengthened experiments during the discussions, justify its acceptance as a poster. I encourage the authors to include the points and discussions raised during the rebuttal and discussion phase.
Decision: Accept (poster)