Title: LLM Interpretability with Identifiable Temporal-Instantaneous Representation
Venue: NeurIPS 2025
Accepted: yes
Keywords: identifiability

causal representation learning

large language models

interpretability
Abstract: Despite Large Language Models' remarkable capabilities, understanding their internal representations remains challenging. Mechanistic interpretability tools such as sparse autoencoders (SAEs) were developed to extract interpretable features from LLMs but lack temporal dependency modeling, instantaneous relation representation, and more importantly theoretical guarantees—undermining both the theoretical foundations and the practical confidence necessary for subsequent analyses. While causal representation learning (CRL) offers theoretically-grounded approaches for uncovering latent concepts, existing methods cannot scale to LLMs' rich conceptual space due to inefficient computation. To bridge the gap, we introduce an identifiable temporal causal representation learning framework specifically designed for LLMs' high-dimensional concept space, capturing both time-delayed and instantaneous causal relations. Our approach provides theoretical guarantees and demonstrates efficacy on synthetic datasets scaled to match real-world complexity. By extending SAE techniques with our temporal causal framework, we successfully discover meaningful concept relationships in LLM activations. Our findings show that modeling both temporal and instantaneous conceptual relationships advances the interpretability of LLMs.


Peer reviews:
comments: SUMMARY:
This paper introduces a framework that integrates sparse autoencoders with causal representation learning to interpret LLMs. It captures temporal and instantaneous causal relationships in model activations, provides theoretical guarantees, and scales to LLMs' high-dimensional concept space, enabling the discovery of meaningful and interpretable features.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper introduces the use of the linear representation hypothesis to provide theoretical guarantees for sparse autoencoders, bridging a gap between empirical interpretability and formal causal representation learning.
2. It effectively combines sparse autoencoders with temporal causal modeling, enabling the discovery of both time-delayed and instantaneous conceptual relationships in LLM activations—something prior methods could not scale to or capture.

Weakness:
1. The paper's objective lacks clarity—while it aims to enhance LLM interpretability, the connection between the linear representation hypothesis and LLMs is not well established. In particular, it remains unclear how variables like $x_t$ and $z_t$ are concretely defined within the LLM context.
2. The use of synthetic time series data undermines the relevance of the evaluation, as it does not accurately reflect the structural or statistical properties of LLM activations, which the method ultimately targets.
3. The paper lacks comparison with strong baseline methods and omits quantitative evaluation metrics, making it difficult to assess the practical effectiveness or improvements of the proposed approach.

QUESTIONS:
1. Will the performance drop after the method is applied?

LIMITATIONS:
This paper does not mention the limitation. As far as I know, the Interpretability will harm the performance.
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applies ideas from causality to improve SAEs by explicitly modeling relations between latents in the SAE within a single token, and relations between tokens in a series. The work provides theoretical guarantees about the soundness of this method when applied to data generated following the linear representation hypothesis (LRH). This method improves the scalability of casuality methods to work with high-dimensional LLM activations and features. The method is validated experimentally on synthetic examples, and qualitatively on data generated from a small LLM.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nThe method provided can give insights into relations between SAE latents both within and across tokens, which is valuable for understanding how latents relate to each other. The method provided has better scalablility characteristics than existing causality based methods, allowing this method to scale to LLMs. Explicitly borrowing ideas from causality is valuable to the interpretability community, given the parallels between LLM interpretability and causality.\n\n### Weaknesses\n\n- While more scalable than existing causality methods, the method seems to still have poor scalability compared to standard SAEs. The paper only evaluates dictionaries up to size 6144, which is miniscule by LLM SAE standards. \n- The paper also does not compare this technique directly with LLM SAEs (e.g. on interpretability benchmarks like SAEBench), so it's hard to gauge if / how much this improves on standard SAEs in practice.\n- The temporal relations are modeled by fixed token distances, which seems inflexible and likely cannot capture real syntac relationships between tokens well.\n\nQUESTIONS:\n- Can this method model hierarchical relationships between latents? E.g. if z1 can only fire if z2 is firing, but z1 can on its own, can this be captured by this model? This would correspond to something like \"z1 = noun\" and \"z2 = dog\", since \"dog\" is a \"noun\". Hierarchical relationships seem to be break standard SAEs (causing feature absorption), so if this method can handle that it would be huge.\n- Can this method only handle relations between 2 latents at a time? e.g. if a latent is and AND of 3 latents, can this method capture that?\n- In Figure 3, is (e) the correlation between A and $\\hat{A}$ not shown? Also, why is estimated $\\hat{B}$ not identical to B? It seems like the method should be able to perfectly recover B in this synthetic setting?\n- In Equation 7, why is there no nonlinearity (presumably ReLU) in the encoder? Should there also be encoder and decoder bias terms?\n- It seems like the $\\hat{A}$ matrix is equivalent to an SAE decoder dictionary - can this matrix be directly used as an SAE? Would you expect this method to learn a more correct dictionary than standard SAE training methods?\n- In the paper the trained LLM models are still very small - is it feasible to train a more standard size version of this method to compare with SAEs, e.g. 32k+ latents?\n- Can this method be directly compared to LLM SAEs using standard SAE benchmarks like sparse probing, or other metrics from SAEBench?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors address the problem of interpretability in large language models (LLMs) through the framework of sparse autoencoders (SAEs). A known limitation of SAE is its lack of shared mechanisms across tokens, which hampers the discovery of interpretable concepts that capture temporal dependencies in LLM activations. Moreover, SAEs typically fail to model correlated concepts, limiting their ability to recover instantaneous relationships between them. The authors propose to overcome these limitations by modeling concepts using linear structural causal models (SCMs), which allows for both temporal and instantaneous causal relationships among concepts. Building on prior results from the causal representation learning (CRL) literature, they establish identifiability guarantees for the learned concepts and causal relationships, and empirically validate these findings on a synthetic dataset. Additionally, the authors present a case study on a real-world LLM dataset, demonstrating that their approach uncovers a range of meaningful temporal and instantaneous relationships.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- While the proposed approach and identifiability analysis builds largely on prior work and are not particularly novel on their own, the application of this temporal CRL framework to LLM interpretability is novel. Temporal relationships are important for LLM interpretability, and to the best of my knowledge, this is the first work to recover them using SAEs.\n\n- The authors have done a good job with the experimental design. Scaling the synthetic dataset to include more concepts is an important step in demonstrating that the proposed approach can operate at the scale required for LLM interpretability tasks. Furthermore, the results in Appendix A.3.2 effectively justify the selection of key hyperparameters, such as the time window ($\\tau$), sparsity penalty ($\\beta$), and the dimensionality of the latent concept space.\n \n\n**Weaknesses**\n\n- The motivation for extending prior temporal CRL approaches to high-dimensional settings feels somewhat unclear. The proposed method relies on the assumption of a linear mixing function between concepts $z$ and observations $x$. Therefore, comparisons to existing CRL methods such as IDOL which are designed to handle the more general nonlinear case do not seem entirely appropriate. I suggest the authors tone down this aspect of their contributions. As currently framed (e.g., lines 57–58), the claim that this work significantly advances prior CRL methods appears overstated.\n\n- The synthetic data results in Section 5.1 serve as a useful sanity check; however, a more comprehensive analysis of concept identifiability using (semi-)synthetic language benchmarks with ground-truth concepts would strengthen the evaluation. For instance, the authors could adapt synthetic benchmarks from SSAE [1] to the temporal domain. This would provide clearer insights into whether LLMs can recover true concepts from language-based synthetic datasets.\n\n- The observation that a sparsity penalty of $\\beta=0.01$ outperforms $\\beta=0.1$ for learning sparse causal mechanisms is somewhat surprising. To better understand this behavior, I recommend the authors conduct experiments with a broader range of $\\beta$ value, such as randomly sampling from the interval $(0.001, 0.1)$  to analyze the trend. Additionally, an important sanity check would be to evaluate the case $\\beta=0$  which removes the sparsity regularizer entirely. Moreover, including ablation studies on the hyperparameter $\\alpha$ would strengthen the paper; it would be valuable to see how variations in $\\alpha$ affect training dynamics and the recovered mechanisms.\n\n- While the case study results are interesting, the authors could further evaluate their approach on related tasks such as finding steering vectors using identifiable concept learning methods with LLMs [1, 2]. It would be valuable to know if the proposed approach can infer accurate steering vectors, and if not, what limitations the authors anticipate.\n\n- The paper overall is well written, however, the writing of the theoretical aspects needs improvement.\n     - The authors should avoid labeling their main theoretical result (Theorem 3) as a “theorem,” since it largely follows from Theorems 1 and 2 in prior work. Referring to it as a proposition or corollary would be more accurate.\n     -  When presenting theoretical results, even from prior works, the authors should clearly define all assumptions and key terms. For example, the notion of component-wise identifiability for latent variable recovery is never formally defined. Similarly, the term “isomorphism” used in Theorem 1 (regarding the estimated Markov network being isomorphic to the ground truth) is not properly explained.\n    - In Theorem 1, the authors never defined what is $\\varepsilon(\\mathcal{M} _{c_t})$?\n\nReferences:\n\n- [1] Joshi, Shruti, Andrea Dittadi, Sébastien Lachapelle, and Dhanya Sridhar. \"Identifiable steering via sparse autoencoding of multi-concept shifts.\" arXiv preprint arXiv:2502.12179 (2025).\n\n- [2] Rajendran, Goutham, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. \"Learning interpretable concepts: Unifying causal representation learning and foundation models.\" arXiv preprint arXiv:2402.09236 (2024).\n\nQUESTIONS:\nPlease refer to the *weaknesses* section above for major questions. In addition, I have the following comments and clarifying questions for the authors:\n\n- There seems to be a typo on line 77, I guess the matrix $A$ should have dimensions $m \\times n$? Since in the theorems ahead authors represent the latent dimension of $z$ with the symbol $n$.\n\n- There seems to be a typo on line 146, authors write $z_{it}$ while it should be $z_{ti}$.\n\n- In Theorem 1, we defined $c_t = ( z_{t-1}, z_t, z_{t+1} )$ while in Theorem 2 we defined $c_t= (z_{t-1}, z_t)$. Why do we change the definition of $c_t$ across theorems?\n\n- On line 131, authors should specify that the diagonal matrix $D$ is also invertible, as not all diagonal matrices are invertible. \n\n- Could the authors explain the intimate neighbor set used in Theorem 1?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors claim four contributions:\n1. A framework to model time-delayed causal relations between concepts and instantaneous constraints, with theoretical guarantees in the direction of reliability and explainability.\n2. Algorithms tailored to causal representation learning for the high-dimensional concept space of LLMs.\n3. Synthetic experiments supporting the results, scaled to match real-world complexity and applied on real LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nOriginality: the authors introduce a linear SEM to model LLM activations. I am unfamiliar with the related mechanistic interpretability literature, but this concept is novel to my knowledge.\n\nQuality and significance: It is shown that the underlying latent variables in the SEM can be identified under some assumptions, including sparsity of latent components. It does not seem to be shown that the SEM can model the distribution of activations; case studies are performed (selecting <=6 examples of specific types of successes in total), but I did not see an objective or large-scale analysis supporting this claim. In other words, evaluation appears to be limited.\n\nClarity: contributions, methodology, theory, and results are stated clearly.\n\nQUESTIONS:\n1. Usually, LLM activations are viewed as attention computations, and there is no sequential time dependence. Why was the modeling choice of sequential time dependence made in this work? Is there evidence support the claim that this is superior to an attention style of activation modeling? Is there a benefit of having an SEM, which induces interventional distributions, rather than a probabilistic model?\n2. Are the latent variables $\\mathbf z_t$ usually confirmed to be identifiable, given that there are many more latent than observed variables? What is the value of showing that identifiability holds in the synthetic setting in Experiment 5.1?\n3. Is there evidence to support the claim that the proposed SEM in equations (1), (2), and (3) is consistent with observational activations generated by real LLMs? For example, how well does the proposed model fit the actual distribution of activations? Is there any baseline for how well a model can fit this distribution, and does the proposed model improve upon it?\n4. Given that the model proposed is an SEM rather than a purely observational model - is there evidence to support the claim that there are the instantaneous and temporal mechanisms can be modeled linearly in the specific way this work models them?\n5. Is there evidence to indicate whether or not the preconditions of Thm 1-3 are met, sparsity in particular?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}