Causal Differentiating Concepts:  Interpreting LM Behavior via Causal Representation Learning
Authors: Navita Goyal
Hal Daumé III
Alexandre Drouin
Dhanya Sridhar
Venue: NeurIPS 2025 spotlight ()
Keywords: LM interpretability
causal representation learning
concept discovery
Primary area: deep_learning
TL;DR: This paper introduces an unsupervised method that disentangles interpretable latent concepts in language model activations that mediate behavior, assuming that sparse changes in these concepts can induce changes in model behavior.

Abstract: Language model activations entangle concepts that mediate their behavior, making it difficult to interpret these factors, which has implications for generalizability and robustness. We introduce an approach for disentangling these concepts without supervision. Existing methods for concept discovery often rely on external labels, contrastive prompts, or known causal structures, which limits their scalability and biases them toward predefined, easily annotatable features. In contrast, we propose a new unsupervised algorithm that identifies causal differentiating concepts—interpretable latent directions in LM activations that must be changed to elicit a different model behavior. These concepts are discovered using a constrained contrastive learning objective, guided by the insight that eliciting a target behavior requires only sparse changes to the underlying concepts. We formalize this notion and show that, under a particular assumption about the sparsity of these causal differentiating concepts, our method learns disentangled representations that align with human-interpretable factors influencing LM decisions. We empirically show the ability of our method to recover ground-truth causal factors in synthetic and semi-synthetic settings. Additionally, we illustrate the utility of our method through a case study on refusal behavior in language models. Our approach offers a scalable and interpretable lens into the internal workings of LMs, providing a principled foundation for interpreting language model behavior.

Causal Differentiating Concepts: Interpreting LM Behavior via Causal Representation Learning
Navita Goyal University of Maryland navita@und, edu
Hal Daumé III
University of Maryland ha130umd, edu
Alexandre Drouin ServiceNow Research cula-Quebec Al Institute
Dhanya Sridhar
Mila-Ques Raebeeu Al Institute Université de Montréal
al alexandre drouin@service.mmg now
dhanya-sri wwwarrmllaaaaaaaaaaaaaaaaaaaaaal quebec
Abstract
Language model activations entangle concepts that mediate their behavior, making it difficult to interpret these factors, which has implications for generalizability and robustness. We introduce approach for disentangling these concepts with- out supervision. Existing methods for concept discovery often rely on external labels, contrastive prompts, or known causal structures, which limits their scala- bility and biases them toward predefined, easily annotatable features. In contrast, we propose a new unsupervised algorithm that identifies causal differentiating concept--inteepretabl latent directions in LMM activations that must be changed to elicit a different model behavior. These concepts are discovered using a constrained contrastive learning objective, guided by the insight that eliciting target behavior requires only sparse changes to the underlying concepts. We formalize this notion and show that, under particular assumption about the sparsity of these causal differentiating concepts, our method learns disentangled representations that align with human-interpretaah factors influencing LM decisions. We empirically show the ability our method to recover ground-truth causal factors in synthetic and illustrate the utility of our method through
semi-synthetie settings. Additionally, a case study on refusal behavior in language models. Our approach offers a scalable and interpretable lens into the internal workings of LMs, providing principled foundation for interpreting language model behavior.
Introduction
As language models (LMs) grow more capable and complex, there is an increasing need for inter- pretability methods to shed light human-interpretable factors that mediate LM behavior a given task. Consider the following running example:
(Income prediction using LMs). In of prompts, each prompt X,, consists of candidate's 's bio followed by an instruction asking an LM to assess whether the candidate earns six- six-figure salary. Suppose p(yes|x,)) high for bios corresponding to high-income occupations (e.g., lawyers and doctors) and low for bios corresponding to low-income occupations (e.g., painters and teachers). But for medium-income occupations (e.g., accountants and lecturers), the model's behavior varies by gender, leading to higher p(yes|x,)) for male-associated bios and lower p(yes) xx)) for female-associated ones.
Example 1
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Concept aligument =(g(K))
plyes (x) d:g
0.55
C
High
B(N)
Occopatiin
G
"Shete bugger person
Language model
Latent ligh-level model .F((0)
Allgommnt between the model activations g(3) and true medharing concepts
Sparse interventions the mediating concepts
Axts-aligaamet causal differentiating concepts
(a)
(b)
(c)
Figure 1: .M represents a high-level model that explains the language model's behavior in terms of latent mediating concepts z, such that sparse interventions to mediating concepts suffice to change model behavior. Instead of assuming access to the high-level model M, our work disentangles the learned representation 2 with the key assumption that causal differentiating concepts are axis-aligned.
In this example, the different ranges of p(yes[x,)) give rise to four distinct behavior classes, and the candidate's 's gender and occupation are the "concepts" that mediate this behavior. These patterns may emerge due correlations present in the data used train the language model. This high-level model of the LM's behavior is illustrated in Figure Tja). In this paper, we focus key aspect of interpretability research: finding alignments between the activations of an LM (e.g., token embeddings at different layers) and mediating concepts (e.g., gender and occupation).
The challenge in directly interpreting individual neurons or embedding dimensions potential concepts is that LM activations generally entangle such concepts (Elhage et al.] 2022: Geiger et al. 2024b). This necessitates mapping that effectively "inverts" activations back into the space of disentangled concepts. The need for such interpretability tools has led to a suite of methods. such as linear probes (Elazar al. (02I), contrastive activation addition (CAA) (Rimsky et al., 2024). and distributed interchange interventions (DII) (Geiger et al. 2024b) that learn mappings from activations ground truth concepts via various forms superviston (c.g., concept labels, contrastive prompts, causal model over concepts). While these methods are effective at finding interpretable concepts encoded in LM activations (Wu et al. 2025). the need for supervision introduces a bias toward inferring simple concepts that we know how to annotate, such as tense, pronoun use, or language. To address these limitations and enable the discovery of behavior-relevant concepts, this paper introduces new method for uncovering such mediating concepts from LM activations without the need for supervision from ground-truth concept labels.
Since fully unsupervised learning is not identifiable (Hyvärinen and Pajunen 1999 Locatello et al. 2019) there are infinitely many correct solutions- many practical approaches unsupervised learning introduce inductive biases. Sparse autoencoders (Huben et al. 2024, SAEs) seek to recover all iunan--nterpreetbble features encoded in LM activations by constraining the feature representation to be sparse. However, SAES require post hoc analysis both to interpret individual features (for instance, by feeding a set of examples that activate feature into a large language model infer its meaning) and to identify which features affect model behavior (for instance, by manipulating different features, one a time, and observing the corresponding effects on model behavior) (Bills et al., 20233 Bricken et al. 2023 Paulo et al. 2024). Moreover, since SAES seek to invert potentially billions of concepts from activations, they may not be able to uniquely recover many concepts (Menon et al. 2025). As one example, SAES could decompose concept (e.g.. "Einstein") into a combination of Teatures (such as "scientist", "Germany", and "famous person") (Leask et al., 2025). Even though in this case the decomposition is interpretable, it may make concepts harder to intervene for causal insights. This makes SAES cumbersome to use for understanding targeted model behavior. this work, place the interpretation of model behavior front and center, aiming to learn concepts the level of granularity that directly relevant to codify the behavior under consideration.
Concretely, we learn mappings from model activations g(x) to features á = r(g(x)), so that the learned features align with the true mediating concepts (e.g., gender and occupation in |Example 1p. To arrive at identifiable objective, we first introduce the idea of "causal differentiating concepts."

---

Put simply for now, causal differentiating concepts are the concepts whose values we must change for any example elicit different model response. In Example we must change the candidate's occupation if we want the model to change p(yes[x.) from a high to a low value. Motivated by work on identifiable representation learning that leverages sparse effects features on outcomes of interest (Lachapelle et al. (023), we make the key assumption that causal differentiating factors are sparse-iee ic., sparse changes suffice to change behavior. We encode this assumption into a constrained contrastive learning objective that we prove recovers disentangled concepts that mediate some model behaviors of interest.
To summarize our contributions: (1) We formalize concept discovery in settings where high-level mediating factors are unknown. We introduce causal differentiating farepts-ffcttrrs that must change to elicit a different model behavior- and propose sparsity assumption that enables their identification. (2) We develop a constrained contrastive learning objective that enforces this assump- tion and can provably recover disentangled, interpretable features. (3) We validate our method in both controlled experiments and real-world case study, where the underlying causal factors are unknown and the assumption is unverifiable, demonstrating the potential of our approach in practice.
2 Problem setting
We consider the setting where a language model takes in an input sequence E and outputs sequence y E y. The fine-grained responses y are categorized into m coarse gaasingr behavior classes f1...., m² so that each input x associated with a discrete behavior label c. For instance, when studying refusal behavior in language models, all queries x to the model that elicit responses such as "I am sorry... " or "I cannot respond..." " are mapped to the same refusal behavior class. These categorizations of fine-grained responses y into coarse-grained behaviors can be provided entirely by a domain expert or by clustering the next-token probabilities p(y|x) x) learned by the model, as with causal feature learning (Chalupka et al. 2017).
Motivated by work on abstracting neural networks (Geiger et al. 202 2024 we assume high- level model M(x) that explains the LM's behavior in terms of & discrete mediating concepts so that x x|z. Figure lillustrate high-level model for Example 1 where the model's 's likelihood of predicting yes is mediated by two latent concepts z: the gender and occupation of candidate.
Problem. To interpret model behavior, the goal is to map model activations g(x) to k: features 2 = =x)) via learned encoder т, so that the learned features 2 align with the true mediating concepts (e.g., gender and occupation).
Given input-behavior pairs (x, c). it would be tempting to simply find the most activated neurons or token embedding dimensions among examples in given class and use these as proxies for the mediating concepts z. However, such activations typically entangle interpretable concepts like gender
occupation (Elhage et al. 2022 Geiger et al. 2024b). To find alignment between activations and concepts, Geiger et al. (2024b) propose using a fully known high-level model M(x) to supervise
the learning of (x).
Key idea. To overcome the need to fully specify M(x), this paper proposes weaker assumptions that we can make about high-level concepts and behaviors that drive identifiable concept learning.
3 Learning causal differentiating concepts
Our method operationalizes two assumptions that we make about the high-level model c M((x)) The key assumption, as illustrated in Figure IJc), says, loosely speaking, that we can change a model's behavior with only a sparse change to the mediating concepts z. We formalize both assumptions and
the resulting learning objective for recovering mediating features 2 ([x()) that align with the true underlying mediating concepts z. Crucially, we show that because of the assumptions, the mapping (g(()) becomes identifiable, meaning that the recovered 2 features are guaranteed to correspond to the mediating concepts z up to permutation and scaling indeterminacies.
Assumption 1. The true conditional probability of behavior given the mediating concepts encoded by the high-level model e M(x) p(C c/z) x exp(w. z). That is, the model's behavior for an input is related to the mediating latent variables by a logit-linear function.
3

---

This assumption, used in other works on disentangled representation learning (Ahuja et al. 2022b). can be motivated by the structure of language models, where the final layer embedding linearly influences next-token logit probabilities. Here, we extend such a logit-linear assumption to behaviors.
To formalize the assumption that sparse feature changes enable behavior changes, define the notion of interchange intervention and causal differentiating concepts. First, we define z as a subset of the vector z that selects elements of the set S, Cs (xic k} as the set that contains inputs with behavior label k, and X E C₁ as an input in this set.
Definition 1. For two imputs X C₁ and x Ci such that i & I, an interchange intervention on a subset of mediating concepts S C {l.....d d} in the high-level model defined as Ma < (xx) where for E S, the r-th component of the high-level representation 2g (associated with the input Xx) is replaced with the corresponding value from 21 from x
Definition 2.
set of latent mediating factors z defined as causal differentiating concepts between two groups Cs and C; if, for some inputs that are labeled with class k, an interchange intervention on S is both necessary and sufficient to change the behavioral label from & to l.
(Necessary condition)
For any X E Cj and X E Ct. Mag' -xs" (xx) # cts
where S' *n$#0
(1)
(Sufficient condition)
For some E Cj and X1 E Cr, Mas+ x (xx) cr.
(2)
A latent mediating factor is thus the 1-sparse causal differentiating concept between a pair of group C. and C if the set of causal mediating concepts between C2 and Ci is S = (r).
Assumption 2. Every dimension 2" is 1-sparse causal differentiating concept for some pair of groups Cs and C₁.
Corollary For the groups Cs and Ct, consider a I-sparse latent mask dur Rd, which is d-dimensional vector with a nonzero value at position r corresponding to the causal differentiating concept between the two groups and zeros elsewhere. It follows from the necessary and the sufficient conditions of causal differentiating concepts that for some X E C and X; E C₁ 2 2 + 8u ?
Method. To learn the mapping à = r(y(x)) from activations g(x) to interpretable concepts, we introduce a constrained contrastive learning objective designed to satisfy the key assumptions (Assumptions and 2 about the true mediating concepts z in the high-level model M(x). We implement& (g(())) ()) as a bottleneck layer top of the language model's final layer, ensuring that the dimension of is less than the dimension of gxx).
Our objective has two components. The first ensures that this bottleneck representation extracts information that is predictive of the labeled behavior c, using log-linear predictor h(r()(x))) to enforce (ssrree/Asummtiinnnnn 1. The bottleneck serves to filter out irrelevant factors, retaining only information necessary for predicting c. Specifically, we minimize the categorical cross entropy loss:
min Epixa)) log (T(()))))))) 7
The second contrastive loss term satisfies Assumption by using Corollary 1, Essentially, for each pair of groups Cs and C₁, the learner guesses 1-sparse perturbation d& and searches for pair of examples x; Cj and x, Ci such that 2, 2; + 8u.. Formally, the objective is,
min T,5
Σl min B[llT(9(x,)) (((((x) -8x113]. 1.1
x.eC X,EC
Since Assumption 2only requires that each concept z" is I 1-sparse causal differentiating concept for at least one pair of groups, we do not expect to find a I-sparse perturbation åg between all pairs
Notttion: We use subscript for enumerating the sample and superseript for enumerating the dimension. So 2; the true latent for the th sample and 2/ is the th dimension this latent variable.
"This is true because if 8u were nonzero in more than one position Vxx E Cs and x; E Ct, the sufficiency condition would be violated. Conversely, if des were zero everywhere, the necessary condition would be violated.

---

of groups. However, we show later, disentangled solution ([(()) achieves a lower objectíve value than an entangled solution, which generally captures fewer axis-aligned changes across pairs of groups. In practice, we search over possible dιt and perform the inner minimization over i and by sampling multiple x; E C2 and x; E Gt. To ensure that the learned perturbations cover all mediating concepts, we constrain span,, (ōk1) = d.
We show that under the above assumptions, our method identifiably recovers interpretable causal factors up to permutation and scaling.
Theorem 1. If the Assumption 1 holds, then the function i that satisfies Equation (3) gives us the true latents up to an affine transformation.
Theorem 2. If(Assumptions and 2 hold, then the function f that satisfies Equations identifies the true latents up to permutation and scaling.
(3) and
Intuitively, we get the result in Theorem 1 because of the log-linearity in the prediction function (Ahuja al. (2022b). For Theorem 2] we leverage the fact that for all causal differentiating concepts of interest, by CCrollary| 1] there exist some pairs x, and x, that are related by a sparse latent shift, allowing us adapt the proof from Ahuja al. (2022a). See the full proofs in Appendix A
4 Experimental details
We evaluate the ability of our method to disentangle mediating concepts in settings where we have some domain knowledge about what the desired mediating concepts are. We compare our method to baselines without disentanglement guarantees and sparse autoencoders (SAEs), and find that our method outperforms these related methods.
Data. We conduct our experiments in three settings: (1) synthetic data, (2) semi-synthetic data with real text and synthetic labels, and (3) non-synthetic data with text and LM outputs.
Synthetic and semi-synthetic datasets allow us control the ground-truth causal factors and their influence on outcomes, enabling a precise evaluation of the ability of our proposed method to recover the true causal factors up to permutation and scaling. The semi-synthetic data with text inputs enables testing our method in the context of language models, assessing whether our method can isolate causally relevant factors from the many encoded during LM pretraining. However, it is limited to naturally occurring features in the text. Fully synthetic data offers more control: we can vary complexity, sparsity, and the number of causal factors. Lastly, we present case study using a dataset with queries with different harmfulness categories and study the language model's 's refusal behavior, demonstrating how our method can be implemented and evaluated in practical, in-the-wild scenarios.
Synthetic data. For synthetic data, we consider true factors z ~ N(4,, σ) C RRd with d = 2 and d 3. 3. We relate factors z behavior labels c, such that the resulting data satisfy Assumption 2 The resulting behavior groups are illustrated in Table I. with each color representing a different group.
We generate IR" given the factor using linear and non-linear mixing functions, Moran et al. (022)) show identifiability up to permutation and scaling for non-linear sparse mixing functions, where each component x depends only subset of factors. We also experiment with non-linear non-sparse mixing functions to assess whether our method yields identifiability when mapping from factors to observations is more complex. The list of mixing functions is included in Appendix B]
Semi-synthetic data. For semi-synthetic data, we consider the data generating process described in Example 1] We use the bios from the BiasBios dataset (De-Arteaga et al. 2019) as the textual input x and generate an outcome y that represents, c.g., a model responding "yes" or "no" to whether or not the candidate makes six figures. We consider two causal factors-binatd gender (male/female) and occupation level (high/medium/low), We restrict our experiments to binary gender because the BiasBios dataset provides binary labels, which we use the gender factor. For occupation, categorize the occupation associated with each bio into three categorics- hhriccs--hghhhh income (e.g.. doctors. lawyers), medium income (e.g., nurses, accountants), and low income (e.g., paralegal, painters) This grouping allows us to test whether our method can recover causal factors that influence the outcome y at the appropriate level of abstraction, rather than solely relying the semantic cues from the bios x.
'These groupings are derived from the U.S Department of Labor's Employment and Earnings by Occupation statistics
5

---

We simulate model behavior such that e is mediated entirely by the true causal factors z. We design p(y|x) so that we can derive labels for high-level model behavior by simply clustering ply x). The resulting groups are the same as in the synthetic data experiments with d = 2 (Tble 1 top). Thus, we get 2 gender and 2² occupation with groups {[O=High); {O=Med, G=Male), {O=Med, G=Female), [O=Low)].
Non-synthetic data. For our case study experiments, we consider refusal behavior in models. We use a collection of harmful prompts, sampled from MALICIOUSINSTRUC
(Huang et al. (024)) 2023). and TDC2023 (Mazeika et al.. HARMBENCH (Mazeika al. 2024), ADVBENCH (Zou et al.,
2022). harmless prompts, sampled from ALPACA (Taori et al. 2023), and pseudo-harmful prompts, sampled from BE-HENCH (Cui et al., (025). We use common refusal patterns in Llama-3.1-88 3.BB model, such as "I am sorry" and "7 cannor" to extract p(refusal| x). We cluster model behavior into three classes based on p(refusal xx), which serve as the behavior classes that we aim to explain. We train r to obtain the aligned hidden representations E IR2. Note that the model has no access to the actual class labels of harmful, aaeudo-harmful, and harmless prompts. We visualize the learned representation z for these three prompt sets to understand what factors affect model refusal behavior.
Models. We conduct the semi-synthetic and non-synthetic experiments with three language models- DestilBert-base (66M) (Sanh et al.) 2020). Llama-3. 1-8BB (Touvron et al. 2023) and Qwen2-7B (Bai ct al. 2023). For synthetic data, where the input x not text, we replace the language model encoder with feedforward neural network. To obtain (xx), we fit a variational autoencoder to x.
We compare our method against two baselines that autoencoding and prediction objectives without the contrastive constraint. The autoencoding baseline trains the bottleneck T to reconstruct x to mimic vanilla contrastive learning without constraints, and the prediction objective encourages discarding irrelevant information but, crucially, both baselines are not guaranteed to identify the true mediating concepts. Additionally, we perform detailed comparisons of our method with sparse autoencoder baselines, discussed in Section 6) More implementation details are included in in Appendix B
Evaluation metrics. We evaluate the effectiveness of our method at recovering the ground-truth causal factors using the
(DCI) metrics (Eastwood and Williams 2018). Briefly, disentanglement measures the extent to which representation disentangles the underly lynng factors of variation, that is, whether each feature capture at most one causal factor. Completeness measures the extent which each causal factor is captured by single learned feature. Informativeness measures the amount of information that a representation as a whole captures about the underlying factors of variation. Thus, a rotated but disentangled representation will have an informativeness score of 1.0. We also include the Mean Correlation Coefficient (MCC) metric (Khemakhem et al. 2020), which computes the maximum linear correlations (accounting for permutations in the learned representations), giving a measure of disentanglement.
Results
Table I shows comparison between our method and the baseline methods for synthetic data. We see that, across all data-generating functions, all methods achieve near- near-perfect informativeness scores. However, expected, the baseline methods entangle the true causal factors, leading to low disentanglemenn and completeness scores. In contrast, our method achieves significantly better disentanglement, with DCI-D scores exceeding 0.89 and MCC scores above 0.86 across all settings.
Next, Table shows results on the semi-synthetic data. We find that in the semi-synthetic setting, the autoencoding baseline shows a low informativeness scores. This is expected since textual data contains a large amount of information, and without additional signals, the autoencoding baseline may struggle to determine which information to retain, potentially discarding information relevant to the true causal factors. In contrast, adding the prediction objective results in a boost in the informativeness score across all models. However, both baselines exhibit low disentanglement and completeness scores. In comparison, incorporating our contrastive constraint consistently improves the disentanglement metries (MCC and DCI-D scores) across all evaluated models. Lastly, for our case-study experiments, since we do not have ground-truth causal factors, we instead visualize the learned latent space for the different sets of prompts. Figure 2 shows the learned latent space = (g())) for Llama-3.1-88 (the findings for Qwen2-7B are consistent and included in Appendix C). Consistent with our previous analysis, we find that the autoencoding baseline is not

---

Table 1: Synthetic data. (Left): Example of data with d 2 (top) and d (bottom) latent factors. (Right): Disentanglement (D), Completeness (C), Informativeness (I), and Mean Correlation Coefficient (MCC) scores with latent dimension d = 2 (top; shaded) and d 3 (bottom; unshaded)
Mixing fn
Method
MCC
D
Autoencoding
0.76
0.30
0.36
1.0
Linear
Autoene+Pred
0.60
0.21
0.21
1.0
Our constraints
0.99
1.0
1.0
1.0
Autoencoding
0.81
0.24
0.33
1.0
Non-linear,
Auloenc+Pred
0.72
0.0
0.0
1.0
sparse
Our constraints
0.91
0.97
0.97
1.0
Autocncoding
0.67
0.0
0.0
Non-lincar,
Autoenc+Pred
0.76
0.02
0.02
non-sparse
Our constraints
0.92
0.90
0.92
Autoeneoding
0.77
0.34
0.35
1.0
Linear
Autocne+Pred
0.65
0.23
0.31
0.87
constraints
0.94
0.99
0.99
Autocncoding
0.74
0.19
0.25
Non-lincar,
Autocnc+Pree
0.77
0.54
0.51
sparse
Our constraints
0.90
0,92
0.95
Autoencoding
0.78
0.51
0.60
1.0
Non-linear,
Autnenc+Pred
0.86
0.54
0.59
1.0
non-sparse
Our constraints
0.89
0.89
0.89
1.0
Table Disentanglement (D), Completeness (C), Informativeness (I), and Mean Correlation Coeffi- cient (MCC) scores on semi-synthetic data
Model
Method
Autocncoding
0.17
0.01
0.01
0.57
Distilbert-baa
Autocne+Pred
0.32
0.18
0.20
0.85
Our constraints
0.92
0.99
Autoencoding
0.32
0.01
0.04
0.57
Qwen2-7B
Autoene+Pred
0.69
0.06
0.06
0.83
constraints
0.79
0,80
0.84
0.97
Autoencoding
0.06
0.0
0.0
0,49
Llama-3.1-8B
Autocnc+Pred
0.54
0.05
0.11
0.84
Our constraints
0.81
0.86
0.88
0.98
Sparse- oparse-autoencoders
0.51
0.49
0.52
0.56
able to distinguish between different data distributions. Adding the prediction constraint for the class labels based on p(y refusal|x) leads distinctive clustering of the harmful, aseudo-harmful, and harmless prompts. However, the learned representation is not axis-aligned. For instance, the representation for harmful and pseudo-harmfin prompts differ along both latent dimensions, similarly for pseudo-harmful harmful and harmless prompts. In contrast, our method yields an axis-aligned latent space. These directions can be interpreted harmfulness (dimension 1) and topic (dimension 2).
6 Comparison to sparse autoencoders
We compare our approach to sparse autoencoders the semi-synthetic setting. We use LlamaS- cope (He al. 2024), a popular resource with 256 SAEs trained on each layer and sublayer of the Llama-3. model, with 32k and 128k features. We restrict our experiments to the residual stream SAES they are reported perform best across all metries evaluated in the original paper, resulting in 64 SAES (one for each 32k and 128k feature dimension and all 32 layers of Llama-3.1-8B))
We perform 3 evaluations for both the 32k- and 128k-feature SAEs. First, measure disentanglement and informativeness for the full feature vectors, which reflect overall sparsity and predictiveness of SAE features. Since the feature dimensionality of SAEs far exceeds the number of
Evaluation.
TTee numbers in bold indicate statistical significance p < 0.05. Details the analysis in Appendix B
7

---

Data class Pseudo-harmhui
ply refusell))
Harmful
Harmless
High
Low
Autoencoding
Auteencoding-Pred
Our constraints
Latent Dimension
6-3
-34
-1.3
Latent Dimension
Latent Dimension 1
Latent Dimension
Figure 2: Latent space for refusal behavior Llama-3, 3,1-8B with autoencoding (left), autoencoding + prediction (center), and our constraints (right). The bascline method entangles the two latent dimensions, but adding the contrastive constraint leads to an aligned latent space.
Diventanglement
Disentanglement
Informativennss
Mean Informativeness
Full feature vector
Two most predictive features for
The most predictive features causal 6d
id
1.9
D:
33
Cr 8:58
0.62
3.8
0.4
0.2
3.3 Layers
Lay 0555
Layers
Figure 3: Disentanglement and informativeness scores for 32k-dimensional SAEs across model layers. Solid lines indicate layer-wise scores, dashed lines denote the mean, and shaded regions represent +1 standard deviation. Each subplot is annotated with the maximum disentanglement score along with its corresponding informativeness score.
ground-truth latent factors, we also train linear classifier on the SAE features to predict c. From this, we select the two most predictive features and compute disentanglement and informativeness scores on this reduced 2-dimensional representation, matching the number ground-truth latents required to predict This evaluation most closely aligns with our setting. which is designed to learn features that best predict c. Finally, we perform ceiling evaluation by training two separate classifiers one for each ground-truth factor-and identifying the most predictive feature for each. We then compute disentanglement and informativeness scores on the resulting 2-dimensional feature vector.
SAE features exhibit high informativeness across multiple layers, with mean and standard deviation of 0.95 0.07. However, disentanglement scores remain relatively low (0.58 + 0.07), with maximum of 0.68.
Results. The results for 32k dikeesnnnsoonal SAEs are shown in Figure 3
When considering only the most informative features for predicting both informativeness (0.57 0.02) and disentanglement scores (0.08 0.12) drop significantly. This suggests that SAFs do isolate the true causal factors into two distinct features. Given the low informativeness in the top-2 features analysis, we hypothesize that SAE features may be more fine-grained than the true causal factors (Leask et al. 2025). To investigate this, we evaluate disentanglement by selecting the top-k most informative features for predicting such that the overall informativeness. is at least 0.95. Even under this setting, disentanglement scores remain low (0.41 + 0.04) with a maximum of only 0.46. Finally, the rightmost plot in Figure 3) shows a ceiling analysis using the most predictive features for each the two ground-truth causal factors. Even under this best-case feature selection, the maximum disentanglement score observed is 0.76, which remains notably lower than the score achieved by our method (0.86) on the 1.ama-3.1-4 model. The results are consistent for the 28k-dimensional SAF, shown in Figure in the appendix.
8

---

Beyond the empirical comparisons, we note some key differences between our method and sparse autoencoders. Notably, SAEs do not offer a natural mechanism for identifying which of the 32k or 128k features, across 4 activations and 1024 tokens, are relevant to a specific model behavior. While we explore several strategies for selecting the most relevant SAF features, the process remains non-trivial and ad hoc. Furthermore, a direct comparison is limited, as SAEs are not explicitly trained to predict the behavior class c. However, in certain settings, such Sahavior-targetee interpretation may be more valuable, which our work is designed for.
7 Sensitivity to assumptions and implementational choices
Choice of latent dimensionality. In the experiments discussed in Section 5| we fix the number of the learned latent dimensions to that of true mediating concepts. However, the number of true mediating concepts is unlikely to be known beforehand. To account for this, perform additional experiments by varying the dimensions of learned latents to assess how over-specification or under- specification affects disentanglement metrics and whether these could be used signal to adjust the specified latent dimensionality. The details of experiments and results are included in in(pppendix C) We find that over-specifying latent dimensionality consistently leads to drop in completeness scores and under-specifying latent dimensionality consistently leads to a lower disentanglement score. These measures can thus serve as useful signals for adjusting the latent dimensionality as hyperparameter.
We restrict this sensitivity analysis to synthetic experiments because a parallel comparison is not feasible in semi-synthetic and non-synthetic cases as the dimensionality of true mediating concepts dataset-dependent and cannot dictated externally. We believe that the insights from synthetic experiments be extended general setting to determine latent dimensionality. Even though non-synthetic settings lack ground-trut truth mediating concepts, not allowing direct evaluation of disen- tanglement and completeness scores, the disentanglement measures can be estimated across multiple model runs as a signal for guiding the latent dimensionality, following Duan et al. 2020).
Choice of intervention layer. In the experiments discussed in Sections 5 and 6 we apply our constrained learning objective to the representations at the final layer of the LLM, where the linearity assumption most naturally satisfied. However, our method can also be applied to intermediate layers. The proposed contrastive learning constraints (Equation (4)) are designed to disentangle linearly mixed concepts, an assumption that may not strictly hold for intermediate representations. Nevertheless, recent studies support the linear representation hypothesis, suggesting that latent concepts are approximately linearly encoded in language model representations (Roeder et al.) 202T]] Jiang et al.] [2024] Park et al. (0224). Based on this, we hypothesize that our method can be used for unsupervised discovery of latent concepts in the model's internal layers. To empirically test this, we extend our refusal case study to intermediate representations and examine whether our method can extract relevant insights from those layers. Detailed results are included in m|Appendix Cj Overall, we find that our approach does not yield disentangled representations in the early layers (up to layer 9 in the Llama-3.1-88 1-BB model), but past layer 9, the structure presented in Figure 2 starts to emerge. That is, we observe that the harmfulness and topicality dimensions identified in experiments can also be disentangled in intermediate model representations.
Violation of 1-sparsity assumption. Our main theoretical and experimental results assume 1- sparsity in causal differentiating concepts. We note that our 1-sparse assumption is more relaxed than that of Ahuja et al.) (2022a) since we do not require that every pair of clusters has only I causal differentiating concept or that the pair of clusters that have 1 causal differentiating concepts are known beforehand, rather that every 2" is 1-sparse for some pair of clusters. We expect our results can be further extended to 2- 2-sparse condition (or generally speaking, p-sparse condition) following the proof strategy as in Ahuja et al. (2022a). We include some preliminary experiments to test this in Appendix C) indicating that our method can be extended beyond 1-sparse settings as well. However, we leave formal extensions of our approach to p-sparse causal differentiating concepts to future work.
8 Related work
Causal representation learning. This work presents an identifiable approach to learning concepts from observed LM activations, extending ideas from the field of causal representation learning (CRL)-
9

---

see Yao et al. 2025) for comprehensive overview. In brief, CRL methods enjoy identifiability
guarantees by leveraging paired datasets (Zhang al. 2023; Ahuja et al. 2024) or samples (see
below), auxiliary labels (Roeder et al. 2021 Khemakhem et al. 2020, Rajendran et al. 2024). or
extra assumptions about the data- generatiig process, such as sparse decoding (Moran et al., 2022
Gresele et al., (02T). In this paper, we take inspiration from line of CRL works that leverage sparsity assumptions such as sparse transitions in latent temporal models (Lachapelle et al. 2022
2024), sparse latent perturbations across samples (Ahuja et al. 2022a Brehmer et al. [2022] Locatello ct al. 2020; Joshi et al. 2025), or sparse dependencies between labels and features (Lachapelle et al.
2023 Here, we introduce
assumption sparse causal differentiating factors, in effect finding "pseudo" R counterfactual pairs of samples x that vary sparsely in concepts.
Causal abstraction. Causal abstraction (also known as causal feature learning) aims to abstract
low-level features (microvariables) into high-level features (macrovariables) such that the causal effect of intervention in the low-level model corresponds to the causal effect of corresponding
interventions in the high-level model (Chalupka et al. 2017: Rubenstein et al., 2017 Beckers
and Halpern 2019; Beckers et al. 2020). Geiger et al. (2020) 2021 adapt causal abstraction for
mechanistic interpretability of neural networks by aligning neurons to high-level features in a human- interpretable hypothesized algorithm. Since concepts are typically distributed across neurons, Geiger
et al. (2024b) propose learning alignments between concepts and LM activations, using a known high-level causal model to supervise the learning, following Geiger et al. (2022); WW et al. (2023).
We take inspiration from causal abstraction in this work to align concepts and LM activations, but require weaker assumptions than assuming that the causal model is known.
Concept discovery and influence. There are a host of methods in machine learning, and language modeling in particular, for interpreting learned concepts and assessing their influence on model behavior. Broadly, these methods fall into two categories-(1) supervised methods, which identify predefined concepts within model activation space, for instance, by aligning model activations with
concepts using examples labeled with predefined concepts (Koh et al. 2020), or identifying concept
directions in activation space (Kim et al. 2018 Elazar et al. 202 Raviogel et al. 2022a b Belrose jet al. (023) using concept- conccppt-ooniiionlll examples, and (2) wnsupervised methods, which discover latent structure in model activations, for instance, by clustering contextual representations (Dalvi ctt al. 2022 Sajjad et al. 2022), or finding examples that highly activate a feature (and those that do
not) and feeding them into an LM to label the feature (Bills et al. 2023 Kalibhat et al. 2023).
Due to polysemanticity in language models (Elhage et al.) 2022). ) however, individual neurons often lack consistent semantic interpretation. To address this, sparse autoencoders disentangle features by learning a sparse intermediate representation (Huben et al.) 2024 Bricken et al. 2023), which can again be interpreted by finding examples that highly activate feature and feeding them into an LLM for labeling. The effect of these features on model behavior is studied using activation patching (Huben et al.) (2024), feature clamping (Bricken et al. (202)). logit weight inspection (Bricken cct al. 2023). training linear probes (Rao et al., 2024). vocabulary projection (Gur-Ande Arich et 2025). Unlike our approach, these methods offer post-hoc, behavior-agnes interpretations. While valuable for general interpretability, they do not provide insights directly tied to specific model behaviors.
9 Discussion and conclusion
In this work, we introduced framework for learning disentangled representations of the latent concepts that mediate language model's behavior. We showed theoretically and empirically that when behavior changes are caused by sparse shifts in these mediating concepts, our proposed method accurately recovers features that align with the true underlying concepts. This sparsity assumption is motivated by identifiable representation learning approaches that leverage sparsity in the mappings from latent features to labels (Lachapelle et al.) 2023) or transitions in temporal data (Lachapelle et al., 2022 2024). While verifying whether such sparsity holds in a given dataset or model is challenging, the disentangled concepts that we found in the case study on refusal behavior of large language models suggest that the assumption is suited to naturally occurring data. We hope that our work can help bridge the gap between theoretical identifiability guarantees and practical interpretability in language models, demonstrating how assumptions like sparsity can yield meaningful and recoverable latent structure in real-works settings. Integrating different assumptions from the identifiability literature to expand the suite of weakly supervised LM interpretability tools and exploring the uses of our approach for stecring are all avenues for future work.
10

---

Acknowledgments
We would like to thank Kartik Ahuja and Sébastien Lachapelle for the very useful discussions and feedback on this work, especially the theoretical elements. This work was initiated at the Second Bellairs Workshop on Causality, held at the McGill University Bellairs Research Institute (January 6-13, 2023). This work was supported by the NSF Institute for Trustworthy AI in Law & Society (TRAILS; Award No. 2229885), NSERC Discovery Grant RGPIN-2023-04866 and a Canada CIFAR AI Chair. Any opinions, findings, or conclusions expressed in this material are those of the author(s) and not necessarily reflect the views of the funding agencies.
References
Kartik Ahuja, Jason $ Hartford, and Yoshua Bengio. 2022a. Weakly supervised representation learning with sparse perturbations In Advances in Neural Information Processing Systems.
Kartik Ahuja, Divyat Mahajan, Vasilis Syrgkanis, and Joannis Mitliagkas. 2022b. Towards efficient representation identification in supervised learning. In Proceedings of the First Conference on Causal Learning and Reasoning.
Kartik Ahuja, Amin Mansouri, and Yixin Wang. 2024. Multi-domain causal representation learning Via weak distributional invariances) In International Conference on Arlificial Intelligence and Statistics.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chenggiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan. Jianwei Zhang. Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report Preprint.
Sander Beckers, Frederick Eberhardt, and Joseph Y. Halpem. 2020. Approximate causal abstractions In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference.
Sander Beckers and Joseph Y. Halpern. 2019. Abstracting causal models In Proceedings of the AAAI Conference on Artificial Intelligence.
Nora Belrose, David Schneider-Joseph. Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. Leace: Perfect linear concept erasure in closed form In Advances in Neural Information Processing Systems.
Steven Bills, Nick Cammarata, Henk Tillman Dan Mossing, Gabriel Goh Leo Gao, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons language
models OpenAl.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. 2022. Weakly supervised causal rrepresenttion learning In Advances in Neural Information Processing Systems.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Brayden McLean
Karina Nguyen, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah, 2023. Towards monosemanticity: Decomposing language models with dictionary learning Blog Post.
Krzysztof Chalupka, Frederick Eberhardt, and Pietro Perona. 2017. Causal feature learning: An overvvew In Behaviormetrika.
Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsich. 2025. benchmark for large language models Preprint.
OR-Bench: An over-refusal
Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2022. Discov cring latent concepts learned in BERT In International Conference Learning Representations.
11

---

Maria Ar-Aaaaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Tronsparency.
Sunny Duan, Loie Matthey, Andre Saraiva, Nick Watters, Chris Burgess, Alexander Lerchner, and
Irina Higgins. 2020. Unsupervised model selection for variational disentangled representation learning In International Conference on Learning Representations.
Cian Eastwood and Christopher K. I. Williams. 2018. A framework for the quantitative evaluation of disentangled representations In International Conference on Learning Representations.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral
explanation with amnesic counterfactuals, In Transactions of the Association for Computational
Linguistics.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodee Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition Blog Post.
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. 2024a.
Causal abstraction: A theoretical foundation for mechanistic interpretability Preprint.
Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. 2021. Causal abstractions of
neural networks In Advances in Neural Information Processing Systems.
Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation, In Proceedings of the Third BlackbexNLP Workshop Analyzing and Interpreting Neural Networks for NLP.
Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. 2022. [Iduciig causal structure for interpretable neural networks) In Proceedings of the 39th International Conference on Machine Learning.
Atticus Geiger, Zhengxuan Wu, Christopher Potts. Thomas Icard, and Noah Goodman. 2024b. Finding alignments between interpretable causal variables and distributed neural representations In Proceedings of the Third Conference on Causal Learning and Reasoning.
Luigi Gresele, Julius Von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. 2021. Independent mechanism analysis, new concept? In Advances in Neural Information Processing Systems.
Yoav Gur-Arieh, Roy Mayan, Chen Agassy, Atticus Geiger, and Mor Geva. 2025. Enhancing automated interpretability with output-centrie feature descriptions Preprint.
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang. Yunhua Zhou, Frances Liu,
Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. [Lama scope Extracting millions of features from 1lama-3.1-88 with sparse autoencoders) Preprint.
Yangsibo Huang, Samyak Gupta. Mengzhou Xia, Kai Li, and Danqi Chen. 2024. Catastrophic jailbreak of open-source LLMs via exploiting generation In The Twelfth International Conference on Learning Representations.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models In The Twelfth International Conference on Learning Representations.
A. Hyvärinen and P. Pajunen. 1999. Nonlinear independent component analysis: Existence and uniqueness results In Neural Networks.
Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. 2024. On the origins of linear representations in large language models. In Proceedings of the 41st International Conference Machine Learning.
12

---

Shruti Joshi, Andrea Dittadi, Sébastien Lachapelle, and Dhanya Sridhar. 2025. Identifiable steering Via sparse autoencoding of multi monti- shifts) Preprint.
Neha Kalibhat, Shweta Bhardwaj, Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. 2023. Identifying interpretable subspaces in image representations. In Proceedings of the 40th International Conference on Machine Learning.
Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. 2020. Ice-beem: Iden- tifiable conditional energy-based deep models based on nonlinear ical In Advances in Neural Information Processing Systems, volume 33, pages 12768-12778. Curran Associates, Inc.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV) In Proceedings of the 35th International Conference on Machine Learning.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Perey Liang. 2020. [Cncept bottleneck models In Proceedings of the 37th International Conference on Machine Learning.
Sebastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. 2023. (Synergies between disentanglement and sparsity Generalization and identifiability in multi-task learning. In Proceedings of the 40th International Conference on Machine Learning.
Sébastien Lachapelle, Pau Rodríguez Lóprzz Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Juliest 2024. Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies, Preprint.
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. 2022. Disentanglement via mechanism sparsity regularization A new principle for nonlinear ICA In Proceedings of the First Conference on Causal Learning and Reasoning.
Patrick Leask, Bart Bussmann, Michael T Pearce. Joseph Isaac Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. 2025. Sparse autoencoders do not find canonical units) of analysis In The Thirteenth International Conference on Learning Representations.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf. and Olivier Bachem. 2019. Challenging common assumptions in the unsupervised learning of ddiennanggedd representations In Proceedings of the 36th International Conference on Machine Learning.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. 2020. WWakly-superviiee disentanglement without compromises In Proceedings of the 37th International Conference on Machine Learning.
Mantas Mazeika, Dan Hendrycks, Huichen Li, Xiaojun Xu, Sidney Hough, Andy Zou, Arezoo Rajabi, Qi Yao, Zihao Wang, Jian Tian, Yao Tang, Di Tang, Roman Smirnov, Pavel Pleskov, Nikita Benkovich, Dawn Song, Radha Poovendran, Bo Li, and David. Forsyth. 2022. The trojan detection challenge In Proceedings of the NeurlPS 2022 Competitions Track.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardiizd evaluation framework for automated red teaming and robust refusal In Proceedings of the st International Conference on Machine Learning.
Abhinay Menon, Manish Shrivastava, David Krueger, and Ekdeep Singh Lubana. 2025. Analyzing (in)abilities of SAEs via formal languages, In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies.
Gemma E Moran, Dhanya Sridhar, Yixin Wang, and David M Blei. 2022. Identifiable deep generative models via sparse decoding In Transaction Machine Learning Research.
13

---

Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning.
Gonçalo Paulo. Alex Mallen, Caden Juang, and Nora Belrose. 2024. Automatically interpreting millions of features in large language models Preprint.
Goutham Rajendran, Simon Buchholz, Bryon Aragam, Berhhard Schölkopf, and Pradeep Ravikumar. 2024. Learning interpretable concepts: Unifying causal representation learning and foundation models Preprint.
Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. 2024. Discover-then-nande name: TTas agnostic concept bottlenecks via automated concept discovery In Proceedings of the 18th European Conference on Computer Vision.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. 2022a. Linear adversarial concept erasure In Proceedings of the 39th International Conference Machine Learning.
Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. 2022b. Adversarial concept erasure in kernel space] In Proceedings of the 2022 Conference on Empirical Methods in Natural
Language Processing.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering Llama 2 via contrastive activation addition In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.
Geoffrey Roeder, Luke Metz, and Durk Kingma. 2021. On linear identifiability of learned representa tions In Proceedings of the 38th International Conference on Machine Learning.
Paul K. Rubenstein, Sebastian Weichwald, Stephan Bongers, Joris M. Mooij, Dominik Janzing, Moritz Grosse- Weestuu, and Bernhard Schölkopf. 2017. Causal consistency of structural equation models In Proceedings of the Annual Conference on Uncertainty in Artificial Intelligence.
Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Firoj Alam, Abdul Khan, and Jia Xu. 2022. Analyzing encoded concepts transformer language models) Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled Version of bert: smaller, faster, cheaper and lighter Preprint.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. [Stanford Alpaca: An instruction Llama model Blog Post.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei.Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosscini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models Preprint.
Zhengxuan WU, Aryaman Arora, Atticus Geiger, Zheng Wang. Jing Huang, Dan Jurafsky, Christo- pher D. Manning, and Christopher Potts. 2025. Axbench: Steering L.MSs? Even simple baselines outperform sparse autoencoders) Preprint.
14

---

Zhengxuan Wu, Karel D' O'Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. 2023. Causal proxy models for concept-based model explanations Proceedings of the 40th International Conference on Machine Learning.
Dingling Yao, Dario Rancati, Riccardo Cadei, Marco Fumero, and Francesco Locatello. 2025.
Unifying causal representation learning with the invariance principle In International Conference Learning Representations.
Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. 2023. Identifiability guarantees for causal disentanglement from soft interventions Advances in Neural Information Processing Systems.
Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau, and Weiyan Shi. 2025. harmfulness and refusal separately Preprint, arXiv:2507.11878.
Llms encode
Andy Zou, Zifan Wang. Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023.
Universal and transferable adversarial attacks on aligned language models] Preprint.
15

---

A Proofs
We repeat the Theorems and 2 here for completeness.
Theorem 1. If Assumption I hold, then the function 7 that satisfies Equation 3 recovers the true
mediating concepts h(x) up an invertible affine transformation, i.e., f(x) Ah(x) b for all
Proof. Let write (x)(x)) f, for simplicity. For logistic linear predictor h, the model family can be written as
cxp(f(x)
p(c[x) ET exp( (())
where q(i) E Id the j-th column the linear prediction matrix. Next, based on Assumption the true posterior of the behavior group can be written as
exp(w
p(cl))
Ej=1 exp(w]22
The categorical cross entropy loss objective Equation (3) (3) minimized when the optimal predictor (x)) matches the posterior probability p(c/z) for all the labels and so we can write
exp w 2) E exp(e exp(f(x)T'((i) j== exp(
(xp(i(x)=
Taking log on both sides, we get
f(x)" q(c) w. b, Ve
where b is the difference of the normalization terms, and independent of Writing
q(x)) = w +
f(x))((11 w. +b
q(x))((m)) = w z +
that if m d, write this
f(x)L. = Wz bb
Az + bb,
where the matrix L is full-rank and invertible, and the last line follows from applying --1 on both sides. Since W is also full-rank and invertible when m d, the linear transformation is also invertible.
If Asssumpttsnss annd hold, then the function that satisfies Equations (3) and (4) identifies true latent up to permutation and scaling, i.e., A DP where D is diagonal scaling matrix and P permutation matrix.
Theorem 2.
Proof. To show this, we follow a similar proof as Ahhuja al. (2022a). Based on Assumption 2 we have for an 21,2,... 2,....dd there exists some Cj and C₁, such that is the causal differentiating factors between C3 and Cj. From Corollary I] we have 2,- Z;- + δus for some X;- E C& and X,. C₁, where du d-dimensional vector which takes a nonzero value at position r and elsewhere. Or simply 2,-
+ b,e,, where vector with at the tth position and 0 elsewhere and b, is nonzero scalar.
[0,.... l...., 0] is d-dimensional identity
16

---

For some pair of groups C₁ and C₁ that are related by a 1-sparse causal differentiating factor, the contrastive objective in Equation 4 yields a zero loss term the learner correctly finds any of the inputs in the set corresponding to i" and
Suppose the learner guesses 2; E C₁ and 2z. E C₁ such that 2,- = + Ōus, where dس is the guessed
perturbation with du c,e,, such that {l....,d dd and c, & O. That is, the learner guesses that the pair of points x; and X,. satisfy the corollary. Our proof follows in two steps: first, considering the case when X; is equivalent to x that is ((((x,) 7(9(x,)) )), or simply put 2; Zi, and second, considering the case when x; is not equivalent to x;-.
Using Theorem
we have
2-- Azj- + b
+ 6yg Alz,- + +81z)
If i i", we can write
2,- + 8R = Alz 8k1) +
2.1 + C.C₂ A(z,' b,c,) +b
2,1 + e = Az, Ae, +b
ege₂ Ar,
Cs b,
e. Ae,
(5)
This implies that the thth column of A is b. es. This is because the jth entry on the right side would be the 8t row of A multiplied by er, Since all values of Cr, except the yth one, are zero, this multiplication would yield the tth entry on right side as Ai,. Therefore, A;, is zero for i = 1...., d, except 8.
However, the learner can attempt to enforce the sufficient condition 2,. 2. pair of points where * i' by learning a axis-misaligned (i.e., linearly entangled) solution for 2 f(g(x)). In this case, when i # i", the necessary condition in Assumption 2 will be violated.
0 de for different
Essentially, if x4 Ck. such that Zy. and z; differ along 8gt, which is not axis-aligned, we can write Zj. Zj + eue where # (depicted in Figure (left)). If this is the case, get
2 Zj- + b,e,
2.- = 2, + b,e, + eue,
A(z;.) + b Alz, + b,e, Ae.) + b
2,- = é, + Ae, + Ae,
By the necessary condition, 2;- and 2, should differ along at least (that is, the learned causal differentiating concept). However, if we choose Zj. such that, Gu = -b, Ar, Am we get (b, + e, Ae,, e.) = O. That is, zi- and 2,. which lead to different behaviors, differ by a factor that is not the guessed causal differentiating factor. This is contradiction. Note that this contradiction does not happen if there is no such Zj Cz. This case is shown in in Figure 4 (right), where the group C₁ does not contain the z, point that could induce the contradiction. In this case, however, the model could use clusters C₁ and C3 to learn the alignment for latent 2² instead.
Now, since the span of both true and guessed perturbations d, we get d equations of the form Equation 5] such that for every there is unique j. Note that the condition does not need to be met for all pairs of groups, but at least some pair of groups along a dimension r' E {l..... d}. Subsequently, applying the above argument to all column of A yiclds A permutation of an identity matrix. Note that, even though this condition derived for pairs of groups that have 1-sparse causal differentiating concepts, since 2 Az b, Vz (based on Theorem 1), if A is permutation of identity matrix for some z, it is true for all z.
The argument in this section requires the learner to enforce sparse perturbations for pairs C₁ and C₁ that actually differ by a 1-sparse causal differentiating factor. However, the learner could cheat by attempting to enforce 1-sparse shifts between the wrong pair of groups. In the next section, we show that by encouraging sparse shifts across all pairs of groups, the objective in Equation (4) finds disentangled solutions.
17

---

22
21
Zi
Z
2z
Figure 4: Visualization of the case when learner guesses causal differentiating concepts that is not axis-aligned.
82
82
82
z?
22
z²
Ca
C2
C
C2
c
C2
C2
C
Figure Base cases
How does the constraint in Equation enforre the correct alignment? In the previous part, we show that if the learner enforces Equation (4) for some Cj and Ct such that is the 1-sparse causal differentiating concept between Cle and Ct. we will have identifiability up to permutation and scaling. What remains is to argue that the optimization in (4) indeed drives the model to choose these correct cluster pairs. This is based on the argument that correctly aligned axes identify strictly more pairs of groups that satisfy the I-sparse causal differentiating concept condition, and therefore yield smaller loss Equation (4) formalize this intuition by induction on the number of groups along a given latent dimension.
Setup. Let us consider case where all but one axis misaligned, say 2! instead of 21. We will show that the number of cluster pairs that have 1-sparse causal differentiating concept as the true latent 2 is strictly larger than along the rotated axis 3t_ This argument then extends identically the remaining d dimensions.
For simplicity, consider a two-dimensional latent space with the true coordinates (= =²) and the learned latent space =²). We call pair C2 and C₁ match-pair if it has a I-sparse causal differentiating concept along the z1. We want show that there are strictly more match pairs along 2² than along =!_
Base cases. We begin with the simplest case where there are only two distinct values along each axis. There are three possible configurations of groups, as illustrated in Figure In the first case, we have match pairs along the axis z!: {C1, C2), (Cs, C4)), but only rotated axis εt: [(C1, Cz)}- Similarly, in the second case, we have two match pairs along the true axis z²: [(C1, C2), (C1, C.)}, but again only match- pair along the rotated axis 21: {(C1, C.) In the last case, however, we have only match-pair along both the true and the rotated axes: {(C2, C and (Ci, G. respectively. Thus, the strict inequality in the number of match pairs holds when there are more than two match pairs along z2-that is, when the variation along z is sufficient to reveal the structure of the causal differentiating concepts.
match-pai along the
Inductive step. Next, we show that adding additional variation along the 21 axis preserves this strict inequality. Consider the scenario depicted in Figure 6 where the shaded (yellow) region represents groupings already included up to iteration Let n denote the number of discrete values taken by 22, i.e., the number of values along the 2² axis. Let m, denote the number of match pairs identified along the true axis z and m denote the number of match pairs identified along the misaligned axis el up iteration i. By the inductive hypothesis, m, m/. We want to show that m,+1 >> m
When additional group of set of groups introduced along 21, three cases occur:
18

---

22
el
gl
Figure 6: Induction cases
(1) A new group Cy is added. This group forms new match pair with at least one existing group for each the n possible values along 2² (and possibly more, depending on the range along 2 Hence, m4+1 2 m, + n. Along the misaligned axis 2 however, the same group can form at most n - match pairs due to the rotation, so m S my + n Thus,
m;+1 m 2 (m, - -m()
= m441 2 m
(2) Multiple new groups are added along z1. Each behaves analogously to case (1), and the inequality above holds again, so M4+11 2 m1
(3) Merged groups. One or more new observations may have similar behavior as the existing observations, leading to merged groups (rightmost case in Figure 6). In this case, new match pairs are created or destroyed, so the inequality from the previous step is preserved.
Therefore, by induction, the number of match pairs along the true axis 2 are strictly greater than along any misaligned axis e Since the objective in Equation (4) sums the contrastive constraint over all pairs of clusters, the alignment : = +()()) that yields more valid match pairs (i.e., the true disentangled alignment) achieves a strictly smaller objective value than any entangled alignment.
B Implementation details
Data. For synthetic, generate 20,000 samples. We consider the following mixing functions for linear, non-linear sparse, and non-linear non-sparse experiments:
d = 2
1220,521 20 21, 220 + 21, 20 + 421, 320 221]
(linear, non-sparse) (non-linear, sparse) (non-linear, non-sparse)
x 20,
3
421, 2 sinn(z)) cos cos(21)]
42
x 22021,20 20
321, 2 21
202,, b 220 + 21, 2
22, 20 + 421 322, 3z0 221 + 522] x [20, 221, 622, 20 + 323 2 + 42 + 522, cos(20). 6sin (21) sin(22)]
x = 2ž0, 52,,322, + 22, 250 +
X [2021 321 + 522, 20 20 sin s22) 250 + 22, 200= =]
For semi-synthetic data, we sample 20,000 bios from BiasBios dataset (De-Arteaga et al. 2019).
which is available under Apache-2.0 license. We assign the high, medium, low occupation class based the 1/3 and 2/3 quantile of median male salary for the respective occupation based on the U.S. Department of Labor's Employment and Earnings by Occupation statistics www.o///ggggg/////aaa//////pppap
For non-synthetic data, we use 500 examples each from harmful, harmless, and pseudo-harmi harmful categories. Harmful examples are sampled uniformly from the MALICIOUSINSTRUCT (Huang et al. (2024), CC BY-SA-4.0 License), HARMBENCE (Mazeika et al. (2024), MIT License), ADVBENCH (Zou et al. (2023). MIT License), and TDC2023 (Mazeika et al. (2022), MIT License) datasets. Pseudo-harmful examples are sampled from OR-BENCH- 80K (Cui et al. (025). CC BY-4.0 License). All datasets follow 70:15:15 train- viii-vvaiaaiio-ttss split.
19

---

Table 3: Compute resources for different experiments. All runtimes were consistent across our method and the two baseline approaches.
Data setting
Model
Compute resources
Approx. time
Synthetic Semi-synthetic Semi-synthetic
Feedforward NN
4 CPU, 32 GB
20 minutes
Distil
1 GPU, 4 CPU, GB
45 minutes
Qwen2-7B
4 GPU, 16 CPU, 256 GB
17 hours
Semi-synthetic
3lama-3.1 1--8B
GPU, 16 CPU, 256 GB 4 GPU, 16 CPU, 256 GB
17 hours
Non-synthetic
Qwen2-7B
2.25 hours
Non-synthetic
-lama-3.1-88
4 GPU, 16 CPU, 256 GB
2.25 hours
Models. In our experiments, the abstraction model M is implemented as feed-forward network with ReLU activations. For synthetic data, the bottleneck consists of two linear layers, with a fixed hidden dimension of 4, except for the final layer in the bottleneck, which has the output dimensionality of d E [2.3]. 3). For semi- comi-synthetic and non-synthetic data, the number of layers and dimensionality of the bottleneck is hyperparameter (with Mayyrrs [2, 4, 8, 16] and ha [64, 128, 256, 512)), with the hidden dimension of the final layers fixed to d. The predictor always consists of a single linear layer to match the loglinear assumption.
For the experiments with synthetic data, we estimate g(x) with a feedforward neural network. Essentially, construct counterpart of the language model in Figure by training an encoder- decoder model using a variational autoencoding objective. We then explain the causal factors in g(x) by learning the alignment =x)(x) using the proposed method detailed in Section 3 The encoder-decoder model is implemented as a stack of 4 linear layers with ReLU activations, each with hidden size of 16.
For the contrastive constraint, we estimate the causal differentiating factor 6μ by searching over the latent dimension d. Specifically, for Ck. we sample examples from another group Cj and define the contrastive loss as min; (x]lf(x)) - f(xx) - 8sa113).
libraries for our experiments. For experiments with synthetic data, we train our models using the Adam optimizer and learning rate scheduler that reduces the learning rate when the validation loss plateaus. The model is trained for 50 epochs and the best checkpoint is selected based on the validation loss.
Training. We use PyTorct and HuggingFace Transformers" To
For semi-synthetic and non-synthetic experiments, we use the default optimizer and scheduler provided in the Transformer training utils (AdamW and a linear learning rate scheduler). The model is trained for epochs. In the semi-synthetic setting, the number and size of layers in the bottleneck modules are treated as hyperparameters, as detailed earlier. Hyperparameter selection is performed with grid search using the Ray Tune library] optimizing for disentanglement score on the validation dataset. For the non-synthetic dataset, where the ground-truth causal factors are not known, we reuse the best best-performing hyperparameters identified in the semi-synthetic setting.
Statistical analysis. For significance testing on experiments with synthetic and semi-synthetic data, we divide the test datasets into 5 splits. For each split, we compute the disentanglement, completeness, informativeness, and MCC scores. Under the assumption that the average scores are approximately normally distributed, we perform independent t-tests assess statistical significance, with a threshold of 0.05.
Compute resources. The experiments in this paper were conducted on machines equipped with Tesla P100-PCIE- 12GB GPUs. The resource usage along with compute time are shown in Table 3
'hatss//pytorch.orrg/
http:://hwggunnnieee..odddddddrr hauns://oossrrgiiieow//eeeeeeeee
20

---

Data class Pacudo-harmful
Ply - retusal)
Autoencoding
Auteencoding-fred
Our constraints
15-
Labent
-20-
Latant Dimension
Latent Dimenston
Latent Dimension
Figure Latent space for refusal behavior the Qwen2-7B model with autoencoding (lef)) autoencoding prediction (center), and constraints (right).
Disentanglement
Mean Disentarglement
informativeness
Mean informativeress
Full Feature vector (d 1294)
Two most predictive teatures for [d 2)
The most predictive features for each causal factor (d==)
0:
0:0:9
1:0.4
Layors
Layers
Layers
Figure 8: Disentanglement and informativeness scores for 128K-dimentt -iseniinnnsi sparse autoencoders across different model layers. Solid lines represent the scores for each layer, with dashed lines indicat- ing the mean, and shaded regions representing 1 standard deviation. The maximum disentanglement score in each subplot is annotated along with its corresponding informativeness score.
C Additional results
Case-study experiments the Qwen2-7B model Figure shows results refusal behavior in language model on the Qwen2-7B model. Similar to the findings on the Llama-3.1-80 1-BB model Figuure 2), we see that the baseline method entangles the two latent dimensions, but the contrastive constraint leads to an aligned latent space with latent dimension I representing the harmfulness factor and the latent dimension 2 representing the topic factor. Notably, even though the model behavior is defined based on p(y refusal x), the learned latent space neatly separates the harmful, pseudo-harmful. and harmless prompts.
Comparison to sparse autoencoder for 128k-dimensional SAE Figure 8 presents results on semi-synthetic dataset for the 128k-dimensional sparse autoencoder. The findings are consistent with those observed for the 32k model, as discussed in Section
Varying latent dimensions As discussed in Section 7 we perform experiments where we vary the dimensionality of the learned latents to assess how over-specification or under-specificatioo affect disentanglement metrics. We consider two cases: (1) when the dimensionality of learned latents is higher than that of true mediating concepts, and (2) when the dimensionality of learned latents is lower than that of true mediating concepts.
In experiment 1, we construct synthetic data with the number of true mediating concepts set to 2 and the number of learned latent concepts set to and 4. We find that we relax the span constraints in Section 3 that is, the learned perturbations do not need to span all latent concepts, the learned
21

---

perturbations correspond to the true causal differentiating concepts. That is, the model disregards any extra latent(s) learning the causal differentiating concepts, and the extra latents are solely some function of one of the other latent variables. Thus, the model achieves high disentanglement score, but a lower completeness score (since more than one latent captures a true causal factor): DCI-D 0.95 + 0.03 and DCI-C 0.81 + 0.14 for 3. and DCI-D 0.94 + 0.04 and DCI-C 0.76 + 0.19 for n = 4. In sum, when the model's latent dimension is mis-specified to be larger than the true latent dimension, we observe degree of robustness in the performance.
With the span constraint, the learned perturbations no longer correspond to the true causal differenti- ating concepts. This leads to degradation in disentanglement scores, alongside lower completeness scores; DCI-D 0.64 + 0.11 and DCI-C 0.81 = 0.14 for n and DCI-D 0.89 + 0.08 and DCI-C 0.57 for n = 4. Interestingly, the disentanglement scores are still higher than the autoencoding (DCI-D 0.19 0.12, D.I-C00244 0.15) and autoencoding prediction (DCI-D 0.08 + 0.07, DCI-C 0.09 + 0.08) baselines. But more importantly, overspecifying latent dimensionality consistently leads to a drop in completeness scores, which can be used as a signal for reducing the latent dimensionality as a hyperparameter.
In experiment 2, we construct synthetic data with the number of true mediating concepts set 3 and the number of learned latent concepts set to 2. We find that under-specifying the latent dimensionality results in significant drop in the disentanglement scores (0.51 = 0.20). This is to be expected since the learned latents cannot separate out the mediating concepts since there are not enough latents. Thus, we conclude that under-specifying latent dimensionality consistently leads to a lower disentanglement score, which can be used as signal for increasing the latent dimensionality as a hyperparameter.
Choice of intervention layers To empirically test the impact of applying our approach to inter- mediate layers of LLMs instead of the last layer, we extend our refusal case study to intermediate representations and assess whether our method can extract relevant insights from intermediate LM layers as well. For simplicity, we consider the representation at the last token position in each layer of the Llama-3.1-88 model. We find that in the earlier layers, our approach does not yield disentangled representations. This is to be expected, as it has been noted before that initial LLM layers capture surface-level encodings of the inputs. However, past layer 9, the structure presented in Figure 2 starts to emerge. That is, we observe that the harmfulness and topicality dimensions identified in our experiments can also be disentangled in intermediate model representations. For quantitative as- sessment, we calculate the average unsupervised disentanglement score between the model's internal representations and the last layer. We observe an unsupervised disentanglement score of 0.78 + 0.02 for layers 10-30 compared to 0.39 + 0.20 for the first 9 layers. Interestingly, these findings align with recent paper (Zhao et al., 2025) that finds that harmfulness and refusal directions can be extracted from LLM representations for Llama2- ChattttBBB around layer 10. It would be interesting for future exploration to study whether there are interesting concepts that can be extracted from intermediate layers that are abstracted away or lost in later layers. Our method, hopefully, provides a useful tool for such discoveries and insights.
Violation of 1- 1-sparse assumption Even though we restrict our discussion to I-sparsity, we expect that our results can be extended to 2-sparse condition as well (or generally speaking, a p-sparse condition) following the proof strategy as in [Ahuja et al. (2022a). That is, if the causal differentiating concepts are p-sparse and non-overlapping, we can expect identifiability up to permutations and block-diagonal transforms, rather than our stronger results on identifiability up to permutations and scaling. Further, causal diffcrentiating concepts are p-sparse and overlapping, we can expect that the latents at the intersection of these blocks will be identified up to permutations and scaling.
To validate this second case in our setting, we construct synthetic data with 2-sparse causal differ- entiating concepts for d = 3. Specifically, imagine three causal variables: occupation, gender, and experience that determine the income prediction behavior of the model, such that the p(y]æ) can be clustered into four classes: [ ((Doctors/aawerrs Male, More experience), (Doctors/Lawyers, Female, Less experience); (Teachers/Nurses, Female, More experience), and (Teachers/Nurses, Male, Less experience)). We see that in this data, groups Cs and Cj have have 2 causal differentiating concepts. For example, for behavior classes (Doy,s,s/lawyers, Male, More experience) and (Teachers/Nurses, Female, More experience), the causal differentiating concepts are occupation and gender.
22

---

We observe that our method can still disentangle the three causal factors. The average disentanglement measures (across 5 runs) are comparable our 1-sparse Prtting-CII-D (0.95 0.02). DCI-C (0.98 + 0.01), and DCI-I (1.0 + 0.0). Note that we get this disentanglement because the 2-sparse causal differentiating concepts are overlapping for different pairs of groups. This indicates that our method can be extended beyond the 1-sparse setting following the intuition from (Ahuja et al. 2022a). We leave more detailed explorations of these cases to future work.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: scope.
Both the abstract and introduction clearly state the paper's contribution and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussion the limitations of our work and applicability of the assumptions that make in Dneecioon 9]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approxinations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
23

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
We include assumptions in Section 3 and proofs
Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include implementation details of our experiments in Section 4 and Appendix B We also include data and code needed to reproduce our experimental results in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
24

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the data and code, with instructions reproduce the experimental results the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
gublic/guides www.ee/Suuiiiiiosoooooo.oooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training details, data splits, and hyperparameter selection are included in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
25

---

Answer: [Yes]
Justification: We include statistical significance of the reported results in respective tables
TTaless T 2).
The detail on the significance analysis available in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details on compute requirements is included in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in this work adheres to NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
26

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We include discussion how this work can aid interpretability of concepts that mediate language model behavior, which can be useful for LM design and applications. To the best of our knowledge, this work does not pose any negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release new data model that have high risk of misuse. Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper includes citations for all data and models used for our experiments.
We also include license information for these resources in Appendix B
27

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We do not introduce any new models datasets. The supplementary material includes the code developed in this work, alongside appropriate documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
28

---

Answer: [NA]
Justification:
This work does not involve any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This work does not involve LLMs in any important, original, non-standard components. We conduct our experiments on large language models, which is appropriately disclosed in the main paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
29


Reviews:
[{"review_id": "ccsbA4P1Tl", "reviewer": "Reviewer_2QmS", "summary": "The authors claim 3 contributions:\n1. \"Causal differentiating concepts\" - factors that much change to eliciting different model behavior - and a sparsity assumption for their identification.\n2. A contrastive learning algorithm to identify the above concepts which is proven recover disentangled features.\n3. Synthetic experiments and a real-world case-study supporting the above results.", "questions": "1. Is there evidence to support the claim that the true conditional probability distribution of concepts can be modeled with the logit-linear function, as in Assumption 1? Are such conditional distributions universally expressive, or does the assumption impose constraints on the distribution that can be learned?\n2. Does Assumption 2 require that every pair must be differentiated by some $z^r$? Does it require that each $z^r$ must differentiate exactly one pair of concepts?", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Originality: The work introduces a novel but simple latent factor model for describing categories of behaviors in LLMs, under certain modeling assumptions.\n\nQuality & significance: The theory appears to be sound, and the work improves significantly upon the baselines in disentanglement and completeness, another measure of disentanglement. However, it would be desirable to see more evaluation on the validity of the underlying assumptions.\n\nClarity: the work is generally clearly written.", "was_revised": true, "final_justification": "The authors satisfactorily addressed my concerns.", "rebuttal": "Thank you for your positive feedback on our work. We appreciate that you found our work to be novel, with sound theory and experiments. Below, we answer the questions posed in your review. \n\n---\n\n#### Expressiveness of log-linear functions  \nGood question. Hyvarinen and Morioka (2016) point out that log-linear prediction on top of flexible, highly non-linear feature extraction still has universal expressiveness. Indeed, this style of modeling is akin to nonlinear prediction via kernel regression or basis expansion, which remain flexible, nonlinear predictors. \n\n--- \n\n#### Does Assumption 2 require that every pair must be differentiated by some $z^r$?  \nEven though this is not explicitly stated in the assumption, it is implicit in the problem definition that a difference in behavior would correspond to some mediating concept(s) $z^r$. \n\n--- \n\n\n#### Does it require that each $z^r$ must differentiate exactly one pair of concepts?  \nNo, it is not necessary that each $z^r$ must differentiate exactly one pair of groups. In the refusal case study, for instance, latent dimension 1 (interpreted as harmfulness) is a causal differentiating concept between harmful ($p(refusal|x)=high$) and pseudo-harmful queries ($p(refusal|x)=high$), and also between harmful and harmless queries ($p(refusal|x)=low$). \n\n\n---\n\n#### References  \n[1] Aapo Hyvärinen, Hiroshi Morioka. Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA. NeurIPS 2016."}, {"review_id": "ETVeXDkYsz", "reviewer": "Reviewer_txCi", "summary": "This paper introduces a novel causal representation learning object which aims to recover the latent factors underlying model behaviour using an labelled dataset. Unlike previous approaches (e.g. causal abstract), this method does not need the hypothesis about the latent factors up front, but rather it uses key properties of the data and assumptions about the latent variables to learn the factors by itself. The authors provide results on both synthetic and real data which validates their method against alternatives without their causal objective, and baselines like sparse autoencoders.", "questions": "- The bottleneck is currently applied to the last layer hidden representations as justified by assumption 1. Is there any setup under which this method could be applied to internal representations, or does assumption 1 render this unjustifiable? This would enable identifying latent factors in model-internal components, which is useful for interpretability. It would also enable an alternative to SAEs.\n- If the dimensionality of the bottleneck doesn't correspond well to the true number of factors, what happens? E.g. if only 1 dimension or 5 dimensions are given to the harmfulness task, are the learned factors still cleanly basis-aligned? I'd be curious if it results in interesting decomposition of the topic factor (latent 2 in figure 2). Some experiments or at least theoretical analysis of this would be informative.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The paper is very mathematically clean and offers a new approach to a real thorny issue in causal abstraction: generating hypotheses about underlying latent variables in the causal model is currently a fully manual process. If we could learn the latent variables that would make causal abstraction far more scalable, which this work seems to make progress towards doing.\n- The experiments involve several levels of complexity (from clean synthetic data to a more complex collection of real-world data on a relevant topic: refusal behaviour) and compare against well-formed baselines. It is great to see the inclusion of currently practical and in-vogue methods like SAEs. Outperforming these baselines builds convincing evidence for the validity of the method.\n\nWeaknesses\n- The real-world dataset is quite cleanly organised. It would be nice if this method could be applied to far messier real-world data like the baseline SAEs that were compared against. But this is not a significant weakness since this may require a wholly new method; this approach seems suitable for the problem the authors put in the scope of the paper.", "was_revised": true, "final_justification": "All questions satisfied with additional experiments, no further questions or objections remain. The method seems solid; future work can test it on more real-world datasets.", "rebuttal": "Thank you for your positive feedback on the theory and experiments presented in our work. We appreciate your recognition of the value of our work in scaling causal abstractions through unsupervised discovery of high-level concepts that affect model behavior. We also appreciate your positive comments regarding our experiments and evaluation. To address the questions raised about the sensitivity to variation in some modeling choices, we ran 3 additional experiments that evaluate misspecification of the latent dimension and assess our ability to extract insights from intermediate layers of LMs. \n\n---\n\n#### Applying the method to intermediate language model layers\nThis is a great point. In our experiments, we restrict ourselves to interpreting the last layer of the language models; however, experimentally, our method can be directly applied to intermediate LM layers as well. Although our theoretical results do not directly translate to intermediate representations, a host of recent literature identifies that latent concepts are linearly encoded in LM representations (linear representation hypothesis; Roeder et al., 2021; Jiang et al., 2024; Park et al., 2024). Since the contrastive learning constraints we develop in Eqn 4 are meant to disentangle linearly mixed concepts, we hypothesize that we can still use the method proposed in this paper for unsupervised discovery of unknown latent concepts present in the model’s internal representations. \n\nTo empirically test this, we extend our refusal case study to intermediate representations to test whether our method can extract relevant insights from intermediate LM layers as well. For simplicity, we consider the representation at the last token position in each layer of the Llama-3.1-8B model, which is a common practice in other work studying internal LLM representations (Zou et al., 2022; Arditi et al., 2024). We find that in the earlier layers, our approach does not yield disentangled representations. This is to be expected, as it has been noted before that initial LLM layers capture surface-level encodings of the inputs. **However, past layer 9, the structure presented in Figure 2 starts to emerge. That is, we observe that the harmfulness and topicality dimensions identified in our experiments can also be disentangled in intermediate model representations.** For a quantitative assessment, we calculate the average unsupervised disentanglement score between the model's internal representations and the last layer. We observe an unsupervised disentanglement score of $0.78 \\pm 0.02$ for layers 10-30 compared to $0.39 \\pm 0.20$ for the first 9 layers. Interestingly, these findings align with a recent paper (Zhao et al., 2025), released earlier this month, that finds that harmfulness and refusal directions can be extracted from LLM representations for Llama2-Chat-7B around layer 10. \n\n---\n\n#### If the dimensionality of the bottleneck doesn't correspond well to the true number of factors, what happens? \nThis is an interesting question. We conducted experiments to test variation in the learned number of latents in synthetic experiment settings. We consider two cases: (1) when the dimensionality of learned latents is **higher** than that of true mediating concepts, and (2) when the dimensionality of learned latents is **lower** than that of true mediating concepts. \n\nIn experiment 1, we consider synthetic data with the number of true mediating concepts set to 2 and the number of learned latent concepts set to 3 and 4. We find that if we relax the span constraints (line 147), that is, the learned perturbations do not need to span all latent concepts, the learned perturbations correspond to the true causal differentiating concepts. That is, the model disregards any extra latent(s) in learning the causal differentiating concepts, and the extra latents are solely some function of one of the other latent variables. Thus, the model achieves a high disentanglement score, but a lower completeness score (since more than one latent captures a true causal factor): DCI-D $0.95 \\pm 0.03$ and DCI-C $0.81 \\pm 0.14$ for $n=3$, and DCI-D $0.94 \\pm 0.04$ and DCI-C $0.76 \\pm 0.19$ for $n=4$.  In sum, when the model's latent dimension is misspecified to be larger than the true latent dimension, we observe a degree of robustness in the performance.\n\nWith the span constraint, the learned perturbations no longer correspond to the true causal differentiating concepts. This leads to degradation in disentanglement scores, alongside lower completeness scores: DCI-D $0.64 \\pm 0.11$ and DCI-C $0.81 \\pm 0.14$ for $n=3$ and DCI-D $0.89 \\pm 0.08$ and DCI-C $0.57 \\pm 0.13$ for $n=4$. Interestingly, the disentanglement scores are still higher than the autoencoding (DCI-D $0.19 \\pm 0.12$, DCI-C $0.24 \\pm 0.15$) and autoencoding + prediction (DCI-D $0.08 \\pm 0.07$, DCI-C $0.09 \\pm 0.08$) baselines. **But more importantly, overspecifying latent dimensionality consistently leads to a drop in completeness scores, which can be used as a signal for reducing the latent dimensionality as a hyperparameter.**\n\nFor experiment 2, we consider synthetic data with the number of true mediating concepts set to 3 and the number of learned latent concepts set to 2. We find that underspecifying the latent dimensionality results in a significant drop in the disentanglement scores ($0.51 \\pm 0.20$). This is to be expected since the learned latents cannot separate out the mediating concepts since there are not enough latents. **In this case, we conclude that underspecifying latent dimensionality consistently leads to a lower disentanglement score, which can be used as a signal for increasing the latent dimensionality as a hyperparameter.**\n\nWe restrict ourselves to synthetic experiments for sensitivity analysis because in semi-synthetic and non-synthetic experiments, a parallel comparison is not feasible since the dimensionality of true mediating concepts is integral to the dataset and is not dictated externally. We believe that findings from synthetic experiments would naturally extend to general cases. Even though non-synthetic settings lack ground-truth mediating concepts, not allowing direct evaluation of disentanglement and completeness scores, the disentanglement measures can be measured across multiple model runs as a signal for guiding the latent dimensionality, following Duan et al. (2021). \n\n---\n\n#### References\n[1] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, Neel Nanda. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2025.   \n[2] Sunny Duan, Loic Matthey, Andre Saraiva, Nick Watters, Chris Burgess, Alexander Lerchner, Irina Higgins. Unsupervised Model Selection for Variational Disentangled Representation Learning. ICLR 2020.   \n[3] Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, Victor Veitch. On the Origins of Linear Representations in Large Language Models. ICML 2024.   \n[4] Kiho Park, Yo Joong Choe, Victor Veitch. The linear representation hypothesis and the geometry of large language models. ICML 2024.   \n[5] Geoffrey Roeder, Luke Metz, Diederik P. Kingma. On Linear Identifiability of Learned Representations. ICML 2021.   \n[6] Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau, Weiyan Shi. LLMs Encode Harmfulness and Refusal Separately. 2025.   \n[7] Andy Zou et al. Representation Engineering: A Top-Down Approach to AI Transparency. 2025."}, {"review_id": "THv8soebiw", "reviewer": "Reviewer_D9ui", "summary": "This paper studies the problem of extracting interpretable variables from LLM activations, under various assumptions. Given a dataset of paired texts x and discrete labels c (coming from an LLM), it's an important problem to be able to extract out the concepts that are responsible for the labels (assuming such a structure exists). The work considers the setting that there exist a set of mediating concepts z that are a function tau(g(x)) of the LLM activations g(x). Furthermore, the work assumes\n1. z is mediating, i.e. c is indepenent of x given z\n2. c is related to z via a logit-linear function.\n3. Every concept is a 1-sparse causal differentiating concept: There exist a pair of classes such that changing this concept is necessary and sufficient to change from one class to the other.\n\nUnder these assumptions, it's shown that z is identifiable up to linear transformations. Moreover, the authors propose a simple loss function to learn these latents, which is a variant of the contrastive loss. Experiments on synthetic data, BiasBios data and LLM text data show the effectiveness of their approach, compared to some prior works such as SAEs, when we look at disentanglement and MCC metrics. The target audience are people interested in interpretability of LLMs.", "questions": "Some questions were raised above.\n\n- It seems like for the proof to go through, it maybe possible to replace 1-sparsity to some sort of \"full rank\" condition on concepts <-> label structure. Could the authors comment on this?", "limitations": "Limitations have been discussed.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "## Strengths:\n\n- The paper approaches the interesting and relevant problem of extract interpretable variables from LLM representations. This is important for many reasons, such as to figure out inherent biases in LLM systems.\n\n## Weaknesses:\n\n- The assumptions seem a bit too strong. For instance, assuming that every relevant concept for a given set of labels is a 1-sparse causal differentiating concept is a very unique structure that is unlikely to hold in practice. It seems like the proof or technique may break if we have 2-sparsity (for an appropriate definition) instead of 1-sparsity. Could the authors comment on this?\n\n- The assumption that the mediating concepts are linearly related to the output variables is also very strong, since it essentially abstracts away the nonlinearity and turns the problem into a standard linear regression setting.\n\n- The theorems are fairly straightforward given the strong assumptions, e.g. theorem 1 follows by simple manipulations once linearity is assumed and theorem 2 essentially follows the proof ideas from Ahuja et al. 2022, so theoretical novelty is quite limited.\n\n- The number labels studied in the experiments seem very small, compared to what the approach or method can handle. It would be nice to have experiments on slightly more complicated real-world datasets with multiple classes.", "was_revised": true, "final_justification": "While the authors have attempted to clarify my questions, the issues I raised are not fully resolved. The authors agree with the weaknesses of the paper and try to justify them with various reasons (which are not too convincing and are rather circumstantial). Therefore, I decide to keep my rating, however I will not oppose of majority of the reviewers like the work and wish to accept the work.", "rebuttal": "Thank you for your feedback on our work. We share some clarifications on the validity of the assumptions and the applicability of our method, which will hopefully help address your concerns. \n\n---\n\n#### Would the proof or technique break if we have 2-sparsity instead of 1-sparsity?   \nEven though we restrict our discussion to 1-sparsity, we expect that our results can be extended to 2-sparse condition as well (or generally speaking, a p-sparse condition) following the proof strategy as Ahuja et al. (2022a). That is, if the causal differentiating concepts are p-sparse and non-overlapping, we can expect identifiability up to permutations and block-diagonal transforms, rather than our stronger results on identifiability up to permutations and scaling. Further, if causal differentiating concepts are p-sparse and overlapping, we expect that the latents at the intersection of these blocks to be identified up to permutations and scaling. \n\nTo validate this second setting in our case, we construct synthetic data with 2-sparse causal differentiating concepts for d=3. Specifically, imagine three causal variables: occupation, gender, and experience that determine the income prediction behavior of the model, such that the $p(y|x)$ can be clustered into four classes: {(Doctors/Lawyers, Male, More experience), (Doctors/Lawyers, Female, Less experience); (Teachers/Nurses, Female, More experience), and (Teachers/Nurses, Male, Less experience). We see that in this data, groups $C_k$ and $C_l$ have 2 causal differentiating concepts. For example, for behavior classes (Doctors/Lawyers, Male, More experience) and (Teachers/Nurses, Female, More experience), the causal differentiating concepts are occupation and gender. \n\nWe observe that our method can still disentangle the three causal factors. The average disentanglement measures (across 5 initializations) are comparable to our 1-sparse setting—DCI-D ($0.95 \\pm 0.02$), DCI-C ($0.98 \\pm 0.01$), and DCI-I ($1.0 \\pm 0.0$). This indicates that our method can be extended beyond the 1-sparse setting as well. \n\nWith respect to the feasibility of 1-sparse assumption, we want to note that our 1-sparse assumption is more relaxed than that of Ahuja et al. (2022a) since we do not require that every pair of clusters have only 1 causal differentiating concept or that the pair of clusters that have 1 causal differentiating concepts are known beforehand, rather that every $z^r$ is 1-sparse for some pair of clusters. This assumption is easier to meet in practice, as unless some points in input space are not observed altogether, 1-sparsity would hold naturally. For instance, in the above example with 2-sparsity, we do not observe data points that are {Doctor/Lawyer, Female, More experience} at all. If we did, we can have one of following cases: (a) this is a separate behavior cluster, in which case this cluster and (Doctors/Lawyers, Male, More experience) will have gender as the causal differentiating concept and this cluster and (Teachers/Nurses, Female, More experience) will have occupation as the causal differentiating concept; (b) we have {Doctor/Lawyer, Female} cluster (including both more and less experience data points), in which case this and (Teachers/Nurses, Female, More experience) will still have occupation as the only causal differentiating concept; and so on. We hope this example helps convey why 1-sparsity is not a particularly strong assumption in practice, as long as sufficient variability in the input space is observed. \n\n---\n\n#### Linearity assumption  \nLog-linearity prediction is ubiquitous not only in causal representation learning literature (e.g., Hyvärinen and Morioka, 2016; Ahuja et al., 2022b) but also in machine learning. The idea is that the non-linearity in data is pushed into the feature extraction, and the outcome (e.g., next-token, label, etc.) is predicted linearly from the extracted features. Thus, the expressivity of such a setup is not akin to linear regression. \n\n\nWhile log-linearity is certainly an assumption, previous work in CRL theoretically proves that unsupervised learning of disentangled representations is fundamentally impossible without inductive biases (Locatello et al., 2019). Thus, certain assumptions are inherent to the goal of disentanglement. In the context of LLMs, the log-linearity assumption may not be as limiting since a host of recent work has found that concepts are linearly encoded internally (linear representation hypothesis; Roeder et al. 2021; Jiang et al., 2024; Park et al., 2024), lending themselves to be discovered by our method without the need of supervision. \n\n---\n\n#### Theoretical novelty  \nThe judgment of novelty is inherently subjective; however, we would like to add that our paper does not claim or promise novelty in proof technique (our proof sketch clearly lays out that the proof naturally follows using techniques from Ahuja et al. (2022b), Hyvärinen and Morioka (2016), and Ahuja et al. (2022a)). A lot of mathematical work, generally, and especially in CRL, focuses on using existing proof techniques to prove theorems in new settings for new insights. The goal of our paper is to develop novel assumptions that take LLM behavior into account, enabling us to use the CRL framework to interpret LLMs without the need for labeled examples, a setting which is majorly under-explored in causal representation learning, with the exception of few recent works (Rajendran et al., 2024; Ahuja et al. 2022b; Moran and Aragam, 2025). \n\nAnother thing to note is that the assumptions that we develop are novel. Reiterating an earlier point, our 1-sparsity assumption is more relaxed than that of Ahuja et al. (2022a) since we do not require that every pair of clusters have only 1 causal differentiating concept or that the pair of clusters that have 1 causal differentiating concept are known beforehand. Thus, the assumptions and proofs from previous work do not out-of-the-box translate to our setting. Rather, we identify more relaxed assumptions that can still use existing proof techniques for identifiable discovery of latent concepts. \n\n---\n\n#### Number of labels  \nCan you clarify why you believe that increasing the number of classes (presumably for the outcome y) could change the conclusion of our experiments, where we already consider behaviors that are non-binary? As we note in the paper, behaviors can be finer-grained than the number of classes, e.g., P(refuse|prompt) can be very high, high, medium, etc. Indeed, in the real-world experiments, we consider behaviors that are continuous valued and bucketed into 3 behavioral classes. \n\nIn terms of comparison with the scope of other interpretability literature, a host of interpretability work spanning SAEs, mechanistic interpretability, mediation analysis, and more study binarized setting, for example, refusal behavior in language models (whether the model refuses a query or not; e.g., Zhao et al., 2025, Yeo et al. 2025), truthfulness in LLMs (whether model output is truthful or not; e.g., Zou et al., 2025), subject-verb agreement (whether subject and verb agree or not; e.g., Marks et al. 2025) and so on. In any of these works and ours, it is not that we restrict the output space of LLMs—they can still produce free-form text—but we interpret a property of this free-form text (refusal, truthfulness, etc.). Thus, our method, similar to existing approaches, can analyze LLMs out-of-the-box, be it a pretrained model, an instruction-tuned model, or a finetuned model. \n\n---\n\n\n#### Replacing 1-sparsity with some “full rank” condition on concepts <-> label structure  \nJust to clarify, by “full rank condition on concepts <-> label structure”, did you mean that if the perturbation matrix $\\Delta$, which is comprised of the latent concepts that differentiate some labels $c_k$ and $c_l$, is full rank? Based on that interpretation, we do not think full rank condition guarantees identifiability. Consider $\\Delta$ as the set of true perturbations and $\\Delta’$ as the set of guessed perturbations. Theorem 1 yields $\\Delta’ = A \\Delta$, where $A$ is an affine transformation matrix. Now, if we assume that $\\Delta$ is full rank, we can conclude that $A$ would be invertible (since both $\\Delta$ and $\\Delta’$ have rank n, so would the matrix $A$). But, it does not say anything about $A$ being an identity matrix (or a permutation of an identity matrix). A similar argument appears in Ahuja et al. (2022a) to prove that full rankness of $\\Delta$ implies that true latents can be identified up to affine transformation, but for identifiability up to permutation and scaling, they use additional sparsity conditions as well. \n\n---\n\n#### References\n[1] Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. NeurIPS 2022.  \n[2] Kartik Ahuja, Divyat Mahajan, Vasilis Syrgkanis, Ioannis Mitliagkas. Towards efficient representation identification in supervised learning. CLeaR 2022.  \n[3] Aapo Hyvärinen, Hiroshi Morioka. Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA. NeurIPS 2016.   \n[4] Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, Aaron Mueller. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. ICLR 2025.   \n[5] Gemma E. Moran, Bryon Aragam. Towards Interpretable Deep Generative Models via Causal Representation Learning. 2025.  \n[6] Goutham Rajendran, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, Pradeep Ravikumar. From Causal to Concept-Based Representation Learning. NeurIPS 2024.  \n[7] Wei Jie Yeo, Nirmalendu Prakash, Clement Neo, Roy Ka-Wei Lee, Erik Cambria, Ranjan Satapathy. Understanding Refusal in Language Models with Sparse Autoencoders. 2025.  \n[8] Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau, Weiyan Shi. LLMs Encode Harmfulness and Refusal Separately. 2025.\n[9] Andy Zou et al. Representation Engineering: A Top-Down Approach to AI Transparency. 2025."}, {"review_id": "9ghO3hyDog", "reviewer": "Reviewer_mRsd", "summary": "The paper introduces a method for discovering interpretable, behaviorally-relevant concepts from language model (LM) activations, called “causal differentiating concepts”. Causal differentiating concepts are defined as latent directions whose change are necessary and sufficient to alter the model’s output behavior. The method operates in an unsupervised manner, relying on a constrained contrastive learning objective and a key sparsity assumption: that behavioral differences between examples can be attributed to sparse shifts in latent concept space. The authors provide theoretical identifiability guarantees under these assumptions, and validate the method on synthetic, semi-synthetic, and real LLM-generated behaviors (e.g., refusal behavior in LLaMA-3.1–8B), showing superior disentanglement performance compared to sparse autoencoders and other baselines.", "questions": "a) Could the authors comment on whether applying the method to intermediate layers (rather than the final LLM layer) could yield more semantically meaningful concepts, even if less directly behavior-predictive?\n\nb) How sensitive is the method to violations of the 1-sparse assumption? Have the authors tested scenarios with 2-sparse or distributed causal shifts?\n\nc) In practice, how does one choose the number of latent dimensions? Is performance sensitive to this choice?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper is well-motivated and addresses an important challenge in interpretability: extracting concise, causal explanations of LLM behavior without relying on predefined labels or concepts. The formalization of \"causal differentiating concepts\" is elegant, and the method is both novel and theoretically grounded. The empirical evaluations are good.\n\nHowever, there are two main concerns. First, the method extracts latent directions from the final layer of the LM, where representations may have already collapsed into task-specific, highly abstract forms. This design choice potentially limits the interpretability of the recovered concepts, since semantic richness is often better preserved in earlier layers. In practice, it is very likely that at least some of the differentiating concepts would collapse at the final layer, especially in more elaborate cases, where such interpretability methods would be most required.", "was_revised": true, "final_justification": "The authors addressed my questions. I am in favour of acceptence, however I am not revising my rating upwards, because I do not know how they intend to integrate their answers into the manuscript.", "rebuttal": "Thank you for your positive feedback on our work. We really appreciate your questions and comments on this work. To address your questions about the sensitivity of our method to specific modeling choices, we ran some additional experiments that evaluate misspecification of the latent dimension, violation of the 1-sparse assumption, and assess our ability to extract insights from additional layers of LMs. \n\n---\n\n#### Applying the method to intermediate layers (rather than the final LLM layer)\nThis is a great point. Indeed, intermediate layers of LLMs can often capture rich semantic information (possibly even richer than the last layer). Even though our experiments assess the last layer of LLMs, implementationally, our method can be directly applied to intermediate LLM layers too. Although our theoretical results do not directly translate to intermediate layers, a host of recent literature finds that latent concepts are linearly encoded in LM representations (linear representation hypothesis; Roeder et al. 2021; Jiang et al., 2024; Park et al., 2024). Since the contrastive learning constraints we develop in Eqn 4 are meant to disentangle linearly mixed concepts, we hypothesize that we can still use the method proposed in this paper for unsupervised discovery of unknown latent concepts present in the model’s internal representations. \n\nTo empirically test this, we extend our refusal case study to intermediate representations to test whether our method can extract relevant insights from intermediate LM layers as well. For simplicity, we consider the representation at the last token position in each layer of the Llama-3.1-8B model. We find that in the earlier layers, our approach does not yield disentangled representations. This is to be expected, as it has been noted before that initial LLM layers capture surface-level encodings of the inputs. **However, past layer 9, the structure presented in Figure 2 starts to emerge. That is, we observe that the harmfulness and topicality dimensions identified in our experiments can also be disentangled in intermediate model representations.** For a quantitative assessment, we calculate the average unsupervised disentanglement score between the model's internal representations and the last layer. We observe an unsupervised disentanglement score of $0.78 \\pm 0.02$ for layers 10-30 compared to $0.39 \\pm 0.20$ for the first 9 layers. Interestingly, these findings align with a recent paper (Zhao et al., 2025), released earlier this month, that finds that harmfulness and refusal directions can be extracted from LLM representations for Llama2-Chat-7B around layer 10. \n\nIt would be interesting for future exploration to study whether there are indeed interesting concepts that can be extracted from intermediate layers that are abstracted away or lost in later layers. Our method, hopefully, provides a useful tool for such discoveries and insights. \n\n---\n\n#### Violations of the 1-sparse assumption\nThis is a great question. Even though we restrict our discussion to 1-sparsity, we expect that our results can be extended to 2-sparse condition as well (or generally speaking, a p-sparse condition) following the proof strategy as Ahuja et al. (2022a). That is, if the causal differentiating concepts are p-sparse and non-overlapping, we can expect identifiability up to permutations and block-diagonal transforms, rather than our stronger results on identifiability up to permutations and scaling. Further, if causal differentiating concepts are p-sparse and overlapping, we expect that the latents at the intersection of these blocks to be identified up to permutations and scaling. \n\nTo validate this second setting in our case, we construct synthetic data with 2-sparse causal differentiating concepts for latent dimension d=3. Specifically, imagine three causal variables: occupation, gender, and experience that determine the income prediction behavior of the model, such that the $p(y|x)$ can be clustered into four classes: {(Doctors/Lawyers, Male, More experience), (Doctors/Lawyers, Female, Less experience); (Teachers/Nurses, Female, More experience), and (Teachers/Nurses, Male, Less experience). We see that in this data, groups $C_k$ and $C_l$ have 2 causal differentiating concepts. For example, for behavior classes (Doctors/Lawyers, Male, More experience) and (Teachers/Nurses, Female, More experience), the causal differentiating concepts are occupation and gender. \n\nWe observe that our method can still disentangle the three causal factors. The average disentanglement measures are comparable to our 1-sparse setting—DCI-D ($0.95 \\pm 0.02$), DCI-C ($0.98 \\pm 0.01$), and DCI-I ($1.0 \\pm 0.0$). The reported numbers are across 5 runs with different initializations. Note that we get this disentanglement because the 2-sparse causal differentiating concepts are overlapping for different pairs of groups. **This indicates that our method can be extended beyond the 1-sparse setting following the intuition from Ahuja et al. (2022a).** However, we leave more detailed explorations of these cases to future work. \n\nCan you please clarify what you mean by “distributed causal shifts”? Does this mean that for any two behavioral classes, all (or multiple) latent concepts are causally differentiating? In this case, we would expect that our method would not yield disentangled representations since some form of inductive biases might be necessary for disentanglement. Please let us know if this does not address your question. \n\n---\n\n#### Choice of latent dimensionality\nThis is an interesting question. We conducted experiments to test variation in the learned number of latents in synthetic experiment settings and to assess factors that can help in choosing the latent dimension in practice. \n\nTo this end, we consider two cases: (1) when the dimensionality of learned latents is **higher** than that of true mediating concepts, and (2) when the dimensionality of learned latents is **lower** than that of true mediating concepts. \n\nIn experiment 1, we consider synthetic experiments with the number of true mediating concepts set to 2 and the number of learned latent concepts set to 3 and 4. We find that if we relax the span constraints (line 147), that is, the learned perturbations do not need to span all latent concepts, the learned perturbations correspond to the true causal differentiating concepts. That is, the model disregards any extra latent(s) in learning the causal differentiating concepts, and the extra latents are solely some function of one of the other latent variables. Thus, the model achieves a high disentanglement score, but a lower completeness score (since more than one latent captures a true causal factor): DCI-D $0.95 \\pm 0.03$ and DCI-C $0.81 \\pm 0.14$ for $n=3$, and DCI-D $0.94 \\pm 0.04$ and DCI-C $0.76 \\pm 0.19$ for $n=4$.  In sum, when the model's latent dimension is misspecified to be larger than the true latent dimension, we observe a degree of robustness in the performance.\n\nWith the span constraint, the learned perturbations no longer correspond to the true causal differentiating concepts. This leads to degradation in disentanglement scores, alongside lower completeness scores: DCI-D $0.64 \\pm 0.11$ and DCI-C $0.81 \\pm 0.14$ for $n=3$ and DCI-D $0.89 \\pm 0.08$ and DCI-C $0.57 \\pm 0.13$ for $n=4$. Interestingly, the disentanglement scores are still higher than the autoencoding (DCI-D $0.19 \\pm 0.12$, DCI-C $0.24 \\pm 0.15$) and autoencoding + prediction (DCI-D $0.08 \\pm 0.07$, DCI-C $0.09 \\pm 0.08$) baselines. **But more importantly, overspecifying latent dimensionality consistently leads to a drop in completeness scores, which can be used as a signal for reducing the latent dimensionality as a hyperparameter.** \n\nFor experiment 2, we consider synthetic experiments with the number of true mediating concepts set to 3 and the number of learned latent concepts set to 2. We find that underspecifying the latent dimensionality results in a significant drop in the disentanglement scores ($0.51 \\pm 0.20$). This is to be expected since the learned latents cannot separate out the mediating concepts since there are not enough latents. **In this case, we conclude that underspecifying latent dimensionality consistently leads to a lower disentanglement score, which can be used as a signal for increasing the latent dimensionality as a hyperparameter.**\n\nWe restrict ourselves to synthetic experiments for sensitivity analysis because in semi-synthetic and non-synthetic experiments, a parallel comparison is not feasible since the dimensionality of true mediating concepts is integral to the dataset and is not dictated externally. We believe that findings from synthetic experiments would naturally extend to general cases. Even though non-synthetic settings lack ground-truth mediating concepts, not allowing direct evaluation of disentanglement and completeness scores, the disentanglement measures can be measured across multiple model runs as a signal for guiding the latent dimensionality, following Duan et al. (2021). \n\n---\n\n#### References  \n[1] Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. NeurIPS 2022.  \n[2] Sunny Duan, Loic Matthey, Andre Saraiva, Nick Watters, Chris Burgess, Alexander Lerchner, Irina Higgins. Unsupervised Model Selection for Variational Disentangled Representation Learning. ICLR 2020.  \n[3] Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, Victor Veitch. On the Origins of Linear Representations in Large Language Models. ICML 2024.  \n[4] Kiho Park, Yo Joong Choe, Victor Veitch. The linear representation hypothesis and the geometry of large language models. ICML 2024.  \n[5] Geoffrey Roeder, Luke Metz, Diederik P. Kingma. On Linear Identifiability of Learned Representations. ICML 2021.  \n[6] Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau, Weiyan Shi. LLMs Encode Harmfulness and Refusal Separately. 2025."}]

Author rebuttal:

Meta-review: This paper is interesting because it uses causal representation learning to interpret the LLM's behavior. The problem is well-defined, and all details are demonstrated clearly in mathematical format. All reviewers agreed that this paper had a solid theoretical contribution, but also found some issues regarding the assumptions used in the theorem. During the rebuttal, the authors addressed these concerns. It has to be admitted that there will be a gap between theoretical assumptions and real cases. However, the assumptions used in this paper are general in the literature, and the authors also conducted the experiments to further verify their results. Thus, it is worth presenting this paper this time. 

All the comments during the rebuttal are excellent, and the authors should merge them into the next version.
Decision: Accept (spotlight)