From Causal to Concept-Based Representation Learning
Authors: Goutham Rajendran
Simon Buchholz
Bryon Aragam
Bernhard Schölkopf
Pradeep Kumar Ravikumar
Venue: NeurIPS 2024 poster (2024)
Keywords: concept learning
causal representation learning
interpretable representation learning
Primary area: causal_inference
TL;DR: We formally study how to extract concepts from data, by utilizing ideas from the causal representation learning and interpretability literatures.

Abstract: To build intelligent machine learning systems, modern representation learning attempts to recover latent generative factors from data, such as in causal representation learning. A key question in this growing field is to provide rigorous conditions under which latent factors can be identified and thus, potentially learned. Motivated by extensive empirical literature on linear representations and concept learning, we propose to relax causal notions with a geometric notion of concepts. We formally define a notion of concepts and show rigorously that they can be provably recovered from diverse data. Instead of imposing assumptions on the "true" generative latent space, we assume that concepts can be represented linearly in this latent space. The tradeoff is that instead of identifying the "true" generative factors, we identify a subset of desired human-interpretable concepts that are relevant for a given application. Experiments on synthetic data, multimodal CLIP models and large language models supplement our results and show the utility of our approach. In this way, we provide a foundation for moving from causal representations to interpretable, concept-based representations by bringing together ideas from these two neighboring disciplines.

From Causal to Concept-Based Representation Learning
Goutham Rajendran
Simon Buchholz
Bryon Aragam
Bernhard Schölkopf
Pradeep Ravikumar
Machine Learning Dept., Carnegie Mellon University, Pittsburgh, USA
Max Planck Institute for Intelligent Systems, Tübingen, Germany University of Chicago, Chicago, USA ELLIS Institute, Tübingen, Germany
Abstract
To build intelligent machine learning systems, modern representation learning attempts to recover latent generative factors from data, such as in causal representa- tion learning. A key question in this growing field is to provide rigorous conditions under which latent factors be identified and thus, potentially learned. Motivated by extensive empirical literature on linear representations and concept learning, we propose relax causal notions with a geometric notion of concepts. We formally define a notion of concepts and show rigorously that they can be provably recov- ered from diverse data. Instead of imposing assumptions latent space, we assume that concepts can be represented linearly in this latent space, The tradeoff is that instead of identifying the "true" generative factors, we identify a subset of desired human-interpretable concepts that are relevant for a given application. Experiments synthetic data, multimodal CLIP models and large language models supplement our results and show the utility of our approach. In this way, we provide foundation for moving from causal representations interpretable, concept-based based representations by bringing together ideas from these two neighboring disciplines.
the "true" generative
Introduction
key goal of modern machine learning is to learn representations of complex data that are human- interpretable and can be controlled. Although existing models are known extract features that useful and intuitive to humans (e.g. color, shape, size), the sensitivity of these models in capturing these features is notorious
[10 71]: The enormous capacity of contemporary models means that it is possible to reconstruct the input data with meaningless representations (e.g. posterior collapse
39 24 ). Accordingly, understanding how and when useful features can be captured, along with providing assurances on the quality of learned representations, is an ongoing endeavor [27]
A natural approach to this problem is to model the input data X (XI,.... Xx_) X fZ)).
where f is a nonlinear transformation that maps structured underlying latent generative factors 7 (7,,.. ... Za.) X, and then to attempt recover the model parameters Z, f from X. This is an appealing approach since implies no restrictions on the data X, and has the interpretation of recovering "ground truth" factors that generated the data. It is well-known that without additional
Equal Contribution
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

concepts of interest to assumptions, this is impossible 46 7T fact which has led to long line of work on nonlinear ICA [2 45] and unsupervised disentanglement 88 62] One approach to resolve this limitation is to assume that Z has an intrinsic causal interpretation. This is known as Causal Representation Learning (CRL) T02 IOT,, which endeavors to identify useful representations through the lens of causality. Structurally, the latent factors Z are assumed to have causal relationships among them, which enables us to reason about effects of interventions and conditioning on these latent factors. CRL studies this setting via an intricate interplay of ideas from causality, latent variable modeling and deep learning, with the main goal being to reconstruct the mixing function f and Z, the true generative factors of data, by leveraging causal assumptions. Recent years have witnessed a surge of rigorous results on provably learning causal representations under different assumptions [53 B3 70 60 78 142 36 [23 49 [15]. For example, long as we have access to interventions on each latent variable Z (a total of at least dz interventions), under weak assumptions on Z and/or /, the causal model over Z well as the model parameters (Z, f) can be uniquely identified [111 [4]
While causal features are intrinsically desirable in many applications, the assumption that we can feasibly perform (dzz) interventions merits relaxing: Indeed, in complex models, the number of true generative factors d₂ = dim ZZ) might be intractably large (e.g. consider all of the latent factors that could be used to describe natural images, video, or text). At the same time, there are yet many other applications where the strict notion of causality may not be needed, and moreover it may not be necessary to learn the full causal model over every causal factor. Is there middle ground where we can simultancously identify a smaller set of interpretable latent representations, without the need for huge number of interventions?
In this paper, we study this problem in detail and provide an alternative setting under which latent representations can be provably recovered. Instead of recovering (potentially huge) number of causal latents with interventions, we settle for recovering a smaller number of interpretable concepts without strict interventions. The basic idea is to recover projections AZ of the generative factors Z that correspond to meaningful, human-interpretable concepts through conditioning instead of intervention. The idea to model concepts as linear projections of the generative factors is derived
from a growing body of literature (e.g. [90 55 130 77 [2 30
118
81 38 75
[03] see
Section for even more references) showing that the embeddings learned by modern, high tegh-performant foundation models are not inherently interpretable, and instead capture interpretable concepts as linear projections of the (apriori) unintelligible embeddings. While this approach sacrifices causal semantics, it makes up for this with two crucial advantages: 1) Instead of strict interventions in the latent space, it suffices condition on the concepts, and 2) When there be learned, only n + 2 < de such concept conditionals are needed.
We validate and utilize our theoretical ideas via experiments. insights on synthetic data, where we use a contrastive algorithm to learn such representations for a given collection of concepts. Moving ahead to real-world data, we probe our theory embeddings learned by multimodal CLIP models [92]. The training scheme for CLIP aligns with our theoretical setting and therefore, it's 's reasonable to ask whether they satisfy that the concepts in the 3d-Shapes dataset approximately lie in hyperplanes, further supporting our theoretical results. Lastly, we show effective application of our framework to alignment of large language models (LLMs) where we extend the alignment technique of (66) to make LLMs more truthful.
First, we validate these theoretical
observations. Indeed, we show
Contributions
In summary, our contributions are:
1. We formalize the notion of distributions induced by abstract concepts in complex domains such as images or text (see Secion 2 for an overview and Section 4.2 for formal defini- tions). Our definition of concept conditional distributions allows both continuous and fuzzy concepts.
2. We prove near-optimal identifiability results for learning a collection of concepts from diverse set of environments in Theorem [ Thus our work can be interpreted new direction for identifiable representation learning in order to study when interpretable concepts can be recovered from data.
3. We then verify our guarantees via contrastive learning algorithm synthetic data. In addition in Section we support our geometric definition of concepts and our identifiability

---

result by analysing image embeddings of CmoP- and we utilize our ideas to improve alignment of LLMs make them more truthful.
2 Overview
In this section, we describe approach and put it in context of prior developments.
Defining concepts geometrically Our starting point is a geometric no- tion that concepts live in linear directions in neural representation space, known as linearity of representations (see extensive references in Section To make this precise assume that for observed data X that has an underlying representation Z with = f() where the latent variables Z follow an arbitrary distribu- tion and f is a (potentially complicated) nonlinear un- derlying mixing map. We do not assume that f and Z correspond to ground truth model or that the latent vari- ables Z themselves are related to causal model are interpretable and instead only assume linearity of repre- sentations (well supported by prior works). In agreement with this hypothesis we define concepts as affine subspaces AZ of the latent space of Zs, i.e., a concept C we assign an affine hyperplane Hc = ZZ E Rd AZ = b} in the embedding space and we say that X = f(Z) satis- fies concept C' if Z E Hc. In this setting the usual goal of CRL. to reconstruct, f and Z seems to unneccesary for many applications. Instead we focus on the more mod- est goal of identifying only a (small) set of concepts we care about, i.e., we want to be able to decide whether datapoint X satisfies a concept C. Our main result shows that is possible to identify n concepts given access to n + 2 concept conditional distributions. We now compare natural assumptions on type of data for causal representation learning and the setting considered here.
The space (X)
= Do
b
Figure 1 Concepts live in affine sub- spaces. The two subspaces in the figure correspond to the same concept but of different valuations.
From interventions to conditioning It is worth contrasting here the difference between viewing a concept as a generic latent generative factor Z; that non-linearly mixes together with other latent factors yield the inputs X, versus the geometric notion above, specifying linear subspace. In the former, the natural way to provide supervision, i.e. define concept distributions, is to simply intervene on specific factor Z; and set it to particular value (see Section 3 for references).nn In the latter however, it is most natural to condition on the concept, i.e., Z E H.
This shift is aligned with the growing interest to relax the notion of interventions, and consequently dilute the notion of causality [5 100 although is still open how to properly achieve this. Two key drivers of this trend are as follows. The first is that the number of additional datasets required is d₂ [46] [ 14), which is infeasible in many settings 2 The second is that the various assumptions that go into these works are often difficult to achieve, such requiring perfect interventions [T]])))
Compared to interventional data, conditional data is often easier to acquire, obtained by conditioning on particular values of the latent factors (see also Appendix B.2). B.2).
Concept conditional distributions We now formalize conditioning on a concept. The obvious approach to define concept conditional distributions is to simply condition Z E Hc. so Pc(Z) Z Z E Hc) where is base distribution of Z on Rdd, However, this suffers from the drawbacks that it is mathematically subtle to condition on sets of measure 0 and this does not account for inherent noise in the learned representations. Therefore we relax this strict conditioning by drawing inspiration from how data collected in practice: We sample X from the base distribution and then keep it satisfies concept C. This leads us to define PC (Z) x p( ((Z)g(ZIC) where is defined to be the probability that Z is perceived to be in H by the data collector and can be chosen to incorporate noise in data gathering scheme. Therefore, this can also be viewed from Bayesian information gathering viewpoint, as well as a stochastic filter standpoint. This is the notion we study in this work
Exceettions are 57 42 which use clever inductive biases to limit the number of environments necded.
3

---

(Definition 3 and we develop theoretical techniques to guarantee identifiability in this formulation. Depending on the specific setting other types of conditional distributions might be utilized to describe the available data and we discuss some options in Appendix
Connection to XAI One important class of applications is human-in-thee -n-o-e--oooo-op (HIL) settings, where the hope is to recover human interpretable concepts. Apriori, there is no reason why the high-dimensional underlying latent Zs should be interpretable, particularly for large dz. To make the situation worse, there may be other potentially composite concepts that humans can conjure up; indeed, there no limit to the number of concepts that humans can conjure, and these almost certainly not going to correspond to the true Zs. Because of this mismatch between the "true" Zs and interpretable concepts, even if we could recover the Zs, they may not be interpretable to humans and therefore less useful for downstream HIL purposes. The second caveat, and which follows from the first as a soft corollary, is that interventions on these will be much harder to obtain, particularly perfect interventions which require to isolate the generative mechanisms of the latent factor with the other factors. Furthermore, some concepts are not manipulable, e.g. psychological traits, biological mechanisms, so obtaining interventional data is not possible in these applications. Accordingly, there has been considerable recent work, especially in the explainable AI (XAI) community on extracting human-interpretablee concepts from latent generative factors, or more generally from complex foundation model representations [55 103 18 137 To do so in an identifiable and automated way with as little supervision as possible is a significant open problem.
3 Related work
Causal representation learning and concept discovery Causal representation learning (CRL)
102 [01 aims to learn generative factors of nigh-dimensional data. This exciting field has seen
significant progress in the last few years [33 106
63 127 A fundamental perspective in this field is to ensure that the model parameters we attempl to recover are identifiable [53 25 [29], We will elaborate more the connection of our framework to CRL
60
78
57
114
36
in Appendix B Concept discovery is an important sub-field of machine learning which extracts human-intepretable concepts from pre-trained models. We do not attempt to list the numerous works in this direction, see e.g.. [003 18 [35 74
89 [14]. However, theoretical progress in this direction is relatively limited. The work [63] studies when concepts can be identified provided the non-linear model is known in advance, whereas we show concept identifiability for unknown non-linearity, while simultaneously allowing entangled concepts. Prior works have also attempted to 61), however their definitions seem specific to the model and domain under consideration, e.g., [85| 52 focus binary concepts via large language
99
58 104
formalize the notion of concepts [30 85 103
model representations of counterfactual word pairs, whereas our general concept definitions are applicable to all domains.
Linearity of representations Sometimes referred to as the linear representation hypothesis, it is commonly believed that well-trained foundation models in multiple domains learn lin- car representations of human-interpretable concepts, with experimental evidence going back at least decade 77
113 55. This has been experimentally observed in computer vision models
90
94
8
31
55
19
30
120, language models [77] 87 5
117 30 large language models [2
79
66
85
38
52). and other intelligent systems 755 [03]. Various works have also
[05]. We take a different angle: Given that this phenomenon has been observed for certain concepts of interest, how does this enable recovery of the concepts themselves? Consequently, our model assumptions are well-founded and our theory applies to multiple domains of wide interest.
attempted to justify why this happens [64 5 35
3 32
Sctup
In this section, we provide formal definition of concepts, which are high-level abstractions present in data. This allows us to develop a theoretical framework for associated data distributions and identifiability theory. For the sake of intuition, we can think of the data images of different objects and the color of the object as a concept.

---

4.1 Generative model
We assume that the observed data X lies in a space X' C Fd of dimension dg and has an underlying representation X = f(Z) for latent variables Z that lie in latent concept space Rd of dimension dg. In contrast to most prior works we do not necessarily assume that 2 represents the true underlying mechanism that generated the data. Instead we simply assume that the latent representation has the geometric property that it maps certain regions of the observation space to linear subspaces of the latent space (motivated by previous work; see Section 3). Our first assumption is standard:
Assumption (Mixing function). The non non-linear f is injective and differentiable.
We make no additional assumptions on f:: The map from Z - X can be arbitrarily non-linear.
We now define concepts living in the latent space Rd= Before presenting the general definition of multidimensional concepts, we outline the basic ideas in the simplified setting of one-dimensional concept. Consider the color "red" concept. Different images have different levels of "redness" in them, so this concept is measured on a continuous scale, represented by a valuation b E IR. An (atomic) concept is then represented by vector a E Rd such that (a, Z) (a, f (X)) encodes the "value" of the concept in X, as measured in the latent space. More precisely. for given valuation b E IR, the set of all observations X that satisfy this concept given by {X - [( Z) b).
For instance, for an object in an image X, if E Id is the concept of red color, b RR could indicate the intensity: then all datapoints X satisfying this concept, i.e., all images with an object that has color red with intensity b, can be characterized as f() where Z satisfies (a, 2) b. For a 3D visualization, see Fig. [ We make this intuition formal below.
Definition (Concepts), A concept C is a linear transformation A Rd= + Rdc. The dimension of the concept will be denoted by dim( C") = dc. A valuation is vector b Rdc and we say that datapoint X satisfies the concept C with valuation b if AZ = b where = f-
In this work, we are interested in learning collection of m concepts Ct,..., Ctth from observed data. By left multiplying by the pseudo-inverse A we can equivalently assume A is a projector matrix. However, the current definition is more suitable for embeddings of real models.
When we talk of learning concepts C, we are in particular interested in learning the evaluation map Af- (z). This is more modest objective than learning the entire map f which is the usual goal in, say, CRL. While the latter typically requires stringent assumptions, particular ((d:) environments are necessary. our weaker identifiability results only need O(dc) < O(dz) environments. To simplify our analysis, we make use of the following definition:
Definition 2 (Atoms). An atom (short for alomic concept) is any concept C' with dim(C) =
The idea is that we can view each concept as being composed of atomic concepts in the following sense: Atomic concepts are fundamental concepts that live in a space of co-dimension 1 in latent space, and thus are equivalently defined by vectors a E Rdd For example, concepts such red color, size of object, etc., may be atomic concepts. Any generic concept is then composed of a collection of atomic concepts, e.g., the concept C of all small dark red objects will correspond to dim (C) = with row corresponding to the atomic concept of red color with large valuation (dark red objects) and row corresponding to the atomic concept of object size with low valuation (small objects).
4.2 Data distributions
We now define the distributions of datasets over concepts. We will predominantly work with distributions of Z over IRd as the resulting distribution of X fZ) over Id can be obtained via simple change of variables.
To build intuition, consider the case where we first collect base dataset with some underlying distribution and then collect concept datasets via filtering. For instance, we could first collect a set of images of all objects and then, collect a dataset of dark red colored objects, we filter them to only keep images of dark red colored objects. We call the former the base distribution and the latter the concept conditional distribution corresponding to our concept.
Fix a nonlinearity f. We assume that the base data distribution is the distribution of X = f(Z) with ~ p. where p is the underlying distribution on Rdd, In what follows, we will abuse notation and use p for both the distribution and the corresponding probability density which we assume exists. In
5

---

contrast to existing work that puts assumptions on (such as linearity, Gaussianity, etc.), we make no assumptions on p since we wish to model real-life datasets which could be very arbitrary.
We now define the concept conditional distribution, which is distribution over X that is induced by noisy observations of particular concept particular valuation. Formally, assume want to condition on some atomic concept E IRd with valuation 6. It is reasonable to assume that this conditioning is noisy operation. For instance, humans are great distilling concepts from noisy images, e.g., they recognize cars in a misty environment. We formalize this by assuming that data collection is based on noisy estimate b = (a, 2} + where is independent of and its density is symmetric distribution with density q(e). Then we consider the distribution
pc(=) - p(=
x p(b b/=)p(=)
g(b (a, p))
where we used Bayes theorem in the last step. This definition directly extends to higher dimensional concepts which are concisely defined as follows.
Definition 3 (Concept conditional distribution). For a concept C with associated linear map A and an arbitrary valuation b E Rdim(()) we define the concept conditional distribution to be the set of observations X respecting this concept, which is defined as the distribution of X f(Z) where Z ~ PC with
dim((C)
(z x p())
II
- -()x)=b)a.)
This is by no means the only possible definition, and we present feasible alternate definitions in Appendix C We remark that our formulation is related to the iVAE setting [53 and the auxiliary variable setting for identifiable ICA in Hyvarinen et al. [48) and we discuss the relation later. The majority of recent identifiability results relied on interventional data while only consider conditional information here.
4.3
Concept learning and identifiability
We are ready to define our main problem of interest.
Problem We are given an observational dataset X = f(Z)) corresponding to the latent base distribution along with datasets X2 ,,. from m environments corresponding to concept conditional datasets for different concepts C¹ ...., Ctm and corresponding valuations b' b'..... btd over the same latent space Rd. with the same mixing f. Under what conditions (and up to which symmetries) can we learn the concepts C¹ C'.....C* Cm which includes the linear maps A .....,.. A and the concept valuations A
Toward this end, a fundamental question is whether this problem is even possible, i.e., whether it is well-defined. This is known as the question of identifiability [53 25 229 57 43
Therefore, make the following definition. Informally, for the setting above, we say that the concepts ("),A')..... (CTm A with associated nonlinearity f are identifiable (and thus Icarnable) if for any other collection of different parameters that fit the data, they are linearly related to the true parameters.
Definition (Identifiability) Given datasets X X X!...., X corresponding to the observa- tional distribution and m concepts C² CI....,CC C with underlying latent base distribution p on _dd nonlinearity f. linear maps A A?,...,A" and valuations 6º "....,bm, we say the concepts are identifible if the following holds: Consider any different collection of parameters J, d...P. concepts (C", ,A),.... (Cmm, A and valuations b¹ bl....,b" h that also generate the same observa- tions x²,X1 "P,Xx,..,,X" ,'..,,",, Then there exists shift w E IRds permutation matrices Pe and invertible diagonal matrices N such that for all e and x,
Ā Ne ww),
i.e. we can evaluate the concept evaluations on the data up to linear reparametrizations. Moreover, there exists linear map T =dd + Rdd such that the concepts and their evaluations satisfy
A = P=PAAA D = p A)w).
(4)

---

Identifiability implies we can identify the nonlinear map w concepts of interest, and therefore we can recover the concepts of interest from our data. That is, certain concepts are identifiable, then we will be able to learn these concept representations up to linearity, even if they can be highly nonlinear functions of our data. Such concept discovery is useful because they can then be used for further downstream tasks such as controllable generative modeling.
within the span of the subspace of the
We emphasize that in contrast to previous work we are not aiming identify f completely and indeed, no stronger identifiability results on f can be expected. First, we cannot hope to resolve the linear transformation ambiguity because the latent space not directly observed. In other words, a concept evaluation can be defined either as (a, Z) or as (Ta, T Z) for an invertible linear map T. For the purposes of downstream tasks, however, this is fine since the learned concepts will still be the same. Second, we cannot expect to recover f- outside the span of the concepts because we do not manipulate the linear spaces outside the span therefore we do not learn this information from our observed data so this is also tight. The permutation matrix captures the fact that the ordering of the concepts does not matter. Therefore, this definition captures the most general identifiability guarantee that we can hope for in our setting and furthermore, this suffices for downstream tasks such as controllable data generation.
Because we will only be interested in recovering the set of concepts up to linear transformations, without loss of generality, we will fix the base collection of atomic concepts. That is, we assume that each concept C" (where 1 S e S m indexes the environment) corresponds to linear map A whose rows are subset of C, where C = {a...... an is a set of atomic concepts that we wish to learn. Moreover, we assume that they are linearly independent, since we want them to encode distinct concepts. This is formalized as follows.
Assumption 2. There exists a set of atomic concepts C = {oi.....,ay of linearly independent vectors such that for each concept C under consideration the rows of the concept matrix A are contained in C, i.e., (A')'e; E C. We denote the indices of the subset of C that appear as rows of A by S and we assume that all concepts in C appear in some environment e (where an environment corresponds a concept conditional distribution), i.e., Ue [m].
Remark 1. Definition 4 limpiies that the atoms can be identified in the sense that there is permutation n and > # O such that for T as in Definition 4 and some X44
5I() 0, TT
(5)
(7x(() (x)) = > ((a1)) + ((),
(6)
i.e., we can evaluate the valuations of the atomic concepts up linear reparametrization.
5 Main Result
In this section, we present our main result on identifying concepts from data. The punchline is that when we have rich datasets, i.e., sufficiently rich concept conditional datasets, then we can recover the concepts. Crucially, we only require a number of datasets that depends only on the number of atoms n we wish to learn (in fact, O(n) datasets), and not on the underlying latent dimension d, of the true generative process. This is a significant departure from many existing works, since the true underlying generative process could have d, 1000, say, whereas we may be interested to learn only = concepts, say. In this case, approaches based on CRL necessitate at least ~ 1000 interventional datasets, whereas we show that n + only want to learn the atomic concepts. We will explain the connection to CRL in Appendix B Let us now discuss our main assumptions.
conditional datasets are enough if we
Assumption 3. The noise distribution is Gaussian, i.e. g NO, ( some o² > 0.
We choose Gaussian noise since is a conventional modeling choice. However, it would be feasible to consider other noise families and we expect similar results to hold (albeit with modified proof techniques). We now relate the concepts C' to the atoms. Recall that we defined the index sets S {i E [n] a E is row of ) of atomic concepts in environment e.
We define the environment-concep matrix M g indexed by environments and atoms by
f 1 if S
Ma
0 otherwise.
7

---

Similarly, we consider the euvironment--aluutio matrix B mR given by
i S and row k of A is ag.
Bei
(8)
0
otherwise.
Our first assumption ensures that the concept conditional distributions sufficiently diverse. Assumption 4 (Environment diversity 1). The environment-come concept matrix M E m has rank and there a vector E R such that u² M and all entries of B are non-zero where B denotes the invironment-valuatio
matrix.
We remark that this assumption can only hold for m 2 n n 1 and indeed is satisfied under mild assumptions the environments if condition on ensures that the concept valuations bg are not equal for all environments e which would prevent identifiability.
1, the following lemma shows. Note that the
Lemma 1. Assumption 4 satisfied aimost-surefy if there are n n+1 concept conditional distributions such that every n rows of the inirromment-concey matrix are linearly independent and the be are drawn independently according to a continuous distribution.
We also assume one additional diversity condition. To motivate this, observe if concepts always occur together, it's information-theoreticaal impossible to distinguish them, e.g., if an agent only sees red large objects (i.e. red objects are large and all large objects are red), it will be unable to disambiguate the "red" concept from the "large" concept. Therefore, we make the following assumption.
Assumption (Environment diversity II). For every pair of atoms a, and aj with + there is an environment e such that E S and j S
We remark that these are the only assumptions about the sets S In particular, we do not need to know the sets S In the proof, we will extract these sets based on the signatures they leave on the datasets. We can now state our main result.
Theorem 1. Suppose we are given m context conditional datasets XI....
...... X and the observational dataset X such that Assumptions 5 hold. Then the concepts are identifibble as in Definition 4
Remark Assumption can only satisfied for m 2 n 1, i.e., the result requires at least n + environments. On the other hand, Lemma assures that + environments are typically sufficiens. We expect that the result could be slightly improved by showing identifiability for n + 1 environments under suitable assumptions. However, this would probably require more advanced techniques from algebraic statistics (28) compared to the techniques we employ here.
As mentioned before, setting somewhat resembles the iVVE setting in Khemakhem et al. (53) and therefore, their proof techniques can also be applied, with several modifications, to derive identifiability results in setting (however our formulation and application are very different). However, this approach will require more environments because their main assumption is that the matrix A (M, B) Rmx2n has rank 2n so that 2n + environments are necessary. Moreover, this rank condition is much stronger than Assumption 4 For completeness and as a warm-up we prove this result in Appendix A The full proof of Theorem is fairly involved and is deferred to Appendix
6 Experiments
In this section, we present experiments to validate and utilize our framework. We first verify our results on synthetic data, via a contrastive learning algorithm for concept learning. Then, we focus on experiments involving real-world settings, in particular on image data using multimodal CLIP models and text data using large language models (LLMs).
End-to-end Contrastive learning algorithm and Synthetic experiments We validate our frame- work on synthetic data as follows. We sample the base distribution from a Gaussian Mixture model and experiment with both linear and nonlinear mixing functions (details deferred to Appendix G The number of concepts is intentionally chosen to be less than the ground truth dimension d, and

---

the number of concept condtional distributions is m - 1 as per our theory. Inspired by [14], we use a contrastive learning algorithm to extract the concepts. Here we sketch the key ideas of the algorithm and defer details to Appendix
The core idea for the algorithm is follows. For each concept conditional distribution X we train a neural network to distinguish concept samples z ~ X from base samples I ~ X Under our assumptions the log-odds of these two distributions have the following simple parametric form in terms of the environment-concepl matrix and the environment-valuatio matrix
In(p - In(p(2)) = Σ
1 Merla,, 2002 + Beilar, 2²) + Ce
1=1
for some constants Ce (see Lemma for the precise statement).
Then, to learn the n atomic concepts up to linearity, we build a neural architecture for this classification problem with the final layer mimicking the log-odds expression above, which can then be trained end-to-end. Because of the careful parametrization of the last layer, this will encourage the model to learn the representations as guaranteed by our results. We can assume without loss of generality that the concept vectors we learn are the first coordinate vectors because concepts are only identifiable up to linear transformations (see Definition +.. In other words, we consider an encoder neural network h" with parameters @ and the valuation of atomic concept is h.. Therefore, for each environment we train classifiers of the form
Ie 9e ar Bi Th, = a
( +
(10)
k=1
using standard cross-entropy loss, where a", Pk, 7% are the parameter of the last layer and 0 parametrizes the decoder.
In Table 1 we report the R² and Mean Corre- lation Coefficient (MCC) metrics [53] [4 with respect to the ground truth concept valuations. In addition we provide results for larger values of de and de in ia Appendix G There are no baselines since we are in novel setting, but our metrics are comparable to and often surpass what's usually reported in such highly nonlinear settings 132 14
Table 1: Linear identifiability when number of concepts n is less than latent dimension dg with observed dimension de, averaged over seeds.
Mixing (f) (n, d2, dz)
R²T
MCC1
Linear
(2, 3, 4)
0.98 + oor
0.98 0.03
Nonlinear (2, 4)
0.94 on6 0.96 0.04
We remark that variations of the contrastive method can be designed for harder synthetic settings and different problems related to concept discovery. However, we will move onto real-life data experiments next.
Linear
(3, 4, 6)
0.99 0or 0.86 0.08
Nonlinear
(3, 4, 6)
0.97 0.92
Linear
(4, 8, 10)
0.97 0.87
Nonlinear
(4, 8, 10)
0.94 0.0 0.87 = 0.06
Probing the theory on multimodal CLIP models A real world example that approximately matches the setting considered in this paper is the training of the multimodal CLIP models [92]. They are trained by aligning the embeddings of images and their captions. We can view the caption as an indicator of the concepts present in the image. Thus the data provides access to several concept conditional distributions such as the collection of all images having the label dog', but also to more complex distributions consisting of more than one atomic concept such as images labeled 'A red flower'. We embed images from the 3d-Shapes Dataset [16] with known factors of variation into the latent space of two different pretrained CLIP models. Using logistic regression we learn atomic concepts for each of the factors of variations (see Appendix [D for details) and then evaluate the concept valuations of the learned atomic concept on held out images. We show the results for the shape attribute in Figure 2 (further results are in Appendix D.2). The results show that there indeed linear subspaces of the embeddings space that represent certain concepts. Moreover, the learned valuations for different models are approximately linearly related as predicted by Theorem We emphasize that, while these observations highlight the relevance of the theory developed in this paper, they do not explain the behavior of CLIP models, as their training objective does not directly align with our theoretical framework.
Alignment of L.MSs of LLMs and improve alignment techniques. In particular, exploit our ideas to improve the
Finally, we show an application of our framework to interpret representations

---

Vit-B/32
RN101
10
6
5
4-
(ashapo- 2)
2
(ashaper
0
5
-2
-10-
-4
-6 Cube Cylinder Ball Ellipsoid Shape
Cube Cylinder
Ellipsoid
Shape
Figure 2: Violin plot of the concept valuations (aShape: 2) for the different shapes and a vision transformer CLIP embedding (left) and residual network CLIP embedding (right). Results show concentration of the concept valuations around the concept planes indicated by the horizontal lines.
Inference- Time Intervention technique [66] to promote LLMs to be more truthful, i.e. the downstream task is to take pre- trainra LLMs and during inference, change the valuation of the truthfulness concept from false to tree, without affecting any other orthogonal concepts. Motivated by our framework, propose replace steering vectors by steering matrices for better alignment. Experiments LLaMA [199 show an improvement of the TruthfulQA dataset [68] accuracy. Additional details, including seff-contained introduction to large language models (LLMs) and the Inference-Time Intervention (ITI) technique are deferred to Appendix
Conclusion
In this work, we study the problem of extracting concepts from data, inspired by techniques from causal representation learning. This an approach to bridge identifiability theory in (causal) repre- sentation learning and the field of concept-based learning. For this, we geometrically define concepts as linear subspaces, well-supported via extensive empirical literature. With this formal definition of concepts, study under what conditions they be provably recovered from data. Our rigorous results show that this is possible under the presence of only conditional dataiiqqu far fewer distributions than the underlying latent dimension. Finally, synthetic experiments, multimodal CLIP experiments and LLM alignment experiments verify and showcase the utility of our ideas.
Acknowledgments We acknowledge the support of AFRL and DARPA via FA8750-23-2 23-2- 1015, ONR via N00014-23-1-2368, NSF via IIS-1909816, 1955555553, IIS-1956330, and NIH R01GM140467. We also acknowledge the support of the Tübingen Al Center, the Deutsche Forschungsgemminsccaatt (DFG, German Research Foundation) under Germany's Excellence Strategy EXC number 2064/1 Project number 390727645, and the Robert H. Topel Faculty Research Fund at the University of Chicago Booth School of Business.
References
[1] K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learn- ing. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
[2] K. Ahuja, A. Mansouri, and Y. Wang. Multi-domain causal representation learning via weak distributional invariances. arXiv preprint arXiv:23 2023.
[3] C. Allen and T. Hospedales. Analogies explained: Towards understanding word embeddings.
In International Conference on Machine Learning, pages 223-231. PMLR, 2019.
[4] M. Arjovsky, Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: arXiv:1907.0289; 2019.
10

---

[5] S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski. A latent variable model approach to pmi- based word embeddings. Transactions of the Association for Computational Linguistics, 4: 385-399, 2016.
[6] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXivv2204.0588
2022.
[7] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022.
[8] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 354116549,, 2017.
[9] Bengio, A. Courville, and Vincent. Representation learning: review and new per- spectives. IEEE transactions pattern analysis and machine intelligence, 35(8): 1798-1888,,, 2013,
[10] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255. 2021.
[11] J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319-38311, 2022.
[12] S. Buchholz and Schölkopf. Robustness of nonlinear representation learning. In Salakhut- dinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Pro- ceedings of Machine Learning Research, pages 4785-482) 4821. PMLR, 21-27 Jul 2024. URL https: pttps://proceedingg .mlr..gr...........m............ http://nmmmm....eee////5555555 pptpess//25/uuuhhooo222222
[13] S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. In A. H. Oh, A. Agarwal. D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL |tttps://ppeneeiiiii net/forum?id--ppaP-- PYBbK
[14] S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing, arXiv preprint arXiv:2306. 02235, 2023.
[15] P. Bühhlann, Invariance, causality and robustness. Statistical Science, 35(3):404 426, 2020.
[16] C. Burgess and H. Kim. 3d shapes dataset. http:///ghhbbbooom/ccccmmo///ahe 2018.
[17] Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint 22.iv:2222.03822 2022.
[18] D. V. Carvalho, E. M. Pereira, and J. S. Cardoso. Machine learning interpretability: A survey methods and metries. Electronics, 888):332, 2019.
[19] Z. Chen, Y. Bei, and C. Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
[20] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, Stoica, and E. Xing. Vicuna: An open-source chatbot impress- ing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsy:: wwwpansssssrrreoeee bbog/bllg/// 2023 113-30vviie
[21] P. Comon. Independent component analysis, a new concept? Signal processing, 36(3):
287-314, 1994.
11

---

[22] A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:/805.0000 2018.
[23] J. Cui, W. Huang, Y. Wang, and Y. Wang. Aggnce: Asymptotically identifiable contrastive learning. In NeurIPS Workshop, 2022.
[24] B. Dai, Z. Wang, and D. Wipf. The usual suspects? reassessing blame for vae posterior collapse. In International conference on machine learning. pages 2313-2322 PMLR, 2020.
[25] A. D' Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. The Journal of Machine Learning Research, 23(1):10237-10297 2022.
[26] N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611 1.02648, 2016.
[27] F. Doshi Velez and B. Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: 1701v01702.08608, 2017.
[28] M. Drton, B. Sturmfels, and S. Sullivant. Lectures on Algebraic Statistics, volume 39 of Oberwolfach Seminars. Springer, 2009. doi: 10.1007/978-3--643--95555
[29] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1, 2021.
[30] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wat- tenberg, and C. Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
https:/ttansfommr-ciiuuuussps putpuuis...uuu2222///o... http:p///2//oooooo///oeee.i.mmma model/index.html.
[31] J. Engel, M. Hoffman, and A. Roberts. Latent constraints: Learning to generate conditionally from unconditional generative models. arXiv preprint arXiv:1711.05772,222 2017.
[32] K. Ethayarajh. D. Duvenaud, and G. Hirst. Towards understanding linear word analogies. arXiv preprint arXiv:1810.04488 2018.
[33] F. Falck, H. Zhang, M. Willetts, G. Nicholson, C. Yau, and C. Holmes. Multi-facet clustering variational autoencoders. Advances in Neural Information Processing Syxtems, 34, 2021.
[34] D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arX1v:2209.07838. 2022.
[35] A. Gittens, D. Achlioptas, and M. W. Mahoney. Skip Skip-gram- zipf+ uniform= vector additivity. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 69-76, 2017.
[36] L. Gresele, J. Von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, a new concept? Advances in Neural Information Processing Systems, 34, 2021.
[37] S. Gupta, Jegelka, D. Lopez-Paz, and K. Ahuja. Context is environment. arXiv e-prints, pages arXiv-2309, 2023.
[38] W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
[39] J. He, D. Spokoyny, G. Neubig, and T. Berg-Kirkpatrick, Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprini 1901::190 2019.
12

---

[40] E. Hernandez, B. Z. Li, and Andreas. Measuring and manipulating knowledge representations in language models. arXiv preprint 2xXiv:3304.00740, 2023.
[41] P. Hitzler and M. Sarker. Human-centered concept explanations for neural networks. Neuro- Symbolic Artificial Intelligence: The State of the Art, 342(337):2, 2022.
[42] D. Horan, E. Richardson, and Y. Weiss. When is unsupervised disentanglement possible? Advances in Neural Information Processing Systems, 34:5150-5161, 2021.
[43] M. Huh, B. Cheung, T. Wang, and P. Isola. Position: The platonic representation hypothesis. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 20617-20642. PMLR, 21-27 Jul 2024.
[44] A. Hyvarinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances neural information processing systems, 29, 2016.
[45] A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):41-4335 2000.
[46] A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439, 1999.
[47] A. Hyvarinen, J. Karhumen, and E. Oja. Independent component analysis. Studies in informatics and control, 11(2)205-2277 2002.
[48] A. Hyvarinen, H. Sasaki, and Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[49] A. Hyvärinen, I. Khemakhem, and R. Monti. Identifiability latent-variable and structural- equation models: from linear nonlinear. arXiv preprint arXiv:2302.02672. 2023.
[50] Y. Jiang and B. Aragam. Learning latent causal graphs with unknown interventions. In Advances in Neural Information Processing Systems, 2023.
[51] Y. Jiang, B. Aragam, and V. Veitch. Uncovering meanings of embeddings via partial orthogo- nality, Advances Neural Information Processing Systems, 2023.
[52] Jiang, G. Rajendran, P. Ravikumar, B. Aragam, and V. Veitch. On the origins of linear representations in large language models. arXiv preprint, 2024.
[53] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[54] I. Khemakhem, R. Monti, D. Kingma, and A. Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33: 1776 2020.
[55] Kim, M. Wattenberg. J. Gilmer, C. Cai, J. Wexler, F. Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 8668-2677 PMLR, 2018.
[56] B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Learning latent causal graphs via mixture oracles. Advances in Neural Information Processing Systems, 34: 14:88087-18101, 2021.
[57] B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687-15701, 2022."
[58] P. W. Koh, T. Nguyen, Y. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang. Concept bottleneck models. In International conference on machine learning, pages 5338-5348, PMLR, 2020.
13

---

[59] L. Kong, S. Xie, W. Yao, Y. Zheng, G. Chen, P. Stojanov, V. Akinwande, and K. Zhang. Partial identifiability for domain adaptation. arXiv preprint arXiv:2306.06510, 2023.
[60] S. Lachapelle, Rodríguez, Y. Sharma, K. Everett, R. Priol, A. Lacoste, and S. Lacoste- Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In B. Schölkopf, C. Uhler, and K. Zhang, editors, Ist Conference on Causal Learning and Reasoning, CLeaR 2022. Sequoia Conference Center, Eureka, CA. USA, 11-13 April, 2022, volume 177 of Proceedings of Machine Learning Research, pages 428 484. PMLR, 2022. URL httpp::/procceednggss mlr http://www......eeee//////////// http://1T///accapeeee222a html
[61] S. Lachapelle, T. Deleu, D. Mahajan, I. Mitliagkas, Bengio, S. Lacoste-Julien, and Q. Bertrand. Synergies between disentanglement and sparsity: Generalization and iden- tifiability in multi-task learning. In International Conference on Machine Learning, pages 18171-18206. PMLR, 2023.
[62] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning, nature, 521(7553):436 2015.
[63] T. Leemann, M. Kirchhof, Y. Rong, E. Kasneci, and G. Kasneci. When are post-hoc conceptual explanations identifiable? Uncertainty in Artificial Intelligence, pages 1207-1218. PMLR, 2023.
[64] O. Levy and Y. Goldberg. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27, 2014.
[65] K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint arXiv:2210. /3382, 2022.
[66] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint arXiv:2306.0334, 2023.
[67] S. Li. B. Hooi, and H. Lee. Identifying through flows for recovering latent representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpss://pperreeeeee..eeee htwp://wwwiieeeeefottmmummm. Sk10UpEYvB
[68] S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint 2xXv:22009.09958, 2021.
[69] P. Lippe. S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves. Biscuit: Causal representation learning from binary interactions. arXiv preprint arXiv:2306.09645 2023.
[70] Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. v. d. Hengel, K. Zhang, and Q. Shi. Identifying weight-variant latent causal models. arXiv preprint arXiv:2208. 14153, 2022.
[71] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning of disentangled representations, In international conference on machine learning, pages 41 4124. PMLR, 2019.
[72] I. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In
5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https //openrevie nett/Forun?id--Stg6555xxx
[73] C. Lu, Y. Wu, J. M. Hernández- and B. Schölkopf. Invariant causal representation learning for -ut-of-distribution generalization. In International Conference on Learning Representations, 2021.
[74] E. Marconato, A. Passerini, and S. Teso. Interpretability is in the mind of the beholder: A causal framework for human-interpretable representation learning. Entropy, 25(12): 2023.
[75] T. McGrath, A. Kapishnikov, N. Tomašev, A. Pearce, M. Wattenberg, D. Hassabis, B. Kim, Paquet, and V. Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, (19(47))c2206225119 2022.
14

---

[76] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Advances Neural Information Processing Systems, 35:17359- 2022.
[77] T. Mikolov, W.-t Yih, and G. Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746-751, 2013.
[78] G. E. Moran, D. Sridhar, Y. Wang, and D. Blei. Identifiable deep generative models via sparse decoding. Transactions on Machine Learning Research, 2022.
[79] L. Moschella, V. Maiorca, M. Fumero, A. Norelli, F. Locatello, and E. Rodola. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:220.114330 2022.
[80] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, Saunders, et al. Webgpt: Browser-assisted question-answeriig with human feedback. arXiv preprint 21..0222...00009933 2021.
[81] N. Nanda, A. Lee, and M. Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint (rXiv:230900094/ 2023.
[82] T. Oikarinen, S. Das, L. M. Nguyen, and T.-W. Weng. Label-free concept bottleneck models. arXiv preprint 0rX1v22304.06/29, 2023.
[83]
Olah. Mechanistic interpretabilityy variables, and the importance of interpretable ciw.urtsssrsssssss.............. bases. https: // transformer- circuits. pud/ 2022/ mech- interp- essay/ index.
html]
2022.
[84] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[85] K. Park, Y. Choe, and V. Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03655 2023.
[86] J. Pearl. Causality. Cambridge university press, 2009.
[87] J. Pennington, Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[88] J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[89] E. Poeta, G. Ciravegna, E. Pastor, T. Cerquitelli, and E. Baralis. Concept-based explainable artificial intelligence: A survey. arXiv preprint (tf.:2332.//93333 2023.
[90] A. Radford, I. Metz, and S. Chintala. Unsupervised representation learning with deep
convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
[91] A. Radford, R. Jozefowicz, and I. Sutskever. Learning generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444, 2017.
[92] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763 PMLR, 2021.
[93] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct pref- erence optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.1822 2023.
[94] M. Raghu, J. Gilmer, J. Yosinski, and Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep understanding and improvement. stat, 1050:19, 2017.
15

---

[95] G. Rajendran, B. Kivva, M. Gao, and B. Aragam. Structure learning in polynomial time: Greedy algorithms, bregman information. and exponential families. Advances in Neural Information Processing Systems, 34:1 8660-18672, 2021.
[96] G. Rajendran, P. Reizinger, W. Brendel, and P. Ravikumar. An interventional perspective on identifiability in gaussian Iti systems with independent component analysis. arXiv preprint arXiv:2311.18045 2023.
[97] N. Reimers and I. Gurevych. Sentence-bers Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 11 2019. URL http::://aaxivv.rggg.
10084
[98] N. Rimsky, N. Gabrieli, J. Schulz, M. Tong. E. Hubinger, and A. M. Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.0668/, 2023.
[99] G. Roeder, L. Metz, and D. Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning, pages 9030-9039. PMLR, 2021.
[100] D. Rothenhäusler, N. Meinshausen, P. Bühlmann, and J. Peters. Anchor regression: Hetero- geneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(2):215 246, 2021.
[101] B. Schölkopf and J. Kügelgen. From statistical causal learning. In Proceedings the International Congress of Mathematicians (ICM). EMS Press, July 2022.
[102] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-63 634, 2021. arXiv:2102.1111
[103] L. Schut, N. Tomasev, T. McGrath,, D. Hassabis, U. Paquet, and B. Kim. Bridging the human-ai knowledge gap: Concept discovery and transfer alphazero. arXiv preprint arXiv:2310.16411 2023.
[104] G. Schwalbe. Concept embedding analysis: A review. arXiv preprint arXi:: 2022.
[105] Y. Seonwoo, S. Park, D. Kim, and A. Oh. Additive compositionality of word vectors. In Proceedings of the 5th Workshop on Noisy User-generated Text WW-NUT2019) pages 387-396, 2019.
[106] Shen, F. Liu, H. Dong, Q. Lian, Z. Chen, and T. Zhang. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23:1-55, 2022.
[107] K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.05567, 2021.
[108] P. Sorrenson, C. Rother, and U. Köthe. Disentanglement by nonlinear ica with general incompressile-flow networks (gin). arXiv preprint arXiv:2001.04872 2020.
[109] P. Spirtes, N. Glymour, and R. Scheines. Causation, prediction, and search. MIT press, 2000.
[110] Squires and C. Uhler. Causal structure learning: a combinatorial perspective. Foundations of Computational Mathematics, pages 1-35, 2022.
[111] C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. Linear causal disentanglement via interven- tions. In A. Krause, Brunskill, K. Cho, B. Engelhardt, S. Sabato, and Scarlett, editors, International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 02540-32560 PMLR, 2023. URL
mll. prwss/ v202 wttpsssuuuraaaaaa hhtmm html
[112] N. Subramani, N. Suresh, and M. E. Peters. Extracting latent steering vectors from pretrained language models. arXiv preprint arXiv:2205.055 124, 2022.
16

---

[113] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6799 2013.
[114] A. Taeb, N. Ruggeri, C. Schnuck, and F. Yang. Provable concept learning for interpretable predictions using variational autoencoders. In ICML 2022 2nd for Science Workshop, 2022.
[115] D. Talon, P. Lippe, S. James, A. Del Bue, and S. Magliacane. Towards the reusability and compositionality of causal representations. In Causal Representation Learning Workshop at NeurlPS 2023, 2023.
[116] R. Taori, I. Gulrajani, Zhang, Y. Dubois, X. Li, C. Guestrin, Liang, and T. B. Hashimoto. Alpaca: A strong, replicable instruction-folloiinng model. Stanford Center for Research on Foundation Models. https://erfmm stanford. edu/2023/03/1 edm//o23/033///pppacc. html, 3(6):7, 2023.
[117] I. Tenneyy D. Das, and F., Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint 1rXiv:/905.059550 2019.
[118] C. Tigges, O. J. Hollinsworth, A. Geiger, and N. Nanda. Linear representations of sentiment in large language models. arXiv preprint arXiv:2310.15154, 2023.
[119] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F, Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[120] M. Trager, P. Perera, L. Zancato, A. Achille, P. Bhatia, and S. Soatto. Linear spaces of mean- ings: compositional structures in vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15395- 2023.
[121] A. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308. 10248, 2023.
[122] B. Varici, K. Shanmugam, P. Sattigeri, and A. Tajer. Intervention target estimation in the presence of latent variables. In Uncertainty in Artificial Intelligence, pages 2013-2023, PMLR. 2022.
[123] B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. Score-based causal represen- tation learning with interventions. arXiv preprint arXiv:2301.088 082200, 2023.
[124] VarIeı, E. Acartürk, K. Shanmugam, and A. Tajer. Score-based causal representation learning: Linear and general transformations. arXiv preprint arXiv:2402.00849 2024.
[125] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[126] J. Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Lo- catello. Self-supervised learning with data augmentations provably isolates content from style. Advances in Neural Information Processing Systems, 34, 2021.
[127] J. von Kügelgen, M. Besserve, W. Liang, L. Gresele, A. Kekić, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. In Advances in Neural Information Processing Systems, 2023.
[128] K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretabilityy in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint 1=XXi:2211000922 2022.
[129] Y. Wang, D. Blei, and J. P. Cunningham. Posterior collapse and latent variable non- identifiability. Advances in Neural Information Processing Systems, 34:5443 -4555, 2021.
[130] Z. Wang, L. Gui, J. Negrea, and V. Veitch. Concept algebra for score-based conditional model. In ICML 2023 Workshop on Structured Probabilistic Inference ({&} Generative Modeling, 2023.
17

---

[131] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 22224444248837 2022.
[132] M. Willetts and B. Paige. I don't need u: Identifiable non-linear ica without side information. arXiv preprint arXiv:2106.052.88 2021.
[133] D. Xu, D. Yao, S. Lachapelle, P. Taslakian, J. von Kügelgen, F. Locatello, and S. Magliacane. A sparsity principle for partially observable causal representation learning. arXiv preprint arXiv:2403.083:
2024.
[134] M. Yang. F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. Causalvae: Disentangled representa- tion learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9593 9602, June 2021.
[135] Y. Yang, A. Panagopoulou, S. Zhou, D. Jin, C. Callison-Burche and M. Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19187-19197, 2023.
[136] D. Yao, D. Xu, S. Lachapelle, S. Magliacane, P. Taslakian, G. Martius, J. von Kügelgen, and F. Locatello. Multi-view causal representation learning with partial observability. arXiv preprint arwiv:2311.044:: 1.04056, 2023.
[137] C.-K. Yeh, B. Kim, S. Arik, C.-L. Li, T. Pfister, and P. Ravikumar. On completeness-aware concept-basee explanations in deep neural networks. Advances in neural information process- ing systems, 33:20554-20565 2020.
[138] F. Zhang and N. Nanda. Towards best practices of activation patching in language models: Metries and methods. arXiv preprint arXiv:2309. 16042, 2023.
[139] J. Zhang, C. Squires, K. Greenewald, A. Srivastava, K. Shanmugam, and C. Uhler. Iden- tifiability guarantees for causal disentanglement from soft interventions. arXiv preprint arXiv:2307.062.0 2023.
[140] Y. Zhang, Y. Du, B. Huang, Z. Wang, J. Wang, M. Fang, and M. Pechenizkiy. Interpretable re- ward redistribution in reinforcement learning: A causal approach. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[141] Y. Zheng, Ng, and K. Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Advances in Neural Information Processing Systems, 15:16411-16422 2022.
[142] S. Zimmermann, Y. Sharma, Schneider, M. Bethge, and Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990. PMLR, 2021.
[143] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin. M. Mazeika, A.-K. Dombrowski, et al. Representation engineering: A top-down approach to transparency. arXiv preprint 1rXiv:2310.01444 2023.
18

---

A Proofs of the main results
In this appendix we provide the proofs of results, in particular the proof of our main result, Theorem [ However, as a warm-up we first start in Appendix [A.] with a proof of the simpler result that can be shown based on the iVAE approach. In Appendix A.2 we prove Theorem I and in Appendix A.3 we prove the additional lemmas that appear in the paper.
A.1 Proof of identifiability with 2n + environments
As warm-up and to provide connection to earlier results we show here how to obtain identifiability by adapting the iVAE framework to our context. Indeed, our mathematical setting related to the setting used in 63 in the sense that the environments are generated by modulation with certain exponential families. Therefore, we can essentially apply their proof techniques to prove identifiability (with some modifications), albeit this requires the suboptimal number of 2m + environments (there are two sufficient statistics for the Gaussian distribution).
Theorem 2. Suppose data satisfies Assumption 2 andFand the environment statistics matrix A has rank 2n. Assume we know the number of atoms n. Then identifiaiility in the sense of Definition holds.
We remark that the rank condition only be satisfied for 2n environments (ohservational distribution and 2n concept conditional distributions. For this theorem the assumption that the filtering distribution is always the same is not necessary. Instead we could consider variances (og)² depending on environment e and row k, i.e., the filtering distribution [(gg) is Gaussian with varying variance. The generalization of the environment-concep matrix M E Rmxx is given by
if S and row 1 of A az otherwise.
Mei
(o)))²
(11)
Similarly the generalization of the emviromment-valuation matrix B E Rm*x is given by
DE
if E S and row i of A is a;
Bai =
(12)
otherwise.
We now prove Theorem We use essentially the same ideas in the proof of Theorem 1 in Khemakhem et al. [53] (followed by the same reasoning in Sorrenson et al. [108) Kivva et al. [57 but since our concepts are not axis aligned and we only extract some information about the mixing we give a complete proof.
Proof of Theorem 2 Suppose there are sets of parameters that generate the same data X X ,...,, X Denote by 7 the latter set of parameters, c.g., X is distributed as f(Z*) where 7e Rdd corresponds the concept class C with distribution 7e io and the same distribution is generated by f(2)) where f and are injective and differentiable. Let C {ai..... an be the set of atomic concepts in the first setting and let غ = {ā...... àn } be the set of atomic concepts in the second setting (here
that is assumed be known). We also consider the transition function - f-1 f and in the following we always write = (Z). The equality f(Z") 2 x 2 f(ž implies (zz)) ze. This implies that for all environments e
p² (Z) |det Jp-1| 11 70 (7) Taking the logarithm and subtracting this for some 1,
(13)
from the base distribution we obtain
In(p(Z)) (2() (n)) In (T)(())) Using the definition rewrite for some constants ce and dc
(14)
Ci
(A - -3
In(p(Z)) In(p((Z)) =
2(
(15)
(ai zed2 - Baclas z3) Co.
19

---

Here we used the environment-conee matrix and the crriioomment-aluuatinn matrix in the second step which were defined in (7 and (8 (in
TTT and (22)
for varying variance). We define the vector In(p (2)). Then we find the relation
z(Z) with components Pe 72) In(p( 2))
(a1. Z)
(a1. Z)
222) = 2 M
B
(16)
(An, Z)2
(an 2)
Together with (44 we conclude that
(a1, 2)
(11, Z)
(ā), 2)2²
(āi, Z)
1 M
B
B
(17)
2
2
(an Z)²
(an: Z)
(ān 2)2
(ān Z)
Since by assumption A (M, E R2x22 has rank 2n there is a vector v such that u M 0 and v T = -e, (e, E Rd denotes the i-th standard basis vector). Thus we find that
(01, 2)
(a1 Z)
1 u T M 2
(a, Z)
=BB B
(18)
(an 2,2
(an, 7)
In other words (ās, Z) can be expressed as a quadratic polynomial Z. We apply the same reasoning
for (ā., 2) i.e., pick vector v' such that _/o²T M and o' B = 0 to obtain a relation
(ā. 2)2 = Σ 1 2) + ((Z)
(19)
for some coefficients 17 and some affine function E of Z. The following reasoning is now the same as in Kivva et al. [57], Sorrenson al. [10]. We thus find that (ū,, Z) and its square can be written as polynimials of degree at most 2 in Z. This implies that in fact (ā,, Z) is an affine function of Z (otherwise its square would be quartic polynomial), i.e., we can write
T 2) 2) + E 2)a Ci.
(20)
Equating the square of this relation with 19 and taking the gradient with respect to Z (as a polynomial the function is differentiable) we find
2 (na 2 Ayaz Σ Xyaj, Z) + w'
(21)
for vectors and w'. The equality (for Z implies w w'. Now linear independence of aj implies that for each
U, Ar Σ Azaz.
(22)
Applying linear independence again we conclude that either = X, for j } r. This implies that there is at most one such that X, * O. The relation (20) and the bijectivity of to implies that there exactly on r(i) such that Av * 0 and therefore
X (ar(1) Z) C..
(23)
Applying the same argument in the reverse direction we conclude that there is permutation π E 6 such that
( 7) = Ai Z) + C,. By linear independence we can find an invertible linear map such that
(24)
(i) = a -
(25)
20

---

(i.e, an(s) = a;) and a vector w E R (the a; are linearly independent) such that
(ān()) 2) (,( 2) + (a,, w)).
(26)
In particular the relations (5) and (6) hold. Now it is straightforward see that if E S i.e, a, is
row of then az(() row of A and vice versa. Indeed, this follows from 17 environment
together with (26) and linear independence of the atoms. Therefore conclude from (25) that there
a permutation p such that
Ā = P A
(27)
Moreover, 26 then implies setting Z = f 7 = f-
(28)
holds for the same permutation matrix P and a diagonal matrix A whose diagonal entries can be related to (26). Let us assume now that row k of A is a, and row k of A is (z())- Now we consider the subspace H c Rd containing all Z such that (Z, 0 for i. Via (26) this implies that (āj, Z) is constant for + (i). Then we conclude from that for Z E
(lai, Z) - DE)z ((āx()) Z) )
+ c
(29)
2(
2( )
for some constant c Using 26 this implies that
((a;, 2)
b2) (Ar((a,, 2) + (0;, w)) - )2
+ c.
(30)
2(ag)²
2 )2
Comparing the quadratic term and the linear term (note that (a,, Z) take any value on H) we find
A?
(31)
2(
( )2
by
b,bL.
(a, w)
(32)
2(0g)2
2(05)
Combining the equation we obtain
T5 2x (at, w))
(33)
This implies then the relation
6 A A'w).
(34)
A.2 Proof of Theorem
In this section prove our main Theorem The proof is structured in several steps: First remove the symmetries of the representation and derive the key relations underlying the proof. Then we show that we can identify the environment-come Once this is done we can complete the proof. We will need the following lemma to conclude the proof.
matrix M and then also the valuations collected in B.
Lemma 2. The relations 3 and 6 in Definition Adefnitionddeeine equivalence relation of representations if we assume that the underlying atoms form a linearly independent set.
The proof of this lemma can be found in Appendix
A.3
Remark 3. Without the assumption on the underlying atoms the lemma is not true. In this case slightly different scaling must be chosen (e.g., (A+) = A p$ pe A instead of T A pe (b -.w))) Since our results address the case of atoms we used the simpler definition the main paper.
21

---

We can allow slightly more general filtering distributions where is Gaussian with variance of if we filter on concept i.e., the variance needs to constant for different environments and the same atom but might depend on the atom. The proof will cover this case, the simple case stated in the main paper obtained by setting of o². Some steps of the proof (e.g., the expressions for the difference of the log-densities) agree with the proof of Theorem To keep the proof self contained we repeat few equations.
Proof of Theorem
7 We proceed in several steps.
Step I:
Reduction to standard form. Let us first transform every possible data representation into a standard form. Recall that we have the set of atomic concepts {a1,..., .a,). Recall that we defined the environmentt-ooccep matrix M m in (7) and note that the natural generalisation reads
az is row of A
Mei
of
(35)
0
otherwise.
We say that concept a, conditioned on the environment e. Note that the nonzero entries of row of M encode the set 8 To pass from A to its rows a, we assume that the e-th row of A is aig. ie.. matrix B which is given by
ag = (AA) eg. Recall also consider the environment-vallse
bg of
if a; is row k of A
Bei
(36)
otherwise.
Denoting by 2aa the centered Gaussian distribution with variance x² we find in environment
dim((C.)
dCm(C.)
(A* 0b)
In(p(2)) (z) =
Σ k=1
In ((1-(=-(77)))) Σ
de
2(
k=1
(37)
a
Mes (ai, za²² (ai, Zz) - Ce.
Now we consider an invertible linear map T Rdd + Rdd such that T a; --=-a for all 1 S S n. Such map exists because we assume that the a, are linearly independent. Moreover, we consider a shift vector E Rd with X; = 0 for > n which we fix later. We define Σ E Rd,xdd to be the diagonal matrix with entries Σn = a; for 1 isis n and Σ. 1 for i n. Now we consider the linear map L(z) E Tz A and a new representation given by
= L(2,, f=foL- C
.n.. T = 1,
A" = A
(38)
((3) p(T. det T-
We also define
b σi
%
X; row A of A is ag
(39)
Define M and B in terms of A", 5t and o? as before. We remark that all entries of M are either 0 or 1 and note that
M MDiag( 01 ....... oit)
(40)
B = BDiag(o ...... "..,.,,,,,.....", )) MDDiag(DI,...... An)
(41)
We claim that this model generates the same observations as the original model. By definition L.P = P (as mentioned before, we slightly abuse notation and here refer to the distributions). Next, we calculate for any 5
-2 2n ((ez, L(z)) 6) = ((e,, - -)2
((ei, ET> A) - -8)2
(TT ei, >; 8)2
(42)
((an,)) o,di 0,0)2
0,
22n ((;, 2) ozd: -,8).
22

---

Using this for 8 = byg and some l such that row k of A is a, we find
ln L()) ),)) -2 2n ( ozAs o,5,) -2 go? bb). (43)
This then implies that for ž (2)
II ( x II log (1A92-15)
(44)
Combining this with the definition 2 and the definition p(z) = (p(-11z) det T we find that for = [(2)
5' x (z)
(45)
f(2%) 2 X Moreover, one directly sees that the two representations are also equivalent in the sense of Definition 4 We now fix the vector À such that each row of B has mean zero. Finally, by changing the sign of & we can in addition assume that for every the first non-zero Bei is positive, Finally we remark that Assumption 4 is still satisfied for M and B. Indeed, w" T M =
and thus )(Z)
implies w M by 44).. But then w B = w BDiag(o ,.....,,. ) by (4T) which has all entries different from zero if this holds for w
T B. In the following we will therefore always assume
that the representation satisfies the properties of the variables and remove the modifier in the following. The plan is now to show that M and B can be identified up to permutations of the rows (under the fixed normalization we derived in this step) and then show that every two representations with the same M and B can be identified.
Step 2: The key identity Let us here restate the key identity based on the difference of the log- densities. As is common in identifiability results for multi-environment data with general mixing we consider the difference in log densities. Consider
(n) (2) =
Σ
Mei (e,, 2)² Beiles, de
2
(46)
1
Met2]
Betz - de
2
for some constant c. Those functions will play a crucial role in the following and we will denote
g (²)= In (z) In (=)
(47)
Note that since the log density changes only by the Jacobian for pushforward measures we find that In - (x) (2) In (f(=)) In (f(=))()))
(f(=))
(48)
Note that the functions G (xr) can be estimated from the distributions of X We remark X might be supported on a submanifold if de and de do not agree making the definition of the density subtle. But we can just consider any chart locally and consider the density of the pushforward with respect the Lebesgue measure. The resulting difference expressed in G will be independent of the chart as the determinant cancels thus G is well defined function. The relation
(x) (²()))) = G*r)
(49)
will be crucial in the following because it shows that properties of gf are closely linked to the identifiable functions G
Step 3: Identifiability of environment-conceet matrix Let now show that we can identify
which concepts are contained in which environment (up to relabeling of the concepts), Recall that
S {i E [m] : a, is row of A and we similarly define Sz U.er Se for all subsets T [m]. The main observation is that this we consider the set
identify |Sr] =
Ucer S for all subsets T [m]. To show
IT argmin Σ (2).
(50)
cET'
23

---

Note that the function 9" are convex functions, and they can be decomposed as sums of functions in 24, i.e. for some functions h?
[の(()=
Σ h (z.) ==1
(51)
<ET'
Now if i Sr then i S for some and thus Mei F for the and hf is the sum of quadratic function in æ; which as strictly convex function has unique minimum z On the other hand, if i & ST then & S for e E T and thus Mes = 0 for all e E T and h (2:) = 0. Thus we conclude that
IT - fz E R = 2 for i E Sr}. This is an affine subspace of dimension dę - -ST The relations G (f(z)) = (z) imply that
(52)
f(IT) = argmin Σ (z)
(53)
x cET
Note that (x) is identifiable from the datasets X and thus the submanifold (by assumption on f) f(IT) is identifiable and by finding its dimension we obtain dN |S Since dg is the dimension of the data manifold f(X) we can indeed identify [ST] for all T' C [ma]. In particular, the total number [m]] identifiable (assuming that all atomic concepts are filtered upon at
of atomic concepts least once). Now, it is standard result that we can identify the matrix M up to permutation of the atomic concepts. Indeed, we can argue by induction in m show this. For m we just have |.s'] atomic concepts appearing in environment 1 and n - | concepts not appearing. For the induction step m - m + we consider the sizes [STu(++)] for T C [m]] Applying the induction hypothesis we can complete Mei for all columns such that M = 1. Similarly, we can consider the sizes |ST| 1STumm+Tmmm to identify the matrix M for concepts not used in environment m +
Thus, we can and will assume after permuting the atomic concepts that M is some fixed matrix.
Step 4: Identifiability of concept valuations Next, we show that we can also identify the matrix B. We do this column by column, i.e., for one atomic concept after another. Assume we consider atomic concept i. Then we consider the set T3 {e Mei 0} of concepts that not filter atomic concept i. By Assumption there is for every i' * i an environment e such that is filtered on, i.c.
Me + O. This implies ST. = [r]] {i}. Then consider as in (52) the set 4r; given by
Ir. {= Rd 2 2 for i' [m] {+))-
(54)
Note that all zy for * are constant on Ir.- Thus we find for any environment e such that i e S
ge(2)
Maj 2]
Bej co
2
j=l
ce + 1
(55)
Mej - Besz2
-a
2
2
j+i
CT. - Beizi
on Ir. for some constant CT.
Now we consider two concepts ei # e2 such that atomic concept is contained in these two environments. Then we consider the set
1 argming" =EIT;
(=) {= E Rd= zz 2 for [n] \ (i), 21 -Bera).
(56)
Note that in the second equality we used that (=)()) (=) depends on through =//2 Begiz so it is minimized at Bej Now we find using (55)
1
(cTies
1 2² --Betzz)
min gd2 min g (z) min CT,ea =ET?? Ir;
Beyi min lr.
+
=Er,)
cT,e2 + _
1
B2
BegiBost
(oTies + 2 B²
(57)
(Beji Boya)²
2
24

---

As before, this quantity is identifiable from observations because f(T) can be identified and we can minimize Gma (²2(x) (x) over f(T.).
This allows us to identify Beg Begi up to a sign. However, we can evaluate this expression over all pairs 61 and e2 and pick the with the maximal difference. Then all remaining values Bei for e such that i is filtered on in must satisfy Bes E [Beii Begi] Together with identifiability of |B Beyil this allows us identify all B& up to one sign indeterminacy and constant shift. However, in the first step we ensured that De Bei = 0 for all i which determines the shift and the sign fixed by our choice of making the first non-zero entry positive. Thus, we can assume that our two representations have the same M and B.
Step Identifiability of concepts We are now ready prove our identifiability result.
Assume we have representations Z', p and Ze, J. and such that the corresponding environment-concepl and cavironment-valuatee matrices agree, i.c., M = M and B. We consider the transition function = f-1 which is by assumption differentiable. What we want to S We now decompose
show is that p0=);; z; for all = E Rd. and concept part and the orthogonal part. We fix 2" Rdd-~ and define the function (²(2) (2 ("),2"),
(=f, =0) into the
the projection ((2", ;,,)))))))))))))))))) = z², and 40 R" + R² given by (z"); = (=9) = ((((), Note that sp differentiable but not necessarily injective. Let us denote by : RRd + R function with coordinates g g and similarly we define G. M Rd Identifiability will be based on the crucial relation
(()(())) G(S(((==))) ((T(
(58)
Here we used in the last step that of is defined in terms of M and B and thus agrees for both representations. Note that g is just quadratic function. Differentiating we obtain
Dig = Mera Bei.
(59)
Concisely this can be written as
Dg MDiag .....,, B.
(60)
Differentiating (88 find
MDiag(a,.......
2n) = (MDiag(()),.... B)))))())
(61)
Let be a vector as in Assumption 4 Denote by M E Ruxm the pscudoinverse of M which has rank because M has. We consider the matrix M Bu+lxn given by
M
M
(62)
Let us multiply the relation 61 1lb by M+ and find that
21
31
M B
M+B B D2p"(z))
(63)
Zx
0
Zn 0
Note that the first n rows of the left hand side are Diag(2).....) .,) M B. This matrix is invertible for almost all values df (21,.... 2n,) because its determinant
non-zero polynomial (the is 1) which vanishes only on set of measure zero. Outside of this
coefficient of the term 21 set the left hand side has rank n. Then the equality (63) implies that also the right hand side has rank n and thus (xf has rank n and thus is invertible. For 2f outside of this set there is up to scaling a unique vector w + (depending on 21,..., 2n such that
21
T
M B
=0
(64)
0
Z
0
25

---

From 63
we conclude using the invertibility of Dip" that
w
M B
= 0.
(655
0
2n
0
Next, we claim that for almost all values of =" the vector w has all entries different from 0 (this property is invariant under rescaling). Actually we need this only for entries 1 to n but the case n + 1 is a bit simpler so we show it first. We show this by proving that for each entry wy there is only a null of such that w; 0. Let w (w', 0) for some w' IR and w' + 0, i.c., Wn+1 O. Then
w
M w (Diagg(a 2n) M B)
(66)
0
2n 0
But this implies that Diag(21..... 2n. M B has non-trivia kernel, i.c., does not have full rank and we have above that this happens only for subset of measure 0 of all af, Next show that the same is true wi = 0. Decompose * w = (0, w'). Then we find
22
0
0 w
M B = WT
- (M+ B)2((+)))
Zn 0
Zn
0
(67)
Thus we conclude that the matrix on the right hand side is not invertible. Its determinant
2n polynomial in .2, and its highest degree term is + ..... Zn (M+ B) By definition
of M+ # we find (M+B)) of the assumption under the transformation of M and B). We find that the determinant is non-zero polynomial and the set its zeros is set of measure 0 of all 22.... on but since it does not depend on 2 this holds true for almost all = The same reasoning for i ..... implies that for every i the set of dd such that w, is a set of measure zero. We have therefore shown that for almost all 2 the rank of the left hand side of (63) is and the corresponding vector & 0 has all entries different from zero. Subtracting (64) and 65 we obtain
= BB * O by Assumption (recall that we showed invariance
21
21
0 0= T
w
= (w1(21 - -1); Wa(2n - -z)).). (68)
Zn 0
Zn
0
Now Wi * 0 implies 2; = 2j. We conclude that for almost all 2 the relation (299 = 2n holds. By continuity this implies that the relation actually holds everywhere. We conclude that T" f-2ff(2"" 2" for a fixed 2" but since was arbitrary the relation holds for all 2" and all 2.. Thus we conclude that for S <
(e, (-))) = (es, (( = (e,,
(69)
holds. This implies that those two representations satisfy B and (with P A Id and T Id). But since this relation is an equivalence relation in our setting by Lemma and since we showed equivalence to representation in standard form in the first step conclude that also any two representations are related through 3 and (4) thus finishing the proof.
A.3 Remaining proofs
Here prove the remaining auxiliary results.
Proof of Lemma 7 Since M E RRxx has rank m n there exactly one vector e R such that v M 0 and + 0. We claim that this vector has all entries different from zero.
26

---

Indeed suppose Um = 0 which then implies T, submatrix of M is invertible (this is equivalent to the rows being linearly independent) that U1:(m-1) = which is a contradiction to * 0. The same reasoning applies to every entry. Note that the assumption on M implies that every column has at least one non-zero entry, i.e., every column of B has one entry sampled from a continuous distribution. But then the probability that is orthogonal to column is zero because this is codimension hyperplane of all valuations of this row (since all entries are non-zen non-zero).
(1:(m-1)) Mi:mm---1)m))) O. But by assumption every conclude
Proof of Lemma 2
Reflexivity is obvious, just pick T = Id,
A P Iddim(Cs)- To show
symmetry we first consider the atoms. Let = T and गे = - Then
(1)
- (1)
TT Tzon (1) T-1 a.T =
(70)
Let Wi be vector such that for all sisn
(a;; w) (Tnn()) w))
(71)
Such a vector exists by linear independence of a. Let 7o = 2z)))-
Then we find that the relation
6
namely
de (lo,, f-)(1)(x) + (as, w))
(72)
implies
(
(Tro fx)())) (a=(1), - J (x)) + (ā. W))
1
(ā.. (x))
1
(ino
(o(())
)#(1)
2z(1)
(73)
remains to be shown that this lifts to the concepts C We first note that the relation 66) together with (7I and 33 implies that
A APeeAPuu - w.
(74)
Let p ((P)-1 and A (A9))-(A9))-pp Then combined with the previous disply implies
(x) = =P=) Atw
= (P
(75)
A (
The relation
-
(76)
a direct consequence of the definitions of Pe and T and (4 and the relation
6" = A pe (T Aww
(77)
follows exactly as 75 The proof of transitivity is similar (first establish the relations on the atomic concepts then lift it to C*).
B Comparison to Causal Representation Learning
In this appendix we describe causal representation learning and discuss the similarities and differences between the viewpoint taken in this paper and the standard setting in causal representation learning.
Causal Representation Learning (CRL) 102 101 aims to learn representations of data that corre- spond to true causal generative processes. More precisely, if we assume that data X is generated as X (() where Z are latent causal factors and f some arbitrary nonlinearity, the goal to learn well the distribution of Z. Since the latent variables Z are assumed to have causal relationships
27

---

among them, many works exploit the presence of interventional data to learn the generative model.
CRL incorporates ideas from the field of causality [109
86
88
95
[10] into the field of latent
variable models and is generalization of nonlinear independent component analysis [2] 45 47
and disentangled representation learning [9
8X
62 The field has seen a surge of advances in
the last few years, e.g., [53, 56 33 770 60,
13,
78 142
36
96 123
50 49 115, 124
136 [33]
As motivated in Schölkopf et al. [022]. CRL enables many desiderata such as robustness, out of
distribution generalization, and in addition enables planning and alignment. CRL has also been robotics
successful in many domains such as computer vision [53
26
73
69 [40 and
genomics [39].
In our work, we take significant inspiration from this framework of causal representation learning and present a relaxed framework that is weaker, but more general and also importantly, aligns better with empirical works on interpretability of large pre-trained models in the literature. We now describe the setup of CRL more formally in Appendix [B.1, Then, in Appendix B.2 we discuss conceptual differences between causal representation learning and our framework.
B.1 Formal setup
We assume that we observe data X E Rd with the generative model X = f(Z) where Z E Rd are the latent variables and f is a deterministic mixing function. The dataset is sampled from a distribution p and the goal is to recover the mixing function f as well as the distributions of the underlying latent variables 7..... Zd To this end, this problem over-parameterized since multiple pairs of Z and f could fit the dataset apriori, so the common practice in CRL is to impose various assumptions that will make this model identifiable. Here, identifiability is the notion that unique set of parameters fit the model (up to trivial transformations). This makes the problem well-defined and feasible, although it could still be a hard problem to solve in practice. Below, we informally summarize two classes of prior works that enable such identifiability guarantees.
1. Disentangled representation learning: In this setting, we assume that the distributions of Zi...., Zd, are jointly independent. Different studies constrain the distribution of the variables Z, Z,...., .Zz., c.g., each Z; is independently sampled from (00, 1). This is also the setting studied in nonlinear independent component analysis [2 45)
2. Causal Representation Learning: This setting is more general than the one above where we relax the independence assumption on the Z4 and instead assume that they have (typically unknown) causal relationships among them. For instance, they could satisfy a linear structural causal model with Gaussian noise, i.c., = AZ E, ~ (0, 1) where A encodes a weighted directed acyclic graph. This setting is generalizes the previous setting, since having no causal relationships (i.e., A = 0) implies joint independence.
As explained earlier, in both these domains, critical notion is that of identifiability which posits that the given dataset(s) are diverse enough for the modeling assumptions, in order to ensure that a unique set of parameters fit the data. It's folklore that the disentangled representation learning model is not identifiable if all Z; are Gaussian [46] 71 However, under appropriate assump- tions, e.g.. distributional, sparsity or observed side-information. the model becomes identifiable,
53 25 129
e.g.. [53 44 11 |06
60
78
[41 57
13
12
142
36 96). In addition, various works have proposed
methods learn them 33 132 26
334
67
23
13
63
B.2
Conceptual differences
In this section, we highlight the conceptual differences between causal representation learning and our framework.
Are causal generative concepts necessarily interpretable? Moreover, we are constantly conjuring new concepts of interest since human-interppetabll concepts are constantly evolving, e.g., the concept of mobile phones did not exist 100 years ago, but is valid concept to learn now. Therefore, as opposed to working with a rigid model in causal representation learning, we take the approach of working with a dynamic representation learning model. Finally, even if individual causal factors are interpretable (which may be the case in certain applications), the perspective that we take in this work is that the number of true generative factors could be prohibitively large so that attempting to extract
28

---

and interpret all of them together is infeasible, whereas the number of desired cammannitterpretabble concepts much smaller and more manageable.
Number of environments needed When the ground truth generative process has ambient latent dimention dz, for causal representation learning to be feasible, we usually require d2 environments or datasets. For instance, in the VAE setting [53] with & sufficient statistics, we require dgk+1 2 dg + environments. This is indeed necessary, as counterexamples show. However, it's not clear what the value of d₂ is for complex datasets, and it could potentially be prohibitively large.
But the question remains, do we need learn the entire generative model for solving downstream tasks? Along these lines, there is a tremendous research effort attempting to relax such requirements by imposing various inductive or domain biases and by building a theory of partial identifiability [57 69 [59] This is for good reason, since even though it would be ideal to learn the full ground truth generative model, may be prohibitively large and moreover it may not be necessary for the downstream tasks we care about, therefore it suffices to learn what is necessary. On this note, the related task of learning only subset of the generative latent variables is also not easy as the latent variables interact in potentially complicated ways.
In this work, we show that if we only wish to learn n < d2 concepts, it suffices to have ((n) environments instead of ((dz) environments. Therefore, our results can be viewed as a result on partial identifiability with a sublinear number of environments.
Multi-node interventions Multi-node interventions are an exciting area of study in CRL, since they are natural extension of existing works and are more useful for modeling various real-life datasets where it can be hard to control precisely one factor of variation. This is easily incorporated in our setting by utilizing non-atomic concepts, since each non-atomic concept a collection of vectors corresponding to atomic concepts and can be modified simultaneously by changing the valuation.
Conditional vs. interventional data In this work we focus on conditional data and identification of concept structure, while recent trend in CRL is to focus on interventional data and identification of the causal structure [TC 122
50 [26]. For causal models, interventions are natural approach to solving the identifiability problem. however, in the absence of an assumed causal model (as in our framework), interventions may not even be formally well-defined. In our framework, we do not think of concepts as being causal variables that are connected by graph. (We note that an interesting approach would be to study learning concepts over a given causal generative model, which is an intriguing direction for future study that we do not pursue in this work).
14
By contrast, conditional data does not require the formal framework of causal models, and is often more frequently available in practice. Conditional data can be obtained by selection through filtering, e.g., patients that are admitted to different hospitals based on the severity of their condition or by the availability of label information as in the CLIP setting [92). Thus conditional data can be obtained On the other hand interventional data requires
by observing the system in different condtions. manipulation of the system which is more difficult to obtain in general.
C Alternate definitions of concept conditional measure
In this section, we present alternate feasible definitions for data distributions than the one we introduced in Section 4.2 While we went with the definition most suited for practice, these alternate definitions are also justifiable in different scenarios and are exciting avenues for further study.
We want to essentially define a concept C' via conditional measure PC where the concept C' is identified with an affine subspace = (Z E Rd: A bCy for some AC E x D E R We consider the shifted parallel linear subspace Co (Z : A 2 0} and the orthogonal splitting Rd= = Co @ V. Suppose we have a distribution qV on the space V which will typically be Gaussian centered around cC V which the unique solution of AC = bc. In addition we have a base distribution on Rd= We will assume that all distributions have smooth density so that conditional probabilities are pointwise well defined. There are least three ways create the context conditional measure Pc-
29

---

1. The first option is to enforce that the distribution of the V marginal pc( Sc, po(e, c) de exactly matches qv(u) (v) while the in-plane distribution pC vo) x pc vo) remains invariant, i.e,, equals p(c|u = vo). Under this condition, there is a unique measure pe given by
p(c,v)
pc(e, & qr(e) Ica p(c', dc
In other words, to get (c, v) we sample v qv and then ~ p(clu) according to the conditional distribution.
2. The second option is again enforce the V marginal but instead of keeping the in plane distribution we average over the V space. Then we obtain
pa(c, v) qv(u) J. p(c,
p(c,,") v')
de'.
This corresponds (vaguely) do(v) operation from causal inference, i.e., we sample according to p(u, c) and then do a random intervention on v with target distribution qv-
3. The third option to take Bayesian standpoint. Then we view p as a prior and ạy as the context dependent acceptance probability, i.e., we sample by p and then accept with probability qy. Then we find
plc, ())v po(c, Jp(c, (e) du x p(c, (v).
(78)
This is probably the closest aligned to practice, so this is the one we study in this work. To justify this option, imagine the following scenario. If we wish to learn the concept of red color, a first step would be to curate a dataset of red objects. To do this, we first consider collection of photos of objects of varying color and then filter out the ones that look red. The concept conditional measure we define aligns with this process. To learn the actual red concept accurately, our theory predicts that it is sufficient to have additional datasets of objects that are not red, from which we can distinguish red objects, thereby learning the concept of red color.
The next question is how to define the measure ςv. When considering a single concept 1 = b the most natural option to consider Nu, IId where e V is the unique solution of A b and a > is a positive constant. This is what we do in this work (note that o² can be set to by scaling the concept and valuation accordingly).
However, we can also use alternate definitions as suggested above. For instance, we can set AZ D (DC Id). Then Z N(v (A A)- However, this runs into some technical issues sketch (and leave to future work to handle this). Consider the intersection of multiple concepts C In this case the concept space is given by the intersection = C and Co ((C*), and we have the orthogonal decomposition dd, + Σ Ve, In general the spaces V are not necessarily orthogonal but it is reasonable to assume that the non-degeneracy condition dim() [ V*) = Σ dim(V*) holds.
Now set V Σ Ve. If we choose just the standard normal distribution for qv- we can define just as in our approach
Φr Nw ,o
(79)
The second option is to enforce that the marginals of gv agree with qv=, i.e., gy (Ilyy= (v) E 0) (O) for O Ve, This results in the set of equations for all
A Idy..
(80)
is likely that this system has a unique solution when non-degeneracy holds for V and this is clearly true for orthogonal spaces but it is not clear how to solve this in general.
D Analysis of pretrained CLIP models
In this section we provide additional experimental details and further results for the analysis of pretrained CLIP models [92].
30

---

RN101 D.1 Experimental Details
We transform the images from the 3d-Shapes dataset to match the CLIP training data, i.e., reshape to images of size 224 and match the channel distributions. Then we calculate the embeddings for all images the dataset using two CLIP models, a model with a vision transformer backbone ('ViT-B/32") and model with Resnet backbone ("RN101 We split the embedded images in to training and test sets of equal size. Then for any factor variation (orientation of the scene, shape and scale of the object, and hue of floor, wall, and object) we perform the following procedure, For each pair of values of factor of variation we run logistic regression on the embeddings for those two values of the concept to classify which value is taken for given embedding. We average the directions of the logistic regression vectors B,, i.e., consider 3 N -i Br. Since the direction defined only up to sign (depending on the order of the two groups) we repeatedly replace 83 by -8; if the scalar product with the current mean is negative (this is heuristic procedure to align (3; with 3). We then use the learned concept vectors a = 3 to evaluate the concept valuations on the held out test data, i... we evaluate (a, Z) where preprocessing to calculate the CLIP image embeddings required few hours on a A A100-GPU. The remaining evaluations were performed on standard notebook.
f-1(f((( (X) the embedding of an image X, The
D.2 Further results
Here we report the mean and standard deviations the per-class concept valuations (a, Z) for the concept vectors learned described in Section D.I The results for the six factors of variation can found Tables 2 and 4 We observe that shape, scale, and orientation are well aligned with linear subspaces. For the hue variables this still holds to some degree the discrepancy might be attributed to hue not being an atomic concept (colours are typically represented by at least two numbers). Moreover, consider the correlation coefficient of the valuastions obtained for different embedding models, i.e., for (a-M,, Z:M1 and (n^ 2,2422 where a and aa are concept vectors for the same concept and two different models and z,Mr and Z,M2 denote the embeddings of the two models Mi and M2 of sample X;. We report these correlation coefficients for the two CLIP models in Table The results indicate that the valuations indeed approximately agree up to linear transformation. Note that for the scene orientation attribute the valuation corresponds to the absolute value of the angle.
Table 2: Mean valuations and standard deviation the test set for the floor hue and wall hue attributes.
Floor hue
Vit-B/32
Wall hue
Vit-B/32
RN101
0.0
1.4 1.4
1.9
0.0
1.1 + 1.3
-1.5 1.4
0.1
4.5 1.5
1.4
0.1
2,8 1.3
1.8
0.2
1.3
0.8
0.2
3.1
1.5
0.3
1.2
3.0
0.3
1.7
0.8
0.4
1.5
2.2
0.4
0.3
0.5
0.5
0.0
0.5
0.5
1.2
1.1
0.6
-.3
0.94
0.6
1.2
1.1
0.7
1.5
1.0
0.7
1.2
1.0
0.8
1.88
0.9
0.8
-1.4+1.1
+1.0+1.0
0.9
1.4
-1.0
0.9
1.2
-2.0 1.1
E Inference-Time Intervention of Large Language Models
In this section, we first briefly describe Large Language Models and the recent Inference- Time Intervention (ITI) technique proposed for LLM alignment, which we build on. Then, we use our framework to provide better intuition some intriguing observations about ITI. including why it works. And then we exploit our ideas to improve the performance of ITI by choosing the steering direction to be matrix instead of vector.
Models are publicly available under https:
31

---

Table 3: Mean valuations and standard deviation on the set for the object hue and scene orientation attributes.
Scene orientation ())
Vit-B/32
RN101
-30.0
4.9 = 1.4
-0.0 1.1
Object hue
Vit-B/32
RN101
-25.7
-4.0 = 1.3
0.4
-21.4
-2.9 = 1.3
-088
0.0
0.3 1.5
-0.1 1.1
-17.1
-0.2
1.4
0.1
4.8 2.1
1.0
-12.9
3.5
1.1
0.2
6.0
2.7
-8.6
7.5
-.77
0.3
1.7
2.7
-4.3
7.2224
-..4
0.4
1.4
2.2
0.0
8.2
1.2
0.5
1.6
0.9
4.3
5.3
1.1
0.6
1.8
0.9
8.6
6.5
-7.0
0.7
1.9
1.0
12.9
1.6
-..7
0.8
1.4
-.9
17.1
-29 1.3
-.2
0.9
1.
1.6
21.4
-4.8 1.3
-.1
25.7
55.7 1.5
1.77
30.0
6.6 = 1.8
1.1
Table 4: Mean valuations and standard deviation on the test set for the scale and shape attributes.
Scale
Vit-B/32
RN101
0.8
10.6 + 2.6
7.0 1.5
Shape
Vit-B/32
RNI01
0.8
8.3 2.1
5.2
0.9
5.0
3.6
Cube
8.2 1.4
6.9 + 0.9
1.0
1.9
1.1
Cylinder
2.9 = 1.6
2.9 1.9:0.9 0.9
1.0
1.8
0.2
Ball
-3.6 1.6
-1.2 + 0.7
1.1
2.0
1.2
Ellipsiod
-1..8 11.8=30 3.1
-5.5 + 1.7
1.2
-7.1
1.2
1.2
2.3
1.3
E.1
Preliminaries
Large Language Models (LLMs) LLMs are large models capable of generating meaningful text given context sentence. Due large-scale training, modern LLMs have shown remarkable capabilities and achieve expert-human-lic performance many benchmarks simultaneously. The architecture of many generative pre-trained transformers (GPT)-style LLMs consists of several transformer layers stacked on top of each other. Since we'll 'l be intervening on them during inference, briefly here. First, the sequence of input tokens (tokens are sub-word units) are encoded into a vector Zu using (learned) text embedding matrix and many cases also positional embedding matrix. Then, series of transformer layers act on this vector which passes through residual stream, to obtain vectors Zo, Zi.... In. The final vector then decoded back into token probabilities with a (learned) unembedding matrix. Each transformer layer consists of a multi-head attention mechanism and a standard multilayer perceptron, which captures the nonlinearity.
we'll describe the transformer architecture 125 29
In the /th layer, each single multi-head attention mechanism can be described as
= Sfaa
= A1t
A=1
Here, P/ and Q are matrices that linearly map the vector to activation space and back respectively, and Att denotes the attention mechanism that allows communication across tokens. Here, we have kept the notation consistent with Li et al. [66] for the sake of clarity.
32

---

Table Correlation coefficients of the evaluations learned for two different CLIP models evaluated on the full dataset.
Concept
p
Floor hue
0.86
Wall hue
0.83
Object hue
0.86
Scale
0.53
Shape 0.95
Orientation
-0.70
In setting, we consider the entire of activations the learnt latent vector Z. That is, the input is r To and the pre-trained model is essentially the function f such that f(x) consists of the concatenation of the vectors h21. the intermediate activations {xf 122 and also the output of the linear transformations { J20: {Q xf 1120- Our theory hinges on the assumption that pre-trained LLMs satisfy the linear representation hypothesis. that is, various relevant concepts can be realized via linear transformations of the latent transformation f(x). Indeed, this has been (see also related works on geometry of representations [51, [2] and references therein). It's a fascinating question why such models trained with next token prediction loss also learn linear representations of various human-interpretable concepts such as sentiment, see Jiang et al. [52) for recent progress on this problem.
empirically observed to hold in many prior works [17 118 81 [7] 66 85 38 52
It's well-known that despite large-scale pretraining and subsequent improvement of pre-trained models via techniques like Reinforcement Learning with Human Feedback (RLHF) and Supervised Fine-Tuning (SFT) 88 [19], significant issues still remain [10]]. c.g., the model can hallucinate or generate incorrect responses (even though the model knows the correct response which can be extracted via other means, e.g.. Chain-of ofain-of-Though prompting 131).. ). Various methods have been proposed to fine-tune the models [84] 6 7 119 [
but many of them are expensive and time- and resource-intensive as they requires huge annotation and computation resources. Therefore, more efficient techniques are highly desired, one of which the category of methods known as activation patching. activation patching (also called activation editing or activation engineering)
40 128
121
138 65 76
Inference-Time Intervention, an activation patching method for truthfulness Activation patch- ing is simple minimally invasive technique align LLMs numan-preferenee Specifically, given various concepts such as truthfulness, activation patching makes modifications to the model during inference time that the desired concepts can be aligned. This technique can be thought of as an application of the emerging field of mechanistic interpretability [83] which aims to interpret the learnt latent vector in terms of human-interree interprattereet concepts, thereby allowing us reverse-engineer what large models learn.
Activation patching has many variants 65 steering vectors to various intermediate layers during intervention [122 121 66 (8]. This means that during inference, the output activations are modified by adding constant vector in order to promote alignment of some concept. The vector will be learnt independently based on separate training data.
40 76
but we'll focus on the simple technique of adding
In particular, a recent technique called Inference-Time Intervention (ITI) was proposed to do this for the specific concept of truthfulness. ITTI focuses on the activation heads { (Attī" (P/ xx) 1120 and add to them stecring vectors in order to promote truthfulness. To learn the steering vectors, a subset of the TruthfulQA dataset [68), namely a dataset of questions qi with annotated true (a,,, 0) and false answers (at.), 1). are prepared as (g,, a,, yi J= For each sample, the question and answer are concatenated as a pair and the corresponding activations the heads x/ (for the final token) are computed via forward passes. Then, a linear probe sigmoid (0, af) )) is independently trained on each activation head to distinguish true from false answers. Finally, the top K heads based the accuracy of this classification task are chosen (for tunable hyperparameter K) and the steering vector 0 for
33

---

the h--th head in layer 1 is chosen to be the mean difference of the activations between the true and false imputs. The intuition that this direction roughly captures the direction towards truthfulness.
Formally, for the lahh head of the Ith layer, I'TI adds the steering vector oa so as to get
H F1+1 x2 + Σ of + of)
x = Au (P
during inference. Here, 0f is the steering vector, of is the standard deviation of the activations of this head along the chosen direction and a is a hyperparameter. That is, the activations are shifted along the truthful directions by multiple of the standard deviation, and this is repeated autoregressively, Note that this does not depend on the specific GPT-like model being used. The intuition is that during inference, the activations are intervened upon to shift towards the truthful direction. The top K heads are chosen to be minimally intrusive and also a design choice based on observations of the probing metrics.
Performance of ITI In Li et al. [66,, ITI was shown to significantly improve the truthfulness of few as few dozen samples, compared to what's
various LLMs after having been trained needed for Reinforcement Learning based techqniues 84 34] ITI was evaluated on the TruthfulQA benchmark [68], which is hard adversarial benchmark to evaluate truthfulness of language models. In particular, it contains 817 questions with a multiple-choice and generation tracks, spanning 38 categories such as logical falsehoods, conspiracies and common points of confusion. For the multiple- choice questions, the accuracy is determined by the conditional probabilities of candidate answers given the question. Evaluating the generation track questions is harder, and it is done by generating model output and then evaluating it via finetuned GPT-3- model [68 80). Moreover, the choice of the intervention strength calibrated so that it's neither too small (to promote truthfulness) nor too large (to ensure the original capabilities of the LLM are not lost). To check if the original capabilies are preserved, [66] compute two additional quantities to measure how far the modified model deviates from the original model. These are the Cross-Entropy (CE) loss, which is standard in
language modeling and the Kullback-Leiblee divergence (KL div.) of the next token probabilities
before and after intervention. To compute these quantities, a subset of Open Web Text is used [91].
Finally, was shown that ITT implemented on the LLaMA [199, Alpaca [1]6] and Vicuna [20] models significantly improved their performance on the TruthfulQA benchmark compared to the baseline models. Moreover, in many cases, also beat other techniques such as few-shot prompting and supervised fine-tuning. Please see Li et al. [66] for additional details.
E.2 Interesting observations of ITI
While the elegant ITI technique was designed to align LLMs towards truthfulness in practice, also raised fascinating and intriguing questions in mechanistic interpretability.. In addition to improving the technique of ITI itself, our work makes progress towards some of these questions via our framework.
1. The authors of Li et al. [66] state in section 2 that although the technique works well in practice, it's not clear what I'TI does to the model's internal representations. In addition, prior
works 17 118 81 79
52 have observed empirically that the latent representations learned by LLMS seem to have interpretable linear directions, which ITT exploits. We use our framework to illustrate in more detail one possible explanation of what ITI does the model representations and why it works, in the next section.
85
2.
The authors visualize the geometry of "truth" representations in section 3.2 of their work via the following experiment: For the most significant head (layer 14, head 18), after finding the first truthful direction via the linear probing technique, they remove it and attempt to find a second probe orthogonal to the first. They find surprisingly that the second probe is also very informative, leading them to predict that the concept of "truth" lies in a subspace, not single direction. Restated in our framework, the concept of truthfulness is a non-atomic concept (as per Definition 2. This served as an inspiration for our proposed technique in the next section, where we propose use steering matrices instead of steering vectors for L.M alignment.
3. As a was increased, the authors observed that truthfulness of the model increased however helpfulness decreased. This suggests that the "truthfulness" and "hcipfulness" concepts
34

---

are not atomic (as per Definition 2 to future work the exciting question of mechanistically extracting such common atomic concepts.
however they share certain atomic concepts. We leave
E.3 The choice of the steering vector
In this section, we will use our theoretical framework to get insights about the ITI technique and use it to improve alignment. First, similar to the multimodal CLIP setting, we will assume that the non-linearity has already been learned up linear transformation (by large-scale training of LLMs). This aligns with our theoretical insights because the training data for powerful LLMs are diverse, so they essentially satisfy our core assumptions (see also the related work (37 that proposes that context is environment in LLM training). Therefore, we simply focus on the downstream tasks, which in this section is LLM alignment. The difficulty, of course, is that do not know the concept matrix nor the valuations.
We will now analyze the truthfulness concept via our framework and give more insight why the mean of the differences is a reasonable choice of steering vector for ITI. Based on our theory, we will then provide a modification to this choice that uses steering matrices instead of steering vectors. Since this section is based on heuristics and informal assumptions, we will refrain from making any formal claims or analyses. Indeed, a formal analysis of concepts in natural language is a hard problem in general and we do not attempt it here. We conclude with ideas for potential extensions that 're worth exploring in future work.
Denote the function h to be the sequence of head activations h(x) = (xf E RRd. Note that while we can study general steering vectors for the entire latent space of representations f(x) learned by LLMs as some works do, ITI focuses only on steering the head activations h(z), so we will apply our framework to this subset representation space. In addition, we will make the simplification that we neglect the effects of the steering vector from bottom layers towards the top layers, which we do because we are dealing with sparse steering vectors and also, each single head shift is minor and does not in isolation change the behavior of the model as verified by experiments [Ap]endd B.1].
Applying our framework, we model the concept of truth via the concept matrix A E Rdcx and two valuations by: by E RRdc corresponding to False and True respectively. In other words, the set of false sentences and true sentences lie respectively in
Sfalse = {a) (xr))(()= = bo);
Strue = {a} Ah(x) b1
Note that they only approximately lie in these spaces because of our notion of concept conditional distribution. However, if we reasonably assume that the Gaussian concentration region is much smaller than the separation between these hyperplanes, then the rest of the arguments in this section should apply.
Now, a steering vector 17 is a vector such that it moves the activations from the false space to the true space, while keeping other concepts unaffected. That is, if we pick false sentence x, i..., Ah(z)= = 60. then the steering vector 77 E Rd essentially steers the activations so that A(h(z) + 77 b, In other words, it moves the sentence from false to true. Indeed, many vectors v do satisfy this equality, because we could move h(()) to any point in the hyperplane (A2- = b1 ). Therefore the goal is to find an optimal 1) that does not (significantly) affect other concepts of interest, i.e., ((h(x) + 7) RV (a) (equivalently Bn 0) for any other concept of interest B. Indeed, natural choice of the steering vector will be A (b, - bo) where A is the pscudoinverse of A. This vector will precisely affect this concept space and will not affect the concept valuations for any concept orthogonal to A. However, there are two issues with this approach: We do not know A and therefore we will approximate this steering vector from training samples and there is guarantee that other concepts of interest are orthogonal to A (note that angles between concepts are not even identifiable).
Previous approaches are based on a collection of counterfactual sentence pairs of of which corre- spond to a false answer and a true answer for the same question q. Consider the ith counterfactual pair of T We will assume the reasonable scenario that the only difference among their concepts is the concept of truthfulness. That is, for any other concept of interest B, for this sample the valuations of B, for these pairs of and of are identical. A common strategy is to use the mean
1
17 = 71
Σ
h(e))) h(cf")
(81)
35

---

as steering vector. Note that if
A(h(c:) - h(c ~ b1 bo:
(82)
i.e. the truthfulness valuation changed as desired for all samples then
Ar b bo-
(83)
Moreover, concepts interest are preserved two prototypical settings. First, if concepts of interest the same for all samples and the new datapoint, i.e., B B, = Bj in which case
Bn = Σ B((h(()) h(c =
(84)
Similarly. if concepts of interest for new point are Bg and the valuations of B, (h(c))) -(("")) of the counterfactual pairs are random, independent, and centered, then we expect them approximately cancel and
B Ri
(85)
Note that in this case, this not true if just a single steering vector h(c)") - h((cf) is used as steering vector.
This explains why the choice of mean of the activation differences across counterfactual pairs is reasonable choice of steering vector. This is precisely the technique used in ITI. While they also experiment with other steering vectors, they found that this works the best their experiments.
Now, we will continue on our insights to analyze whether we can build better steering vectors v/. We present two crucial insights based on our analysis so far.
1. Looking at our desired equations. any weighted combination of n, = =h(c,) h(c,") will satisfy Ah(x) = 66 A(h(x) + 7) b1 exactly.
2. We could potentially choose the steering vector 7 to be function of instead of being constant vector, provided v)(x) is efficiently computable during inference time.
Exploiting first insight, conclude that choosing any weighted combination of the ny should be a reasonable choice steering vector provided we can control its effeets on the spaces orthogonal to A. That is, we choose
17 Σ WiTT Σ w. (h(ct) h(cf"))
steering vector. This gives the extra freedom to tune the weights w1, u2.... based other heuristics. Note that this also captures the choice of the top principal component of the steering vector as experimented in (118).
Our second observation suggests that even the steering vector could be a function of x, namely 72x), provided it's efficiently computable during inference. Therefore, this suggests the usage of
(Tx) = Σ (x)())((())) -M(()
as our steering vector where the weights w; (z) depend on x.
Based on these two observations, we propose our ITI modification. We choose the steering vector to be dependent the context æ, with weights chosen to be w; (A(x), A(cF)) for sentence embedding (such as Mentence-BERY (77). That is,
<Tx)) ((EE) - -h(cf))
Indeed, this is reasonable if a context x is close cf for specific training sample i in terms of their sentence embeddings (xx) and X(ef), then this particular sample's steering vector should be upsampled. In other words, we can think of the training sample contexts as voting on their respective counterfactual steering vector, with weights determined by the similarity between the representation
36

---

of the test context and the representation of the sample context. A justification would be that B(x) (the relevant concepts for a datapoint) depend smoothly on a (proximity is measured by similarity of embeddings) so it makes sense to upweight close points to enforce that preserves similar concepts.
Finally, we need to argue that can compute this efficiently during inference. For this, we exploit the structure of our steering vector representation as follows.
( Σ M(MO) h(cF))
- ((())
Mh(x))
where v' denotes the tranposed vector. We remark that the weights wx)x) used could potentially be negative but this is not issue since the projection of the corresponding counterfactual vector in the direction of B is still random and we finally normalize nz2)) so the magnitude doesn't matter.
for the matrix M = (.(( McF))) -
Therefore, this stecring can be done efficiently by precomputing the steering matrix M and then during inference, we simply compute the steering vector ()() as 77z)= Ml))).
In Table 6 we show the results of our experiments with steering matri- ces. We use the open-soeneee source large lan- guage model LLaMA [119] with 7 bil- lion parameters (open sourced version from Hugging Face) and the sentence transformer SBERT (97] for the sen- tence embedding. We report the ac- curacy of the multiple-choice track of TruthfulQA [66] over random seeds and also the Cross-Entropy Loss and KL divergence of the model pre- and post-intervention. All hyperparameters tuned as per [66] and the experiments are performed on eight A6000 GPUs Higher accuracy is better and lower CE loss, and KI. divergence indicate that the original model has not been significantly modified. Here, the baselines are the unmodified model, random direction intervention, Contrast-Consisten Search (CCS) direction (17 and two different direction choices using vanilla ITT; and 2-fold cross validation is used.
Table 6: Comparison of steering vectors for LLM alignment
Technique
Acc.
CE loss
KL div.
Baseline
0,257 +0,00005
2.16
0,0+0.00
Random direction
20 0,258
2.19 40.02
0,02 400002
CCS direction
5 0,262
2.21
0.06
ITI: Probe weight dir. ITT: Mass mean shift
15 0.270 #0.004
2.21 20.02
0.06 20.00s
20 0.288 10.004
2.41
20.08
0.27
Steering matrices (ours) 15 0.295 +0002
2.61 #0.07 0.41 +0.04
We see that the multiple-choice accuracy improved, showcasing the potential of our steering matrices technique which novel in the field of LLM aligmment to the best of our knowledge. This meant to be proof of concept and not meant to be comprehensive study of this specific technique. For exploratory purposes, we outline potential modifications to our technique below, which could potentially improve the performance, both in terms of accuracy as well as in terms of invasiveness. These form an exciting direction for a more comprehensive study of our proposed ideas, which we leave for future work.
Implementation considerations We briefly note down some design choices we made in our implementation of the above method.
1. Since r7(x)) is function of æ. the standard deviation of the activation projection on this direction, i.e., (x) cannot be precomputed (as Li et al. [66] do), therefore we compute them dynamically during inference, which takes little overhead with fast tensorization operations (in particular, this is not the slow step).
2. We opted to go with evaluating the model only on the multiple-choiee questions. This is partly because evaluate the generated text, the recommended method is to use fine-tuned GPT-3-13B models but OpenAl have retired many of their older models as of this year, and therefore, the entire batch of experiments would have to be rerun with their newer models which could potentially change the baselines, and also because this work is a proof-of-concept rather than a comprehensive evaluation.
37

---

3. For computing the sentence embeddings, we only use the question prompts, as they contain all relevant contexts, And we normalize (2)) during inference time.
Additional ideas for improvement We re-iterate that our experimental exploration is not exhaustive and the preliminary experiments are merely meant to be proof-of -oonccep... this section, building on our insights, we outline some further ideas to improve the performance of ITTI We leave to future work to comprehensively explore these techniques in order to extract better performance towards LLM alignment.
Note that we opted to go with the weights (A(z), X(ef") where was chosen to be a sentence transformer embedding [97]. While this is reasonable choice, similarity metrics could be measured in other ways, c.g., with other sentence embedding models.
2. Going further, the weights do not have to be similarity scores and could be chosen via other heuristics. For instance, they could be chosen to be constants but potentially be optimized using hold-out test set.
3. As Li al. [66] noted, the ITI technique could be applied top of fine-tuned models in order to further improve their performance. Therefore, our proposed modification could also potentially be applied on top of fine-tuned models.
F Contrastive algorithm for end-to-end concept learning
In this section, we present an end-to-end framework based on contrastive learning to learn the nonlinearity as well as concepts from data. This inspired by the methods of the CRL work [14. The model architecture is designed based on our concept conditional distribution parametrization. The core idea is as follows. For each concept conditional distribution XS, we train a neural network to distinguish concept samples log-ode oog- for this problem. Then, to learn the n atomic concepts up to linearity, we build a neural architecture for this classification problem with the final layer mimicking the log-odd expression above, which can then be trained end-to-end Because of the careful parametrization of the last layer, this will encourage the model learn the representations as guaranteed by results.
X from base samples
X In Lemma 3 we derive the
First, we will derive the computation of the true log-odds.
Lemma 3. For any concept index e, there exist some constants Ce such that
In(p((2) In(((2) = Σ - Merla, + Beslas, z3)) +
(86)
where M, B are the environment-concepp matrix and the environment-valuatio matrix defined in and 88.
Proof. This follows from Eq. 15 in the proof of Theorem
From our main identifiability results, we can assume without loss of generality that the concept vectors we learn are coordinate vectors. In other words, we consider neural network h" with parameters o with output neurons h!..... h such that the n atomic concepts will now correspond the concept vectors e en (which is reasonable as they are only identifiable up to linear transformations). Therefore, for each environment e, we can train classifiers of the form
.......h,
(X, Bh. Nh, 0) = a Σ 3 (HL(())) + Ench((X)
(87)
equipped with standard cross-entropy loss, for hyperparameters a", ,8, 7h. O. Indeed, this is reason- able since if the training reaches the global optima in the ideal case, then the loss function will corre- spond to the Bayes optimal classifier and therefore, Ie (X, a², ,3% 7E, 0) In(p(()) which along with Lemma will suggest that the learnt network is linearly related to the function A Aff-!, as desired. Lastly, we choose the loss function to be the aggregated CE loss and an extra
In(p( Z)).
38

---

regularization term. That is,
(In el)me8((X) (0e( +
L
(o
(88)
CE environment
for regularization hyperparameter 77-
Sampling from concept conditional distributions common task in controllable generative modeling is being able to generate data from known concept. Note that this is not straightforward in our setting because the normalization term Eq. (2) is not efficiently computable. To do this efficiently, we also outline simple algorithm (Algorithm T in Appendix H to sample from the concept conditional distribution for known concept. Our proposed algorithm is based on rejection sampling and the algorithm as well as the complexity analysis is deferred to Appendix H
G Additional details about the synthetic setup
In this section, we detail the synthetic setup in Section 6 The base distribution is sampled from a Gaussian mixture model with 3 components whose parameters chosen randomly. The weights randomly chosen from Unif(O.3, 1) (and then normalized), the entries of the means are chosen from Unif(-1, 1) and the covariance is chosen be diagonal matrix with entries in Unif(0.01, 0.015) (note that the diagonal nature doesn't It really matter since map f will be applied to this distribution). The mixing function f is chosen be either (i) linear or (ii) nonlinear with I-layer MLP containing 16 hidden neurons and LeakyReLU((O..
activations.
The number of concepts is intentionally chosen to be less than the ground truth dimension d, and the number of concepts is m = n + 1 as per our theory. The concepts are taken to be atomic, with the concept vectors and valuations chosen randomly, where each entry of the concept vector is chosen i.i.d from Unif(- 0.3), and the resampling distribution is chosen to be a Gaussian with variance 0.005. Finally, we choose 5000 samples per environment, sampled via the rejection sampling Algorithm[] [ For the contrastive algorithm, we choose the architecture to either be linear or nonlinear with a 2-layer MLP with 32 hidden neurons each layer, with the final parametric layer chosen based on the known concept, to have the form described above. We train for 100 epochs, on single A6000 GPU, with 7) 0.0001 and use Adam optimizer with learning rates 0.5 for the parametric layer and 0.005 for the non non-parametric layer, with Cosine Annealing schedule (72).
Additional results for higher dimensional settings can be found in Table
Table Linear identifiability of synthetic settings, averaged over seeds. Same setup in Table with larger values of d₂ and dz.
Mixing (f) (n, d2, d.) l²t
MCC1
Linear
(4, 15, 18)
0.98 = no1 0.98 =: 0.03
Nonlincar
(4, 15, 18)
0.89 = nos 0.84 0.08
Linear
(4, 20, 22) 0.94 + oos 0,80 + 0.06
Nonlinear
(4. 20, 22)
0.93 + eas 0.84 0.08
Linear
(4, 25, 28)
0.85 =:22 0.3 0.79 0.07
Nonlincar
(4, 25, 28) 0.76 0.11 0.72 0.15
H Controllable generative modeling via rejection sampling
In this section, we will describe how to sample from concept conditional distribution with a known concept. Once the concepts are learned in framework, we use this technique generate new data satisfying various desired concepts, which will aid controllable generative modeling.
Consider the base distribution 2 E Rd= with density p(Z). Suppose we wish to sample from a concept given by AZ = and resampling distribution g. We additionally assume that y is efficiently computable and an upper bound is known for its density, i.c., L > max(g).
39

---

Recall that the desired density is defined
Pc(Z) p(Z)
II igdim(C
g((AZ b):)
Note that it's infeasible to compute the normalization constant for such complex distributions. However, we bypass this by using rejection sampling. We describe the procedure in Algorithm |
Algorithm 1: Rejection sampling for controllable generative modeling
Input:
Base distribution
Resampling distribution with upper bound L 2 max(4)
Concept C with transformation and valuation C
Output: Returns a single sample from PZ
M ddim(C)
// Repeat trials until condition is met
while True do
Z = yield(p)
yield(Unif((O, 1))
R M II (C) (((AZ - b).)
if R 2 U then return Z
Informally, we first sample ~ p (we overload notation for both density and the distribution) and an independent variable U ~ Un, 1). the uniform distribution on (0, 1). We accept the variable Z if
1
II (Sdim(C)
((AA b):) 2 U
M
for a predetermined upper bound M on the quantity IIgaim(c) 8((AZ- b),). the inequality is false, we simply reject the sample and repeat.
Now we will argue why this algorithm is correct, which is accomplished Theorem 3
Let
No [ v(z) II
o((AZ-b)
(Sdim(C)
be the normalization constant in the definition of (Z))). Therefore
po(Z)
p(Z)
II
a((AZ b);)
Nc
iEdim(C)
Lemma 4. Let M > max(g) (Cm dam(() The acceptance probability of each iteration of the while loop in Algorithm 7 is Pr accepted
Ne M
Proof. We have
Pr[Z accepted] = Pru.z
U M
II
d((AZ - -b);)
iSdim(C)
q((AZ b)x)
= Pru.z
U
since M > maz(g) dim(C)
max(g)
is (Cdim(()
= Pru
((AAZ - b)())
=
US
p(Z) dz as U, Z are independent
max(g)
iSdim(C)
40

---

d(AAZ - b)1)
(((AZ - -b).)
((Z) dz
since
always
max(g)
max(g)
(S)
/ (C
dz
M
Nc
M
Before we prove correctness, will remark the expected number of trials needed for accepting each sample.
Corollary 1. The expected number of trials needed to generate a single sample is
M No
Proof. Note that each iteration of the while loop is independent, therefore the number of trials until acceptance is distributed geometric random variable whose expectation is the inverse of the parameter.
This suggests that for algorithm to be efficient in practice, M should be chosen as small as possible, i.e., estimates of max(q) should be as tight as possible.
Theorem 3. Algorithm 7 yields samples from the concept conditional distribution pc-
Proof. The proof is at heart the proof of correctness of rejection sampling. For arbitrary parameters ti...., to E R, let's compute the cumulative density of the samples output by Algorithm [ and show that it matches the cumulative distribution function of pC (Z(Z) evaluated at t1,... tdd which will complete the proof. That is, we wish to calculate
PriZi S .1,..., Zd, S ld, Z accepted)
Pr[Zi S st..... Zd, S ta, accepted]
Pr[ Z accepted]
We already computed the denominator in Lemma Therefore,
Pr(Zi $ 41,
Zd, S ta, 1 accepted]
Pr[Zz S 11 Zd, S Id Z accepted]
Nc
M
Ez
12,sh ... 1ze, Std. Ey [1 acccpted]]
Ne
M
Ez
12,sh ... 12, Staz
II q((AZ - b));)
from the proof of Lemma
No
M is iSdim((C
-f LZ1sh ... ...12a, Std. Nc
II
q((AZ b):)p(Z) (2) dz
iSdim(C)
-f Sta Pc) dz
which is precisely the cumulative distribution function of (7() evaluated at l1,...,4a, la.
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See, c.g., Section and 4 for the general framework, Section55oo the main result and Section for the experiments (with details in the Appendix).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The assumptions for this work are stated in Assumptions I in Sections
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
42

---

Justification:
Assumptions are stated in Section 45 and the proof can be found in Ap-
pendix A.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details for the synthetic dataset can be found in Appendix F The remaining experiments rely on publicly available pretrained models and the experimental details are in
Appendix andd
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
43

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The codes, along with instructions on how to run them, attached in supple- mentary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https ctspps.cc/pullic//ge httpa///ggiieee/eeeeuubbiiiiiii Cotm//umeeeeeeeeeeeeeee ssi onslicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Appendices D
F
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Tables and 6
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
44

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendices D B for the individual experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics sectpa:/nneurpps... http:///wrpp...ccool/////ccccaaa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The focus of paper is theoretical and no immediate societal impact expected.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
45

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No models or data are released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Appendices experimental details.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
46

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
47


Reviews:
[{"review_id": "fu1EppI2wx", "reviewer": "Reviewer_KRMD", "summary": "The authors argue the shift from causal representation learning (CRL) to concept-based representation learning since the current CRL framework relies on strong requirements such as interventional datasets and stands far from realistic, practical use-cases. The paper formalizes the notion of concepts and establishes a theoretical foundation on the identifiability of concepts. The experiments demonstrate the utility of the framework.", "questions": "- (line 208) Can you elaborate? I mean isn’t $A$ a projector matrix?\n- Is there any way to quantitatively measure concepts learned by two different models are how much linearly-related to each other?\n- The proposed method using contrastive learning should be described in the main section in more detail. Currently, it appears at the appendix, but I think that the algorithm is a key part of the paper which illustrate the practical utility of the proposed framework.\n\n(minor)\n\n- (line 1171-1172) Parenthesis is not closed.\n- (line 370, 1491) “the number of concepts $m$” should be “the number of environments $m$”", "limitations": ".", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The motivation is convincing and the framework is novel. It also provides a rigorous foundation for the notion of concept and its identifiability.\n- The writing is clear and easy to follow. The paper provides a thorough literature review which makes it very helpful to understand the paper positioning and key contributions.\n- Experimental results on CLIP and LLMs are interesting. It supports the paper’s motivation to move from CRL to concept-based representation learning.", "weaknesses": "Datasets $X^e$ from each environment is associated with different concept $C^e$ and corresponding valuation $b^e$. The proposed method using contrastive learning requires how the dataset is partitioned into each environment, i.e., $X^0, \\cdots, X^m$. This implies that the framework is naturally more useful for **discrete** concepts (i.e., discrete valuation), as showcased in the experiments where the authors use discrete labels. However, as the motivation suggests, concepts could be continuous in many cases (e.g., intensity of the color). Therefore, I have doubt on the practical utility of the proposed framework since it cannot handle such continuous concepts. In other words, the requirement of data partition $X^0, \\cdots, X^m$ goes against the motivation of the proposed framework of handling continuous concept valuations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive review and are glad they find the paper novel, well-written and rigorous. We address the weakness below.\n\n> In other words, the requirement of data partition $X^0, \\ldots, X^m$ goes against the motivation of the proposed framework of handling continuous concept valuations.\n\nApologies for the confusion, but we would like to clarify that our setting indeed handles continuous concepts (as claimed) as follows.\nThe actual valuations could be continuous (e.g. we allow Gaussian noise) and our identifiability results do hold in such settings. \nIn other words, the concept conditional distribution do not condition on a fixed value but rather allow for noise. \nLet us consider this using the example of intensity which was brought up in the review. \nAssume that we have datasets consisting of images taken at different times of the day. Then the intensity of the colors will fluctuate within each dataset due to slight variations of the time or weather conditions, but they will fluctuate around different mean valuations for each dataset. This matches our assumptions for this concept with continuous valuations.\n\nWe also note that the experiments we conduct also involve some continuous valuations, since the latent points lie in only an approximate hyperplane.\nHowever we acknowledge that our methods may have not been fully probed in complex non-discrete settings and we leave this exciting direction for future work.\n\nWe now address the questions raised.\n\n> (line 208) Can you elaborate? I mean isn’t $A$ a projector matrix?\n\nThe formal definition 1 allows $A$ to be any linear transformation (e.g. it can be scaled), but as the reviewer noticed, there is no loss in generality in assuming it's a projector matrix, and we choose this definition for technical convenience.\n\n> Is there any way to quantitatively measure concepts learned by two different models are how much linearly-related to each other?\n\nYes, one option are the $R^2$ metric and the Mean Correlation Coefficient (L373-375) that we report in our synthetic experiments (while we compute them against the ground truth since we know it in the case of synthetic data, these can also be computed across models). \nIndeed, in our CLIP experiments, we use the correlation coefficient to measure to what degree the learned concepts for two different models are linearly related (L1177-1188). \n\n> The proposed method using contrastive learning should be described in the main section in more detail. Currently, it appears at the appendix, but I think that the algorithm is a key part of the paper which illustrate the practical utility of the proposed framework.\n\nWe appreciate the reviewer's acknowledgment of the contrastive learning method. However, we have regrettably deferred the details to Appendix F due to lack of space. With the additional page available in the final version, we are happy to include this in the main paper.\n\nWe thank the reviewer for the additional typographic suggestions and welcome additional feedback to improve the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o5Gfr9EJw5", "reviewer": "Reviewer_jSX2", "summary": "This work takes a step toward learning human-interpretable concepts while relaxing the restrictions of (interventional) causal representation learning, and they do so inspired by the linear representation hypothesis.\n\nThe authors claim that learning the generative process and the \"true\" causal factors $f^{-1}, Z$ from observations $X$ using interventions has important caveats: 1) One needs many interventions ($\\Omega(d_z)$) for identifiability of $f^{-1}, Z$ which might be too much of a requirement in many cases, 2) There's no reason that a priori such latent representations are interpretable, 3) Interventions in many examples might not be possible at all 4) One might not need the whole $f^{-1}, Z$, and there are cases where we can seek only a handful of interpretable concepts for an application without learning the full encoder and latent representation.\n\nThe authors then introduce the geometric notion of concepts as linear subspaces in the latent space of $Z$. This is inspired by the abundant evidence on the linear representation hypothesis. Based on this notion, they define the concept conditional distributions as a source of supervision for learning concepts which will replace interventional distributions as the source of supervision for learning causal representations $Z$. Concept conditional distributions are simply defined by filtering the dataset with samples that are $\\textit{perceived}$ to satisfy a concept (see eq. 1)\n\nThe problem then becomes whether given an observational distribution $X^0$ and a set of concept-conditional distributions $X^1, \\dots, X^m$ corresponding to $m$ concepts, one can identify the linear subspaces $A^mf^{-1}(x)$ corresponding to those concepts.\n\nThe main theorem then proves the identifiability (according to definition 4) of those concepts given linear independence of concepts, as well as some diversity constraints on the environments.\n\nThe authors then try to validate the claim using 3 experiments:\n1) Synthetic experiments with various linear and non-linear mixing $f$, and different dimensions for $Z,X$.\n2) Evaluating the linearity of the valuations of the concepts learned via multi-modal CLIP (inspired by the similarity of CLIP objective to their contrastive algorithm\n3) Showing that concepts can be used to steer LLM outputs.", "questions": "- Could you connect/contrast this work the recent advances in sparse autoencoders (SAEs)? While the problems are somewhat differently motivated, the resulting concepts bear resemblance to the learned sparse features from activations of transformers. Could the authors comment on this please?\n- If we care about a few features, why not use CRL methods that guarantee weaker identification requiring much less interventions? (See question below).\n- I generally agree with the motivation and direction of the paper, but I feel like the CRL community has been aware of these restrictions and recently took steps to address them; for instance, from what I understand the multi-node intervention line of work (cited by the authors) alleviates the challenge of perfect interventions and relaxes the requirement of learning all of $Z$, instead weaker and more general notions of identifiability have been introduced. A proper discussion on this would be helpful (more than what is in the appendix). Wouldn't it make sense to leverage such methods in tasks where we have a prior that there is some underlying causal representation?\n- Does equation 2 come from the independence of concepts (asking because that is introduced later). Why is not $k=dim(C)$?\n- Line 294 onwards, is $e$ properly defined before and used here?\n- Line 304, is $S_n$ a typo? $n$ was used as superscript before that, here it's used as a subscript.\n- Could assumption 4 be explained in words as well? Where does it come from? When/in what situations does it break?\n- How should one interpret table 2 of the appendix? Is there a reason why it's not plotted, and presented as numbers? Shouldn't we expect a linear plot?\n- Line 165, what do you mean by entangled concepts, is there an experiment to show that? Or do you mean the superposition of atomic concepts?\n- (Not a question impacting the score) Related to independence of concepts: Can the theory say anything about hierarchical concepts similar to hierarchical representations?\n\nRemark: I am willing to increase my score if some of the questions and weaknesses are discussed since I find the direction of this work quite interesting and important.", "limitations": "See questions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- I find this work original and novel. There have been attempts at learning concepts recently, but from my understanding (as well as the author's mention of the related work) such attempts have been limited to specific domains, while the work at hand seems to be addressing that challenge in a broad way. \n- Moreover, I believe they have nicely translated the linear representation hypothesis to concept learning, and relaxing the restrictions of interventional causal representation learning is an important endeavor (see questions though) as is also nicely motivated in multiple places in the paper.\n- I find the theoretical result insightful and important; not only does this work move away from interventions as a not-so-ideal tool, but also clearly demonstrates the theoretical advantage of concepts vs. interventions (only theoretically though).\n- The experiments touch upon different modalities showing the versatility of the claims.\n- The presentation and arguments are generally well-constructed (up until page 8)", "weaknesses": "- In the synthetic experiments, I was expecting to see large $d_z$ and small $n$ to match the claims that had been made earlier as to the advantage of concept conditional distributions, but the dimensions are quite small. I can see that they show a proof of concept, but still, it would have been nice to be consistent with the claims made earlier (unless there's a reason why the authors didn't do so)\n\n- Could the authors think of any experiment to contrast concept learning to CRL? Maybe with simple datasets like CLEVR or 3d-shapes? Isn't there a way to try to learn causal representations and concepts, and show empirically that one is easier to be achieved? I understand that the premise was that $Z$ is not always interpretable in the first place, but I think there would exist situations where it will be. If I understand correctly, do the authors think such an experiment would add to the empirical evidence for their method?\n\n- I'm also a bit unclear about where this work is taking us, and would have liked it if it was explained better. In particular, are we hoping to change our representation learning towards learning concepts? If so, are you proposing this for reasoning tasks? For alignment? Interpretability? Or what else? The reason I'm asking is that wouldn't we probably still need some causal representations in some reasoning tasks, say in vision? Basically a short discussion of when we would prefer concepts over causal representations (or else) would be helpful.\n\n----\nWriting and Clarity:\n\n- The notion of environment in the context of concepts appeared out of the blue on page 7 (same with its notation that followed)\n- Not a weakness of the method - but the learning method seems like an important component of the paper which is deferred altogether to the appendix, i.e., one would learn about the identifiability, but there is no mention of the actual method to learn such (identifiable) concepts which might hide the difficulties and challenges associated with it.\n\n\n\nPlease also see the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and nice summary of the paper. We also appreciate their insightful comments on the importance of this work.\n\n> In the synthetic experiments, I was expecting to see large $d_z$ and small $n$\n\nThanks for the suggestion, we ran additional experiments and included the results in the global response. We kept $n$ as $4$ and increased $d_z$. While the metrics naturally degrade a bit, they're still comparable to other nonlinear CRL works.\n\n> Could the authors think of any experiment to contrast concept learning to CRL?\n\nThis is an interesting suggestion and would potentially add to the empirical evidence. However, it is beyond the scope of our work to compare against CRL comprehensively, partly because we're studying settings where CRL is not directly applicable (e.g., due to lack of interventional data). Moreover, our work is primarily theoretical and we leave for future work to extensively compare concept learning methods to CRL techniques empirically.\n\n> I'm also a bit unclear about where this work is taking us, and would have liked it if it was explained better. In particular, are we hoping to change our representation learning towards learning concepts?\n\nOur hope with this work is to provide concept learning a rigorous footing via the theory of identifiability. We do not focus on a specific task and instead build a conceptual bridge between causal representations and concept learning.\nWhile causal representations may be ideal or even necessary for some tasks, our work presents a middle ground for many contemporary settings where this may not be possible (please see also L43-50).\nAs reviewer `ohha` points out, \"Bridging identifiability in (causal) representation learning and concept-based learning is an open problem\" and this is one of the main motivations of our work.\nWe will include this discussion in the paper.\n\n> The notion of environment in the context of concepts appeared out of the blue on page 7\n\nThank you, we will make sure to introduce this terminology carefully.\n\n> Not a weakness of the method - but the learning method seems like an important component of the paper which is deferred altogether to the appendix\n\nWe agree that the contrastive learning method is an important component of the work, however, we have regrettably deferred the details to Appendix F due to lack of space. With the additional page available in the final version, we are happy to include this in the main paper.\n\nWe next address the questions in order.\n\n> Could you connect/contrast this work the recent advances in sparse autoencoders (SAEs)?\n\nWe're briefly aware of works on SAEs that learn interpretable features in models. However, to the best of our knowledge, they do not provide theoretical identifiability guarantees for learning concepts, which our work endeavors.\n\n> If we care about a few features, why not use CRL methods that guarantee weaker identification requiring much less interventions?...the multi-node intervention line of work (cited by the authors) alleviates the challenge of perfect interventions and relaxes the requirement of learning all of $Z$\n\nCould the reviewer please clarify which specific work they mean? We're more than happy to comment further and have a proper discussion on this in the paper as well. \n\n> Does equation 2 come from the independence of concepts. Why is not $k = dim(C)$?\n\nYes, equation 2 arises if the noise for the noisy estimates is independent for all atomic concepts.\nThe product runs from 1 to $\\mathrm{dim}(C)$ so our expression is the same as\n$\\prod_{k=1}^{\\mathrm{dim}(C)}$.\n\n> Line 294 onwards, is $e$ properly defined before and used here?\n\n We will clarify that $e$ is just an environment label for a concept conditional distribution.\n\n> Line 304, is $S_n$ a typo?  $n$ was used as superscript before that, here it's used as a subscript.\n\nNo, here $S_n$ corresponds to the permutation group on $n$ elements, we will use a different font to make the distinction clear.\n\n> Could assumption 4 be explained in words as well? Where does it come from? When/in what situations does it break?\n\nThis condition is similar to other diversity conditions in identifiability theory. It ensures that there are sufficiently many, non-redundant datasets. It breaks, e.g., \nif there are fewer datasets than atomic concepts of interest or when several of the concept conditional datasets disagree.\n\n> How should one interpret table 2 of the appendix? Is there a reason why it's not plotted, and presented as numbers? Shouldn't we expect a linear plot?\n\n Note that for the hue variables the different numbers correspond to different colors but the numbers\n do not correspond to meaningful concept valuations, but are just discrete labels for the different colors. Therefore, we do not expect\n to recover a linear relation between the evaluated concept valuations and the label index.\n This is different for attributes such as, e.g., size or orientation where the value should correspond approximately to the valuation (potentially up to a non-linear transformation).\n Note that it is moreover not clear whether color is an atomic concept as we assume here (e.g., standard color representations are 2 dimensional). \nWe nevertheless observe a high correlation coefficient between the representations learned by different models (see Table 5).\n\n> Line 165, what do you mean by entangled concepts...Related to independence of concepts: Can the theory say anything about hierarchical concepts similar to hierarchical representations?\n\nWe would like to clarify that in assumption 2, we talk about linear independence (and not statistical independence) of atomic concepts. However, the concepts we actually allow can each consist of multiple atomic concepts and different non-atomic concepts can overlap (not superposition), which can be interpreted as hierarchical or entangled concepts as well.\n\nWe thank the reviewer for their suggestions and welcome additional feedback to improve the text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7mUgCzDOV", "reviewer": "Reviewer_ohha", "summary": "This paper proposes a theory to identify latent concepts from model representations. In contrast to previous work in the concept-based models' field, concepts are expected to be linearly represented in the model representations, and a linear transformation A is associated with such concepts. The paper is theory-driven and presents rigorous results for the identifiability of such concepts. The authors also present some experiments related to the theory, one of which verifies in a synthetic setup the validity of the proposed theory, one on CLIP and one on LLMs.", "questions": "**Recovering the concepts without post-hoc analysis?**\nAs I understood correctly (correct if I am wrong), the identifiability class studied in the paper presupposes recovering the latent concepts (the matrix A) up to an invertible transformation $T$. Thus, the theory offers only guarantees to which a linear probe on the latent representations would recover the latent concepts, is it correct? One in practice has still to find the matrix $A$ related to the concept, and that cannot be done without concept annotation. Is that the case?\n\n**Typo?** In Def  It seems that $d_Z$ and $\\tilde{d_Z}$ have to be the same to guarantee the inverse exists (and how it is used in the proofs). Is that the case? Would the theory hold also for models with different latent dimensions $d_Z \\neq \\tilde{d_Z}$?\n\nI asked other questions in the weakness part.", "limitations": "All assumptions are natural limitations of the proposed theoretical results. It is not clear whether foundation models are trained under the conditions that the author found for assessing identifiability of the concepts\nand the connection to LLMs seems a bit weak.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "### **Originality** \n\nThe paper proposes a new perspective on concept-based learning by providing also identifiability guarantees. The notion of concepts as linear subspaces in the representation space is interesting although not entirely new, see [1]. The results on identifiability are new and potentially useful for follow-up works: it is a valid idea to learn such concepts from conditional distributions. \n\n### **Quality and Clarity**\n\nThe paper is of high quality, providing new important results for identifiability of latent concepts, and addresses the important problem of learning high-level concepts from data. The presentation is somehow clear, although it can be improved.\n\n### **Significance**\n\nBridging identifiability in (causal) representation learning and concept-based learning is an open problem. The main contribution by the authors is indicating a viable route to achieve it by relating with theory. It also can be related to visual-language models, like CLIP, that are typically learned without any concept supervision but, due to interpretability concerns, it is often of interest to know whether concepts are linearly represented and used by the model. The theory seems solid and is of interest for advancing research in concept-based models.\n\n\n[1] Concept Whitening for Interpretable Image Recognition, Chen et al. (2020)", "weaknesses": "## Major \n\n### **Experiments**\n\nOne thing that is particularly dangling in the paper is the experimental section and the supposed evidence in support of the theory: \n\n1) The experiments on CLIP and LLMs seem to me unrelated to the theory devised by the authors and rather support on one hand that concepts can be found in the representations in CLIP to some extent (which was previously observed, e.g. [2,3]), on the other hand, that it is possible to improve the steering of LLMs predictions with some matrix operation rather than vector addition. How is this related to the theory the authors propose? How is it that CLIP training aligns with the assumptions that would lead to the identifiability of latent concepts? LLMs like LLaMA or GPT are next-token predictors learning with a different objective, see [4], how are even these models related to the theory proposed?\n\n2) It would have been more useful to provide experimental evidence of the proposed contrastive learning method on semi-synthetic datasets like MPI3D or Shapes3D. \n Some real-world datasets are of particular interest to the community in concept-based models, being more tackling, like CUB200 and OAI [5], CelebA [6], and many more [7,8].\n\n3) The synthetic experiment is impossible to understand from the main text and many details concerning the data, the learning procedure proposed (which is also very detailed), and the metrics are confined to appendices rendering it necessary to consult them in length. Nonetheless, it seems that under the working assumptions the model trained in a contrastive manner on the synthetic data and environments captures the right latent concepts. How do you evaluate $A^e$? \n\n### **Assumptions**\n\nIt would be beneficial to present the assumptions and discuss the intuition behind them. Assumption 1 is fine and is common in studying identifiability, but if the aim is to identify concepts, it seems unnaturally restricting to consider only invertible functions $f$. One could hope to extend the results also to non-invertible functions. However, this is not a serious limitation given the novelty of the results.  \\\nAssumption 2 takes the concepts to be linearly independent, what happens if for some of them is not the case? \\\nAssumption 3 requires a Gaussian distribution for the noise, it is remarked that other distributions work as well (which distributions?) but there is no citation. \\\nHow should Assumptions 4 and 5 be understood, what data are expected to be collected? Are these expensive to obtain in practice? They seem to presuppose a lot of knowledge on what latent concepts should be identified, which may not be the case when concepts are not known apriori.\n\n\n## Minor\n\n### **Conditional distributions in practice**\n\nIt is a bit puzzling how data should be gathered for the theory to work, and thus the scaling of the proposed method. The synthetic experiment offers a proof of principle but it does not show how the model behaves in settings where more concepts (and structured ones, like the color of an object) and conditional distribution are to be considered.  \n\n\n### **Related work**\n\nCausal Representation Learning seems more of an inspiration to the paper rather than having a tight connection to the theory. The authors consider conditional distributions for latent concepts and the proof techniques are inspired by the iVAE work (2019), so there is no clear link to what should be the causal aspects that should be taken into consideration. \n\nIt seems more that works about identifiability in Causal Representation Learning become relevant if one wants to extend authors' theory to causal variables, not being essential to support their claims and the connection. \n\nI was also expecting to see a comparison to Concept-based models' current practices, which require dense supervision of the concepts in cases [1, 7, 8],  partial [6], or language-guided [2,3]. Some approaches aim to learn concepts only by leveraging supervision on the classification task, see [9], and seemingly related work on continuous latent variable identification [10]. \n\n## Summary\n\nThe experiments do not complement the theory and dilute the message by showing two post-hoc analyses on CLIP and LLMs. I struggle to see how both constitute valid evidence for the theory proposed. On the other hand, additional experiments on known semi-synthetic datasets or real ones would highlight the extent of the theory to the community in concept-based interpretability. \n\nThe presentation of the material also requires some clarification around the assumptions the authors make and their validity in practice.\n\n[2] Label-free Concept Bottleneck Models, Oikarinen et al. (2023) \\\n[3] Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification, Yang et al. (2023) \\\n[4] On Linear Identifiability of Learned Representations, Roeder et al. (2021) \\\n[5] Concept Bottleneck Models, Koh et al. (2020) \\\n[6] GlanceNets: Interpretable, Leak-proof Concept-based Models, Marconato et al. (2022)  \\\n[7] Concept embedding analysis: A review, Schwalbe (2022) \\\n[8] Concept-based explainable artificial intelligence: A survey, Poeta et al. (2023) \\\n[9] Provable concept learning for interpretable predictions using variational autoencoders, Taeb et al., (2022) \\\n[10] Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning, Lachapelle (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review and are glad that they appreciate our contributions towards the important open problem of bridging causal representation learning and concept based representation learning.\n\n> The experiments on CLIP and LLMs seem to me unrelated to the theory...How is this related to the theory the authors propose?\n\nThe CLIP experiments showcase \nthat the learned representations \nof concepts indeed lie approximately within an affine hyperplane which is an illustration of our key hypothesis.\nMoreover, the training data approximately satisfies our assumptions if we view the caption as the environment, i.e., images with similar/equal captions form a concept conditional distribution and the joint image distribution is the observational distribution. \nOur experiments also show that  the representations of different CLIP models agree up to linear transformation.\nAs noted by the reviewer,\na more rigorous connection would be to establish if CLIP models learn a representation such that concepts are represented by affine hyperplanes and whether concept conditional distributions are approximately given by eq. (2).\nIn such a case, we can apply our identifiability theory \nto conclude that different CLIP models necessarily learn approximately the same representation\nup to linear transformation.\nProving these facts rigorously for the representation based on the CLIP-loss is a challenging problem we leave for future work.\n\nThe LLM experiment is a bit more exploratory and the connection to our theory is via our conceptualisation of concepts. Indeed, this helps us to obtain intuition what the right steering vector should look like and guided our construction.\n\n> It would have been more useful to provide experimental evidence ... on semi-synthetic datasets like MPI3D or Shapes3D. Some real-world datasets ... like CUB200 and OAI [5], CelebA [6], and many more [7,8].\n\nWe appreciate the nice suggestions. Since the primary thrust of the work is theoretical and it is currently quite challenging to scale these type of methods to large datasets, we leave this interesting direction for future work.\n\n> The synthetic experiment is impossible to understand from the main text and many details ... are confined to appendices...How do you evaluate $A^e$?\n\nWe apologize that this part is difficult to understand as details were deferred due to lack of space.\nIn order to evaluate $A^e$ in our synthetic experiments, we compute the linear correlation metrics $R^2$ and MCC between the ground truth $Z$ (restricted to the projection space) and the predicted $Z$ (from our model), which we report in our tables.\nAs outlined to other reviewers, we will use additional space to add experimental details to the main text.\n\nAbout assumptions in general, we believe our assumptions could likely be further relaxed at the expense of more technical work, which we leave for future work.\n\n> Assumption 1 is fine and is common in studying identifiability. One could hope to extend the results also to non-invertible functions. However, this is not a serious limitation given the novelty of the results.\n\nWe agree.\n\n> Assumption 2 takes the concepts to be linearly independent, what happens if for some of them is not the case?\n\nThen we can in general not identify \nthe concept matrix, think, e.g., of the case where two concepts are collinear.\n\n> Assumption 3 requires a Gaussian distribution for the noise, it is remarked that other distributions work as well (which distributions?) but there is no citation.\n\nThe result could be extended to exponential families, we will clarify this and add a reference (see e.g. Khemakhem et al. [50]).\n\n> How should Assumptions 4 and 5 be understood, what data are expected to be collected?...how data should be gathered for the theory to work, and thus the scaling of the proposed method.\n\nAssumptions 4 and 5 state that the environments need to be sufficiently diverse. We do not think that the approach should be used to collect data in practice because this would indeed require a lot of prior knowledge which could probably be directly used to learn the concepts. Instead we think that often real-world data comes with subtle heterogeneity that allows us to identify  concepts. \n\n> Causal Representation Learning seems more of an inspiration to the paper... not being essential to support their claims and the connection.\n\nWe generally agree with the reviewer\nand refer, e.g., to the discussions\nin L43-50, 60-63, for this remark. We're happy to add more details to clarify this.\n\n> I was also expecting to see a comparison to Concept-based models' current practices\n\nWe appreciate the relevant references and are happy to include them in the paper. In short, some of these works study very useful empirical methods whereas we focus on rigorous theoretical contributions, whereas the others differ in the kind of assumptions and settings studied, i.e. they're related but not directly comparable at a technical level.\n\n> Recovering the concepts without post-hoc analysis?...One in practice has still to find the matrix $A$ related to the concept, and that cannot be done without concept annotation. Is that the case?\n\nYes, our identifiability is only upto linear transformations. After having learned the nonlinearity, in order to recover the transformation itself, additional information such as concept annotation may be utilized in practice.\n\n> It seems that $d_Z$ and $\\tilde{d_Z}$ have to be the same to guarantee the inverse exists (and how it is used in the proofs). Is that the case? Would the theory hold also for models with different latent dimensions $d_Z \\neq \\tilde{d_Z}$?\n\nYes, for technical reasons this is the case.  To consider $d_Z\\neq \\tilde{d_Z}$ we need to relax the injectivity assumptions otherwise $d_Z$ has to correspond to the dimension of the data manifold.\n\nWe hope the rebuttal answered the questions raised, and are also happy to take additional feedback to improve the writing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "smbw5Z1jRk", "reviewer": "Reviewer_kVe2", "summary": "The paper focus on recover human-interpretable concepts from observation. It proposes a concept based representation learning method, which relax causal notions with a geometric notion of concepts. Experiments on synthetic data, multimodal CLIP models and  large language models supplement their results and show the utility of their approach.", "questions": "Please see the weaknesses.", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The authors hope to find a middle ground where they can simultaneously identify a smaller set of interpretable latent representations, which is an interesting idea.\n2.This work can be interpreted as a new direction for identifiable representation learning in order to study when interpretable concepts\ncan be recovered from data.\n3.Experiments on synthetic data, multimodal CLIP models and large language models supplement the results and show the utility of the approach.", "weaknesses": "1.This approach sacrifices causal semantics. This can be particularly problematic in situations where a deep understanding of causality is crucial, such as in root cause analysis, where the goal is to identify the fundamental reasons behind a problem or an event. Without causal semantics, one might only address the symptoms rather than the core issues, leading to temporary or ineffective solutions.\n2.The author has made numerous assumptions within the article, which could potentially impact the universal applicability of the theory presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review. We address their comments below.\n\n> This approach sacrifices causal semantics. This can be particularly problematic in situations where a deep understanding of causality is crucial\n\nIn this work, we focus on the many applications where a relaxation of causal semantics is allowed or necessary, such as when interventional data is out of reach (L46-50, L118-124).\nWhile a deep understanding of causality would be ideal, concept learning is itself a mature field independent of causality and we present a middle ground that is useful in contemporary applications.\n\n> The author has made numerous assumptions within the article, which could potentially impact the universal applicability of the theory presented.\n\nWe would like to highlight that without any assumptions, we cannot have identifiability (L29-31), \ntherefore we make natural assumptions to prove our results.\nHowever, we tried to keep the assumptions as general as possible, e.g. we allow non-linear mixing (Assumption 1) and use Gaussian noise (Assumption 3).\n\nWe welcome additional suggestions to improve the work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the thoughtful reviews and suggestions by the reviewers. 
We are glad that the reviewers found our approach original and novel (reviewers KRMD, jSX2, kVe2), well-written and rigorous (reviewers ohha, KRMD) and significant (reviewers jSX2, ohha).
The reviewers appreciated our theoretical results and its significance, e.g. "I find the theoretical result insightful and important ... also clearly demonstrates the theoretical advantage of concepts vs. interventions", ""The paper is of high quality, providing new important results for identifiability of latent concepts"".
We first address repeated comments below.

**On additional experiments:**
We appreciate the interesting suggestions to extend our method to semi-synthetic datasets and other real-world datasets. Following the suggestion of reviewer jSX2, we have also scaled up our synthetic experiments, please see attached pdf.
However the main contribution of this work is theoretical.
Indeed, not all of the theory community is aware of the empirical support for linearity of representations, and this is a key motivation for our work.
We leave it to future work to comprehensively study experimental methods towards concept learning via our framework.

**On relation between theory and experiment:**
Our synthetic experiments serve to verify the theory whereas the other experiments are a bit more exploratory and serve to probe the different assumptions and conclusions of our theory (see also the response to reviewer ohha).

In the individual responses, we have addressed the weaknesses and answered the questions raised by the reviewers.

Meta-review: In this submission the authors argue for a shift from causal approaches to representation learning to concept-based approaches. One of the key questions that the authors are interested in is whether such representations are identifiable. Instead of learning causal representations, where we require access to interventional distributions, the authors assume access to a set of concept-conditional distributions. They then define a geometric notion of concepts as linear subspaces of the laten space and prove that these subspaces are identifiable under the assumption of linear independence of concepts, as well as some assumptions about data diversity. The authors evaluate the proposed framework in the context synthetic experiments, multi-modal CLIP models, and LLMs.

After discussion, all reviewers support acceptance and overall this submission appears above the bar. Some comments from reviewers that should be addressed in the camera ready include moving the algorithm block from the appendix to the main text, addressing some points where terminology could be introduced more carefully and providing a greater degree of intuition behind some of the mathematical assumptions. The AC would also recommend that the authors revise their presentation of the experiments to address some of the questions raised by reviewer ohha regarding the connection between experiments and the developed theory.
Decision: Accept (poster)