From Causal to Concept-Based Representation Learning

Authors: goutham  rajendran
simon  buchholz
bryon  aragam
bernhard  schölkopf
pradeep  ravikumar
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:45:32

Abstract: To build intelligent machine learning systems, modern representation learning attempts to recover latent generative factors from data, such as in causal representation learning. A key question in this growing field is to provide rigorous conditions under which latent factors can be identified and thus, potentially learned. Motivated by extensive empirical literature on linear representations and concept learning, we propose to relax causal notions with a geometric notion of concepts. We formally define a notion of concepts and show rigorously that they can be provably recovered from diverse data. Instead of imposing assumptions on the "true" generative latent space, we assume that concepts can be represented linearly in this latent space. The tradeoff is that instead of identifying the "true" generative factors, we identify a subset of desired human-interpretable concepts that are relevant for a given application. Experiments on synthetic data, multimodal CLIP models and large language models supplement our results and show the utility of our approach. In this way, we provide a foundation for moving from causal representations to interpretable, concept-based representations by bringing together ideas from these two neighboring disciplines.

Main paper: 
 introduction: 
 A key goal of modern machine learning is to learn representations of complex data that are humaninterpretable and can be controlled. Although existing models are known to extract features that are useful and intuitive to humans (e.g. color, shape, size), the sensitivity of these models in capturing these features is notorious [10, 25, 71] : The enormous capacity of contemporary models means that it is possible to reconstruct the input data with meaningless representations (e.g. posterior collapse [39, 24, 129] ). Accordingly, understanding how and when useful features can be captured, along with providing assurances on the quality of learned representations, is an ongoing endeavor [27] .A natural approach to this problem is to model the input data X = (X 1 , . . . , X dx ) as X = f (Z), where f is a nonlinear transformation that maps structured underlying latent generative factors Z = (Z 1 , . . . , Z dz ) to X, and then to attempt to recover the model parameters Z, f from X. This is an appealing approach since it implies no restrictions on the data X, and has the interpretation of recovering "ground truth" factors that generated the data. It is well-known that without additional assumptions, this is impossible [46, 71] , a fact which has led to a long line of work on nonlinear ICA [21, 45] and unsupervised disentanglement [9, 88, 62] . One approach to resolve this limitation is to assume that Z has an intrinsic causal interpretation. This is known as Causal Representation Learning (CRL) [102, 101] , which endeavors to identify useful representations through the lens of causality. Structurally, the latent factors Z are assumed to have causal relationships among them, which enables us to reason about effects of interventions and conditioning on these latent factors. CRL studies this setting via an intricate interplay of ideas from causality, latent variable modeling and deep learning, with the main goal being to reconstruct the mixing function f and Z, the true generative factors of data, by leveraging causal assumptions. Recent years have witnessed a surge of rigorous results on provably learning causal representations under different assumptions [53, 33, 70, 60, 78, 142, 36, 123, 49, 115] . For example, as long as we have access to interventions on each latent variable Z j (a total of at least d z interventions), under weak assumptions on Z and/or f , the causal model over Z as well as the model parameters (Z, f ) can be uniquely identified [111, 14] .While causal features are intrinsically desirable in many applications, the assumption that we can feasibly perform Ω(d z ) interventions merits relaxing: Indeed, in complex models, the number of true generative factors d z = dim(Z) might be intractably large (e.g. consider all of the latent factors that could be used to describe natural images, video, or text). At the same time, there are yet many other applications where the strict notion of causality may not be needed, and moreover it may not be necessary to learn the full causal model over every causal factor. Is there a middle ground where we can simultaneously identify a smaller set of interpretable latent representations, without the need for a huge number of interventions?In this paper, we study this problem in detail and provide an alternative setting under which latent representations can be provably recovered. Instead of recovering a (potentially huge) number of causal latents with interventions, we settle for recovering a smaller number of interpretable concepts without strict interventions. The basic idea is to recover projections AZ of the generative factors Z that correspond to meaningful, human-interpretable concepts through conditioning instead of intervention. The idea to model concepts as linear projections of the generative factors is derived from a growing body of literature (e.g. [90, 55, 130, 77, 5, 22, 30, 17, 118, 81, 38, 75, 103] , see Section 3 for even more references) showing that the embeddings learned by modern, high-performant foundation models are not inherently interpretable, and instead capture interpretable concepts as linear projections of the (apriori) unintelligible embeddings. While this approach sacrifices causal semantics, it makes up for this with two crucial advantages: 1) Instead of strict interventions in the latent space, it suffices to condition on the concepts, and 2) When there are n concepts of interest to be learned, only n + 2 ≪ d z such concept conditionals are needed.We validate and utilize our theoretical ideas via experiments. First, we validate these theoretical insights on synthetic data, where we use a contrastive algorithm to learn such representations for a given collection of concepts. Moving ahead to real-world data, we probe our theory on embeddings learned by multimodal CLIP models [92] . The training scheme for CLIP aligns with our theoretical setting and therefore, it's reasonable to ask whether they satisfy our observations. Indeed, we show that the concepts in the 3d-Shapes dataset approximately lie in hyperplanes, further supporting our theoretical results. Lastly, we show an effective application of our framework to alignment of large language models (LLMs) where we extend the alignment technique of [66] to make LLMs more truthful.Contributions In summary, our contributions are:1. We formalize the notion of distributions induced by abstract concepts in complex domains such as images or text (see Secion 2 for an overview and Section 4.2 for formal definitions). Our definition of concept conditional distributions allows both continuous and fuzzy concepts. 2. We prove near-optimal identifiability results for learning a collection of concepts from a diverse set of environments in Theorem 1. Thus our work can be interpreted as a new direction for identifiable representation learning in order to study when interpretable concepts can be recovered from data. 3. We then verify our guarantees via a contrastive learning algorithm on synthetic data. In addition in Section 6, we support our geometric definition of concepts and our identifiability result by analysing image embeddings of CLIP-models and we utilize our ideas to improve alignment of LLMs to make them more truthful.i = h(c T i ) -h(c F i ) will satisfy Ah(x) = b 0 , A(h(x) + η) = b 1 exactly. 2.We could potentially choose the steering vector η to be a function of x instead of being a constant vector, provided η(x) is efficiently computable during inference time.Exploiting our first insight, we conclude that choosing any weighted combination of the η i should be a reasonable choice of steering vector provided we can control its effects on the spaces orthogonal to A. That is, we can chooseη = i w i η i = i w i (h(c T i ) -h(c F i ))as our steering vector. This gives us the extra freedom to tune the weights w 1 , w 2 , . . . based on other heuristics. Note that this also captures the choice of the top principal component of the steering vector as experimented in [118] .Our second observation suggests that even the steering vector η could be a function of x, namely η(x), provided it's efficiently computable during inference. Therefore, this suggests the usage ofη(x) = i w i (x)(h(c T i ) -h(c F i ))as our steering vector where the weights w i (x) depend on x.Based on these two observations, we propose our ITI modification. We choose the steering vector to be dependent on the context x, with weights chosen to be w i = ⟨λ(x), λ(c F i )⟩ for a sentence embedding λ (such as Sentence-BERT [97] ). That is,η(x) = i ⟨λ(x), λ(c F i )⟩(h(c T i ) -h(c F i ))Indeed, this is reasonable as if a context x is close to c F i for a specific training sample i in terms of their sentence embeddings λ(x) and λ(c F i ), then this particular sample's steering vector should be upsampled. In other words, we can think of the training sample contexts as voting on their respective counterfactual steering vector, with weights determined by the similarity between the representation of the test context and the representation of the sample context. A justification would be that B(x) (the relevant concepts for a datapoint) depend smoothly on x (proximity is measured by similarity of embeddings) so it makes sense to upweight close points to enforce that x preserves similar concepts.Finally, we need to argue that we can compute this efficiently during inference. For this, we exploit the structure of our steering vector representation as follows.η(x) = i ⟨λ(x), λ(c F i )⟩(h(c T i ) -h(c F i )) = i (h(c T i ) -h(c F i ))λ(c F i ) ′ h(x) = M h(x) for the matrix M = i (h(c T i ) -h(c F i ))λ(c F i ) ′, where v ′ denotes the tranposed vector. We remark that the weights w i (x) as used could potentially be negative but this is not an issue since the projection of the corresponding counterfactual vector in the direction of B is still random and we finally normalize η(x), so the magnitude doesn't matter.Therefore, this steering can be done efficiently by precomputing the steering matrix M and then during inference, we simply compute the steering vector η(x) as η(x) = M h(x). [97] for the sentence embedding. We report the accuracy of the multiple-choice track of TruthfulQA [66] over 3 random seeds and also the Cross-Entropy Loss and KL divergence of the model pre-and post-intervention. All hyperparameters are tuned as per [66] and the experiments are performed on eight A6000 GPUs. Higher accuracy is better and lower CE loss, and KL divergence indicate that the original model has not been significantly modified. Here, the baselines are the unmodified model, random direction intervention, Contrast-Consistent Search (CCS) direction [17] and two different direction choices using vanilla ITI; and 2-fold cross validation is used.We see that the multiple-choice accuracy improved, showcasing the potential of our steering matrices technique which is novel in the field of LLM alignment to the best of our knowledge. This is meant to be a proof of concept and not meant to be a comprehensive study of this specific technique.For exploratory purposes, we outline potential modifications to our technique below, which could potentially improve the performance, both in terms of accuracy as well as in terms of invasiveness. These form an exciting direction for a more comprehensive study of our proposed ideas, which we leave for future work. 
overview: 
 In this section, we describe our approach and put it in context of prior developments.Defining concepts geometrically Our starting point is a geometric notion that concepts live in linear directions in neural representation space, known as linearity of representations (see extensive references in Section 3). To make this precise we assume that for observed data X that has an underlying representation Z with X = f (Z) where the latent variables Z follow an arbitrary distribution and f is a (potentially complicated) nonlinear underlying mixing map. We do not assume that f and Z correspond to a ground truth model or that the latent variables Z themselves are related to a causal model or are interpretable and instead only assume linearity of representations (well supported by prior works). In agreement with this hypothesis we define concepts as affine subspaces AZ = b of the latent space of Zs, i.e., to a concept C we assign an affine hyperplane H C = {Z ∈ R dz : AZ = b} in the embedding space and we say that X = f (Z) satisfies a concept C if Z ∈ H C . In this setting the usual goal of CRL to reconstruct f and Z seems to be unneccesary for many applications. Instead we focus on the more modest goal of identifying only a (small) set of concepts we care about, i.e., we want to be able to decide whether a datapoint X satisfies a concept C. Our main result shows that it is possible to identify n concepts given access to n + 2 concept conditional distributions. We now compare natural assumptions on type of data for causal representation learning and the setting considered here.From interventions to conditioning It is worth contrasting here the difference between viewing a concept as a generic latent generative factor Z i that non-linearly mixes together with other latent factors to yield the inputs X, versus the geometric notion above, as specifying a linear subspace. In the former, the natural way to provide supervision, i.e. define concept distributions, is to simply intervene on a specific factor Z i and set it to a particular value (see Section 3 for references).In the latter however, it is most natural to condition on the concept, i.e., Z ∈ H. This shift is aligned with the growing interest to relax the notion of interventions, and consequently dilute the notion of causality [15, 100, 4] , although it is still open how to properly achieve this. Two key drivers of this trend are as follows. The first is that the number of additional datasets required is d z [46, 71, 53, 14] , which is infeasible in many settings 2 . The second is that the various assumptions that go into these works are often difficult to achieve, such as requiring perfect interventions [111, 14] . Compared to interventional data, conditional data is often easier to acquire, obtained by conditioning on particular values of the latent factors (see also Appendix B.2).We now formalize conditioning on a concept. The obvious approach to define concept conditional distributions is to simply condition on Z ∈ H C , so p C (Z) = p(Z|Z ∈ H C ) where p is a base distribution of Z on R dz . However, this suffers from the drawbacks that it is mathematically subtle to condition on sets of measure 0 and this does not account for inherent noise in the learned representations. Therefore we relax this strict conditioning by drawing inspiration from how data is collected in practice: We sample X from the base distribution and then keep it if it satisfies our concept C. This leads us to define p C (Z) ∝ p(Z)q(Z|C) where q is defined to be the probability that Z is perceived to be in H by the data collector and can be chosen to incorporate noise in our data gathering scheme. Therefore, this can also be viewed from a Bayesian information gathering viewpoint, as well as a stochastic filter standpoint. This is the notion we study in this work (Definition 3) and we develop theoretical techniques to guarantee identifiability in this formulation. Depending on the specific setting other types of conditional distributions might be utilized to describe the available data and we discuss some options in Appendix C.Connection to XAI One important class of applications is human-in-the-loop (HIL) settings, where the hope is to recover human interpretable concepts. Apriori, there is no reason why the high-dimensional underlying latent Zs should be interpretable, particularly for large d z . To make the situation worse, there may be other potentially composite concepts that humans can conjure up; indeed, there is no limit to the number of concepts that humans can conjure, and these almost certainly are not going to correspond to the true Zs. Because of this mismatch between the "true" Zs and interpretable concepts, even if we could recover the Zs, they may not be interpretable to humans and therefore less useful for downstream HIL purposes. The second caveat, and which follows from the first as a soft corollary, is that interventions on these will be much harder to obtain, particularly perfect interventions which require us to isolate the generative mechanisms of the latent factor with the other factors. Furthermore, some concepts are not manipulable, e.g. psychological traits, biological mechanisms, so obtaining interventional data is not possible in these applications. Accordingly, there has been considerable recent work, especially in the explainable AI (XAI) community on extracting human-interpretable concepts from latent generative factors, or more generally from complex foundation model representations [55, 103, 18, 41, 137] . To do so in an identifiable and automated way with as little supervision as possible is a significant open problem. 
related work: 
 Causal representation learning and concept discovery Causal representation learning (CRL) [102, 101] aims to learn generative factors of high-dimensional data. This exciting field has seen significant progress in the last few years [53, 11, 106, 60, 78, 57, 114, 14, 36, 1, 127, 63] . A fundamental perspective in this field is to ensure that the model parameters we attempt to recover are identifiable [53, 25, 129] . We will elaborate more on the connection of our framework to CRL in Appendix B. Concept discovery is an important sub-field of machine learning which extracts human-intepretable concepts from pre-trained models. We do not attempt to list the numerous works in this direction, see e.g., [103, 18, 135, 74, 82, 99, 58, 104, 89, 114] . However, theoretical progress in this direction is relatively limited. The work [63] studies when concepts can be identified provided the non-linear model is known in advance, whereas we show concept identifiability for unknown non-linearity, while simultaneously allowing entangled concepts. Prior works have also attempted to formalize the notion of concepts [130, 85, 103, 61] , however their definitions seem specific to the model and domain under consideration, e.g., [85, 52] focus on binary concepts via large language model representations of counterfactual word pairs, whereas our general concept definitions are applicable to all domains.Linearity of representations Sometimes referred to as the linear representation hypothesis, it is commonly believed that well-trained foundation models in multiple domains learn linear representations of human-interpretable concepts, with experimental evidence going back at least a decade [77, 113, 5] . This has been experimentally observed in computer vision models [90, 94, 8, 31, 55, 19, 130, 120] , language models [77, 87, 5, 22, 117, 30] , large language models [17, 118, 81, 79, 66, 85, 38, 52] , and other intelligent systems [75, 103] . Various works have also attempted to justify why this happens [64, 5, 35, 3, 32, 105] . We take a different angle: Given that this phenomenon has been observed for certain concepts of interest, how does this enable recovery of the concepts themselves? Consequently, our model assumptions are well-founded and our theory applies to multiple domains of wide interest. 
setup: 
 In this section, we provide a formal definition of concepts, which are high-level abstractions present in data. This allows us to develop a theoretical framework for associated data distributions and identifiability theory. For the sake of intuition, we can think of the data as images of different objects and the color of the object as a concept.We assume that the observed data X lies in a space X ⊆ R dx of dimension d x and has an underlying representation X = f (Z) for latent variables Z that lie in a latent concept space R dz of dimension d z . In contrast to most prior works we do not necessarily assume that Z represents the true underlying mechanism that generated the data. Instead we simply assume that the latent representation has the geometric property that it maps certain regions of the observation space to linear subspaces of the latent space (motivated by previous work; see Section 3) . Our first assumption is standard: Assumption 1 (Mixing function). The non-linear f is injective and differentiable.We make no additional assumptions on f : The map from Z → X can be arbitrarily non-linear.We now define concepts living in the latent space R dz . Before presenting the general definition of multidimensional concepts, we outline the basic ideas in the simplified setting of a one-dimensional concept. Consider the color "red" as a concept. Different images have different levels of "redness" in them, so this concept is measured on a continuous scale, represented by a valuation b ∈ R. An (atomic) concept is then represented by a vector a ∈ R dz such that ⟨a, Z⟩ = ⟨a, f -1 (X)⟩ encodes the "value" of the concept in X, as measured in the latent space. More precisely, for a given valuation b ∈ R, the set of all observations X that satisfy this concept is given by {X = f (Z)|⟨a, Z⟩ = b}. For instance, for an object in an image X, if a ∈ R dz is the concept of red color, b ∈ R could indicate the intensity; then all datapoints X satisfying this concept, i.e., all images with an object that has color red with intensity b, can be characterized as X = f (Z) where Z satisfies ⟨a, Z⟩ = b. For a 3D visualization, see Fig. 1 We make this intuition formal below. Definition 1 (Concepts). A concept C is a linear transformation A : R dz → R d C . The dimension of the concept will be denoted by dim(C) = d C . A valuation is a vector b ∈ R d C and we say that a datapoint X satisfies the concept C with valuation b if AZ = b where Z = f -1 (X).In this work, we are interested in learning a collection of m concepts C 1 , . . . , C m from observed data. By left multiplying by the pseudo-inverse A + , we can equivalently assume A is a projector matrix. However, the current definition is more suitable for embeddings of real models.When we talk of learning concepts C, we are in particular interested in learning the evaluation map Af -1 (x). This is a more modest objective than learning the entire map f which is the usual goal in, say, CRL. While the latter typically requires stringent assumptions, in particular Ω(d z ) environments are necessary, our weaker identifiability results only need O(d C ) ≪ O(d z ) environments. To simplify our analysis, we make use of the following definition: Definition 2 (Atoms). An atom (short for atomic concept) is any concept C with dim(C) = 1.The idea is that we can view each concept as being composed of atomic concepts in the following sense: Atomic concepts are fundamental concepts that live in a space of co-dimension 1 in latent space, and thus are equivalently defined by vectors a ∈ R dz . For example, concepts such red color, size of object, etc., may be atomic concepts. Any generic concept is then composed of a collection of atomic concepts, e.g., the concept C of all small dark red objects will correspond to dim(C) = 2 with row 1 corresponding to the atomic concept of red color with large valuation (dark red objects) and row 2 corresponding to the atomic concept of object size with low valuation (small objects).We now define the distributions of datasets over concepts. We will predominantly work with distributions of Z over R dz , as the resulting distribution of X = f (Z) over R dx can be obtained via a simple change of variables.To build intuition, consider the case where we first collect a base dataset with some underlying distribution and then collect concept datasets via filtering. For instance, we could first collect a set of images of all objects and then, to collect a dataset of dark red colored objects, we filter them to only keep images of dark red colored objects. We call the former the base distribution and the latter the concept conditional distribution corresponding to our concept. Fix a nonlinearity f . We assume that the base data distribution is the distribution of X = f (Z) with Z ∼ p, where p is the underlying distribution on R dz . In what follows, we will abuse notation and use p for both the distribution and the corresponding probability density which we assume exists. We make no further assumptions on p since we do not wish to model the collection of real-life datasets that have been collected from nature and which could be very arbitrary.We now define the concept conditional distribution, which is a distribution over X that is induced by noisy observations of a particular concept at a particular valuation. Formally, assume we want to condition on some atomic concept a ∈ R dz with valuation b. It is reasonable to assume that this conditioning is a noisy operation. For instance, humans are great at distilling concepts from noisy images, e.g., they recognize cars in a misty environment. We formalize this by assuming that data collection is based on a noisy estimate b = ⟨a, z⟩ + ϵ where ϵ is independent of z and its density is a symmetric distribution with density q(ϵ). Then we consider the distributionEQUATIONwhere we used Bayes theorem in the last step. This definition directly extends to higher dimensional concepts which are concisely defined as follows.Definition 3 (Concept conditional distribution). For a concept C with associated linear map A and an arbitrary valuation b ∈ R dim(C) , we define the concept conditional distribution to be the set of observations X respecting this concept, which is defined as the distribution of X = f (Z) whereEQUATIONThis is by no means the only possible definition, and we present feasible alternate definitions in Appendix C. We remark that our formulation is related to the iVAE setting [53] and the auxiliary variable setting for identifiable ICA in Hyvarinen et al. [48] and we discuss the relation later.The majority of recent identifiability results relied on interventional data while we only consider conditional information here.We are ready to define our main problem of interest. Problem 1. We are given an observational dataset X 0 = f (Z 0 ) corresponding to the latent base distribution p along with datasets X 1 , . . . , X m from m environments corresponding to concept conditional datasets for different concepts C 1 , . . . , C m and corresponding valuations b 1 , . . . , b m over the same latent space R dz with the same mixing f . Under what conditions (and up to which symmetries) can we learn the concepts C 1 , . . . , C m , which includes the linear maps A 1 , . . . , A m , and the concept valuationsA 1 f -1 (x), . . . , A m f -1 (x)?Toward this end, a fundamental question is whether this problem is even possible, i.e., whether it is well-defined. This is known as the question of identifiability [53, 25, 129, 57, 43] . Therefore, we make the following definition. Informally, for the setting above, we say that the concepts (C 1 , A 1 ), . . . , (C m , A m ) with associated nonlinearity f are identifiable (and thus learnable) if for any other collection of different parameters that fit the data, they are linearly related to the true parameters. Definition 4 (Identifiability). Given datasets X 0 , X 1 , . . . , X m corresponding to the observational distribution and m concepts C 1 , . . . , C m with underlying latent base distribution p on R dz , nonlinearity f , linear maps A 1 , . . . , A m and valuations b 1 , . . . , b m , we say the concepts are identifiable if the following holds: Consider any different collection of parameters f , d z , p, concepts ( C 1 , A 1 ), . . . , ( C m , A m ) and valuations b 1 , . . . , b m that also generate the same observations X 0 , X 1 , . . . , X m . Then there exists a shift w ∈ R dz , permutation matrices P e and invertible diagonal matrices Λ e such that for all e and x, A e f -1 (x) = Λ e P e A e (f -1 (x) + w),i.e., we can evaluate the concept evaluations on the data up to linear reparametrizations. Moreover, there exists a linear map T : R dz → R dz such that the concepts and their evaluations satisfy Identifiability implies we can identify the nonlinear map f -1 within the span of the subspace of the concepts of interest, and therefore we can recover the concepts of interest from our data. That is, if certain concepts are identifiable, then we will be able to learn these concept representations up to linearity, even if they can be highly nonlinear functions of our data. Such concept discovery is useful because they can then be used for further downstream tasks such as controllable generative modeling.We emphasize that in contrast to previous work we are not aiming to identify f completely and indeed, no stronger identifiability results on f can be expected. First, we cannot hope to resolve the linear transformation ambiguity because the latent space is not directly observed. In other words, a concept evaluation can be defined either as ⟨a, Z⟩ or as ⟨T a, T -⊤ Z⟩ for an invertible linear map T . For the purposes of downstream tasks, however, this is fine since the learned concepts will still be the same. Second, we cannot expect to recover f -1 outside the span of the concepts because we do not manipulate the linear spaces outside the span therefore we do not learn this information from our observed data so this is also tight. The permutation matrix captures the fact that the ordering of the concepts does not matter. Therefore, this definition captures the most general identifiability guarantee that we can hope for in our setting and furthermore, this suffices for downstream tasks such as controllable data generation.Because we will only be interested in recovering the set of concepts up to linear transformations, without loss of generality, we will fix the base collection of atomic concepts. That is, we assume that each concept C e (where 1 ≤ e ≤ m indexes the environment) corresponds to a linear map A e whose rows are a subset of C, where C = {a 1 , . . . , a n } is a set of atomic concepts that we wish to learn. Moreover, we assume that they are linearly independent, since we want them to encode distinct concepts. This is formalized as follows. Assumption 2. There exists a set of atomic concepts C = {a 1 , . . . , a n } of linearly independent vectors such that for each concept C e under consideration the rows of the concept matrix A e are contained in C, i.e., (A e ) t e i ∈ C. We denote the indices of the subset of C that appear as rows of A e by S e and we assume that all concepts in C appear in some environment e (where an environment corresponds to a concept conditional distribution), i.e., e S e = [n]. Remark 1. Definition 4 implies that the atoms can be identified in the sense that there is a permutation π ∈ S n and λ i ̸ = 0 such that for T as in Definition 4 and some λ iEQUATIONi.e., we can evaluate the valuations of the atomic concepts up to linear reparametrization. 
main result: 
 In this section, we present our main result on identifying concepts from data. The punchline is that when we have rich datasets, i.e., sufficiently rich concept conditional datasets, then we can recover the concepts. Crucially, we only require a number of datasets that depends only on the number of atoms n we wish to learn (in fact, O(n) datasets), and not on the underlying latent dimension d z of the true generative process. This is a significant departure from many existing works, since the true underlying generative process could have d z = 1000, say, whereas we may be interested to learn only n = 5 concepts, say. In this case, approaches based on CRL necessitate at least ∼ 1000 interventional datasets, whereas we show that ∼ n + 2 = 7 conditional datasets are enough if we only want to learn the n atomic concepts. We will explain the connection to CRL in Appendix B. Let us now discuss our main assumptions. Assumption 3. The noise distribution q is Gaussian, i.e. q ∼ N (0, σ 2 ) for some σ 2 > 0.We choose Gaussian noise since it is a conventional modeling choice. However, it would be feasible to consider other noise families and we expect similar results to hold (albeit with modified proof techniques). We now relate the concepts C e to the atoms. Recall that we defined the index sets S e = {i ∈ [n] : a i ∈ C is a row of A e } of atomic concepts in environment e.We define the environment-concept matrix M ∈ R m×n indexed by environments and atoms byEQUATIONSimilarly, we consider the environment-valuation matrix B ∈ R m×n given byEQUATION)Our first assumption ensures that the concept conditional distributions are sufficiently diverse. Assumption 4 (Environment diversity I). The environment-concept matrix M ∈ R m×n has rank n and there is a vector v ∈ R m such that v ⊤ M = 0 and all entries of v ⊤ B are non-zero where B denotes the environment-valuation matrix.We remark that this assumption can only hold for m ≥ n + 1 and indeed is satisfied under mild assumptions on the environments if m = n + 1, as the following lemma shows. Note that the condition on B ensures that the concept valuations b e k are not equal for all environments e which would prevent identifiability. Lemma 1. Assumption 4 is satisfied almost-surely if there are n + 1 concept conditional distributions such that every n rows of the environment-concept matrix are linearly independent and the b e are drawn independently according to a continuous distribution.We also assume one additional diversity condition. To motivate this, observe if two concepts always occur together, it's information-theoretically impossible to distinguish them, e.g., if an agent only sees red large objects (i.e. all red objects are large and all large objects are red), it will be unable to disambiguate the "red" concept from the "large" concept. Therefore, we make the following assumption. Assumption 5 (Environment diversity II). For every pair of atoms a i and a j with i ̸ = j there is an environment e such that i ∈ S e and j / ∈ S e .We remark that these are the only assumptions about the sets S e . In particular, we do not need to know the sets S e . In the proof, we will extract these sets based on a the signatures they leave on the datasets. We can now state our main result. Theorem 1. Suppose we are given m context conditional datasets X 1 , . . . , X m and the observational dataset X 0 such that Assumptions 1-5 hold. Then the concepts are identifiable as in Definition 4. Remark 2. Assumption 4 can only be satisfied for m ≥ n + 1, i.e., the result requires at least n + 2 environments. On the other hand, Lemma 1 assures that n + 2 environments are typically sufficient. We expect that the result could be slightly improved by showing identifiability for n + 1 environments under suitable assumptions. However, this would probably require more advanced techniques from algebraic statistics [28] compared to the techniques we employ here.As mentioned before, our setting somewhat resembles the iVAE setting in Khemakhem et al. [53] and therefore, their proof techniques can also be applied, with several modifications, to derive identifiability results in our setting (however our formulation and application are very different). However, this approach will require more environments because their main assumption is that the matrix Λ = (M, B) ∈ R m×2n has rank 2n so that 2n + 1 environments are necessary. Moreover, this rank condition is much stronger than Assumption 4. For completeness and as a warm-up we prove this result in Appendix A. The full proof of Theorem 1 is fairly involved and is deferred to Appendix A. 
experiments: 
 In this section, we present experiments to validate and utilize our framework. We first verify our results on synthetic data, via a contrastive learning algorithm for concept learning. Then, we focus on experiments involving real-world settings, in particular on image data using multimodal CLIP models and text data using large language models (LLMs).End-to-end Contrastive learning algorithm and Synthetic experiments We validate our framework on synthetic data as follows. We sample the base distribution from a Gaussian Mixture model and experiment with both linear and nonlinear mixing functions (details deferred to Appendix G).The number of concepts n is intentionally chosen to be less than the ground truth dimension d z and the number of concept condtional distributions is m = n + 1 as per our theory. Inspired by [14] , we use a contrastive learning algorithm to extract the concepts. Here we sketch the key ideas of the algorithm and defer details to Appendix F.The core idea for the algorithm is as follows. For each concept conditional distribution X e , we train a neural network to distinguish concept samples x ∼ X e from base samples x ∼ X 0 . Under our assumptions the log-odds of these two distributions have the following simple parametric form in terms of the environment-concept matrix and the environment-valuation matrixEQUATIONfor some constants c e (see Lemma 3 for the precise statement).Then, to learn the n atomic concepts up to linearity, we build a neural architecture for this classification problem with the final layer mimicking the log-odds expression above, which can then be trained end-to-end. Because of the careful parametrization of the last layer, this will encourage the model to learn the representations as guaranteed by our results. We can assume without loss of generality that the concept vectors we learn are the first coordinate vectors because concepts are only identifiable up to linear transformations (see Definition 4) . In other words, we consider an encoder neural network h θ with parameters θ and the valuation of atomic concept i is h θ i . Therefore, for each environment e, we train classifiers of the formEQUATIONusing standard cross-entropy loss, where α e , β e k , γ e k are the parameter of the last layer and θ parametrizes the decoder. In Table 1 , we report the R 2 and Mean Correlation Coefficient (MCC) metrics [53, 54] with respect to the ground truth concept valuations. In addition we provide results for larger values of d x and d z in Table 7 in Appendix G. There are no baselines since we are in a novel setting, but our metrics are comparable to and often surpass what's usually reported in such highly nonlinear settings [132, 14] . We remark that variations of the contrastive method can be designed for harder synthetic settings and different problems related to concept discovery. However, we will move onto real-life data experiments next.Probing the theory on multimodal CLIP models A real world example that approximately matches the setting considered in this paper is the training of the multimodal CLIP models [92] . They are trained by aligning the embeddings of images and their captions. We can view the caption as an indicator of the concepts present in the image. Thus the data provides access to several concept conditional distributions such as the collection of all images having the label 'A dog', but also to more complex distributions consisting of more than one atomic concept such as images labeled 'A red flower'. We embed images from the 3d-Shapes Dataset [16] with known factors of variation into the latent space of two different pretrained CLIP models. Using logistic regression we learn atomic concepts for each of the factors of variations (see Appendix D.1 for details) and then evaluate the concept valuations of the learned atomic concept on held out images. We show the results for the shape attribute in Figure 2 (further results are in Appendix D.2). The results show that there are indeed linear subspaces of the embeddings space that represent certain concepts. Moreover, the learned valuations for different models are approximately linearly related as predicted by Theorem 1. We emphasize that, while these observations highlight the relevance of the theory developed in this paper, they do not explain the behavior of CLIP models, as their training objective does not directly align with our theoretical framework.Alignment of LLMs Finally, we show an application of our framework to interpret representations of LLMs and improve alignment techniques. In particular, we exploit our ideas to improve the Cube Cylinder Ball Ellipsoid Shape Inference-Time Intervention technique [66] to promote LLMs to be more truthful, i.e. the downstream task is to take pre-trained LLMs and during inference, change the valuation of the truthfulness concept from false to true, without affecting any other orthogonal concepts. Motivated by our framework, we propose to replace steering vectors by steering matrices for better alignment. Experiments on LLaMA [119] show an improvement of the TruthfulQA dataset [68] accuracy. Additional details, including a self-contained introduction to large language models (LLMs) and the Inference-Time Intervention (ITI) technique are deferred to Appendix E. 
conclusion: 
 In this work, we study the problem of extracting concepts from data, inspired by techniques from causal representation learning. This is an approach to bridge identifiability theory in (causal) representation learning and the field of concept-based learning. For this, we geometrically define concepts as linear subspaces, well-supported via extensive empirical literature. With this formal definition of concepts, we study under what conditions they can be provably recovered from data. Our rigorous results show that this is possible under the presence of only conditional data,requiring far fewer distributions than the underlying latent dimension. Finally, synthetic experiments, multimodal CLIP experiments and LLM alignment experiments verify and showcase the utility of our ideas.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Please see Tables 1 and 6 . Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: The focus of the paper is theoretical and no immediate societal impact is expected.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: No models or data are released. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: See Appendices on experimental details. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In this appendix we provide the proofs of our results, in particular the proof of our main result, Theorem 1. However, as a warm-up we first start in Appendix A.1 with a proof of the simpler result that can be shown based on the iVAE approach. In Appendix A.2 we prove Theorem 1 and in Appendix A.3 we prove the additional lemmas that appear in the paper.As a warm-up and to provide a connection to earlier results we show here how to obtain identifiability by adapting the iVAE framework to our context. Indeed, our mathematical setting is related to the setting used in [53] in the sense that the environments are generated by modulation with certain exponential families. Therefore, we can essentially apply their proof techniques to prove identifiability (with some modifications), albeit this requires the suboptimal number of 2m + 1 environments (there are two sufficient statistics for the Gaussian distribution). Theorem 2. Suppose data satisfies Assumption 1, 2, and 3 and the environment statistics matrix Λ has rank 2n. Assume we know the number of atoms n. Then identifiability in the sense of Definition 4 holds.We remark that the rank condition can only be satisfied for 2n + 1 environments (observational distribution and 2n concept conditional distributions. For this theorem the assumption that the filtering distribution is always the same is not necessary. Instead we could consider variances (σ e k ) 2 depending on environment e and row k, i.e., the filtering distribution q (σ e k ) 2 is Gaussian with varying variance. The generalization of the environment-concept matrix M ∈ R m×n is given byEQUATIONSimilarly the generalization of the environment-valuation matrix B ∈ R m×n is given byEQUATION)We now prove Theorem 2. We use essentially the same ideas as in the proof of Theorem 1 in Khemakhem et al. [53] (followed by the same reasoning as in Sorrenson et al. [108] , Kivva et al. [57] but since our concepts are not axis aligned and we only extract some information about the mixing we give a complete proof.Proof of Theorem 2. Suppose there are 2 sets of parameters that generate the same data X 0 , X 1 , . . . , X m . Denote by . the latter set of parameters, e.g., X e is distributed as f ( Z e ) where Z e ∈ R dz corresponds to the concept class C e with distribution Z e ∼ p e and the same distribution is generated by f (Z e ) where f and f are injective and differentiable. Let C = {a 1 , . . . , a n } be the set of atomic concepts in the first setting and let C = { a 1 , . . . , a n } be the set of atomic concepts in the second setting (here we use that n is assumed to be known). We also consider the transition function φ = f -1 fEQUATIONHere we used the environment-concept matrix and the environment-valuation matrix in the second step which were defined in ( 7) and (8) (in ( 11) and ( 12) for varying variance). We define the vector p(Z) with components p e (Z) = ln(p(Z)) -ln(p e (Z)). Then we find the relationEQUATIONTogether with (14) we conclude thatEQUATIONSince by assumption Λ = ( M , B) ∈ R m×2n has rank 2n there is a vector v such that v ⊤ M = 0 and v ⊤ B = -e i (e i ∈ R dz denotes the i-th standard basis vector). Thus we find thatEQUATIONIn other words ⟨ a i , Z⟩ can be expressed as a quadratic polynomial in Z. We apply the same reasoning for ⟨ a i , Z⟩ 2 , i.e., pick a vector v ′ such that 1 2 v ′⊤ M = e i and v ′⊤ B = 0 to obtain a relationEQUATIONfor some coefficients η j and some affine function ℓ of Z. The following reasoning is now the same as in Kivva et al. [57] , Sorrenson et al. [108] . We thus find that ⟨ a i , Z⟩ and its square can be written as polynimials of degree at most 2 in Z. This implies that in fact ⟨ a i , Z⟩ is an affine function of Z (otherwise its square would be a quartic polynomial), i.e., we can writeEQUATION)Equating the square of this relation with (19) and taking the gradient with respect to Z (as a polynomial the function is differentiable) we findEQUATIONfor two vectors w and w ′ . The equality (for Z = 0) implies w = w ′ . Now linear independence of a j implies that for each r η r a r = λ r j λ j a j .Applying linear independence again we conclude that either λ r = 0 or λ j = 0 for all j ̸ = r. This implies that there is at most one r such that λ r ̸ = 0. The relation (20) and the bijectivity of φ implies that there is exactly on r(i) such that λ r(i) ̸ = 0 and thereforeEQUATION)Applying the same argument in the reverse direction we conclude that there is a permutation π ∈ S n such thatEQUATIONBy linear independence we can find an invertible linear map T such thatEQUATION(i.e, T ⊤ a π(i) = a i ) and a vector w ∈ R dz (the a i are linearly independent) such thatEQUATIONIn particular the relations ( 5) and ( 6) hold. Now it is straightforward to see that if i ∈ S e , i.e., a i is a row of A e then a π(i) is a row of A e and vice versa. Indeed, this follows from (17) for environment e together with (26) and linear independence of the atoms. Therefore we conclude from ( 25) that there is a permutation P e such thatEQUATIONMoreover, (26) then implies settingEQUATIONholds for the same permutation matrix P e and a diagonal matrix Λ e whose diagonal entries can be related to (26) . Let us assume now that row k of A e is a i and row k ′ of A e is a π(i) . Now we consider the subspace H ⊂ R dz containing all Z such that ⟨Z, a j ⟩ = 0 for j ̸ = i. Via (26) this implies that ⟨ a j , Z⟩ is constant for j ̸ = π(i). Then we conclude from ( 17) that for Z ∈ HEQUATION)for some constant c e k . Using (26) this implies thatEQUATIONComparing the quadratic term and the linear term (note that ⟨a i , Z⟩ can take any value on H) we findEQUATIONEQUATIONCombining the equation we obtainEQUATIONThis implies then the relationEQUATIONA.2 Proof of Theorem 1In this section we prove our main Theorem 1. The proof is structured in several steps: First we remove the symmetries of the representation and derive the key relations underlying the proof. Then we show that we can identify the environment-concept matrix M and then also the valuations collected in B.Once this is done we can complete the proof. We will need the following lemma to conclude the proof. Lemma 2. The relations (3) and (6) in Definition 4 define an equivalence relation of representations if we assume that the underlying atoms form a linearly independent set.The proof of this lemma can be found in Appendix A.3. Remark 3. Without the assumption on the underlying atoms the lemma is not true. In this case a slightly different scaling must be chosen (e.g., (Λ e ) -1 b e = Λ e P e b e -P e A e w instead of b e = Λ e P e (b e -A e w)). Since our results address the case of atoms we used the simpler definition in the main paper.We can allow slightly more general filtering distributions where q is Gaussian with variance σ 2 i if we filter on concept i, i.e., the variance needs to be constant for different environments and the same atom but might depend on the atom. The proof will cover this case, the simple case stated in the main paper is obtained by setting σ 2 i = σ 2 . Some steps of the proof (e.g., the expressions for the difference of the log-densities) agree with the proof of Theorem 2. To keep the proof self contained we repeat a few equations.Proof of Theorem 1. We proceed in several steps.Step 1: Reduction to standard form. Let us first transform every possible data representation into a standard form. Recall that we have the set of atomic concepts C = {a 1 , . . . , a n }. Recall that we defined the environment-concept matrix M ∈ R m×n in (7) and note that the natural generalisation readsEQUATION)We say that concept a n is conditioned on the environment e. Note that the nonzero entries of row e of M encode the set S e . To pass from A e to its rows a i we assume that the e-th row of A e is a i e j , i.e., a i e j = (A e ) ⊤ e j . Recall also consider the environment-valuation matrix B which is given byEQUATION)Denoting by q σ 2 the centered Gaussian distribution with variance σ 2 we find in environment eEQUATIONNow we consider an invertible linear map T : R dz → R dz such that T -⊤ a i = e i for all 1 ≤ i ≤ n. Such a map exists because we assume that the a i are linearly independent. Moreover, we consider a shift vector λ ∈ R dz with λ i = 0 for i > n which we fix later. We define Σ ∈ R dz×dz to be the diagonal matrix with entries Σ ii = σ i for 1 ≤ i ≤ n and Σ ii = 1 for i > n. Now we consider the linear map L(z) = Σ -1 T z -λ and a new representation given byEQUATIONWe also defineEQUATION)Define M and B in terms of A e , b e and σ 2 i as before. We remark that all entries of M are either 0 or 1 and note thatEQUATIONEQUATIONWe claim that this model generates the same observations as the original model. By definition L * p = p (as mentioned before, we slightly abuse notation and here refer to the distributions). Next, we calculate for any δEQUATIONUsing this for δ = b e k and some k such that row k of A e is a i we findEQUATIONThis then implies that for z = L(z)EQUATION)Combining this with the definition (2) and the definition p( zEQUATIONand thus f (Z e )D = f (Z e ) D= X e . Moreover, one directly sees that the two representations are also equivalent in the sense of Definition 4. We now fix the vector λ such that each row of B has mean zero. Finally, by changing the sign of z i we can in addition assume that for every i the first non-zero B ei is positive. Finally we remark that Assumption 4 is still satisfied for M and B. Indeed, w ⊤ M = 0 implies w ⊤ M = 0 by (40) . But then w ⊤ B = w ⊤ BDiag(σ -1 1 , . . . , σ -1 n ) by ( 41) which has all entries different from zero if this holds for w ⊤ B. In the following we will therefore always assume that the representation satisfies the properties of the Z variables and we remove the modifier in the following. The plan is now to show that M and B can be identified up to permutations of the rows (under the fixed normalization we derived in this step) and then show that every two representations with the same M and B can be identified.Step 2: The key identity Let us here restate the key identity based on the difference of the logdensities. As is common in identifiability results for multi-environment data with general mixing we consider the difference in log densities. ConsiderEQUATION)for some constant c ′ e . Those functions will play a crucial role in the following and we will denote g e (z) = ln p 0 (z) -ln p e (z)Note that since the log-density changes only by the Jacobian for pushforward measures we find thatEQUATIONNote that the functions G e (x) can be estimated from the distributions of X e . We remark X might be supported on a submanifold if d z and d x do not agree making the definition of the density subtle. But we can just consider any chart locally and consider the density of the pushforward with respect to the Lebesgue measure. The resulting difference expressed in G e will be independent of the chart as the determinant cancels thus G e is a well defined function. The relationEQUATIONwill be crucial in the following because it shows that properties of g e are closely linked to the identifiable functions G e .Step Note that the function g e are convex functions, and they can be decomposed as sums of functions in z i , i.e., for some functions h T i e∈TEQUATIONNow if i ∈ S T then i ∈ S e for some e and thus M ei ̸ = 0 for the e and h T i is the sum of quadratic function in x i which as a strictly convex function has a unique minimum z T i . On the other hand, if i / ∈ S T then i / ∈ S e for e ∈ T and thus M ei = 0 for all e ∈ T and h T i (z i ) = 0. Thus we conclude that Thus, we can and will assume after permuting the atomic concepts that M is some fixed matrix.EQUATIONStep 4: Identifiability of concept valuations Next, we show that we can also identify the matrix B. We do this column by column, i.e., for one atomic concept after another. Assume we consider atomic concept i. Then we consider the set T i = {e : M ei = 0} of concepts that not filter on atomic concept i. By Assumption 5 there is for every i ′ ̸ = i an environment e such that i ′ is filtered on, i.e., M ei ′ ̸ = 0. This implies S Ti = [n] \ {i}. Then we consider as in (52) the set I Ti given byEQUATIONNote that all z i ′ for i ̸ = i ′ are constant on I Ti . Thus we find for any environment e such that i ∈ S e .EQUATIONon I Ti for some constant c Ti . Now we consider two concepts e 1 ̸ = e 2 such that atomic concept i is contained in these two environments. Then we consider the setEQUATION)Note that in the second equality we used that g e1 (z) depends on z i through z 2 i /2 -Be 1 iz i so it is minimized at B e1i . Now we find using (55)EQUATION)As before, this quantity is identifiable from observations because f (T i ) can be identified and we can minimize G e2 (x) over f (T i ).This allows us to identify B e1i -B e2i up to a sign. However, we can evaluate this expression over all pairs e 1 and e 2 and pick the one with the maximal difference. Then all remaining values B ei for e such that i is filtered on in e must satisfy B ei ∈ [B e1i , B e2i ]. Together with identifiability of |B ei -B e1i | this allows us to identify all B ei up to one sign indeterminacy and a constant shift. However, in the first step we ensured that e B ei = 0 for all i which determines the shift and the sign is fixed by our choice of making the first non-zero entry positive. Thus, we can assume that our two representations have the same M and B.Step 5: Identifiability of concepts We are now ready to prove our identifiability result.Assume we have two representations Z e , f , p and Z e , f , and p such that the corresponding environment-concept and environment-valuation matrices agree, i.e., M = M and B = B. We consider the transition function φ = f -1 • f which is by assumption differentiable. What we want to show is that φ(z) i = z i for all z ∈ R dz and 1 ≤ i ≤ n. We now decompose z = (z c , z o ) into the concept part and the orthogonal part. We fix z o ∈ R dz-n and define the functionι o (z c ) = (z c , z o ), the projection π c ((z c , z o )) = z c , and φ o : R n → R n given by φ o (z c ) i = φ(ι o (z c ) = φ((z c , z o )) i .Note that φ o is differentiable but not necessarily injective. Let us denote by g : R dz → R m the function with coordinates g e = g e and similarly we define G : M → R d . Identifiability will be based on the crucial relationEQUATIONHere we used in the last step that g e is defined in terms of M and B and thus agrees for both representations. Note that g is just a quadratic function. Differentiating we obtainEQUATIONConcisely this can be written asEQUATIONDifferentiating ( 58) we findEQUATIONLet v be a vector as in Assumption 4. Denote by M + ∈ R n×m the pseudoinverse of M which has rank n because M has. We consider the matrix M + ∈ R n+1×m given byEQUATIONLet us multiply the relation ( 61) by M + and find that    z 1 0 . . . 0 z n 0 . . . 0     -M + B =         z 1 0 . . . 0 z n 0 . . . 0     -M + B     Dφ o (z c ) (63)Note that the first n rows of the left hand side are Diag(z 1 , . . . , z n ) -M + B. This matrix is invertible for almost all values of z c = (z 1 , . . . , z n ) ⊤ because its determinant is a non-zero polynomial (the coefficient of the term z 1 • . . . z n is 1) which vanishes only on a set of measure zero. Outside of this set the left hand side of has rank n. Then the equality (63) implies that also the right hand side has rank n and thus Dφ o (z c ) has rank n and thus is invertible. For z c outside of this set there is up to scaling a unique vector w ̸ = 0 (depending on z 1 , . . . , z n such thatw ⊤         z 1 0 . . . 0 z n 0 . . . 0     -M + B     = 0 (64)From (63) we conclude using the invertibility of Dφ o (z c ) thatw ⊤         z 1 0 . . . 0 z n 0 . . . 0     -M + B     = 0. (65)Next, we claim that for almost all values of z c the vector w has all entries different from 0 (this property is invariant under rescaling). Actually we need this only for entries 1 to n but the case n + 1 is a bit simpler so we show it first. We show this by proving that for each entry w i there is only a null set of z c such that w i = 0. Let w = (w ′ , 0) for some w ′ ∈ R n and w ′ ̸ = 0, i.e., w n+1 = 0. ThenEQUATIONBut this implies that Diag(z 1 , . . . , z n ) -M + B has non-trivial kernel, i.e., does not have full rank and we have seen above that this happens only for a subset of measure 0 of all z c . Next we show that the same is true if w 1 = 0. Decompose 0 ̸ = w = (0, w ′ ). Then we findEQUATIONThus we conclude that the matrix on the right hand side is not invertible. Its determinant is a polynomial in z 2 , . . . , z n and its highest degree term is ±z2 • . . . • z n • ( M + B) (n+1),1. By definition of M + B we find ( M + B) (n+1),1 = (v ⊤ B) 1 ̸ = 0 by Assumption 4 (recall that we showed invariance of the assumption under the transformation of M and B). We find that the determinant is a non-zero polynomial and the set of its zeros is a set of measure 0 of all z 2 , . . . , z n but since it does not depend on z 1 this holds true for almost all z c . The same reasoning for i = 2, . . . , n implies that for every i the set of z c such that w i = 0 is a set of measure zero. We have therefore shown that for almost all z c the rank of the left hand side of ( 63) is n and the corresponding vector w ̸ = 0 has all entries different from zero. Subtracting ( 64) and ( 65) we obtain0 = w ⊤     z 1 0 . . . 0 z n 0 . . . 0     -w ⊤     z 1 0 . . . 0 z n 0 . . . 0     = (w 1 (z 1 -z 1 ), . . . w n (z n -z n ), 0) . (68)Now w i ̸ = 0 implies z i = z i . We conclude that for almost all z c the relation φ o (z c ) = z c holds. By continuity this implies that the relation actually holds everywhere. We conclude that π c f -1 f ((z c , z o )) = z c for a fixed z o but since z o was arbitrary the relation holds for all z o and all z c . Thus we conclude that forEQUATIONholds. This implies that those two representations satisfy (3) and (4) (with P e = Λ e = Id and T = Id). But since this relation is an equivalence relation in our setting by Lemma 2 and since we showed equivalence to a representation in standard form in the first step we conclude that also any two representations are related through (3) and (4) thus finishing the proof.Here we prove the remaining auxiliary results.Proof of Lemma 1. Since M ∈ R m×n has rank n and m = n + 1 there is exactly one vector v ∈ R m such that v ⊤ M = 0 and v ̸ = 0. We claim that this vector has all entries different from zero.Indeed suppose v m = 0 which then implies v ⊤ 1:(m-1) M 1:(m-1) = 0. But by assumption every n × n submatrix of M is invertible (this is equivalent to the rows being linearly independent) so we conclude that v 1:(m-1) = 0 which is a contradiction to v ̸ = 0. The same reasoning applies to every entry. Note that the assumption on M implies that every column has at least one non-zero entry, i.e., every column of B has one entry sampled from a continuous distribution. But then the probability that v is orthogonal to a column is zero because this is a codimension 1 hyperplane of all valuations of this row (since all entries of v are non-zero).Proof of Lemma 2. Reflexivity is obvious, just pick T = Id, w = 0, Λ e = P e = Id dim(C e ) . To show symmetry we first consider the atoms. Let T = T -1 and π = π -1 . ThenEQUATIONLet w be a vector such that for all 1 ≤ i ≤ nEQUATIONSuch a vector exists by linear independence of a i . Let λ i = λ -1 π(i) . Then we find that the relation ( 6), namelyEQUATIONimpliesEQUATIONIt remains to be shown that this lifts to the concepts C e . We first note that the relation ( 6) together with ( 71) and (3) implies that Λ e P e A e w = -A e w.Let P e = (P e ) -1 and Λ e = (P e ) -1 (Λ e ) -1 P e . Then (3) combined with the previous disply impliesA e f -1 (x) = (P e ) -1 (Λ e ) -1 A e f -1 (x) -A e w= Λ e P e A e f -1 (x) + (P e ) -1 (Λ e ) -1 A w = Λ e P e A e ( f -1 (x) + w).The relationEQUATIONis a direct consequence of the definitions of P e and T and ( 4) and the relationEQUATIONfollows exactly as in (75) . The proof of transitivity is similar (first establish the relations on the atomic concepts then lift it to C e ).In this appendix we describe causal representation learning and discuss the similarities and differences between the viewpoint taken in this paper and the standard setting in causal representation learning.Causal Representation Learning (CRL) [102, 101] aims to learn representations of data that correspond to true causal generative processes. More precisely, if we assume that data X is generated as X = f (Z) where Z are latent causal factors and f is some arbitrary nonlinearity, the goal is to learn f as well as the distribution of Z. Since the latent variables Z are assumed to have causal relationships among them, many works exploit the presence of interventional data to learn the generative model. CRL incorporates ideas from the field of causality [109, 86, 88, 95, 110] into the field of latent variable models and is a generalization of nonlinear independent component analysis [21, 45, 47] and disentangled representation learning [9, 88, 62] . The field has seen a surge of advances in the last few years, e.g., [53, 56, 33, 70, 60, 13, 78, 142, 36, 96, 123, 50, 49, 115, 124, 136, 133] .As motivated in Schölkopf et al. [102] , CRL enables many desiderata such as robustness, out of distribution generalization, and in addition enables planning and alignment. CRL has also been successful in many domains such as computer vision [53, 126, 2] , robotics [73, 11, 69, 140] and genomics [111, 139] .In our work, we take significant inspiration from this framework of causal representation learning and present a relaxed framework that is weaker, but more general and also importantly, aligns better with empirical works on interpretability of large pre-trained models in the literature. We now describe the setup of CRL more formally in Appendix B.1. Then, in Appendix B.2, we discuss conceptual differences between causal representation learning and our framework.We assume that we observe data X ∈ R dx with the generative model X = f (Z) where Z ∈ R dz are the latent variables and f is a deterministic mixing function. The dataset X is sampled from a distribution p and the goal is to recover the mixing function f as well as the distributions of the underlying latent variables Z 1 , . . . , Z dz . To this end, this problem is over-parameterized since multiple pairs of Z and f could fit the dataset apriori, so the common practice in CRL is to impose various assumptions that will make this model identifiable. Here, identifiability is the notion that a unique set of parameters fit the model (up to trivial transformations). This makes the problem well-defined and feasible, although it could still be a hard problem to solve in practice. Below, we informally summarize two classes of prior works that enable such identifiability guarantees.1. Disentangled representation learning: In this setting, we assume that the distributions of Z 1 , . . . , Z dz are jointly independent. Different studies constrain the distribution of the variables Z 1 , . . . , Z dz , e.g., each Z i is independently sampled from N (0, 1). This is also the setting studied in nonlinear independent component analysis [21, 45] . 2. Causal Representation Learning: This setting is more general than the one above where we relax the independence assumption on the Z i , and instead assume that they have (typically unknown) causal relationships among them. For instance, they could satisfy a linear structural causal model with Gaussian noise, i.e., Z = AZ + ϵ, ϵ ∼ N (0, I) where A encodes a weighted directed acyclic graph. This setting is generalizes the previous setting, since having no causal relationships (i.e., A = 0) implies joint independence.As explained earlier, in both these domains, a critical notion is that of identifiability [53, 25, 129] , which posits that the given dataset(s) are diverse enough for the modeling assumptions, in order to ensure that a unique set of parameters fit the data. It's folklore that the disentangled representation learning model is not identifiable if all Z i are Gaussian [46, 71] . However, under appropriate assumptions, e.g., distributional, sparsity or observed side-information, the model becomes identifiable, see e.g., [53, 44, 11, 106, 60, 78, 141, 57, 13, 12, 142, 36, 96] . In addition, various works have proposed methods to learn them [33, 132, 26, 134, 67, 23, 13, 63, 14] .In this section, we highlight the conceptual differences between causal representation learning and our framework.Are causal generative concepts necessarily interpretable? Moreover, we are constantly conjuring new concepts of interest since human-interpretable concepts are constantly evolving, e.g., the concept of mobile phones did not exist 100 years ago, but is a valid concept to learn now. Therefore, as opposed to working with a rigid model as in causal representation learning, we take the approach of working with a dynamic representation learning model. Finally, even if individual causal factors are interpretable (which may be the case in certain applications), the perspective that we take in this work is that the number of true generative factors could be prohibitively large so that attempting to extract and interpret all of them together is infeasible, whereas the number of desired human-interpretable concepts is much smaller and more manageable.When the ground truth generative process has ambient latent dimention d z , for causal representation learning to be feasible, we usually require d z environments or datasets. For instance, in the iVAE setting [53] with k sufficient statistics, we required z k +1 ≥ d z +1environments. This is indeed necessary, as counterexamples show. However, it's not clear what the value of d z is for complex datasets, and it could potentially be prohibitively large.But the question remains, do we need to learn the entire generative model for solving downstream tasks? Along these lines, there is a tremendous research effort attempting to relax such requirements by imposing various inductive or domain biases and by building a theory of partial identifiability [57, 69, 59] . This is for good reason, since even though it would be ideal to learn the full ground truth generative model, it may be prohibitively large and moreover it may not be necessary for the downstream tasks we care about, therefore it suffices to learn what is necessary. On this note, the related task of learning only a subset of the generative latent variables is also not easy as the latent variables interact in potentially complicated ways.In this work, we show that if we only wish to learn n ≪ d z concepts, it suffices to have O(n) environments instead of Ω(d z ) environments. Therefore, our results can be viewed as a result on partial identifiability with a sublinear number of environments.Multi-node interventions Multi-node interventions are an exciting area of study in CRL, since they are a natural extension of existing works and are more useful for modeling various real-life datasets where it can be hard to control precisely one factor of variation. This is easily incorporated in our setting by utilizing non-atomic concepts, since each non-atomic concept is a collection of vectors corresponding to atomic concepts and can be modified simultaneously by changing the valuation.In this work we focus on conditional data and identification of concept structure, while a recent trend in CRL is to focus on interventional data and identification of the causal structure [110, 122, 14, 50, 126] . For causal models, interventions are a natural approach to solving the identifiability problem, however, in the absence of an assumed causal model (as in our framework), interventions may not even be formally well-defined. In our framework, we do not think of concepts as being causal variables that are connected by a graph. (We note that an interesting approach would be to study learning concepts over a given causal generative model, which is an intriguing direction for future study that we do not pursue in this work).By contrast, conditional data does not require the formal framework of causal models, and is often more frequently available in practice. Conditional data can be obtained by selection through filtering, e.g., patients that are admitted to different hospitals based on the severity of their condition or by the availability of label information as in the CLIP setting [92] . Thus conditional data can be obtained by observing the system in different condtions. On the other hand interventional data requires manipulation of the system which is more difficult to obtain in general.In this section, we present alternate feasible definitions for data distributions than the one we introduced in Section 4.2. While we went with the definition most suited for practice, these alternate definitions are also justifiable in different scenarios and are exciting avenues for further study.We want to essentially define a concept C via a conditional measure p C where the concept C is identified with an affine subspaceC = {Z ∈ R dz : A C Z = b C } for some A C ∈ R k×dz , b C ∈ R k .We consider the shifted parallel linear subspace C 0 = {Z : A C Z = 0} and the orthogonal splitting R dz = C 0 ⊕ V . Suppose we have a distribution q V on the space V which will typically be a Gaussian centered around v C ∈ V which is the unique solution of A C v C = b C . In addition we have a base distribution p on R dz . We will assume that all distributions have a smooth density so that conditional probabilities are pointwise well defined. There are at least three ways to create the context conditional measure p C .1. The first option is to enforce that the distribution of the V marginal p C (v) = C0 p C (v, c) dc exactly matches q V (v) while the in-plane distribution p C (c|v = v 0 ) ∝ p C (c, v 0 ) remains invariant, i.e., equals p(c|v = v 0 ). Under this condition, there is a unique measure p C given byp C (c, v) ∝ q V (v) p(c, v) C0 p(c ′ , v) dc ′ .In other words, to get (c, v) we sample v ∼ q V and then c ∼ p(c|v) according to the conditional distribution. 2. The second option is to again enforce the V marginal but instead of keeping the in plane distribution we average over the V space. Then we obtainp C (c, v) ∝ q V (v) V p(c, v ′ ) dv ′ .This corresponds (vaguely) to a do(v) operation from causal inference, i.e., we sample according to p(v, c) and then do a random intervention on v with target distribution q V . 3. The third option is to take a Bayesian standpoint. Then we view p as a prior and q V as the context dependent acceptance probability, i.e., we sample by p and then accept with probability q V . Then we findEQUATION)This is probably the closest aligned to practice, so this is the one we study in this work. To justify this option, imagine the following scenario. If we wish to learn the concept of red color, a first step would be to curate a dataset of red objects. To do this, we first consider a collection of photos of objects of varying color and then filter out the ones that look red.The concept conditional measure we define aligns with this process. To learn the actual red concept accurately, our theory predicts that it is sufficient to have additional datasets of objects that are not red, from which we can distinguish red objects, thereby learning the concept of red color.The next question is how to define the measure q V . When considering a single conceptA C Z = b C the most natural option to consider N (v C , σ 2 Id V ) where v C ∈ V is the unique solution of A C v C = b Cand σ > 0 is a positive constant. This is what we do in this work (note that σ 2 can be set to 1 by scaling the concept and valuation accordingly).However, we can also use alternate definitions as suggested above. For instance, we can setAZ D = N (b C , Id). Then Z ∼ N (v C , (A ⊤ A) -1). However, this runs into some technical issues we sketch (and leave to future work to handle this). Consider the intersection of multiple concepts C e . In this case the concept space is given by the intersection C = C e and C 0 = (C e ) 0 and we have the orthogonal decomposition R dz = C 0 ⊕ V e . In general the spaces V e are not necessarily orthogonal but it is reasonable to assume that the non-degeneracy condition dim( V i ) = dim(V e ) holds. Now set V = V e . If we choose just the standard normal distribution for q V e we can define just as in our approachEQUATION)The second option is to enforce that the marginals of q V agree with q V e , i.e., q V (Π V e (v) ∈ O) = q V e (O) for O ⊂ V e . This results in the set of equations for all iEQUATION)It is likely that this system has a unique solution when non-degeneracy holds for V e and this is clearly true for orthogonal spaces but it is not clear how to solve this in general.In this section we provide additional experimental details and further results for the analysis of pretrained CLIP models [92] .We transform the images from the 3d-Shapes dataset to match the CLIP training data, i.e., reshape to images of size 224 and match the channel distributions. Then we calculate the embeddings for all images in the dataset using two CLIP models, a model with a vision transformer backbone ('ViT-B/32') and a model with a Resnet backbone ('RN101') 3 . We split the embedded images in to training and test sets of equal size. Then for any factor of variation (orientation of the scene, shape and scale of the object, and hue of floor, wall, and object) we perform the following procedure. For each pair of values of a factor of variation we run logistic regression on the embeddings for those two values of the concept to classify which value is taken for a given embedding. We average the directions of the logistic regression vectors β i , i.e., consider β = N -1 N i=1 β i . Since the direction is defined only up to a sign (depending on the order of the two groups) we repeatedly replace β i by -β i if the scalar product with the current mean is negative (this is a heuristic procedure to align β i with β). We then use the learned concept vectors a = β to evaluate the concept valuations on the held out test data, i.e., we evaluate ⟨a, Z⟩ where Z = f -1 (X) is the embedding of an image X. The preprocessing to calculate the CLIP image embeddings required few hours on a A100-GPU. The remaining evaluations were performed on a standard notebook.Here we report the mean and standard deviations of the per-class concept valuations ⟨a, Z⟩ for the concept vectors learned as described in Section D.1. The results for the six factors of variation can be found in Tables 2, 3 , and 4. We observe that shape, scale, and orientation are well aligned with linear subspaces. For the hue variables this still holds to some degree the discrepancy might be attributed to hue not being an atomic concept (colours are typically represented by at least two numbers). Moreover, we consider the correlation coefficient of the valuastions obtained for different embedding models, i.e., for ⟨a M1 , Z M1 i ⟩ and ⟨a M2 , Z M2 i ⟩ where a M1 and a M2 are concept vectors for the same concept and two different models and Z M1 i and Z M2 i denote the embeddings of the two models M 1 and M 2 of sample X i . We report these correlation coefficients for the two CLIP models in Table 5 . The results indicate that the valuations indeed approximately agree up to a linear transformation. Note that for the scene orientation attribute the valuation corresponds to the absolute value of the angle. Table 2 : Mean valuations and standard deviation on the test set for the floor hue and wall hue attributes.Vit-B/32 RN101 0.0 -1.4 ± 1.4 -0.3 ± 0.9 0.14.5 ± 1.5 1.4 ± 0.8 0.2 4.3 ± 1.3 3.2 ± 0.8 0.3 2.2 ± 1.4 3.0 ± 0.8 0.4 1.2 ± 1.5 2.2 ± 0.8 0.5 0.0 ± 1.1 0.5 ± 0.8 0.6 -2.8 ± 1.3 -0.4 ± 0.9 0.7 -5.8 ± 1.5 -2.0 ± 1.0 0.8 -3.8 ± 1.4 -1.3 ± 0.9 0.9 -3.2 ± 1.4 -1.0 ± 0.8 Wall hue Vit-B/32 RN101 0.0 1.1 ± 1.3 -1.5 ± 1.4 0.1 2.8 ± 1.3 1.8 ± 1.0 0.2 3.3 ± 1.1 1.5 ± 0.9 0.3 1.7 ± 1.0 0.8 ± 0.8 0.4 0.8 ± 1.3 0.5 ± 0.9 0.5 -0.6 ± 1.2 -0.6 ± 1.1 0.6 -3.3 ± 1.2 -2.3 ± 1.1 0.7 -3.6 ± 1.2 -3.7 ± 1.0 0.8 -1.4 ± 1.1 -2.0 ± 1.0 0.9 -0.6 ± 1.2 -2.0 ± 1.1In this section, we first briefly describe Large Language Models and the recent Inference-Time Intervention (ITI) technique proposed for LLM alignment, which we build on. Then, we use our framework to provide better intuition on some intriguing observations about ITI, including why it works. And then we exploit our ideas to improve the performance of ITI by choosing the steering direction to be a matrix instead of a vector. 5.0 ± 1.9 3.6± 1.3 1.0 1.9 ± 1.9 1.8 ± 1.1 1.0 -1.3 ± 1.8 0.2 ± 1.1 1.1 -4.3 ± 2.0 -1.4 ± 1.2 1.2 -7.1 ± 2.1 -2.8 ± 1.2 1.2 -9.3 ± 2.3 -3.9 ± 1.3 Shape Vit-B/32 RN101 Cube 8.2 ± 1.46.9 ± 0.9 Cylinder 2.9 ± 1.6 2.9 ± 0.9 Ball -3.6 ± 1.6 -1.2 ± 0.7 Ellipsiod -11.8 ± 3.1 -5.5 ± 1.7Large Language Models (LLMs) LLMs are large models capable of generating meaningful text given a context sentence. Due to large-scale training, modern LLMs have shown remarkable capabilities and achieve expert-human-like performance in many benchmarks simultaneously. The architecture of many generative pre-trained transformers (GPT)-style LLMs consists of several transformer layers stacked on top of each other. Since we'll be intervening on them during inference, we'll describe the transformer architecture [125, 29] briefly here. First, the sequence of input tokens (tokens are sub-word units) are encoded into a vector x 0 using a (learned) text embedding matrix and in many cases also a positional embedding matrix. Then, a series of transformer layers act on this vector which passes through a residual stream, to obtain vectors x 0 , x 1 , . . . , x n . The final vector x n is then decoded back into token probabilities with a (learned) unembedding matrix. Each transformer layer consists of a multi-head attention mechanism and a standard multilayer perceptron, which captures the nonlinearity.In the lth layer, each single multi-head attention mechanism can be described asx l+1 = x l + H h=1 Q h l x h l , x h l = Att h l (P h l x l )Here, P h l and Q h l are matrices that linearly map the vector to an activation space and back respectively, and Att denotes the attention mechanism that allows communication across tokens. Here, we have kept the notation consistent with Li et al. [66] for the sake of clarity. In our setting, we consider the entire set of activations as the learnt latent vector Z. That is, the input is x = x 0 and the pre-trained model is essentially the function f such that f (x) consists of the concatenation of the vectors {x l } l≥1 , the intermediate activations {x h l } l≥0 and also the output of the linear transformations {P h l x l } l≥0 , {Q h l x h l } l≥0 . Our theory hinges on the assumption that pre-trained LLMs satisfy the linear representation hypothesis, that is, various relevant concepts can be realized via linear transformations of the latent transformation f (x). Indeed, this has been empirically observed to hold in many prior works [17, 118, 81, 79, 66, 85, 38, 52] (see also related works on geometry of representations [51, 52] and references therein). It's a fascinating question why such models trained with next token prediction loss also learn linear representations of various human-interpretable concepts such as sentiment, see Jiang et al. [52] for recent progress on this problem.It's well-known that despite large-scale pretraining and subsequent improvement of pre-trained models via techniques like Reinforcement Learning with Human Feedback (RLHF) and Supervised Fine-Tuning (SFT) [84, 6, 119] , significant issues still remain [107] , e.g., the model can hallucinate or generate incorrect responses (even though the model knows the correct response which can be extracted via other means, e.g., Chain-of-Thought prompting [131] ). Various methods have been proposed to fine-tune the models [84, 6, 7, 119, 93] but many of them are expensive and timeand resource-intensive as they requires huge annotation and computation resources. Therefore, more efficient techniques are highly desired, one of which is the category of methods known as activation patching. activation patching (also called activation editing or activation engineering) [40, 128, 112, 121, 143, 138, 65, 76] .Inference-Time Intervention, an activation patching method for truthfulness Activation patching is a simple minimally invasive technique to align LLMs to human-preferences. Specifically, given various concepts such as truthfulness, activation patching makes modifications to the model during inference time so that the desired concepts can be aligned. This technique can be thought of as an application of the emerging field of mechanistic interpretability [83] , which aims to interpret the learnt latent vector in terms of human-interpretable concepts, thereby allowing us to reverse-engineer what large models learn.Activation patching has many variants [65, 40, 76] , but we'll focus on the simple technique of adding steering vectors to various intermediate layers during intervention [112, 121, 66, 98] . This means that during inference, the output activations are modified by adding a constant vector in order to promote alignment of some concept. The vector will be learnt independently based on separate training data.In particular, a recent technique called Inference-Time Intervention (ITI) was proposed to do this for the specific concept of truthfulness. ITI focuses on the activation heads {Att h l (P h l x l )} l≥0 and add to them steering vectors in order to promote truthfulness. To learn the steering vectors, a subset of the TruthfulQA dataset [68] , namely a dataset of questions q i with annotated true (a i,j , 0) and false answers (a i,j , 1), are prepared as {q i , a i , y i } i=1,2,... . For each sample, the question and answer are concatenated as a pair and the corresponding activations of the heads x h l (for the final token) are computed via forward passes. Then, a linear probe sigmoid(⟨θ, x h l ⟩) is independently trained on each activation head to distinguish true from false answers. Finally, the top K heads based on the accuracy of this classification task are chosen (for a tunable hyperparameter K) and the steering vector θ h l for the h-th head in layer l is chosen to be the mean difference of the activations between the true and false inputs. The intuition is that this direction roughly captures the direction towards truthfulness.Formally, for the hth head of the lth layer, ITI adds the steering vector ασ h l θ h l so as to getx l+1 = x l + H h=1 Q h l (x h l + ασ h l θ h l ), x h l = Att h l (P h l x l )during inference. Here, θ h l is the steering vector, σ h l is the standard deviation of the activations of this head along the chosen direction and α is a hyperparameter. That is, the activations are shifted along the truthful directions by a multiple of the standard deviation, and this is repeated autoregressively. Note that this does not depend on the specific GPT-like model being used. The intuition is that during inference, the activations are intervened upon to shift towards the truthful direction. The top K heads are chosen to be minimally intrusive and also a design choice based on observations of the probing metrics.Performance of ITI In Li et al. [66] , ITI was shown to significantly improve the truthfulness of various LLMs after having been trained on as few as a few dozen samples, compared to what's needed for Reinforcement Learning based techqniues [84, 34] . ITI was evaluated on the TruthfulQA benchmark [68] , which is a hard adversarial benchmark to evaluate truthfulness of language models. In particular, it contains 817 questions with a multiple-choice and generation tracks, spanning 38 categories such as logical falsehoods, conspiracies and common points of confusion. For the multiplechoice questions, the accuracy is determined by the conditional probabilities of candidate answers given the question. Evaluating the generation track questions is harder, and it is done by generating a model output and then evaluating it via a finetuned GPT-3-13B model [68, 80] . Moreover, the choice of the intervention strength α is calibrated so that it's neither too small (to promote truthfulness) nor too large (to ensure the original capabilities of the LLM are not lost). To check if the original capabilies are preserved, [66] compute two additional quantities to measure how far the modified model deviates from the original model. These are the Cross-Entropy (CE) loss, which is standard in language modeling and the Kullback-Leibler divergence (KL div.) of the next token probabilities before and after intervention. To compute these quantities, a subset of Open Web Text is used [91] . Finally, it was shown that ITI implemented on the LLaMA [119] , Alpaca [116] and Vicuna [20] models significantly improved their performance on the TruthfulQA benchmark compared to the baseline models. Moreover, in many cases, it also beat other techniques such as few-shot prompting and supervised fine-tuning. Please see Li et al. [66] for additional details.While the elegant ITI technique was designed to align LLMs towards truthfulness in practice, it also raised fascinating and intriguing questions in mechanistic interpretability. In addition to improving the technique of ITI itself, our work makes progress towards some of these questions via our framework.1. The authors of Li et al. [66] state in section 2 that although the technique works well in practice, it's not clear what ITI does to the model's internal representations. In addition, prior works [17, 118, 81, 79, 85, 52] have observed empirically that the latent representations learned by LLMs seem to have interpretable linear directions, which ITI exploits. We use our framework to illustrate in more detail one possible explanation of what ITI does to the model representations and why it works, in the next section. 2. The authors visualize the geometry of "truth" representations in section 3.2 of their work via the following experiment: For the most significant head (layer 14, head 18), after finding the first truthful direction via the linear probing technique, they remove it and attempt to find a second probe orthogonal to the first. They find surprisingly that the second probe is also very informative, leading them to predict that the concept of "truth" lies in a subspace, not a single direction. Restated in our framework, the concept of truthfulness is a non-atomic concept (as per Definition 2). This served as an inspiration for our proposed technique in the next section, where we propose to use steering matrices instead of steering vectors for LLM alignment. 3. As α was increased, the authors observed that truthfulness of the model increased however helpfulness decreased. This suggests that the "truthfulness" and "helpfulness" concepts are not atomic (as per Definition 2) however they share certain atomic concepts. We leave to future work the exciting question of mechanistically extracting such common atomic concepts.In this section, we will use our theoretical framework to get insights about the ITI technique and use it to improve alignment. First, similar to the multimodal CLIP setting, we will assume that the non-linearity has already been learned up to a linear transformation (by large-scale training of LLMs). This aligns with our theoretical insights because the training data for powerful LLMs are diverse, so they essentially satisfy our core assumptions (see also the related work [37] that proposes that context is environment in LLM training). Therefore, we simply focus on the downstream tasks, which in this section is LLM alignment. The difficulty, of course, is that we do not know the concept matrix nor the valuations.We will now analyze the truthfulness concept via our framework and give more insight on why the mean of the differences is a reasonable choice of steering vector for ITI. Based on our theory, we will then provide a modification to this choice that uses steering matrices instead of steering vectors. Since this section is based on heuristics and informal assumptions, we will refrain from making any formal claims or analyses. Indeed, a formal analysis of concepts in natural language is a hard problem in general and we do not attempt it here. We conclude with ideas for potential extensions that're worth exploring in future work.Denote the function h to be the sequence of head activations h(x) = (x h l ) l,h ∈ R d . Note that while we can study general steering vectors for the entire latent space of representations f (x) learned by LLMs as some works do, ITI focuses only on steering the head activations h(x), so we will apply our framework to this subset representation space. In addition, we will make the simplification that we neglect the effects of the steering vector from bottom layers towards the top layers, which we do because we are dealing with sparse steering vectors and also, each single head shift is minor and does not in isolation change the behavior of the model as verified by experiments [66] Note that they only approximately lie in these spaces because of our notion of concept conditional distribution. However, if we reasonably assume that the Gaussian concentration region is much smaller than the separation between these hyperplanes, then the rest of the arguments in this section should apply. Now, a steering vector η is a vector such that it moves the activations from the false space to the true space, while keeping other concepts unaffected. That is, if we pick a false sentence x, i.e., Ah(x) = b 0 , then the steering vector η ∈ R d essentially steers the activations so that A(h(x) + η) = b 1 . In other words, it moves the sentence from false to true. Indeed, many vectors η do satisfy this equality, because we could move h(x) to any point in the hyperplane {AZ = b 1 }. Therefore the goal is to find an optimal η that does not (significantly) affect other concepts of interest, i.e., B(h(x) + η) ≈ Bh(x) (equivalently Bη = 0) for any other concept of interest B. Indeed, a natural choice of the steering vector will be A + (b 1 -b 0 ) where A + is the pseudoinverse of A. This vector will precisely affect this concept space and will not affect the concept valuations for any concept orthogonal to A. However, there are two issues with this approach: We do not know A and therefore we will approximate this steering vector from training samples and there is no guarantee that other concepts of interest are orthogonal to A (note that angles between concepts are not even identifiable).Previous approaches are based on a collection of counterfactual sentence pairs c F i , c T i which correspond to a false answer and a true answer for the same question q i . Consider the ith counterfactual pair c F i , c T i . We will assume the reasonable scenario that the only difference among their concepts is the concept of truthfulness. That is, for any other concept of interest B i for this sample the valuations of B i for these pairs c F i and c T i are identical. A common strategy is to use the meanEQUATIONas a steering vector. Note that ifEQUATIONi.e., the truthfulness valuation is changed as desired for all samples thenEQUATIONMoreover, concepts of interest are preserved in two prototypical settings. First, if concepts of interest are the same for all samples and the new datapoint, i.e., B = B i = B j in which caseEQUATIONSimilarly, if concepts of interest for a new point x are B x and the valuations ofB x (h(c T i ) -h(c F i ))of the counterfactual pairs are random, independent, and centered, then we expect them to approximately cancel andEQUATIONNote that in this case, this is not true if just a single steering vector h(c T i ) -h(c F i ) is used as a steering vector. This explains why the choice of mean of the activation differences across counterfactual pairs is a reasonable choice of steering vector. This is precisely the technique used in ITI. While they also experiment with other steering vectors, they found that this works the best for their experiments. Now, we will continue on our insights to analyze whether we can build better steering vectors η. We present two crucial insights based on our analysis so far.We briefly note down some design choices we made in our implementation of the above method.1. Since η(x) is a function of x, the standard deviation of the activation projection on this direction, i.e., σ h l (x) cannot be precomputed (as Li et al. [66] do), therefore we compute them dynamically during inference, which takes little overhead with fast tensorization operations (in particular, this is not the slow step). 2. We opted to go with evaluating the model only on the multiple-choice questions. This is partly because to evaluate the generated text, the recommended method is to use fine-tuned GPT-3-13B models but OpenAI have retired many of their older models as of this year, and therefore, the entire batch of experiments would have to be rerun with their newer models which could potentially change the baselines, and also because this work is a proof-of-concept rather than a comprehensive evaluation. for a regularization hyperparameter η.Sampling from concept conditional distributions A common task in controllable generative modeling is being able to generate data from a known concept. Note that this is not straightforward in our setting because the normalization term in Eq. ( 2) is not efficiently computable. To do this efficiently, we also outline a simple algorithm (Algorithm 1 in Appendix H) to sample from the concept conditional distribution for a known concept. Our proposed algorithm is based on rejection sampling and the algorithm as well as the complexity analysis is deferred to Appendix H.In this section, we detail the synthetic setup in Section 6. The base distribution is sampled from a Gaussian mixture model with 3 components whose parameters are chosen randomly. The weights are randomly chosen from Unif(0.3, 1) (and then normalized), the entries of the means are chosen from Unif(-1, 1) and the covariance is chosen to be a diagonal matrix with entries in Unif(0.01, 0.015) (note that the diagonal nature doesn't really matter since a map f will be applied to this distribution).The mixing function f is chosen to be either (i) linear or (ii) nonlinear with a 1-layer MLP containing 16 hidden neurons and LeakyReLU(0.2) activations.The number of concepts n is intentionally chosen to be less than the ground truth dimension d z and the number of concepts is m = n + 1 as per our theory. The concepts are taken to be atomic, with the concept vectors and valuations chosen randomly, where each entry of the concept vector is chosen i.i.d from Unif(-0.3, 0.3), and the resampling distribution is chosen to be a Gaussian with variance 0.005. Finally, we choose 5000 samples per environment, sampled via the rejection sampling Algorithm 1. For the contrastive algorithm, we choose the architecture to either be linear or nonlinear with a 2-layer MLP with 32 hidden neurons in each layer, with the final parametric layer chosen based on the known concept, to have the form described above. We train for 100 epochs, on a single A6000 GPU, with η = 0.0001 and use Adam optimizer with learning rates 0.5 for the parametric layer and 0.005 for the non-parametric layer, with a Cosine Annealing schedule [72] .Additional results for higher dimensional settings can be found in Table 7 . In this section, we will describe how to sample from a concept conditional distribution with a known concept. Once the concepts are learned in our framework, we can use this technique to generate new data satisfying various desired concepts, which will aid in controllable generative modeling.Consider the base distribution on Z ∈ R dz with density p(Z). Suppose we wish to sample from a concept C given by AZ = b and resampling distribution q. We additionally assume that q is efficiently computable and an upper bound L is known for its density, i.e., L ≥ max(q). Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The codes, along with instructions on how to run them, is attached in supplementary material. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Exceptions are[57,42], which use clever inductive biases to limit the number of environments needed.Models are publicly available under https://github.com/openai/CLIP 


Decision: In this submission the authors argue for a shift from causal approaches to representation learning to concept-based approaches. One of the key questions that the authors are interested in is whether such representations are identifiable. Instead of learning causal representations, where we require access to interventional distributions, the authors assume access to a set of concept-conditional distributions. They then define a geometric notion of concepts as linear subspaces of the laten space and prove that these subspaces are identifiable under the assumption of linear independence of concepts, as well as some assumptions about data diversity. The authors evaluate the proposed framework in the context synthetic experiments, multi-modal CLIP models, and LLMs.

After discussion, all reviewers support acceptance and overall this submission appears above the bar. Some comments from reviewers that should be addressed in the camera ready include moving the algorithm block from the appendix to the main text, addressing some points where terminology could be introduced more carefully and providing a greater degree of intuition behind some of the mathematical assumptions. The AC would also recommend that the authors revise their presentation of the experiments to address some of the questions raised by reviewer ohha regarding the connection between experiments and the developed theory.

Reviews: review: paper_summary: paper_summary: The authors argue the shift from causal representation learning (CRL) to concept-based representation learning since the current CRL framework relies on strong requirements such as interventional datasets and stands far from realistic, practical use-cases. The paper formalizes the notion of concepts and establishes a theoretical foundation on the identifiability of concepts. The experiments demonstrate the utility of the framework.
strength_weakness: strength_weakness: Datasets $X^e$ from each environment is associated with different concept $C^e$ and corresponding valuation $b^e$. The proposed method using contrastive learning requires how the dataset is partitioned into each environment, i.e., $X^0, \cdots, X^m$. This implies that the framework is naturally more useful for **discrete** concepts (i.e., discrete valuation), as showcased in the experiments where the authors use discrete labels. However, as the motivation suggests, concepts could be continuous in many cases (e.g., intensity of the color). Therefore, I have doubt on the practical utility of the proposed framework since it cannot handle such continuous concepts. In other words, the requirement of data partition $X^0, \cdots, X^m$ goes against the motivation of the proposed framework of handling continuous concept valuations.
questions: questions: - (line 208) Can you elaborate? I mean isn’t $A$ a projector matrix?
- Is there any way to quantitatively measure concepts learned by two different models are how much linearly-related to each other?
- The proposed method using contrastive learning should be described in the main section in more detail. Currently, it appears at the appendix, but I think that the algorithm is a key part of the paper which illustrate the practical utility of the proposed framework.

(minor)

- (line 1171-1172) Parenthesis is not closed.
- (line 370, 1491) “the number of concepts $m$” should be “the number of environments $m$”
limitations: limitations: .
paper_summary: paper_summary: This work takes a step toward learning human-interpretable concepts while relaxing the restrictions of (interventional) causal representation learning, and they do so inspired by the linear representation hypothesis.

The authors claim that learning the generative process and the "true" causal factors $f^{-1}, Z$ from observations $X$ using interventions has important caveats: 1) One needs many interventions ($\Omega(d_z)$) for identifiability of $f^{-1}, Z$ which might be too much of a requirement in many cases, 2) There's no reason that a priori such latent representations are interpretable, 3) Interventions in many examples might not be possible at all 4) One might not need the whole $f^{-1}, Z$, and there are cases where we can seek only a handful of interpretable concepts for an application without learning the full encoder and latent representation.

The authors then introduce the geometric notion of concepts as linear subspaces in the latent space of $Z$. This is inspired by the abundant evidence on the linear representation hypothesis. Based on this notion, they define the concept conditional distributions as a source of supervision for learning concepts which will replace interventional distributions as the source of supervision for learning causal representations $Z$. Concept conditional distributions are simply defined by filtering the dataset with samples that are $\textit{perceived}$ to satisfy a concept (see eq. 1)

The problem then becomes whether given an observational distribution $X^0$ and a set of concept-conditional distributions $X^1, \dots, X^m$ corresponding to $m$ concepts, one can identify the linear subspaces $A^mf^{-1}(x)$ corresponding to those concepts.

The main theorem then proves the identifiability (according to definition 4) of those concepts given linear independence of concepts, as well as some diversity constraints on the environments.

The authors then try to validate the claim using 3 experiments:
1) Synthetic experiments with various linear and non-linear mixing $f$, and different dimensions for $Z,X$.
2) Evaluating the linearity of the valuations of the concepts learned via multi-modal CLIP (inspired by the similarity of CLIP objective to their contrastive algorithm
3) Showing that concepts can be used to steer LLM outputs.
strength_weakness: strength_weakness: - In the synthetic experiments, I was expecting to see large $d_z$ and small $n$ to match the claims that had been made earlier as to the advantage of concept conditional distributions, but the dimensions are quite small. I can see that they show a proof of concept, but still, it would have been nice to be consistent with the claims made earlier (unless there's a reason why the authors didn't do so)

- Could the authors think of any experiment to contrast concept learning to CRL? Maybe with simple datasets like CLEVR or 3d-shapes? Isn't there a way to try to learn causal representations and concepts, and show empirically that one is easier to be achieved? I understand that the premise was that $Z$ is not always interpretable in the first place, but I think there would exist situations where it will be. If I understand correctly, do the authors think such an experiment would add to the empirical evidence for their method?

- I'm also a bit unclear about where this work is taking us, and would have liked it if it was explained better. In particular, are we hoping to change our representation learning towards learning concepts? If so, are you proposing this for reasoning tasks? For alignment? Interpretability? Or what else? The reason I'm asking is that wouldn't we probably still need some causal representations in some reasoning tasks, say in vision? Basically a short discussion of when we would prefer concepts over causal representations (or else) would be helpful.

----
Writing and Clarity:

- The notion of environment in the context of concepts appeared out of the blue on page 7 (same with its notation that followed)
- Not a weakness of the method - but the learning method seems like an important component of the paper which is deferred altogether to the appendix, i.e., one would learn about the identifiability, but there is no mention of the actual method to learn such (identifiable) concepts which might hide the difficulties and challenges associated with it.



Please also see the questions.
questions: questions: - Could you connect/contrast this work the recent advances in sparse autoencoders (SAEs)? While the problems are somewhat differently motivated, the resulting concepts bear resemblance to the learned sparse features from activations of transformers. Could the authors comment on this please?
- If we care about a few features, why not use CRL methods that guarantee weaker identification requiring much less interventions? (See question below).
- I generally agree with the motivation and direction of the paper, but I feel like the CRL community has been aware of these restrictions and recently took steps to address them; for instance, from what I understand the multi-node intervention line of work (cited by the authors) alleviates the challenge of perfect interventions and relaxes the requirement of learning all of $Z$, instead weaker and more general notions of identifiability have been introduced. A proper discussion on this would be helpful (more than what is in the appendix). Wouldn't it make sense to leverage such methods in tasks where we have a prior that there is some underlying causal representation?
- Does equation 2 come from the independence of concepts (asking because that is introduced later). Why is not $k=dim(C)$?
- Line 294 onwards, is $e$ properly defined before and used here?
- Line 304, is $S_n$ a typo? $n$ was used as superscript before that, here it's used as a subscript.
- Could assumption 4 be explained in words as well? Where does it come from? When/in what situations does it break?
- How should one interpret table 2 of the appendix? Is there a reason why it's not plotted, and presented as numbers? Shouldn't we expect a linear plot?
- Line 165, what do you mean by entangled concepts, is there an experiment to show that? Or do you mean the superposition of atomic concepts?
- (Not a question impacting the score) Related to independence of concepts: Can the theory say anything about hierarchical concepts similar to hierarchical representations?

Remark: I am willing to increase my score if some of the questions and weaknesses are discussed since I find the direction of this work quite interesting and important.
limitations: limitations: See questions.
paper_summary: paper_summary: This paper proposes a theory to identify latent concepts from model representations. In contrast to previous work in the concept-based models' field, concepts are expected to be linearly represented in the model representations, and a linear transformation A is associated with such concepts. The paper is theory-driven and presents rigorous results for the identifiability of such concepts. The authors also present some experiments related to the theory, one of which verifies in a synthetic setup the validity of the proposed theory, one on CLIP and one on LLMs.
strength_weakness: strength_weakness: ## Major 

### **Experiments**

One thing that is particularly dangling in the paper is the experimental section and the supposed evidence in support of the theory: 

1) The experiments on CLIP and LLMs seem to me unrelated to the theory devised by the authors and rather support on one hand that concepts can be found in the representations in CLIP to some extent (which was previously observed, e.g. [2,3]), on the other hand, that it is possible to improve the steering of LLMs predictions with some matrix operation rather than vector addition. How is this related to the theory the authors propose? How is it that CLIP training aligns with the assumptions that would lead to the identifiability of latent concepts? LLMs like LLaMA or GPT are next-token predictors learning with a different objective, see [4], how are even these models related to the theory proposed?

2) It would have been more useful to provide experimental evidence of the proposed contrastive learning method on semi-synthetic datasets like MPI3D or Shapes3D. 
 Some real-world datasets are of particular interest to the community in concept-based models, being more tackling, like CUB200 and OAI [5], CelebA [6], and many more [7,8].

3) The synthetic experiment is impossible to understand from the main text and many details concerning the data, the learning procedure proposed (which is also very detailed), and the metrics are confined to appendices rendering it necessary to consult them in length. Nonetheless, it seems that under the working assumptions the model trained in a contrastive manner on the synthetic data and environments captures the right latent concepts. How do you evaluate $A^e$? 

### **Assumptions**

It would be beneficial to present the assumptions and discuss the intuition behind them. Assumption 1 is fine and is common in studying identifiability, but if the aim is to identify concepts, it seems unnaturally restricting to consider only invertible functions $f$. One could hope to extend the results also to non-invertible functions. However, this is not a serious limitation given the novelty of the results.  \
Assumption 2 takes the concepts to be linearly independent, what happens if for some of them is not the case? \
Assumption 3 requires a Gaussian distribution for the noise, it is remarked that other distributions work as well (which distributions?) but there is no citation. \
How should Assumptions 4 and 5 be understood, what data are expected to be collected? Are these expensive to obtain in practice? They seem to presuppose a lot of knowledge on what latent concepts should be identified, which may not be the case when concepts are not known apriori.


## Minor

### **Conditional distributions in practice**

It is a bit puzzling how data should be gathered for the theory to work, and thus the scaling of the proposed method. The synthetic experiment offers a proof of principle but it does not show how the model behaves in settings where more concepts (and structured ones, like the color of an object) and conditional distribution are to be considered.  


### **Related work**

Causal Representation Learning seems more of an inspiration to the paper rather than having a tight connection to the theory. The authors consider conditional distributions for latent concepts and the proof techniques are inspired by the iVAE work (2019), so there is no clear link to what should be the causal aspects that should be taken into consideration. 

It seems more that works about identifiability in Causal Representation Learning become relevant if one wants to extend authors' theory to causal variables, not being essential to support their claims and the connection. 

I was also expecting to see a comparison to Concept-based models' current practices, which require dense supervision of the concepts in cases [1, 7, 8],  partial [6], or language-guided [2,3]. Some approaches aim to learn concepts only by leveraging supervision on the classification task, see [9], and seemingly related work on continuous latent variable identification [10]. 

## Summary

The experiments do not complement the theory and dilute the message by showing two post-hoc analyses on CLIP and LLMs. I struggle to see how both constitute valid evidence for the theory proposed. On the other hand, additional experiments on known semi-synthetic datasets or real ones would highlight the extent of the theory to the community in concept-based interpretability. 

The presentation of the material also requires some clarification around the assumptions the authors make and their validity in practice.

[2] Label-free Concept Bottleneck Models, Oikarinen et al. (2023) \
[3] Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification, Yang et al. (2023) \
[4] On Linear Identifiability of Learned Representations, Roeder et al. (2021) \
[5] Concept Bottleneck Models, Koh et al. (2020) \
[6] GlanceNets: Interpretable, Leak-proof Concept-based Models, Marconato et al. (2022)  \
[7] Concept embedding analysis: A review, Schwalbe (2022) \
[8] Concept-based explainable artificial intelligence: A survey, Poeta et al. (2023) \
[9] Provable concept learning for interpretable predictions using variational autoencoders, Taeb et al., (2022) \
[10] Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning, Lachapelle (2023)
questions: questions: **Recovering the concepts without post-hoc analysis?**
As I understood correctly (correct if I am wrong), the identifiability class studied in the paper presupposes recovering the latent concepts (the matrix A) up to an invertible transformation $T$. Thus, the theory offers only guarantees to which a linear probe on the latent representations would recover the latent concepts, is it correct? One in practice has still to find the matrix $A$ related to the concept, and that cannot be done without concept annotation. Is that the case?

**Typo?** In Def  It seems that $d_Z$ and $\tilde{d_Z}$ have to be the same to guarantee the inverse exists (and how it is used in the proofs). Is that the case? Would the theory hold also for models with different latent dimensions $d_Z \neq \tilde{d_Z}$?

I asked other questions in the weakness part.
limitations: limitations: All assumptions are natural limitations of the proposed theoretical results. It is not clear whether foundation models are trained under the conditions that the author found for assessing identifiability of the concepts
and the connection to LLMs seems a bit weak.
paper_summary: paper_summary: The paper focus on recover human-interpretable concepts from observation. It proposes a concept based representation learning method, which relax causal notions with a geometric notion of concepts. Experiments on synthetic data, multimodal CLIP models and  large language models supplement their results and show the utility of their approach.
strength_weakness: strength_weakness: 1.This approach sacrifices causal semantics. This can be particularly problematic in situations where a deep understanding of causality is crucial, such as in root cause analysis, where the goal is to identify the fundamental reasons behind a problem or an event. Without causal semantics, one might only address the symptoms rather than the core issues, leading to temporary or ineffective solutions.
2.The author has made numerous assumptions within the article, which could potentially impact the universal applicability of the theory presented.
questions: questions: Please see the weaknesses.
limitations: limitations: The authors adequately addressed the limitations.

Comments: title: Author-Reviewer Discussion Phase (ends Aug 13 midnight AOE)
Reply to authors
comment: Thanks again to the authors for their helpful clarifications and explanations.

If I remember correctly, the results in [2] apply to general diffeomorphisms, and the number of required environments grows slower than $\Omega(d_z)$, however, I totally agree that the goals are a bit different and that is why I liked the paper in the first place. Relaxing the uncovering of the unmixing for the identification of concepts is quite nice, and I agree (with the help of the author's clarification) that an apples-to-apples comparison might not be possible or fair.

Thanks again, and I'm optimistic that the authors will take the various feedback from all the reviewers into account to update the manuscript and the presentation, and I'd like to see this work accepted, therefore, I'm raising my score.
Thanks for your comments concurring with the positioning of this paper in relation to the existing literature. We also appreciate your open-mindedness and willingness to allow some of the experimental concerns to be sorted out in more dedicated future work. And yes, the work [1] you cited is along those lines (we briefly highlight this work in L163-165). 

We again thank the reviewer for their great effort in reviewing the paper and are happy to take additional questions or suggestions.
> The works that I can recall are: https://proceedings.mlr.press/v238/ahuja24a/ahuja24a.pdf, https://arxiv.org/pdf/2311.12267, https://arxiv.org/pdf/2406.05937 (there are probably more)

Thanks for clarifying. We have taken another look at these papers, and to the best of our knowledge, these papers still require a number of environments that is lower bounded by the dimension of the latent space $d_z$ (with few notable exceptions with purely observational data, on which we comment below, please see also the footnote in page 3 where we state this). Thus, these papers do not seem to support the claim that there are "CRL methods that guarantee weaker identification requiring _much less interventions_". If we are mistaken, and the reviewer can pinpoint a specific result in one of these papers that accomplishes this, we would be happy to discuss further.

It is true that the work https://proceedings.mlr.press/v238/ahuja24a/ahuja24a.pdf (reference [2] in the paper) and similarly [54, 40] show identifiability from just a single environment as we state in the footnote in the paper. Note, however, that they all make restrictive assumptions on the mixing function and on the distribution of all variables of the latent space, e.g., in the case of [2] it is assumed that the mixing function is polynomial and the support of the latent variables is the Cartesian product of bounded intervals. 
Therefore, neither these results nor their techniques extend to general mixing functions 
or settings where we only make assumptions on the distribution on some of the latent variables. This contrasts with our work, where we make minimal assumptions on the mixing function and only make assumptions on the latent distributions with respect to the concepts of interest.

> However, since the paper contrasts itself a number of times to CRL requiring many interventions, I don't see it totally beyond the scope of this work to make a minimal effort at contrasting to basic CRL methods. I agree with the authors that the motivation for concept-based learning is to go beyond where CRL is not applicable, but still would find it compelling if there could be a setup where the advantage (in terms of the fewer number of domains required) could be showcased.

Thank you for the question. Making an apples-to-apples comparison to CRL is not straightforward because the goals are different. However, we can say the following:
- Experimentally, we would like to clarify that in cases where _CRL is also applicable_, our algorithm, which is similar to Buchholz et al. [13], can always be applied. Indeed we clarify in the paper that our algorithm is inspired by their works, please see L370-371, 1451.
However, in the case of sublinear number of environments in our setting, standard CRL techniques are not applicable whereas our work is applicable, and this is the main motivation for our paper.
- Theoretically, if we can intervene as per the concept distribution we work with (as also described in L129-134), then our results show that $2n$ concept interventions suffice to learn $n\ll d_z$ concepts, and existing methods would not handle this setting to the best of our knowledge (please see also Appendix C for alternate technicalities). Thus, in our setting, current CRL results require $\Omega(d_z)$ interventions, whereas we only need $o(d_z)$ interventions.

Of course, part of our contribution is to propose a different goal (learning concepts) under different assumptions (conditioning vs. intervention), which makes such comparisons difficult. Nonetheless, we hope this offers some insight to understand how our work compares.
Thank you for your efforts and for the clarifications.

Also, thanks for carrying out experiments with larger $d_z$ and smaller $n$. Although I originally brought it up so there is an experiment aligning with the premise of the paper about very large $d_z$, I find this additional experiment fine as well.

I appreciate the following comment
> This is an interesting suggestion and would potentially add to the empirical evidence. However, it is beyond the scope of our work to compare against CRL comprehensively, partly because we're studying settings where CRL is not directly applicable (e.g., due to lack of interventional data). Moreover, our work is primarily theoretical and we leave for future work to extensively compare concept learning methods to CRL techniques empirically.

However, since the paper contrasts itself a number of times to CRL requiring many interventions, I don't see it totally beyond the scope of this work to make a minimal effort at contrasting to basic CRL methods. I agree with the authors that the motivation for concept-based learning is to go beyond where CRL is not applicable, but still would find it compelling if there could be a setup where the advantage (in terms of the fewer number of domains required) could be showcased.


> Could the reviewer please clarify which specific work they mean? We're more than happy to comment further and have a proper discussion on this in the paper as well.

The works that I can recall are: https://proceedings.mlr.press/v238/ahuja24a/ahuja24a.pdf, https://arxiv.org/pdf/2311.12267, https://arxiv.org/pdf/2406.05937 (there are probably more)

Thanks again for your rebuttal. I like this work and its direction and will maintain my score for now and adjust it if needed after discussion with other reviewers.
Dear reviewers,

Thank you for your efforts so far! The authors have followed up with each of you to respond to your reviews and have also submitted an overall response (which includes a table of experimental evaluations in the PDF). Could you follow up with them to ask any clarifying questions that may further inform your opinion on the paper?

With thanks, AC
Thank you for your clarifications. After reading the reply and other reviews I feel some parts of the paper still present the limitations we pointed out (experimental section).  I have some comments for your rebuttal:

> We appreciate the nice suggestions. Since the primary thrust of the work is theoretical and it is currently quite challenging to scale these type of methods to large datasets, we leave this interesting direction for future work.

I understand that scaling to real-world datasets is challenging and a bit out of scope. However, it should be shown that the method can applied to semisynthetic datasets (even MNIST, which is 28x28 dimensional), otherwise leaving me puzzled about the practical utility of the theory and method proposed. Given the theoretical focus of the authors, I accept their decision to do that in future work.

> The CLIP experiments showcase that the learned representations of concepts indeed lie approximately within an affine hyperplane which is an illustration of our key hypothesis.

This is ok. It seems to me more of a verification of an interesting empirical fact rather than a consequence or an explanation that is already captured by the theory. 

> In such a case, we can apply our identifiability theory to conclude that different CLIP models necessarily learn approximately the same representation up to linear transformation.  Proving these facts rigorously for the representation based on the CLIP-loss is a challenging problem we leave for future work.

That would be interesting, but I would expect the theory would require several adjustments to consider the captions to train visual representations. 

> The LLM experiment is a bit more exploratory and the connection to our theory is via our conceptualisation of concepts. Indeed, this helps us to obtain intuition what the right steering vector should look like and guided our construction.

Similarly, this is an interesting empirical fact, but not explained by the theory. 

> As outlined to other reviewers, we will use additional space to add experimental details to the main text.

Thank you. 

> We do not think that the approach should be used to collect data in practice because this would indeed require a lot of prior knowledge which could probably be directly used to learn the concepts. Instead we think that often real-world data comes with subtle heterogeneity that allows us to identify concepts.

What do you mean by real-world data having this heterogeneity, is it something that could be verified? It seems to me that you have first to assume what are your concepts of interest (being linearly related to generative variables) and then to have those environments. Both points seem challenging.

> We generally agree with the reviewer and refer, e.g., to the discussions in L43-50, 60-63, for this remark. We're happy to add more details to clarify this.

Thank you.

> In short, some of these works study very useful empirical methods whereas we focus on rigorous theoretical contributions, whereas the others differ in the kind of assumptions and settings studied, i.e. they're related but not directly comparable at a technical level.

I also agree with this. Common practice has been shifting towards using concept-supervised examples (which trivially address the identifiability of concepts) which is a limitation and your theory is very relevant in relaxing it. 
 
> Yes, our identifiability is only up to linear transformations. After having learned the nonlinearity, in order to recover the transformation itself, additional information such as concept annotation may be utilized in practice.

Thank you for the clarification. It would be interesting to connect to [1] to see if the same can be done without concept supervision. 

[1] When are Post-hoc Conceptual Explanations Identifiable? Leeman, UAI 2023.