Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data

Authors: charles  jin
mit  csail
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 06:32:45

Abstract: As language models (LMs) deliver increasing performance on a range of NLP tasks, *probing classifiers* have become an indispensable technique in the effort to better understand their inner workings. A typical setup involves (1) defining an auxiliary task consisting of a dataset of text annotated with labels, then (2) supervising small classifiers to predict the labels from the representations of a pretrained LM as it processes the dataset. A high probing accuracy is interpreted as evidence that the LM has learned to perform the auxiliary task as an unsupervised byproduct of its original pretraining objective. Despite the widespread usage of probes, however, the robust design and analysis of probing experiments remains a challenge. We develop a formal perspective on probing using *structural causal models* (SCM). Specifically, given an SCM which explains the distribution of tokens observed during training, we frame the central hypothesis as whether the LM has learned to represent the latent variables of the SCM. Empirically, we extend a recent study of LMs in the context of a synthetic grid-world navigation task, where having an exact model of the underlying causal structure allows us to draw strong inferences from the result of probing experiments. Our techniques provide robust empirical evidence for the ability of LMs to induce the latent concepts underlying text.

Main paper: 
 introduction: 
 As large LMs pretrained on massive amounts of unlabeled text continue to reach new heights in NLP tasks (and beyond), the question of what kinds of information such models encode about their training data remains a topic of intense discussion and research. One prominent technique is to supervise small probing classifiers to extract some linguistically relevant property from the representations of the pretrained LM (Shi et al., 2016; Adi et al., 2017; Alain & Bengio, 2018) , with the intuition being that the success of the probe reveals the LM has, in fact, learned to encode the property of interest as a byproduct of its training.Despite their widespread usage, however, probes themselves are also an active area of research, with a number of interconnected open questions in the design and interpretation of probing experiments (Belinkov, 2022) , including:(Q1) Control and interpretation. Given that the probe itself is directly supervised to perform the auxiliary task, the observed outcomes could depend not only on the information inherently encoded in the LM but also the ability of the probe to extract the information itself. For instance, researchers have found that training probes to predict randomized labels can often yield comparably high accuracies on certain tasks, calling into question the significance of prior results (Hewitt & Liang, 2019) . As a result, drawing robust conclusions from the classification accuracy of a probe remains up for debate.(Q2) Classifier selection and training. To combat the risk of measuring the probe's capacity to learn the auxiliary task, researchers often limit probes to low capacity architectures such as linear classifiers (Maudslay et al., 2020) . However, other works have countered with evidence that LMs encode more complex concepts using non-linear representations, which can only be accurately measured using higher capacity classifiers (Belinkov & Glass, 2019; Li et al., 2022) . A related question which has received little attention is how the training procedure itself (e.g., optimizer selection, training hyperparameters, auxiliary dataset size) interacts with the outcome of the probing experiment.(Q3) Auxiliary task design. Finally, as large, pretrained LMs have progressed from producing human-like text to exhibiting increasingly "intelligent" behaviors such as reasoning and in-context learning (Brown et al., 2020) , there is an emerging need to better understand the limitations and capabilities of LMs along dimensions such world knowledge and theory of mind. These domains present a distinct set of challenges compared to traditional linguistic tasks such as part-of-speech tagging and dependency parsing.The theoretical section of this paper develops a formal perspective on probing using the language of structural causal models (SCM). Specifically, given a causal model which explains the distribution of tokens observed during training, we pose the central hypothesis as determining whether the LM has learned to represent the latent variables of the SCM: concepts that explain how the text was generated, but are never directly observed during training. We then introduce probes as a means of empirically testing such hypotheses, by extracting the value of the latent concepts given only the LM representations as input. Our setting naturally captures broader questions about the inductive bias of LMs trained solely on text, and the latent concepts they acquire over the course of training (Q3).Next, by extending the SCM beyond the data generation process to cover the training of the LM (unsupervised) and probe (supervised), we further show that Q1 and Q2 can be understood as the mediating and moderating effects of the probe, respectively. We propose a general technique based on causal mediation analysis which isolates the causal path through the LM while excluding the probe's influence. Our analysis yields clear, testable conditions for accepting or rejecting our hypotheses based on a probing experiment's outcomes.Finally, we conduct an empirical study that extends Jin & Rinard (2024) , who use probes to quantify the extent to which LMs are capable of learning "meaning" from text, as operationalized by the semantics of a synthetic programming language for grid-world navigation. By leveraging the proposed latent causal probing framework, our experiments allow us to draw precise conclusions about the causal relationship between the latent dynamics that generated the training data and what is learned by the LM. In particular, we find evidence that (1) the LM has, in fact, learned to represent the latent variables corresponding to the underlying semantics of the language, and (2) the LM representations exhibit an inductive bias that generalizes to novel action sequences. Our study marks the first rigorous empirical evaluation of the hypothesis that language models are latent concept learners, revealing intriguing insights into how language models might acquire an understanding of language. Code to reproduce our experiments is available at https://github.com/charlesjin/emergent-semantics. 
structural causal models of text: 
 This section introduces the setting of our framework for probing, which is based on the idea that the text used to train LMs may exhibit latent causal structure; we formalize these concepts using the approach of structural causal models.Structural causal models are graphical models which represent causal relationships in a data generation process as directed graphs (Pearl et al., 2000) . We refer the reader to Pearl (2010) for a comprehensive overview. Suppose that we are interested in the effect the weather has on employees bringing an umbrella to work. In this case, we may hypothesize a SCM like the one in Figure 1a . Each node represents a random variable: the weather, the weather forecast, whether the employee's morning gets off to a late start, and whether the employee brings an umbrella to work. Nodes without parents are exogenous variables, whose causes are left unexplained; they are often used to model nature or randomness. Nodes with a parent indicate the possibility of a causal relationship, e.g., the edge from weather to forecast indicates that the weather might influence the forecast. A standard assumption of causal analysis is that the underlying causal graph is Markovian (or acyclic).The SCM hypothesizes 3 possible causes for how the weather affects employees bringing an umbrella to work: the weather, the weather forecast, and having a late start. The forecast is a mediator because total causal effect of the weather on umbrella in partially transferred by the path-specific effect over the weather-forecast-umbrella pathway (Avin et al., 2005; Imai et al., 2010) . A natural question is how much the forecast is responsible for the increase in likelihood that an employee brings an umbrella to work when, for instance, the weather changes from sunny to rainy. This can be analyzed via necessary indirect effects, which quantify how much the presence of the causal path through the mediator contributes to the total measured effect (Weinberger, 2019) :NIE rain,sun (Forecast) = E[Umbrella | Weather = rain] -E[Umbrella | Weather = rain, do(Forecast = sun)],where do(Forecast = sun) is a causal intervention that can be conceptualized as forcing the weather station to forecast sun regardless of the weather, thereby severing the weatherforecast-umbrella pathway. Figure 1b depicts the SCM post-intervention.The late start variable is a moderator of the weather-umbrella causal effect: variables that do not directly mediate a causal effect, but affect the strength (and possibly direction) of another causal path (Baron & Kenny, 1986) . For instance, the forecast's effect (i.e., NIE rain,sun (Forecast)) might be lower if the employee has a late start and rushes out the door without checking the forecast.Case study: causal structure in programming languages Jin & Rinard (2024) propose an experiment to study whether LMs are able to ground a sequence of actions into a sequence of states, having only seen instances of the initial and final state during training. Specifically, they train a 350M parameter Transformer (Vaswani et al., 2017) on a corpus of specification-program examples using a standard causal language modeling objective. The programs are strings in a grid-world navigation language with 5 actions (move, turn_right, turn_left, put_marker, pick_marker), sampled uniformly between lengths 6 and 10, inclusive. The specifications consist of the initial and final state, which are 8x8 grids. Executing the program navigates a single robot in the initial state to the final state. We refer to Jin & Rinard (2024) for more details about the language.Figure 2 displays an SCM of the data generation process (along with an example assignment of values to each variable). The exogeneous variables are the initial state and the program actions. Each action produces a latent state (green), save for the last action, which is observed as the final state. A training sample consists of the sequence: s 0 , s n , p 1 , . . . , p n , where each grid world is converted to text by scanning in row order, with one token per entry.Consider now modeling a distribution of text drawn from this SCM. In particular, for each sample x there is an assignment e to the exogenous variables in the SCM M such that M(e) = x. One strategy would be to learn a model of the SCM depicted in Figure 2 , and integrate the latent variables during inference. For instance, knowing that the robot is one space away from s n in s n-1 could help a learner predict p n = move.More generally, given observations generated according to some unknown causal mechanism, a learner could propose various SCMs of the underlying causal mechanism consistent with the observations, then use these SCMs to inform future predictions, an approach known as causal learning (Schölkopf et al., 2021) . A major challenge in the foregoing approach is the problem of latent variable induction, or inferring the latent variables over which candidate SCMs are to be defined. In this work, we focus on the causal structure of programming languages, where the underlying causal dynamics are governed by a precise formal semantics, and the latent variables are given by program states. Having formally defined semantics and latent variables enables us interpret the results of our probing experiments in an unambiguous way; we refer the reader to Sloman (2005); Feder et al. (2022) for surveys of causal structure in natural language. 
latent causal probing: 
 We present latent causal probing, a formal framework for empirically testing the hypothesis Language models are latent concept learners.At a high level, given an SCM that models the training data as the observed variables, we probe the LM for representations of the latent variables of the SCM. Our main insight, as illustrated in Figure 2 , is that knowing the latent value of s n-1 could help predict the observed value of p n ; hence, an LM trained to predict p n might eventually induce the existence of the latent variable s n-1 .We begin by defining the auxiliary task and dataset for probing. Fix some structural causal model M, and let v M be the latent variable of interest. Given some text x, we use v M (x) to denote the value of the latent variable in the SCM of text x. For instance, the value of v M = s 1 in the sample x from Figure 2 is the grid depicted in the s 1 node. We assume that the value of each latent variable is uniquely determined by x and M.Given a language model LM with parameters θ, we denote an arbitrary representation function as LM(x; θ). The auxiliary dataset consists of input features {LM(x; θ) | x ∈ X} and labels {v M (x) | x ∈ D}, where D = {x i } N i=1 is a corpus of text. We then split D into two auxiliary datasets: one for calibration and one for measurement. The probe is trained to predict v M (x) given LM(x; θ) on the calibration data, and the accuracy is taken over the measurement data. We next discuss the design and interpretation of these two datasets. Bound vs. free latent variable outcomes. In general, there may exist several causal dynamics that explain the data equally well. For instance, the following dynamics could also generate the data in Figure 2 :put_marker Jump to a random location.turn_right Return to the last position, put a marker, then turn right.These dynamics assign a different value to s 1 , but explain the observed variables equally well. Assuming the training corpus consists entirely of this single example, it would be impossible to distinguish between M and M ′ on the basis of data alone. In other words:1. M and M ′ share the same set of set of latent, observed, and exogenous variables;2. M and M ′ agree on the observed data; and3. there exists an assignment e to the exogenous variables such that v M (x)̸ = v M ′ (x ′ ) for x = M(e) and x ′ = M ′ (e).In this case, we say that the latent variable v is free over the assignment e. More generally, given a hypothesis class M of SCMs over the same set of variables, denote the LM training data as D train and define M| train to be the subset of SCMs that generate D train . The free latent variable outcomes consist of pairs of latent variables and assignments (v, e) such that there exist M, M ′ ∈ M| train where v M (M(e)) ̸ = v M ′ (M ′ (e)). Any latent variable outcome (v, e) which is not free is bound, i.e., its value is fully determined by the training data.Probing with free vs. bound splits. Table 1 details four possible probing setups when separating the auxiliary dataset D into free and bound splits. In particular, when calibration and measurement occur on the same split, the probe quantifies the knowledge, or information content, that can be extracted from the LM representations; conversely, probing with different splits measures the transferability of the representations across different splits, which is a bias. Additionally, because the bound variables outcomes, can, by definition, be deduced from the given data (and hypothesis class M), measuring on the bound split relates to the deductive ability of the LM; conversely, measuring on the free split is inherently an inductive process. We highlight that the inductive bias can be understood as quantifying the capacity of the LM representations to infer values in unseen data by applying theories derived from known data, a form of inductive inference; while the deductive bias measures the extent to which the LM representations produce theories of unseen data that are consistent with the observed data, a key tenet of deductive logic.We next turn to controlling for the probe (Q1). Intuitively, the challenge is any measurement using a supervised probe conflates the LM's representation of the auxiliary labels with the probe's ability to learn the auxiliary task (Hewitt & Liang, 2019) . While there exist a number of proposals for controlling for the contribution of the probe, such techniques typically do not provide any formal guarantees, rendering their correct application and interpretation a challenge (Belinkov, 2022) .We propose a method of disentangling the two effects using the formal framework of causal mediation analysis, and specifically, path-specific effects, which analyze how causal effects decompose over multiple causal paths (Avin et al., 2005; Imai et al., 2010) . To begin, we extend the SCM of the data generation process to include (1) the LM training, (2) the probe calibration, and (3) the probe measurement. Figure 3 illustrates an example where the hypothesis class M consists of 3-variable SCMs with a single exogenous, observed variable s 0 , a latent variable v M , and another observed variable s 1 .Observe that there are three causal paths from the true SCM of the data generation process to the auxiliary task accuracy, each of which is mediated by a different set of the latent variables v M : (1) during LM training, the LM is trained on a dataset whose text is causally affected by v M ; (2) during probe calibration, the probe is calibrated using v M directly; and(3) during probe measurement, the probe is evaluated for accuracy on v M directly. However, we only care to measure the effect over the first of these causal paths, i.e.:To what extent can the auxiliary task performance be attributed to what LM learns from the latent variables in its training data?This question can be posed formally as the necessary indirect effect of the paths mediated by the LM's learned representation for some baseline causal dynamics M ′ :NIE M,M ′ (θ LM ) = E accuracy | LM is trained on M, probe is calibrated and measured on M -E accuracy | do(LM is trained on M ′ ), probe is calibrated and measured on M],Although path-specific effects offer a crisp conceptual framework for isolating the contribution of the LM in probing experiments, actually computing NIE is not straightforward. First, picking a proper baseline M ′ is critical: if we pick an inappropriate M ′ , then the NIE will measure the difference between the data generated by M and M ′ in addition to the latent variables hypothetically mediated by the LM representations. For instance, we could have M ′ produce gibberish text. Then the LM wouldn't learn anything, biasing the NIE to be positive. Second, measuring the effect requires training a new LM with the baseline M ′ , which would be prohibitively expensive for large pretrained LMs.Let acc aux (M 0 , M 1 ) denote the (expected) auxiliary task accuracy after the LM is trained using the SCM M 0 and the probe is calibrated and measured on M 1 . The following result addresses these challenges (proof in Appendix C). Definition 3.1 (Valid baseline). M ′ is a valid baseline for M ifEQUATIONEQUATION)Proposition 3.2. Let M ′ be a valid baseline for M. Thenacc aux (M, M) -acc aux (M, M ′ ) > 0 implies both NIE M,M ′ (θ LM ) > 0 and NIE M ′ ,M (θ ′ LM ) > 0.Intuitively, M ′ is a valid baseline when measuring M ′ is easier than measuring M under both normal or intervened circumstances. The conclusion then states that, so long as acc aux (M, M)acc aux (M, M ′ ) > 0 (which can be evaluated by running probe calibration and measurement twice rather than training the LM twice), there is no bias in which SCM is used to train the LM and which is the baseline: the LM representations always mediate a positive amount of the measured effect. Intuitively, given (1) an LM trained on either M or M ′ and (2) some neutral text x that is equally likely to have been generated by either M or M ′ , we could distinguish which data the LM was trained on purely on the basis of which latent concepts it assigns to the neutral text x. A positive NIE now also has a rigorous interpretation as the LM having induced latent concepts, as some positive amount of causal effect is transferred through the representations of the LM. For instance, a positive mediated measurement for inductive bias implies thatThe presence of latent causal variables in the pretraining data causes the LM to learn representations that generalize to unknown data.We summarize the latent causal probing framework as follows:1. Fix the exogenous, latent, and observed variables and design a hypothesis class M.2. Pick a target SCM M ∈ M and a set of latent variables v ∈ M to test.3. Construct the auxiliary dataset and create the bound vs. free splits (if possible). 4. Identify a valid baseline M ′ and perform the mediation analysis.A significant measurement acc aux (M, M)acc aux (M, M ′ ) > 0 is interpreted evidence that the LM encodes the latent concepts in its representations. We conclude with some remarks.Choice of hypothesis class. One requirement of our framework is to pick an explicit hypothesis class M, which yields a separation of the latent variables into bound vs. free splits. Conceptually, the choice of hypothesis class forces the experimenter to make explicit their prior about what a language model can "deduce" from data. In our case, we will assume that language models can perfectly memorize the behavior of previously seen programs, but cannot extrapolate this information to any previously unseen programs. Our construction of the hypothesis class reflects this assumption, and the deductive knowledge measures the ability of the language model to directly recall this information from the training data.Interventions, and probing for non-causal latent variables. Our mediation technique requires knowing "what would the text have been if the underlying dynamics were different?", which could be difficult (especially in non-synthetic domains). Similarly, for non-causal latent variables, such as part-of-speech, producing a hypothesis class M with more than one SCM may seem unnatural: what would the data look like in a counterfactual world in which "dog" is actually an adverb? Our analysis suggests that a baseline M ′ which induces a different distribution of text is a necessary precondition, since otherwise NIE M,M ′ (θ LM ) and NIE M ′ ,M (θ ′ LM ) cannot both be positive (as M and M ′ are indistinguishable when used to train the LM and hence θ LM = θ ′ LM ). Intuitively, we interpret this result as saying any measurement is inherently biased when the auxiliary task has only one "right" answer.Probe architecture and hyperparameters. Our framework also explicates the role of the probe's architecture and other hyperparameters in the training process, such as the optimizer, learning rate, dataset size, etc., as potential moderators, but not mediators (Q2). In other words, so long as there exists hyperparameters such that the NIE is positive, the analysis concludes that there exists a causal effect mediated by the model's parameters (although crucially, these choices must not introduce new confounders). Practically speaking, our framework also offers a novel way to interpret (and justify) complex probes (Voita & Titov, 2020; Pimentel & Cotterell, 2021) . 
experiments: 
 We conduct an empirical study of whether an LM, trained from scratch on a corpus of program data, induces the latent concepts in the underlying data generation process. We obtained the original LM checkpoints from Jin & Rinard (2024) . As the LM achieves 92.4% accuracy on generating semantically correct programs for unseen specifications by the end of training, we consider the LM to be well trained in the sense that it has successfully fit not only the explicit objective of minimizing the next-token prediction loss, but also the implicit task encoded by the SCM, i.e., generating programs that correctly implement specifications. We thus evaluate the hypothesis that a well trained language model is a latent concept learner. Appendix A.2 contains further experiments and analyses, including two additional valid baselines and a study of the moderating effect of the probe architecture and training.We describe the key steps according to the framework in Section 3.3; Appendix A.1 contains full experimental details (e.g., LM and probe architecture and training, LM representations).Hypothesis class. The exogenous variables are the initial state and program. The latent variables are the intermediate states, and the observed variables are the initial and final state and the program. For the hypothesis class M, we will assume that each program is a purely symbolic string that represents a sequence of instructions, but that the instructions are not necessarily compositional. For instance, the string "turn_right" could map to the function [[move move]] while the string "turn_right turn_left" could map to the function [[move] ]. Then to execute the program "turn_right turn_left" on an initial state s 0 , the robot would first execute "turn_right" and visit s 1 = move(move(s 0 )), and then execute "turn_right turn_left" which would result in the final state being s 2 = move(s 0 ). Note that the function represented by each string can be inferred from input-output examples, but that the intermediate states in the execution cannot.Target SCM and latent variables. The target SCM M ∈ M is the true data generation process in Figure 2 . The target latent variables consist of the robot's position, facing direction, and whether the robot is facing a rock for each intermediate state.Auxiliary dataset splits. For the auxiliary dataset, we use the same data generation process, except that programs range in length between 1 and 15, and we replace the final state in the specification with the initial state. We assume the LM observes all combinations of the exogenous variables. The bound latent variables are s 6 to s 10 (they correspond to the final state of programs observed during training). The free latent variables are s 1 to s 5 and s 11 to s 15 . Note that we overapproximate the bound latent variables (the input-output behavior for some programs of length 6 to 10 might not be completely known given the training data), but the free latent variables are guaranteed to be free (i.e., the training data does not contain any input-output behavior for programs of length less than 6 or greater than 10).Valid baseline. We construct a valid baseline by using the same causal structure as the target SCM, but permuting the causal dynamics of the turn_right, turn_left, and move actions (e.g., the robot turns left when executing a turn_right action). Note that the valid baseline still has compositional semantics. As M and M ′ are clearly symmetric from a language modeling perspective, Definition 3.1 (and hence Proposition 3.2) holds.Figure 4 plots the main results. For all four measurements (deductive knowledge, inductive bias, deductive bias, and inductive knowledge) and across all three probes (linear, 1-layer MLP, 2-layer MLP), the mediated measurements are significantly positive (each green region is entirely above the corresponding red region) by the end of training. We thus conclude that a positive fraction of the observed measurements of latent concepts can be attributed to what is learned by the LM's representations. We make several additional observations:Spurious features dominate early training. Early in training, both raw and mediated measurements show significant variance and can even decrease with additional training. For example, the mediated measurement for inductive knowledge starts high for all three probes but drops to a minimum around halfway through training. Furthermore, for both the deductive bias and inductive knowledge, the highest raw measurement over the entirely of training is achieved in the first quarter of training by the baseline, rather than the target, SCM. Finally, when using the linear probe to measure the deductive bias, the highest raw measurement for the target SCM does not even occur at the end of training, but at the beginning. We attribute such behavior to spurious features, which are not the result of training but rather exist in random initialization of the LM parameters. These observations suggest that (1) spurious features early in training can lead to uninformative results for probing experiments, and (2) LMs can unlearn spurious features even when correlated with the target SCM, before relearning them again later in training.Deeper probes are (generally) more accurate. For almost every raw measurement across the entirety of training, deeper probes exhibit better auxiliary accuracy. Furthermore, at the end of training, the linear probe exhibits the lowest mediated measurements while the 2-layer MLP exhibits the highest mediated measurement in two of the four tasks. Only in the case of the deductive bias does the 1-layer MLP achieve a substantially higher measurement than the 2-layer MLP. The improved mediated measurements suggests that probes with greater capacity can yield clearer signals and, more generally, highlights the importance of probing frameworks that can robustly account for the design of different probes.Raw measurements can be biased. At the end of training, the raw measurements for inductive knowledge are approximately 52% and 58%, compared to 48% and 51% for deductive knowledge, using a 1-layer and 2-layer MLP, respectively. However, for the mediate measurements, the inductive knowledge shows roughly 4% and 7%, while deductive knowledge shows 12% and 15%, respectively. This reversal demonstrates how raw measurements can be confounded by how easy or hard the auxiliary task is for the probe to fit. 
related work: 
 Causal interpretability of LMs. Several prior lines of work apply causal techniques to the interpretability of LMs. These works typically intervene on either the model's representations (Elazar et al., 2021; Geiger et al., 2021; Meng et al., 2022; Abraham et al., 2022; Li et al., 2022) or the model's inputs (Kaushik et al., 2020; Vig et al., 2020; Gangal & Hovy, 2020; Amini et al., 2023) , and analyze the causal effect on the LM's outputs. In contrast, we present a formal framework that, conceptually, intervenes on the model's training data, and measures the causal effect on the LM's internal representations. Elazar et al. (2022) also study the causal relationship between the LM and its training data, but they focus on the causal effect of dataset statistics on the factuality of LM's outputs.World models in LMs. The hypothesis that LMs can induce latent concepts is related to evidence of world models in LMs, or the extent to which LMs are capable of grounding their inputs to (some representation of) reality. For instance, Li et al. (2021) find that LMs perform entity state tracking over the course of simple stories. Li et al. (2022) show that an LM trained on Othello transcripts develops a representation of the underlying board state. Our work is based on Jin & Rinard (2024) , who show that an LM can develop representations of the intermediate world states underlying a sequence of instructions, given only instances of the initial and final states. However, none of these works provide theoretical guarantees for controlling the classifier's ability to fit the auxiliary objective. This leaves open the possibility that the LM might simply represent the text as is, with either the text directly encoding the world model or the probe learning to infer the latent world model from the text. Our formulation of world models according to the underlying data generation process is also highly related to the position developed by Andreas (2022) , who argues that LMs could act as "agent models" that model properties of agents that are likely to have generated the language in their training data.A number of works have proposed frameworks toward a more rigorous understanding of probing. One line takes an information-theoretic view on the information represented by the LM (Zhu & Rudzicz, 2020; Pimentel et al., 2020; Voita & Titov, 2020; Pimentel & Cotterell, 2021) . Our work leverages causal analysis to attribute the presence of knowledge and biases in the LM's learned representations. Immer et al. (2022) propose an interpretation of probing as quantifying the inductive bias of pretrained representations for downstream tasks, but their framework differs significantly from ours in that the model is understood as a representation-probe pair. In contrast, our approach explicitly aims to separate the LM training and probe calibration procedures using causal mediation analysis, and interprets probes as a means of measuring causal effects. Our analyses also reveals settings in which prior techniques, such as control tasks (Hewitt & Liang, 2019) and the use of untrained or otherwise random baselines (Zhang & Bowman, 2018; Belinkov, 2022) , can yield misleading estimates of the intended auxiliary measurement. 
conclusion: 
 This paper presents latent causal probing, a probing framework that studies whether LMs induce latent variables as a byproduct of the language modeling objective. Our framework offers robust tools for interpreting experiment results through the lens of causal analysis, and in particular, rigorously controls for the probe's contribution in learning the auxiliary task. Experimentally, we extend a previous study of whether We next ablate the probe architecture and hyperparameters by adopting the settings used in Jin & Rinard (2024) . The differences are: no dropout, a batch size of 1024, training the probe for 10000000 steps, and using 100000 samples in the auxiliary dataset. We use the same valid baseline as in Figure 4 of the main text.The results are plotted in Figure 7 . We observe that the general trends are preserved, and all four target measurements are above the baseline measurements by the end of training (i.e., the mediated measurements are positive). However, we note that both deductive and inductive knowledge measure slightly lower, which is an example of the moderating effect of the probe architecture and training hyperparameters. We attribute the effect to the increased batch size and lack of dropout, which could encourage the probe to converge more quickly to a global optimum, given that the risk of overfitting is low (due to the large size and high quality of the training dataset). This is also consistent with (1) the general intuition that simpler (or less optimal) probes are a proxy for "ease of extraction," which is often interpreted as evidence that the representations are "more aligned" with the target features (Hewitt & Liang, 2019) , and (2) the theoretical findings in Pimentel et al. (2020) , who conclude that probes of infinite capacity are most informative for measuring (syntactic) knowledge. 
 
 Appendix: In this section, we highlight several key departures from the experimental design in Jin & Rinard (2024) .First, they do not split their auxiliary dataset into bound and free latent variables, and hence their results do not yield fine-grained interpretations about probing with different calibration and measurement datasets.Second, our analysis reveals the presence of possible confounders in the design of their interventional baseline, leading to uncontrolled effects. In particular, the auxiliary dataset is constructed using programs generated by the LM itself, rather than randomly sampled as we do. Intuitively, this means that the LM "sees" both s 0 and s n , which reveals information about the original casual dynamics. Formally, the representations of the LM used for probing mediates all 3 causal pathways, rather than the simple causal pathway from the LM training data (as in Figure 3 ), and hence their interventional baseline is not a proper measurement of the causal effect mediated by the LM representations. Our solution is to use randomly sampled programs and replace the occurrence of s n with s 0 in the construction of the auxiliary dataset, which breaks this causal dependence.Finally, Jin & Rinard (2024) do not verify that their interventional baselines satisfy the conditions in Equations ( 1) and (2). In particular, one of their baselines map the put_marker and pick_marker actions to turn_right and turn_left, respectively, in addition to permuting the turn_right, turn_left, and move actions. Because the extracted features all relate to the position and direction of the robot, the new dynamics could present a more difficult task (for both the LM and the probe) due to replacing what were effectively no-ops (put_marker and pick_marker) with new operations that affect the position or direction (turn_right, turn_left, and move). Hence, the observed drop in accuracy post-intervention could be attributable to increased task difficulty, rather than the learned representations of the LM.Proof of Proposition 3. ≥ acc aux (M, M)acc aux (M, M ′ ).  


Decision: The paper revisits an old chestnut in the NLP community -- probing. What excited me most about the paper was the application of SCMs. All reviewers highlighted the causal framework the paper introduces as a strength. The only reasons to reject given by the reviewers seem to focus on the number of experiments (after investigating the experiments myself, I disagree), and a limited number of latent variables. I think, however, that the importance of the problem trumps those concerns. Understanding and getting into the causal mechanisms that underlie language models is an important endeavor.

Reviews: review: paper_summary: paper_summary: The paper proposes integrating Structural Causal Models (SCM) with the existing probing techniques in NLP. This method aims to refine how we interpret probing results, providing insights into the information that language models (LMs) encode from their training data.

One key contribution of this paper is its focus on the probe's capacity, addressing a common shortcoming in traditional probing studies. By using causal mediation analysis, the paper effectively distinguishes the information genuinely encoded by the LM from what is learned independently by the probe. This approach is crucial as it aims to ensure that high probing accuracy reflects the LM's capabilities rather than the probe's.

The paper also advances beyond merely associating high probing accuracy with an LM's ability to understand causal structures. Instead, it uses causal models to rigorously test hypotheses about the causal information that LMs might capture, providing a more nuanced interpretation of probing results.

However, the reliance on SCMs assumes clear causal relationships within the training data, which may not always be evident, especially in natural language settings where such relationships can be complex and less defined. This assumption is a significant limitation and could restrict the method's application to more naturalistic datasets where causal relationships are not explicitly outlined.

The paper demonstrates its approach in controlled experimental setups. However, extending this framework to broader, real-world datasets and demonstrating its effectiveness in these contexts could further validate its practicality and impact.
strength_weakness: strength_weakness: 1. **Assumptions on causal relationships**: The method assumes clear causal relationships within the training data for language models. In practice, especially with complex real-world data, these relationships are often unclear or confounded by other factors. 

2. **Limited experiments**: The paper primarily validates its approach in controlled settings. To strengthen the case for publication, it would need to demonstrate effectiveness with real-world datasets, which are generally less structured.

3. **Overfitting**: There is a concern that the causal models could be overfitted to specific characteristics of the training data, given the high dimensionality of language model embeddings. This risk could lead to incorrect conclusions about what the language models are learning.
paper_summary: paper_summary: The authors propose a probing framework based on Structural Causal Models (SCMs) in order to uncover the latent variables LMs learn. As their main contributions, the authors argue that they show that LMs learn the "underlying semantics of the language" (page 2), and they can learn to generalize to unseen data (page 2).

While I agree with the authors that there is much to be explored when it comes to developing robust and reliable probing mechanisms and I appreciate the presentation of SCMs, I fail to see how this work contributes on top of already existing work and what its main audience is.

[edit of the review during discussion period: I adjusted my overall score from 4 to 6.]
strength_weakness: strength_weakness: My main concern is that the overall goal of the paper is unclear to me. 

If the paper is about investigating whether LMs learn "the underlying semantics of the language" (page 2), then this is very well-established already. It is a direct consequence of the good performance of LMs on relational similarity prediction (king-man+woman=queen; see e.g., Vylomova, Rimell, Cohn, Baldwin 2016), using semantic representations for semantic cluster analyses, and vision-language representation space alignment work. If the paper aims to establish overall that LMs latently encode a variety of semantic variables (which I'm again not sure adds much to existing work), this should be evaluated in a suite of experiments. It is my intuition that this isn't the main goal. However, especially the framing of the results, and also the narrative in the abstract and introduction appear to put a significant focus here instead of using this task as a "proof-of-concept".

If the paper is intended as an argument for using this particular SCM-based framework as a probing mechanism, the paper needs to focus on establishing it against alternative probing methods, elaborating on how to use it for a variety of potential downstream tasks, as well as establishing its effectiveness. For example, in the results, the paper solely focuses on the implications for the model under investigation but under this framing, I would instead expect a focus on establishing the method. 

Lastly, I'm confused about the way in which the authors present this work as "interven[ing] on the model's training data" [page 9], as opposed to other work that intervenes on model representations. However, intervening on training data to understand the "causal effect of dataset statistics" [page 9] is the general idea of traditional data augmentation and model generalizability setups (e.g., Perez and Wang 2017; Liu, Kusner, and Blunsom 2021). How does this setup situate within the established data augmentation and generalizability frameworks?
questions: questions: Very minor note: There are multiple cases where the text says "casual" where it's supposed to say "causal" (I assume).

The authors might find the following paper interesting to make connections to given the parallels to SCMs, a focus on the training data, the navigational instruction task, and connections to generalizability and data augmentations (Inducing Causal Structure for Interpretable Neural Networks; Geiger, Wu, Lu et al.; ICML 2022).
paper_summary: paper_summary: This paper proposes latent causal probing, a new probing framework that leverages latent causal structure to design auxiliary tasks. In particular, the task is to assess whether the language model has learned to represent the latent variables of the structural causal model (SCM). A method is proposed to disentangle the causal effects using the causal mediation analysis framework. Theoretical justifications are provided, and empirical results also support the claims of the paper. Overall, the proposed idea is novel, and the paper is well motivated and clearly written.
strength_weakness: strength_weakness: 1. This work only focuses on causal structure of programming languages. It is unclear whether it is easy to extend the proposed latent probing framework to natural language. Additional discussions should be provided in the paper.

2. Experiments only involve an LM trained from scratch. It would be helpful to include evaluations on pretrained LMs.

3. There are some typos in the paper, such as "we refer the ready to" in Section 2.1.
paper_summary: paper_summary: The paper proposes a formal probing framework to test whether LMs have learned the right laten variables to behave as a “structural causal model” (SCM). An SCM forms a directed graph of  token representations as a causal pathway from an initial state to a final state. The proposed methods tries to separate what is learned during pretraining (LM) from the training (calibration) for probing (SCM) by differentiating free (outside the calibration) and bound (seen during pretraining and calibration) variables  Their method is tested using a Code-generation model that is trained on a corpus of specification-program pairs. Their empirical test shows that the LM representation positively contribute to the causal predictive accuracy.
strength_weakness: strength_weakness: - the code generation “language” is very constrained (5 actions) and it is not clear how this can be used for LLMs that learn far more complex and diverse representations that may interfere  in many different ways.
- the section on the results is only 9 lines and way to short. Figure-4 is too small to read and is hardly addressed in the results section. It is not clear what the figure a - h stand for and what the different graph patterns show.
- critical discussion lacks on how this method can be extended to more divers and complex LLMs.

Comments: comment: Thank you for addressing my concerns.

While I still would have liked to see additional experiments in more natural settings, this paper is insightful in its current state and I support accepting it.
Thank you for the update!

> The only remaining concerns are with the limited presentation of the results in the paper and that I believe the paper would benefit from the extra discussion the authors indicate they intend to include.

We agree these suggestions make for a stronger paper and will be sure to do this.
Thank you for the follow up!

 >  If the extra space is used to explain and discuss the results more, I would be more inclined to accept the paper.

We agree this will help make the contributions more clear, and will be sure to dedicate the extra content page to (1) a more in depth discussion of the empirical results and (2) how the methodology can be extended to broader domains (e.g., natural language), including any limitations.
Thank you for replying to my review. If the extra space is used to explain and discuss the results more, I would be more inclined to accept the paper. I also propose to discuss more in depth the fact that on true natural language data and models the method may not give conclusive results due to the enormous complexity and variation in causal reasoning that are required for natural language based tasks.
The authors have provided detailed responses to address my questions. I would like to maintain my positive rating of this paper.
I thank the authors for engaging with my review. Upon revisiting the paper after reading the author response, my questions were largely resolved and I changed my overall rating accordingly. I can see now how this work comes together considering this overall framing. The only remaining concerns are with the limited presentation of the results in the paper and that I believe the paper would benefit from the extra discussion the authors indicate they intend to include.