Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data
Authors: Charles Jin
Venue: COLM (2024)
Keywords: probing, causality, interpretability, understanding, world models
TL;DR: We propose a framework for probing based on structural causal models, which analyses the extent to which LMs are grounded over the latent variables of the data generation process.

Abstract: As language models (LMs) deliver increasing performance on a range of NLP tasks, *probing classifiers* have become an indispensable technique in the effort to better understand their inner workings. A typical setup involves (1) defining an auxiliary task consisting of a dataset of text annotated with labels, then (2) supervising small classifiers to predict the labels from the representations of a pretrained LM as it processes the dataset. A high probing accuracy is interpreted as evidence that the LM has learned to perform the auxiliary task as an unsupervised byproduct of its original pretraining objective. Despite the widespread usage of probes, however, the robust design and analysis of probing experiments remains a challenge. We develop a formal perspective on probing using *structural causal models* (SCM). Specifically, given an SCM which explains the distribution of tokens observed during training, we frame the central hypothesis as whether the LM has learned to represent the latent variables of the SCM. Empirically, we extend a recent study of LMs in the context of a synthetic grid-world navigation task, where having an exact model of the underlying causal structure allows us to draw strong inferences from the result of probing experiments. Our techniques provide robust empirical evidence for the ability of LMs to induce the latent concepts underlying text.

Published as a conference paper at COLM 2024
Latent Causal Probing:
A Formal Perspective on Probing with Causal Models of Data
Charles Jin"
MIT CSAIL ccjecsail.mit.eer
Abstract
As language models (LMs) deliver increasing performance on a range of NLP probing classifiers have become an indispensable technique in the effort to better understand their inner workings. A typical setup involves (1) defining an auxiliary task consisting of a dataset of text annotated with labels, then (2) supervising small classifiers to predict the labels from the high
representations of pretrained LM as processes the dataset. probing accuracy is interpreted as evidence that the LM has learned to perform the auxiliary task as an unsupervised byproduct of its original pretraining objective. Despite the widespread usage of probes, however, the robust design and analysis of probing experiments remains a challenge. We develop formal perspective on probing using structural causal models (SCM). Specifically, given an SCM which explains the distribution of tokens observed during training,
frame the central hypothesis whether the
LM has learned to represent the latent variables of the SCM. Empirically, we extend recent study of LMs in the context of a synthetic grid-world navigation task, where having an exact model of the underlying causal structure allows us to draw strong inferences from the result of probing experiments. Our techniques provide robust empirical evidence for the ability of LMs to induce the latent concepts underlying text.
Introduction
As large LMs pretrained on massive amounts of unlabeled text continue to reach new heights in NLPP tasks (and beyond), the question of what kinds of information such models encode about their training data remains a topic of intense discussion and research. One prominent technique is to supervise small probing classifiers to extract some linguistically relevant property from the representations of the pretrained LM (Shi et al., 2016; Adi et al., 2017; Alain & Bengio, 2018), with the intuition being that the success of the probe reveals the LM has, in fact, learned to encode the property of interest as a byproduct of its training.
Despite their widespread usage, however, probes themselves are also an active area of research, with a number of interconnected open questions in the design and interpretation of probing experiments (Belinkov, 2022), including:
(Q1) Control and interpretation. Given that the probe itself is directly supervised to perform the auxiliary task, the observed outcomes could depend not only on the information inherently encoded in the LM but also the ability of the probe to extract the information itself. For instance, researchers have found that training probes to predict randomized labels can often yield comparably high accuracies on certain tasks, calling into question the significance of prior results (Hewitt & Liang, 2019). As result, drawing robust conclusions from the classification accuracy of a probe remains up for debate.
(Q2) Classifier selection and training. To combat the risk of measuring the probe's capacity to learn the auxiliary task, researchers often limit probes to low capacity architectures such
We gratefully acknowledge the contributions of senior author Martin Rinard, who was unfortu- nately left off the official author list due to late-night clerical error.
1

---

Published as a conference paper at COLM 2024
as linear classifiers (Maudslay et al., 2020). However, other works have countered with evidence that LMs encode more complex concepts using non-linear representations, which can only be accurately measured using higher capacity classifiers (Belinkov & Glass, 2019; Li et al., 2022). A related question which has received little attention is how the training procedure itself (e.g., optimizer selection, training hyperparameters, auxiliary dataset size) interacts with the outcome of the probing experiment.
(Q3) Auxiliary task design. Finally, as large, pretrained LMs have progressed from produc- ing human-like text to exhibiting increasingly "intelligent" behaviors such as reasoning and in-context learning (Brown et al., 2020), there is an emerging need better understand the limitations and capabilities of LMs along dimensions such world knowledge and theory of mind. These domains present a distinct set of challenges compared to traditional linguistic tasks such as part-of-speech tagging and dependency parsing.
The theoretical section of this paper develops a formal perspective on probing using the language of structural causal models (SCM). Specifically, given causal model which explains the distribution of tokens observed during training, we pose the central hypothesis as determining whether the LM has learned to represent the latent variables of the SCM: concepts that explain how the text was generated, but are never directly observed during training. We then introduce probes as means of empirically testing such hypotheses, by extracting the value of the latent concepts given only the LM representations as input. Our setting naturally captures broader questions about the inductive bias of LMs trained solely on text, and the latent concepts they acquire over the course of training (Q3).
Next, by extending the SCM beyond the data generation process cover the training of the LM (unsupervised) and probe (supervised), we further show that Q1 and Q2 can be understood as the mediating and moderating effects of the probe, respectively. We propose a general technique based on causal mediation analysis which isolates the causal path through the LM while excluding the probe's influence. Our analysis yields clear, testable conditions for accepting or rejecting our hypotheses based on probing experiment's outcomes.
Finally, we conduct an empirical study that extends Jin & Rinard (2024), who use probes to quantify the extent to which LMs are capable of learning "meaning" from text, as opera- tionalized by the semantics of synthetic programming language for grid-world navigation. By leveraging the proposed latent causal probing framework, our experiments allow us to draw precise conclusions about the causal relationship between the latent dynamics that gener- ated the training data and what is learned by the LM. In particular, we find evidence that (1) the LM has, in fact, learned to represent the latent variables corresponding to the underlying semantics of the language, and (2) the LM representations exhibit an inductive bias that gen- eralizes to novel action sequences. Our study marks the first rigorous empirical evaluation of the hypothesis that language models are latent concept learners, revealing intriguing insights into how language models might acquire an understanding of language. Code to reproduce our experiments is available at https: http:///gtthtit /ggtthu........ . com/ charlesjind eeergggntteeeeeeeeeeeeeeeeeeeeee semantics.
2 Structural causal models of text
This section introduces the setting of our framework for probing, which is based on the idea that the text used to train LMs may exhibit latent causal structure; we formalize these concepts using the approach of structural causal models.
2.1 Background: structural causal models
Structural causal models are graphical models which represent causal relationships in a data generation process as directed graphs (Pearl et al., 2000). We refer the reader to Pearl (2010) for a comprehensive overview. Suppose that we are interested in the effect the weather has on employees bringing an umbrella to work. In this case, we may hypothesize SCM like the one in Figure 1a. Each node represents a random variable: the weather, the weather forecast, whether the employee's 's morning gets off to late start, and whether the employee brings an umbrella to work. Nodes without parents are exogenous variables, whose causes are left unexplained; they are often used to model nature or randomness. Nodes with
2

---

Published as a conference paper at COLM 2024
Forecast
Late start
Forecaatt sun)
Late start
Weather
Umbrella
Weather
Umbrella
(a) The original SCM.
(b) Intervening on the forecast.
Figure An SCM for bringing an umbrella to work.
parent indicate the possibility of causal relationship, e.g,, the edge from weather to forecast indicates that the weather might influence the forecast. A standard assumption of causal analysis is that the underlying causal graph is Markovian (or acyclic).
Mediators and moderators. The SCM hypothesizes 3 possible causes for how the weather affects employees bringing an umbrella to work: the weather, the weather forecast, and having a late start. The forecast is a mediator because total causal effect the weather on umbrella in partially transferred by the path-specific effect over the cattherfforecst-ummrre pathway (Avin et al., 2005; Imai et al., 2010). A natural question how much the forecast is responsible for the increase in likelihood that employee brings umbrella to work when, for instance, the weather changes from sunny to rainy. This can be analyzed via necessary indirect effects, which quantify how much the presence of the causal path through the mediator contributes to the total measured effect (Weinberger, 2019):
NTE rain,ssu (arransuuFoorrca) (Forecast) E[Umbrella Weather rain]
|E[Umbrella Weather rain, do (Forecast sun)],
where do sun) is a causal intervention that can be conceptualized as forcing the weather station to forecast sun regardless of the weather, thereby severing the weather- forecast-umbrell pathway. Figure 1b depicts the SCM post-intervention.
The late start variable is a moderator the weather-umbretd causal effect: variables that do not directly mediate causal effect, but affect the strength (and possibly direction) of another causal path (Baron & Kenny, 1986). For instance, the forecast's effect (i.e., NTE (Forecast)) might be lower if the employee has late start and rushes out the door without checking the forecast.
2.2 Case study: causal structure in programming languages
Jin & Rinard (2024) propose an experiment to study whether LMs are able to ground sequence of actions into sequence of states, having only seen instances of the initial and final state during training. Specifically, they train 350M parameter Transformer (Vaswani al., 2017) on a corpus of speciffcntion- language modeling objective. The programs are strings in grid-world navigation language with 5 actions (move, turn_right, turn_left, put__arkee between lengths and 10, inclusive. The specifications consist of the initial and final state, which are 8x8 grids. Executing the program navigates a single robot in the initial state to the final state. We refer to Jin & Rinard (2024) for more details about the language.
examples using standard causal
pick_marker" sampled uniformly
Figure 2 displays an SCM of the data generation process (along with an example assignment of values to each variable). The exogeneous variables are the initial state and the program actions. Each action produces a latent state (green), save for the last action, which is observed as the final state. A training sample consists of the sequence: 50,5n, Pi.... Pn, where each grid world is converted to text by scanning in row order, with one token per entry.
Consider now modeling distribution of text drawn from this SCM. In particular, for each sample x there an assignment e to the exogenous variables in the SCM M such that M(e) One strategy would be to learn model of the SCM depicted in Figure 2, and
3

---

Published as a conference paper at COLM 2024
intermediate states
Initial state
final state
narker
P2 tuze, righht
Ph no7e
Figure An SCM of the data generation process for the grid world corpus. The exogenous variables are the initial state and the actions; latent variables are green and observed variables are gray. The training corpus consists programs length between and 10.
integrate the latent variables during inference. For instance, knowing that the robot is one space away from in 8n could help learner predict pn move.
More generally, given observations generated according to some unknown causal mecha- nism, learner could propose various SCMs of the underlying causal mechanism consistent with the observations, then use these SCMs to inform future predictions, an approach known as causal learning (Schölkopf et al., 2021). A major challenge in the foregoing approach is the problem latent variable induction, or inferring the latent variables over which candidate SCMs are to be defined. In this work, we focus on the causal structure of programming languages, where the underlying causal dynamics are governed by a precise formal semantics, and the latent variables given by program states. Having formally defined semantics and latent variables enables us interpret the results of our probing experiments in an unam- biguous way; refer the reader to Sloman (2005); Feder al. (2022) for surveys of causal structure in natural language.
Latent causal probing
We present latent causal probing, a formal framework for empirically testing the hypothesis
Language models are latent concept learners.
At a high level, given an SCM that models the training data as the observed variables, probe the LM for representations of the latent variables of the SCM. Our main insight, as illustrated in Figure 2, is that knowing the latent value of sn could help predict the observed value of Pxi hence, an LM trained to predict pn might eventually induce the existence of the latent variable 5n-1-
3.1 Probing for latent concepts
We begin by defining the auxiliary task and dataset for probing. Fix some structural causal model M, and let UM be the latent variable of interest. Given some text we use (x) to denote the value of the latent variable in the SCM of text x. For instance, the value of UM = in the sample x from Figure 2 is the grid depicted in the node. We assume that the value of each latent variable uniquely determined by x and M.
Given a language model LM with parameters f, we denote an arbitrary representation function as [M(x; The auxiliary dataset consists of input features (LM(x;; f) E X) and labels (x) x E D}, where = { is a corpus of text. We then split into two auxiliary datasets: one for calibration and one for measurement. The probe is trained to predict (x) given (Mxx;;) on the calibration data, and the accuracy is taken over the measurement data. We next discuss the design and interpretation of these two datasets.

---

Published as a conference paper at COLM 2024
calibration
measurement
bound
free
bound
deductive knowledge
inductive bias (inference) inductive knowledge
free
deductive bias (consistency)
Table 1: Interpreting probing with different calibration and measurement datasets.
Bound vs. free latent variable outcomes. general, there may exist several causal dynamics that explain the data equally well. For instance, the following dynamics could also generate the data in Figure 2:
put_marker Jump to random location.
turn_right Return the last position, put marker, then turn right.
These dynamics assign different value to 51, but explain the observed variables equally well. Assuming the training corpus consists entirely of this single example, it would be impossible to distinguish between M and M' on the basis of data alone. In other words:
1. M and M' share the same set of set of latent, observed, and exogenous variables;
2. M and M' agree on the observed data; and
3. there exists an assignment e to the exogenous variables such that (x) & aM (x') for x M(e) and x' M' (c).
In this case, we say that the latent variable v is free over the assignment e. More generally, given a hypothesis class M of SCMs over the same set of variables, denote the LM training data as Dtrain and define M train to be the subset of SCMs that generate Dtrain- The free latent variable outcomes consist pairs of latent variables and assignments (2,e) such that there exist M, M' E M|train where PM (M(e)) # uM (M' (e)). Any latent variable outcome (0,e) which is not free is bound, ie., its value is fully determined by the training data.
Probing with free vs. bound splits. Table details four possible probing setups when separating the auxiliary dataset into free and bound splits. In particular, when calibration and measurement occur the same split, the probe quantifies the knowledge, informa- tion content, that can be extracted from the LM representations; conversely, probing with different splits measures the transferability of the representations across different splits, which is a bias. Additionally, because the bound variables outcomes, can, by definition, be deduced from the given data (and hypothesis class M), measuring on the bound split relates to the deductive ability of the LM; conversely, measuring on the free split is inherently an inductive process. We highlight that the inductive bias can be understood as quantifying the capacity of the LM representations to infer values in unseen data by applying theories derived from known data, form of inductive inference; while the deductive bias measures the extent to which the LM representations produce theories of unseen dala that are consistent with the observed data, a key tenet of deductive logic.
3.2 Causal mediation analysis of probing
We next turn controlling for the probe (Q1). Intuitively, the challenge any measurement using a supervised probe conflates the LM's representation of the auxiliary labels with the probe's s ability learn the auxiliary task (Hewitt & Liang, 2019). While there exist a number of proposals for controlling for the contribution of the probe, such techniques typically do not provide any formal guarantees, rendering their correct application and interpretation challenge (Belinkov, 2022).
We propose a method of disentangling the two effects using the formal framework of causal mediation analysis, and specifically, path-specific effects, which analyze how causal effects decompose over multiple causal paths (Avin et al., 2005; Imai et al., 2010). To begin, we extend the SCM of the data generation process to include (1) the LM training, (2) the probe
5

---

Published as a conference paper at COLM 2024
8g
PLM
LM
LM training
VM
N
8g
LMM 0)
Probe calibration
M
PM (5g
gecbbe
probe
N'
8g
[M(
Probe
measurement
Pa (5g)
accuracy
N
Figure 3: An SCM depicting the LM training, probe calibration, and probe measurement. We use plate notation for repeated iid samples, e.g,, we draw N samples for LM training.
calibration, and (3) the probe measurement. Figure 3 illustrates an example where the hypothesis class M consists of 3-variable SCMs with a single exogenous, observed variable 8p, latent variable VM, and another observed variable s1.
Observe that there are three causal paths from the true SCM of the data generation process to the auxiliary task accuracy, each of which is mediated by different set of the latent variables MM: (1) during LM training, the LM trained on a dataset whose text is causally affected by UMi (2) during probe calibration, the probe is calibrated using UM directly; and (3) during probe measurement, the probe evaluated for accuracy on WM directly. However, only care measure the effect over the first of these causal paths, i.e.:
To what extent can the auxiliary task performance be attributed to what LM learns from the latent variables in its training data?
This question can be posed formally as the necessary indirect effect of the paths mediated by the LM's learned representation for some baseline causal dynamics M':
NIEM, (fLM)
E [accuracy
LM is trained on M, probe is calibrated and measured on M]
EE [accuracy do(LM is trained on M'), probe is calibrated and measured on M),
Although path-specific effects offer a crisp conceptual framework for isolating the contribu- tion of the LM in probing experiments, actually computing NIE is not straightforward. First, picking a proper baseline M' is critical: if we pick an inappropriate M', then the NIE will measure the difference between the data generated by M and M' in addition to the latent variables hypothetically mediated by the LM representations. For instance, we could have M' produce gibberish text. Then the LM wouldn't learn anything, biasing the NIE to be positive. Second, measuring the effect requires training a new LM with the baseline M', which would be prohibitively expensive for large pretrained LMs.
Let acCaux (MCCauxMM... M1) denote the (expected) auxiliary task accuracy after the LM is trained using the SCM Mg and the probe is calibrated and measured on Mj. The following result addresses these challenges (proof in Appendix C).
Definition 3.1 (Valid baseline). M' is a valid baseline for M if
acCaux (M', M') > acCaux (M, M)
(1)
acCaux (M, M²) acCaux (M', M).
(2)

---

Published as a conference paper at COLM 2024
Proposition 3.2. Let M' be valid baseline for M. Then
ACCaux (M, M) acCaux (M, M') > 0
implies both NIE (6rM) > 0 and M > 0.
Intuitively, M' is valid baseline when measuring M' is easier than measuring M under both normal or intervened circumstances. The conclusion then states that, so long as (cCaux(M,,) M) aCCaux (M, M') > 0 (which can be evaluated by running probe calibration and measurement twice rather than training the LM twice), there is no bias in which SCM is used to train the LM and which is the baseline: the LM representations always mediate positive amount of the measured effect. Intuitively, given (1) an LM trained on either M or M' and (2) some neutral text x that is equally likely to have been generated by either M or M', we could distinguish which data the LM was trained on purely on the basis of which latent concepts it assigns to the neutral text x. A positive NIE now also has a rigorous interpretation as the LM having induced latent concepts, as some positive amount of causal effect is transferred through the representations of the LMM. For instance, a positive mediated measurement for inductive bias implies that
The presence of latent causal variables in the pretraining data causes the LM to learn representations that generalize to unknown data.
3.3 Discussion
We summarize the latent causal probing framework follows:
1. Fix the exogenous, latent, and observed variables and design a hypothesis class M.
2. Pick a target SCM M E .M and a set of latent variables v E M to test.
3. Construct the auxiliary dataset and create the bound vs. free splits (if possible).
4. Identify a valid baseline M' and perform the mediation analysis.
A significant measurement acCaux (M, M) ACCaux (M, M') > O is interpreted evidence that the LM encodes the latent concepts in its representations. We conclude with some remarks.
Choice of hypothesis class. One requirement of our framework is to pick an explicit hypothesis class M, which yields a separation of the latent variables into bound vs. free splits. Conceptually, the choice of hypothesis class forces the experimenter to make explicit their prior about what language model can "deduce" from data. In our case, we will assume that language models can perfectly memorize the behavior of previously seen programs, but cannot extrapolate this information to any previously unseen programs. Our construction of the hypothesis class reflects this assumption, and the deductive knowledge measures the ability of the language model to directly recall this information from the training data.
Interventions, and probing for non-causal latent variables. Our mediation technique re- quires knowing "what would the text have been if the underlying dynamics were different?", which could be difficult (especially in non-synthetic domains). Similarly, for non-causal latent variables, such as part-of-speech, producing a hypothesis class M with more than one SCM may seem unnatural: what would the data look like in a counterfactual world in which "dog" is actually an adverb? Our analysis suggests that a baseline M' which induces a different distribution of text is a necessary precondition, since otherwise M. (fLM) and NIE M (f(M) cannot both be positive (as M and M' are indistinguishable when used to train the LM and hence LMM = B(y)) Intuitively, we interpret this result as saying any measurement is inherently biased when the auxiliary task has only one "right" answer.
Probe architecture and hyperparameters. Our framework also explicates the role of the probe's s architecture and other hyperparameters in the training process, such as the optimizer, learning rate, dataset size, etc., as potential moderators, but not mediators (Q2). In other words, so long as there exists hyperparameters such that the NIE is positive, the analysis concludes that there exists a causal effect mediated by the model's parameters (although crucially, these choices must not introduce new confounders). Practically speaking, our
7

---

Published as a conference paper at COLM 2024
framework also offers novel way to interpret (and justify) complex probes (Voita & Titov, 2020; Pimentel & Cotterell, 2021).
Experiments
We conduct an empirical study of whether an LM, trained from scratch on corpus of program data, induces the latent concepts in the underlying data generation process. We obtained the original LM checkpoints from Jin & Rinard (2024), As the LM achieves 92.4% accuracy on generating semantically correct programs for unseen specifications by the end of training, we consider the LM to be well trained in the sense that it has successfully fit not only the explicit objective of minimizing the next-token prediction loss, but also the implicit task encoded by the SCM, i.e., generating programs that correctly implement specifications. We thus evaluate the hypothesis that well trained language model is a latent concept learner. Appendix A.2 contains further experiments and analyses, including two additional valid baselines and a study of the moderating effect of the probe architecture and training.
4.1 Methods
We describe the key steps according to the framework in Section 3.3; Appendix A.1 contains full experimental details (e.g,, LM and probe architecture and training, LM representations).
Hypothesis class. The exogenous variables are the initial state and program. The latent variables are the intermediate states, and the observed variables are the initial and final state and the program. For the hypothesis class M, we will assume that each program is purely symbolic string that represents sequence of instructions, but that the instructions are not necessarily compositional. For instance, the string "t "turn_right" could map to the function [[move move]] while the string "turn_right turn_left" could map to the function [[move]]. Then to execute the program "turn_right turn_left" on an initial state so, the robot would first execute "turn_right" and visit $ move (ooeemove(ss) "turn_right turn_left" which would result in the final state being $2 = move move(so). Note that the function represented by each string can be inferred from input-output examples, but that the intermediate states in the execution cannot.
and then execute
Target SCM and latent variables. The target SCM M M is the true data generation process in Figure 2. The target latent variables consist of the robot's position, facing direction, and whether the robot is facing rock for each intermediate state.
Auxiliary dataset splits. For the auxiliary dataset, we use the same data generation process, except that programs range in length between 1 and 15, and we replace the final state in the specification with the initial state. We assume the LM observes all combinations of the exogenous variables. The bound latent variables are % to $10 (they correspond state of programs observed during training). The free latent variables are 91 to $5 and $11 to $15. Note that we overapproximate the bound latent variables (the input-output behavior for some programs of length 6 to 10 might not be completely known given the training data), but the free latent variables are guaranteed to be free (i.e., the training data does not contain any input-output behavior for programs of length less than 6 greater than 10).
the final
Valid baseline. We construct a valid baseline by using the same causal structure as the target SCM, but permuting the causal dynamics of the turn_right turn_left, and move actions (e-g., the robot turns left when executing a turn_right action). Note that the valid baseline still has compositional semantics. As M and M' are clearly symmetric from language modeling perspective, Definition 3.1 (and hence Proposition 3.2) holds.
4.2 Results
Figure 4 plots the main results. For all four measurements (deductive knowledge, inductive bias, deductive bias, and inductive knowledge) and across all three probes (linear, 1-layer MLP, 2-layer MLP), the mediated measurements are significantly positive (each green region is entirely above the corresponding red region) by the end of training. We thus conclude
8

---

Published as a conference paper at COLM 2024
45%
50%
40%
Probe Mcasurement
Probe Measurement
35%
45%
30%
40%
25%
20%
35%
15%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Training Progress
Training Progress
(a) Deductive knowledge.
(b) Inductive bias.
60%
Probe Measurement
Probe Measurement
30%
55%
20%
50%
45%
10%
40%
0%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Training Progress
Training Progress
(c) Deductive bias.
(d) Inductive knowledge.
Figure 4: Experimental results. Solid, dashed, and dotted lines plot the median accuracy of a linear, 1-layer MLP, and 2-layer MLP probes, respectively. Each shaded area represents the full range over 5 random seeds. Green is the target SCM and red is the baseline.
that positive fraction the observed measurements of latent concepts can be altributed to what is learned by the LM's representations. We make several additional observations:
Spurious features dominate early training. Early in training, both raw and mediated measurements show significant variance and can even decrease with additional training. For example, the mediated measurement for inductive knowledge starts high for three probes but drops to minimum around halfway through training. Furthermore, for both the deductive bias and inductive knowledge, the highest raw measurement over the entirely of training is achieved in the first quarter of training by the baseline, rather than the ta target, SCM. Finally, when using the linear probe to measure the deductive bias, the highest raw measurement for the target SCM does not even occur at the end of training, but the beginning. We attribute such behavior to spurious features, which are not the result of training but rather exist in random initialization of the LM parameters. These observations suggest that (1) spurious features early in training can lead to uninformative results for probing experiments, and (2) LMs can unlearn spurious features even when correlated with the larget SCM, before relearning them again later in training.
Deeper probes are (generally) more accurate. For almost every raw measurement across the entirety of training, deeper probes exhibit better auxiliary accuracy. Furthermore, at the end of training, the linear probe exhibits the lowest mediated measurements while the 2-layer MLP exhibits the highest mediated measurement in two of the four tasks. Only in the case the deductive bias does the 1-layer MLP achieve a substantially higher measurement than the 2-layer MLP. The improved mediated measurements suggests that probes with greater capacity can yield clearer signals and, more generally, highlights the importance of probing frameworks that can robustly account for the design of different probes.

---

Published as a conference paper at COLM 2024
Raw measurements can be biased. At the end of training, the raw measurements for inductive knowledge are approximately 52% and 58%, compared to 48% and 51% for de- ductive knowledge, using 1-layer and 2-layer MLP, respectivll However, for the mediate measurements, the inductive knowledge shows roughly 4% and 7%, while deductive knowl- edge shows 12% and 15,, respectively. This reversal demonstrates how raw measurements can be confounded by how easy or hard the auxiliary task is for the probe to fit.
5 Related work
Causal interpretability of LMs. Several prior lines of work apply causal techniques to the interpretability of LMs. These works typically intervene on either the model's representations (Elazar et al., 2021; Geiger et al., 2021; Meng et al., 2022; Abraham et al., 2022; Li al., 2022) or the model's inputs (Kaushik et al., 2020; Vig et al., 2020; Gangal & Hovy, 2020; Amini et al., 2023), and analyze the causal effect on the LM's outputs. In contrast, we present a formal framework that, conceptually, intervenes on the model's fraining data, and measures the causal effect on the LM's internal representations. Elazar et al. (2022) also study the causal relationship between the LM and its training data, but they focus on the causal effect of dataset statistics on the factuality of LM's outputs.
World models in LMs. The hypothesis that LMs can induce latent concepts is related to evidence of world models in LMs, or the extent to which LMs are capable of grounding their inputs to (some representation of) reality. For instance, Li et al. (2021) find that LMs perform entity state tracking over the course of simple stories. Li al. (2022) show that an LM trained on Othello transcripts develops a representation of the underlying board state. Our work is based on Jin & Rinard (2024), who show that an LM can develop representations of the intermediate world states underlying sequence of instructions, given only instances of the initial and final states. However, none of these works provide theoretical guarantees for controlling the classifier's ' ability to fit the auxiliary objective. This leaves open the possibility that the LM might simply represent the text as is, with either the text directly encoding the world model or the probe learning to infer the latent world model from the text. Our formulation of world models according to the underlying data generation process is also highly related to the position developed by Andreas (2022), who argues that LMs could act as "agent models" that model properties of agents that are likely to have generated the language in their training data.
Frameworks for probing. A number of works have proposed frameworks toward more rigorous understanding of probing. One line takes an nnformation-theorree view on the information represented by the LM (Zhu & Rudzicz, 2020; Pimentel et al., 2020; Voita & Titov, 2020; Pimentel & Cotterell, 2021). Our work leverages causal analysis to attribute the presence of knowledge and biases in the LM's learned representations. Immer et al. (2022) propose an interpretation of probing as quantifying the inductive bias of pretrained representations for downstream tasks, but their framework differs significantly from ours in that the model is understood as ippresentation-probe pair. In contrast, our approach explicitly aims to separate the LM training and probe calibration procedures using causal mediation analysis, and interprets probes as a means of measuring causal effects. Our analyses also reveals settings in which prior techniques, such as control tasks (Hewitt & Liang, 2019) and the use of untrained or otherwise random baselines (Zhang & Bowman, 2018; Belinkov, 2022), can yield misleading estimates of the intended auxiliary measurement.
Conclusion
This paper presents latent causal probing, a probing framework that studies whether LMs induce latent variables as byproduct of the language modeling objective. Our framework offers robust tools for interpreting experiment results through the lens of causal analysis, and in particular, rigorously controls for the probe's contribution in learning the auxiliary task. Experimentally, we extend a previous study of whether Transformers can infer the intermediate states that underlie sequence of actions. Our results provide strong empirical evidence that LMs can induce latent concepts from textual pretraining.
10

---

Published as a conference paper at COLM 2024
Acknowledgements
We sincerely thank Jacob Andreas, Armando Solar-Lezama, and the anonymous COLM reviewers for their insightful feedback on earlier drafts of this manuscript. We also grate- fully acknowledge support from DARPA Grants HR001120C0015,, HR001120C0191, and N6600120C4025. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the United States Government.
References
Eldar David Abraham, Karel D'Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. CEBab: Estimating the causal effects of real-world concepts on NLP model behavior. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022. URL https: //ppr//v/eeeeeeieweeeeeeeee wtt.:t/ooooomm.d-aaiiiiilgl
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations, 2017. URL
BJh6Ztux1.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018
Afra Amini, Tiago Pimentel, Clara Meister, and Ryan Cotterell. Naturalistic Causal Probing for Morpho-Syntax. Transactions of the Association for Computational Linguistics, 11:384-403, 05 2023. ISSN 2307-387X. doi: 10.1162 /t//tacllaa a_00554. URL https:://oi..org/////// cac1_a_00554.
Jacob Andreas. Language models as agent models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5769-5779, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ /v1/ htt/2222.inningss-ese. laaclannholoog.orr//22200 wttp:e..ffinniiii findings-emmlp... 423.
URL https:
Chen Avin, llya Shpitser, and Judea Pearl. Identifiability of path-specific effects. In IJCAI International Joint Conference on Artificial Intelligence, pp. 357-363, 2005.
effects
Reuben M Baron and David A Kenny. The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations, Journal of personality and social psychology, 51 (6):1177, 1986.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Com- putational Linguistics, 48(1):207-219 March 2022. doi: 10.1162/coli, a_00422. URL
https://aaclanthologgy /ttp c1-1..7 c1-1.7.
Yonatan Belinkov and James Glass. Analysis Methods in Neural Language Processing: A Survey. Transactions of the Association for Computational Linguistics, 7:49-72, 04 2019. ISSN 2307-387X. doi: 10.1162 tacl 00254. URL h/tpp:://oi. //tp::///ooooo/...///////// 1112/tacl_a00222
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Adoances in neural information processing systems, 33:1877-1901, 2020.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 03 2021. ISSN 2307-387X. doi: 10.1162 tacl/aclla. a 00359, URL https: //doi..rg//e..0 116.1....1116/11 1162/taccl 003399
11

---

Published as a conference paper at COLM 2024
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Mar- ius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. Measuring causal effects of data statistics on language mfdel'sfactual infactualppredicttin arXiv preprint ar arXi::2207.144 14251,, 2022.
Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. Causal inference in natural language processing: Estimation, prediction, interpretation pro and beyond. Transactions of the Association for Computational Linguistics, 10:1138-1158, 2022.
Varun Gangal and Eduard Hovy. BERTering RAMS:
What and how much does BERT already know about event arguments? a study on the RAMS dataset. In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupata, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings of the Third BlackboxNIP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 1-10, Online, November 2020. Association for Computational Linguistics. doi: 10. 1t///////2220.baaaooo/llpppppp 2020. blackboxnlp-1..1
URL
org/
Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. Causal abstractions of neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:/ net/ forum?id-RmuXDtjDnn..
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP)), pp. 2733-2743, 2019.
2019.
Kosuke Imai, Luke Keele, and Dustin Tingley. A general approach to causal mediation analysis. Psychological methods, 15(4):309, 2010.
Alexander Immer, Lucas Torroba Hennigen, Vincent Fortuin, and Ryan Cotterell. Probing as quantifying inductive bias. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics ( (Volume 1: Long Papers), pp. 1839-1851, 2022.
Charles Jin and Martin Rinard. Emergent representations of program semantics in language models trained on programs. In Forty-first International Conference on Machine Learning, 2024. URL https: / /ttpss://operreview.. hwtp:/nee/foooooooooo
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes difference with counterfactullly-aggeeete data. In International Conference Learning Representations, 2020. URL https://opennew///
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 1813-1827, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ /wwp///.acllooogg...... 143. URL https: //a:: hat.s//holaaanooo.o.ooo.o org. grgg/221.... acl-long. 143.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task. In The Fleventh International Conference on Learning Representations, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL http::///pppnneeeeeeeefffrrrro Bkg6RiCqY7.
Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, and Ryan Cotterell. A tale of a probe and a parser. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Compulational
12

---

Published as a conference paper at COLM 2024
Linguistics, pp. 7389-73995 Online, July 2020. Association for Computational Linguistics. 10 doi: 10.18653 w2/2020..cllaa mcl-mmain6659 URL https: httts:://allanthooogg... org/ orgg/020. acl-main. 659.
Kevin Meng, Mevin David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations gpt. Advances Neural Information Processing Systems, 35:17359-17372, 2022.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou,Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. ICLR, 2023.
Judea Pearl. An introduction to causal inference. The international journal of biostatistics, 6(2), 2010.
Judea Pearl al. Models, reasoning and inference. Cambridge, UK: CambriigeUnnioerstyppresss 19(2):3, 2000.
Tiago Pimentel and Ryan Cotterell. A Bayesian framework for information-theorrtie probing. In Mranii-r Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2869-2887, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 11001/66///211..eeeeeee //aclanthologg.orr .aceey..rr//2// org/2021. httg//22.e.mmmmmmai...... 229.
URL https:
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and probing for linguistic structure. In Proceedings of the
Ryan Cotterell. Information- 58th Annual Meeting of the Association for Computational Linguistics, pp. 4609-4622, 2020.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the TEEE, 109(():612-66)) 2-34,, 2021.
Xing Shi, Inkit Padhi, and Kevin Knight. Does string-based neural mt learn source syntax? In Proceedings of the 2016 conference on empirical methods in natural language processing, pp- 1526-1534, 2016.
Steven Sloman. Locating Causal Structure in Language. In Causal Models: How Peo- ple Think about the World and Its Allernatives. Oxford University Press, 08 2005. ISBN URL https:///oi... org/
9780195183115. doi: 10.1093 000331115.0000.000001001 0011.
10. 10910093//// /093//cprof::.. 978 95183115
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 01112388-12000 2020.
Elena Voita and Ivan Titov. Information-theoree probing with minimum description length. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 183-196, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/vil org/2020. emnlp-main. 14.
2020.emnlp-main. 14. URL
Naftali Weinberger. Path-specific effects. British Journal for the Philosophy of Science, 70(1): 53-76, 2019. doi: 10.1093 /0.1093/bjps//ae bts////spp//xxx////////
Thomas Wolf, Lysandre Debut, Victor Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Syl- vain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical
13

---

Published as a conference paper at COLM 2024
Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v v v 2t01/2220.eemmm w00.0eemml--dmmm....... URL http::://aclaathollleeee/ org/ 2020. emnnpp-dmoss.e 6.
Kelly Zhang and Samuel Bowman. Language modeling teaches you more than translation Lessons learned through auxiliary syntactic task analysis. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 359-361, Brussels, Belgium, Novem- ber 2018. Association for Computational Linguistics. doi: 10. 20.1663/v11W1885444 URL / hctts://aclanthhon
does:
pp.
o 8 0rr//8885548.
Zining Zhu and Frank Rudzicz. An information theoretic view selecting linguistic probes. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 9251-9262 2020.
14

---

Published as a conference paper at COLM 2024
A
Additional experimental details and results
A.1 Language model and probe details
Following Jin & Rinard (2024), the language model is 350M parameter CodeGen model (Nijkamp et al., 2023) taken from the HuggingFacc Transformers library (Wolf et al., 2020). The model was trained for 2.5 billion tokens, which was roughly 6 passes or 80000 training batches over the training corpus. We refer to Jin & Rinard (2024) for further details.
We next describe the design and training of the probing classifiers; these notes apply to all the probing experiments, unless otherwise noted. The linear probe is single linear layer. The MLP probes have ReLU, batch norm, then 2ropout(p=2) after each linear layer. The hidden dimensions of the 1-layer and 2-layer MLP probes were (256,) and (256, 1024), respectively. The auxiliary datasets consisted of 500000 randomly selected samples. To extract representations from the LM, we use the same strategy Jiin & Rinard (2024), averaging the LM hidden states over the layer dimension after processing each program token. Probes were trained using A (Loshchilov Hutter, 2019) with weight decay of le-4. The learning rate starts at 0.01, then decays by .1 at 75% and 90% through training. All probes are trained for 2000000 steps using batch size of 256.
For the mediated results reported in Figure 4, we generated the auxiliary dataset using an SCM that maps turn_right turn_left, turn_left move, and move to turn_right.
A.2
Ablation studies
This section present some ablation studies on the set up of the probing experiments.
A.2.1 Valid baseline selection
To test the sensitivity of the mediated results (and hence, overall conclusions) on the choice of valid baseline, we generate two additional auxiliary datasets with the following SCMs:
1. swap move and turn_left
2. swap turn_right and turn_left
The results are plotted in Figure 5 and Figure 6, respectively. We find that the mediated measurements in the first case are nearly identical to those in Figure 4, despite only swapping two actions (instead of permuting three). However, in the second case, the mediated measurements are essentially noise, with almost complete overlap between the target and baseline auxiliary measurements. We attribute this to the fact that the resulting labels are extremely similar, as, in most cases, the robot is simply reflected along the starting axis.
We emphasize that, strictly speaking, no amount of negative empirical results can defini- tively prove that LMs are never capable of latent concept learning. Conversely, a single positive result with a valid baseline does constitute evidence that LMs can sometimes induce latent concepts from text.
15

---

Published as a conference paper at COLM 2024
45%
50%
40%
Probe Mcasurement
Probe Measurement
35%
45%
30%
25%
20%
35%
15%
0%
20%
40%
60%
80%
100%
20%
60%
80%
100%
Training Progress
Training Progress
(a) Deductive knowledge.
(b) Inductive bias.
40%
60%
Probe Mcasurement
Probe Measurement
30%
55%
20%
50%
10%
45%
40%
0%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Training Progress
Training Progress
(c) Deductive bias.
(d) Inductive knowledge.
Figure 5: Mediating with the valid baseline that swaps move and turn_left.
16

---

Published as a conference paper at COLM 2024
45%
50%
40%
Probe Mcasurement
Prohe Measurement
35%
45%
30%
25%
20%
35%
15%
0%
20%
40%
60%
80%
100%
20%
60%
80%
100%
Training Progress
Training Progress
(a) Deductive knowledge.
(b) Inductive bias.
40%
Probe Mcasurement
Probe Measurement
55%
30%
50%
20%
45%
10%
40%
0%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Training Progress
Training Progress
(c) Deductive bias.
(d) Inductive knowledge.
Figure 6: Mediating with the valid baseline that swaps turn_right and turn_left.
17

---

Published as a conference paper at COLM 2024
A.2.2
Probe architecture and hyperparameters
45%
60%
40%
Probe Mcasurement
Probe Measurement
35%
50%
30%
25%
20%
15%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Training Progress
Training Progress
(a) Deductive knowledge.
(b) Inductive bias.
50%
70%
65%
Probe Mcasurement
40%
Probe Mcasurement
60%
30%
55%
20%
50%
10%
45%
40%
0%0 6%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Training Progress
Training Progress
(c) Deductive bias.
(d) Inductive knowledge.
Figure 7: Mediating with the valid baseline from the main text. Both the original and the mediated measurements are retaken using the probe architecture and hyperparameters from Jin & Rinard (2024).
We next ablate the probe architecture and hyperparameters by adopting the settings used in Jin & Rinard (2024). The differences are: no dropout, batch size of 1024, training the probe for 10000000 steps, and using 100000 samples in the auxiliary dataset. We use the same valid baseline as in Figure 4 of the main text.
The results are plotted in Figure 7. We observe that the general trends are preserved, and all four target measurements are above the baseline measurements by the end of training (i.e., the mediated measurements are positive). However, we note that both deductive and inductive knowledge measure slightly lower, which is an example of the moderating effect of the probe architecture and training hyperparameters. We attribute the effect to the increased batch size and lack of dropout, which could encourage the probe to converge more quickly to a global optimum, given that the risk of overfitting is low (due to the large size and high quality of the training dataset). This is also consistent with (1) the general intuition that simpler (or less optimal) probes are proxy for "ease of extraction," which is evidence that the representations are "more aligned" with the target
often interpreted features (Hewitt & Liang, 2019), and (2) the theoretical findings in Pimentel et al. (2020), who conclude that probes of infinite capacity are most informative for measuring (syntactic) knowledge.
18

---

Published as a conference paper at COLM 2024
B Comparison with Jinn & Rinard (2024)
In this section, we highlight several key departures from the experimental design in Jin & Rinard (2024).
First, they do not split their auxiliary dataset into bound and free latent variables, and hence their results do not yield fine-grained interpretations about probing with different calibration and measurement datasets.
Second, our analysis reveals the presence of possible confounders in the design of their interventional baseline, leading to uncontrolled effects. In particular, the auxiliary dataset is constructed using programs generated by the LM itself, rather than randomly sampled as we do. Intuitively, this means that the LM "sees" both so and Sn, whicch reveals information about the original casual dynamics. Formally, the representations of the LM used for probing mediates all 3 causal pathways, rather than the simple causal pathway from the LM training data (as in Figure 3), and hence their interventional baseline is not proper measurement of the causal effect mediated by the LM representations tions. Our solution is to use randomly sampled programs and replace the occurrence of Sn with So in the construction of the auxiliary dataset, which breaks this causal dependence.
Finally, Jin & Rinard (2024) do not verify that their interventional baselines satisfy the condi- tions in Equations (1) and (2). In particular, one of their baselines map the put_marker and pick_marker actions to turn_right and turn_left, respectively, in addition to permuting the turn_right, turn_left, and move actions. Because the extracted features all relate to the position and direction of the robot, the new dynamics could present more difficult task (for both the LM and the probe) due replacing what were effectively no-ops (put_marker and pick_marker) with new operations that affect the position or direction (turn_right, turn_left, and move). Hence, the observed drop accuracy post-intervention could be attributable to increased task difficulty, rather than the learned representations of the LM.
C Proofs
Proof of Proposition 3.2. The proof follows directly from substituting the appropriate assump- tions into the definitions of NIE. Recall that
MIEM,M' (0LM) =
acCaux (M, M) - acc (M',M)
(3)
NIEM MM (f(M) := acCaux (M', MM) accaux (M, M'),
(4)
and, by Definition 3.1, M is valid baseline for M if
acCaux(M', (a'uxMM,,M'' M') 2 acCaux (M, M) acCaux (M,,ux,M,M' M') acCaux (M'' M).
(5)
(6)
Applying Equation (5) to the definitions of NIE, NIEM, (fLM) S acCaux (M', "T,' accaux (M', M)
(7)
NIEM,, (0(M)
(8)
Applying Equation (6) to the definition of NIE, MIE (0LM) acCaux (M, M) acCaux (M', MM)
(9)
> ACCaux (eaauxx(M. (M, M) acCaux (M, M').
(10)
Hence,
acCaux (M, M) - acCaux (M, < &NIE (0LM) S NIEMMMMPH
(11)
19


Reviews:
[{"review_id": "kkuhrqYwDx", "reviewer": "Reviewer_Jppb", "summary": "The paper proposes integrating Structural Causal Models (SCM) with the existing probing techniques in NLP. This method aims to refine how we interpret probing results, providing insights into the information that language models (LMs) encode from their training data.\n\nOne key contribution of this paper is its focus on the probe's capacity, addressing a common shortcoming in traditional probing studies. By using causal mediation analysis, the paper effectively distinguishes the information genuinely encoded by the LM from what is learned independently by the probe. This approach is crucial as it aims to ensure that high probing accuracy reflects the LM's capabilities rather than the probe's.\n\nThe paper also advances beyond merely associating high probing accuracy with an LM's ability to understand causal structures. Instead, it uses causal models to rigorously test hypotheses about the causal information that LMs might capture, providing a more nuanced interpretation of probing results.\n\nHowever, the reliance on SCMs assumes clear causal relationships within the training data, which may not always be evident, especially in natural language settings where such relationships can be complex and less defined. This assumption is a significant limitation and could restrict the method's application to more naturalistic datasets where causal relationships are not explicitly outlined.\n\nThe paper demonstrates its approach in controlled experimental setups. However, extending this framework to broader, real-world datasets and demonstrating its effectiveness in these contexts could further validate its practicality and impact.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad the reviewer appreciates our contributions to the state of probing! We hope our rebuttal can address the remaining concerns\n\n> Assumptions on causal relationships\n\nWhile we agree this makes it difficult to apply the technique directly on real world data, we do want to provide another perspective, which is that our assumptions put the risks of probing on real world data in a *mathematically precise language*. Specifically, the lack of a clear causal relationship means that we cannot hope for a valid baseline in the exact sense of Proposition 1, so extra care should be taken when interpreting results against a baseline measurement.\n\n> Limited experiments\n\nWe chose to test our methodology in controlled settings so that we could follow the methodology as faithfully as possible. We felt that, given that the methodology is itself a major contribution of the paper, conducting our experiments under “ideal” circumstances would allow us to demonstrate the benefits of the new framework as cleanly as possible and minimize concerns around whether the results could be due to a misapplication of the methodology.\n\nWe would also like to emphasize that our experiments are, to the best of our knowledge, the first to show that LMs *can* infer free latent variables from text. We think there’s an argument for conducting experiments in a controlled environment where the complexity of real-world data does not intrude on experimental design.\n\n> Overfitting\n\nOur understanding of this point is that the high dimensionality of the LM representations may allow the probe to “find something which is not really there”. However, having a valid baseline (Proposition 1) actually controls for this already: loosely speaking, a valid baseline would benefit equally from the high dimensionality of the LM representations.\n\nIf this is not the correct reading, please let us know and we would be happy to respond in the discussion!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Methods**: Integrating Structural Causal Models (SCM) with language model probing is both novel and interesting. This methodology introduces a new way to interpret what language models learn from data, focusing on understanding causal relationships rather than accuracy metrics. \n\n2. **Tackling limitations of probing**: The paper tackles the limitations of current probing practices by focusing on the probe's capacity to learn independently of the language model. This is important for obtaining more accurate interpretations of model behaviors and capabilities, addressing a gap in how probes are currently used.", "reasons_to_reject": "1. **Assumptions on causal relationships**: The method assumes clear causal relationships within the training data for language models. In practice, especially with complex real-world data, these relationships are often unclear or confounded by other factors. \n\n2. **Limited experiments**: The paper primarily validates its approach in controlled settings. To strengthen the case for publication, it would need to demonstrate effectiveness with real-world datasets, which are generally less structured.\n\n3. **Overfitting**: There is a concern that the causal models could be overfitted to specific characteristics of the training data, given the high dimensionality of language model embeddings. This risk could lead to incorrect conclusions about what the language models are learning.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A0USTEcDiF", "reviewer": "Reviewer_rGMR", "summary": "The authors propose a probing framework based on Structural Causal Models (SCMs) in order to uncover the latent variables LMs learn. As their main contributions, the authors argue that they show that LMs learn the \"underlying semantics of the language\" (page 2), and they can learn to generalize to unseen data (page 2).\n\nWhile I agree with the authors that there is much to be explored when it comes to developing robust and reliable probing mechanisms and I appreciate the presentation of SCMs, I fail to see how this work contributes on top of already existing work and what its main audience is.\n\n[edit of the review during discussion period: I adjusted my overall score from 4 to 6.]", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to engage with our submission! If there are any remaining concerns we can address, please do let us know in the discussion period.\n\n> If the paper is about investigating whether LMs learn \"the underlying semantics of the language\" ...\n\nOur paper establishes a framework for studying the question of whether **LMs are capable of inducing the causal structure which is latent in their training data**. While in the setting of our experiments, the causal structure happens to come from the formal semantics of the programming language, in general, causal structure and semantics are distinct concepts. \n\nTo the best of our knowledge, **our results are the first to show that LMs can infer free latent variables from text**, which suggests that LMs have the ability to perform causal induction. Note that none of the examples in the review relate to an LM’s ability to perform causal induction.\n\n> If the paper is intended as an argument for using this particular SCM-based framework.. against alternative probing methods\n\nWe summarize some major implications of our framework for probing in the remarks titled “Interventions, and probing for non-causal latent variables” on page 7. Briefly, Proposition 1 presents precise, formal conditions which guarantee the probe is properly controlled. The converse is also true: if the conditions laid out by Proposition 1 are NOT satisfied, then the results *could* be measuring either 1) the ability of the probe to learn the task (confounding) OR 2) the “naturalness” of the original task (bias). Indeed, existing probing frameworks produce baselines that are biased (“Frameworks for probing” in the Related works on page 9).\n\nWe will commit to dedicating a portion of the extra content page to a more detailed discussion that situates our contributions against other probing methodologies if the paper is accepted.\n\n> How does this setup situate within the established data augmentation and generalizability frameworks?\n\nAt a high level, the works cited in the review are generally concerned with “(Why / how) does including interventions (data augmentation) in the training data improve generalization?”\n\nIn contrast, interventions are simply a technique we employ to answer the question “Can LMs learn latent causal variables?”  A model is only ever trained on either the original OR the intervened data, NEVER both; and in fact a positive result requires the model to perform poorly on the intervened data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The overall topic is relevant to the NLP community. The paper is clearly structured and SCMs as well as probing mechanisms are intuitively introduced.", "reasons_to_reject": "My main concern is that the overall goal of the paper is unclear to me. \n\nIf the paper is about investigating whether LMs learn \"the underlying semantics of the language\" (page 2), then this is very well-established already. It is a direct consequence of the good performance of LMs on relational similarity prediction (king-man+woman=queen; see e.g., Vylomova, Rimell, Cohn, Baldwin 2016), using semantic representations for semantic cluster analyses, and vision-language representation space alignment work. If the paper aims to establish overall that LMs latently encode a variety of semantic variables (which I'm again not sure adds much to existing work), this should be evaluated in a suite of experiments. It is my intuition that this isn't the main goal. However, especially the framing of the results, and also the narrative in the abstract and introduction appear to put a significant focus here instead of using this task as a \"proof-of-concept\".\n\nIf the paper is intended as an argument for using this particular SCM-based framework as a probing mechanism, the paper needs to focus on establishing it against alternative probing methods, elaborating on how to use it for a variety of potential downstream tasks, as well as establishing its effectiveness. For example, in the results, the paper solely focuses on the implications for the model under investigation but under this framing, I would instead expect a focus on establishing the method. \n\nLastly, I'm confused about the way in which the authors present this work as \"interven[ing] on the model's training data\" [page 9], as opposed to other work that intervenes on model representations. However, intervening on training data to understand the \"causal effect of dataset statistics\" [page 9] is the general idea of traditional data augmentation and model generalizability setups (e.g., Perez and Wang 2017; Liu, Kusner, and Blunsom 2021). How does this setup situate within the established data augmentation and generalizability frameworks?", "questions_to_authors": "Very minor note: There are multiple cases where the text says \"casual\" where it's supposed to say \"causal\" (I assume).\n\nThe authors might find the following paper interesting to make connections to given the parallels to SCMs, a focus on the training data, the navigational instruction task, and connections to generalizability and data augmentations (Inducing Causal Structure for Interpretable Neural Networks; Geiger, Wu, Lu et al.; ICML 2022).", "extra_scores": {}}, {"review_id": "04y4VHJxvD", "reviewer": "Reviewer_nisC", "summary": "This paper proposes latent causal probing, a new probing framework that leverages latent causal structure to design auxiliary tasks. In particular, the task is to assess whether the language model has learned to represent the latent variables of the structural causal model (SCM). A method is proposed to disentangle the causal effects using the causal mediation analysis framework. Theoretical justifications are provided, and empirical results also support the claims of the paper. Overall, the proposed idea is novel, and the paper is well motivated and clearly written.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding the paper novel and clearly written! We address the main concerns here, and would be happy to address any remaining questions in the discussion period.\n\n> It is unclear whether it is easy to extend the proposed latent probing framework to natural language. Additional discussions should be provided in the paper\n\nIn general, extending latent probing to any new language or domain simply requires having a model of the underlying causal structure. We provide two references to two surveys on the topic of causal structure in natural language (end of section 2.2, page 4); please let us know if you would like further details in the discussion period.\n\nPractically speaking, outside of synthetic datasets, the underlying causal structure of real / natural language data is likely to be much more complex. While we agree this makes it difficult to apply the technique directly on real world data, we do want to provide another perspective, which is that our assumptions put the risks of probing on real world data in a mathematically precise language. Specifically, the lack of a clear causal relationship means that we cannot hope for a valid baseline in the exact sense of Proposition 1, so extra care should be taken when interpreting results against a baseline measurement (see also the discussion in Section 3.3).\n\nWe will include a discussion of this point in the extra content page if the paper is accepted.\n\n> Experiments only involve an LM trained from scratch. It would be helpful to include evaluations on pretrained LMs.\n\nWe chose to use an LM trained because we felt this would allow us to demonstrate our framework most cleanly with the separation between “free” and “bound” latent variables. For pre-trained LMs, it is impossible to know what the LM actually saw during training (which means it is impossible to know whether a latent variable or concept should be considered free or bound). Training our own LM gives us complete control over the training data, so that we can properly test whether LMs are capable of performing causal induction (i.e., inferring the value of free latent variables vs. just deducing the value of bound latent variables)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper presents a new probing framework from the perspective of causality. The idea is well motivated. The proposed latent probing framework helps address some limitations in existing work. \n\n2. The authors introduced the background of structural causal model in Section 2. Some motivating examples and use cases are discussed and explained. \n\n3. The authors proposed a very interesting idea, i.e., causal mediation analysis of probing.", "reasons_to_reject": "1. This work only focuses on causal structure of programming languages. It is unclear whether it is easy to extend the proposed latent probing framework to natural language. Additional discussions should be provided in the paper.\n\n2. Experiments only involve an LM trained from scratch. It would be helpful to include evaluations on pretrained LMs.\n\n3. There are some typos in the paper, such as \"we refer the ready to\" in Section 2.1.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NwOfVgfMR9", "reviewer": "Reviewer_B6Ky", "summary": "The paper proposes a formal probing framework to test whether LMs have learned the right laten variables to behave as a “structural causal model” (SCM). An SCM forms a directed graph of  token representations as a causal pathway from an initial state to a final state. The proposed methods tries to separate what is learned during pretraining (LM) from the training (calibration) for probing (SCM) by differentiating free (outside the calibration) and bound (seen during pretraining and calibration) variables  Their method is tested using a Code-generation model that is trained on a corpus of specification-program pairs. Their empirical test shows that the LM representation positively contribute to the causal predictive accuracy.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide us this feedback! We respond to the main points here. If any concerns require further addressing, we would appreciate the opportunity in the discussion period\n\n> the code generation “language” is very constrained (5 actions) and it is not clear how this can be used for LLMs that learn far more complex and diverse representations that may interfere in many different ways.\n\nOur methodology can (in principle) be applied to any language, regardless of how complex the LLM is, so long as you have a model for how the latent variable of interest causally affects the distribution of data. \nWe chose a very constrained setting so that we could follow the methodology as faithfully as possible: conducting our experiments under “ideal” circumstances allows us to demonstrate the benefits of the new framework as cleanly as possible and minimize concerns around whether the results could be due to a misapplication of the methodology.\n\nPractically speaking, the underlying causal structure could be less clear in more complex language data. While we agree this makes it difficult to apply the technique to such data, we do want to provide another perspective, which is that our assumptions put the risks of probing on complex data in a mathematically precise language. Specifically, the lack of a clear causal relationship means that we cannot hope for a valid baseline in the exact sense of Proposition 1, so extra care should be taken when interpreting results against a baseline measurement.\n\n> the section on the results ... It is not clear what the figure a - h stand for and what the different graph patterns show.\n\nWe will use the extra content page to include an extended discussion of the results in the revision. Briefly, each of the 4 columns shows a different combination of the calibration / measurement split (from left to right: bound / bound, bound / free, free / bound, and free / free, respectively). The top row displays the unmediated results; the bottom row displays the mediated results. The x-axis in all graphs is over LM training.\n\nThe most important pattern is that the mediated measurement (bottom) row trends up and positive over the course of LM training, i.e., the LM learns the latent causal variables above a valid baseline.\n\n> critical discussion lacks on how this method can be extended to more divers and complex LLMs.\n\nSee our response to point 1 - we will be sure to include this discussion in the paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a probing frame work that goes beyond testing solely on the elicited output but tries to trace the causal relations between implicit latent variables that lead to such output\n- provides evidence that the LM learns representations that support such causal structures", "reasons_to_reject": "- the code generation “language” is very constrained (5 actions) and it is not clear how this can be used for LLMs that learn far more complex and diverse representations that may interfere  in many different ways.\n- the section on the results is only 9 lines and way to short. Figure-4 is too small to read and is hardly addressed in the results section. It is not clear what the figure a - h stand for and what the different graph patterns show.\n- critical discussion lacks on how this method can be extended to more divers and complex LLMs.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper revisits an old chestnut in the NLP community -- probing. What excited me most about the paper was the application of SCMs. All reviewers highlighted the causal framework the paper introduces as a strength. The only reasons to reject given by the reviewers seem to focus on the number of experiments (after investigating the experiments myself, I disagree), and a limited number of latent variables. I think, however, that the importance of the problem trumps those concerns. Understanding and getting into the causal mechanisms that underlie language models is an important endeavor.
Decision: Accept