SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CLADDER: Assessing Causal Reasoning in Language Models
Zhijing Jin $^{1,2,*}$ , Yuen Chen $^{1,*}$ , Felix Leeb $^{1,*}$ , Luigi Gresele $^{1,*}$ , Ojasv Kamal $^{3}$ , Zhiheng Lyu $^{4}$ , Kevin Blin $^{2}$ , Fernando Gonzalez $^{2}$ , Max Kleiman-Weiner $^{5}$ , Mrinmaya Sachan $^{2}$ , Bernhard Schölkopf $^{1}$
$^{1}$ MPI for Intelligent Systems, Tübingen  $^{2}$ ETH Zürich  $^{3}$ IIT Kharagpur  $^{4}$ University of Hong Kong  $^{5}$ University of Washington jinzhi@ethz.ch chenyuen0103@berkeley.edu fleeb@tue.mpg.de luigi.gresele@tue.mpg.de
The ability to perform causal reasoning is widely considered a core feature of intelligence. In this work, we investigate whether large language models (LLMs) can coherently reason about causality. Much of the existing work in natural language processing (NLP) focuses on evaluating commonsense causal reasoning in LLMs, thus failing to assess whether a model can perform causal inference in accordance with a set of well-defined formal rules. To address this, we propose a new NLP task, causal inference in natural language, inspired by the "causal inference engine" postulated by Judea Pearl et al. We compose a large dataset, CLADDER, with 10K samples: based on a collection of causal graphs and queries (associational, interventional, and counterfactual), we obtain symbolic questions and ground-truth answers, through an oracle causal inference engine. These are then translated into natural language. We evaluate multiple LLMs on our dataset, and we introduce and evaluate a bespoke chain-of-thought prompting strategy, CAUSALCOT. We show that our task is highly challenging for LLMs, and we conduct an in-depth analysis to gain deeper insights into the causal reasoning abilities of LLMs.<sup>1</sup>
Once we really understand the logic behind causal thinking, we could emulate it on modern computers and create an "artificial scientist".
— Pearl and Mackenzie [2018]
Causal reasoning is believed to be one of the hallmarks of human intelligence [29, 68]. The ability to draw causal inferences from available information is crucial for scientific understanding and rational decision-making: for example, knowing whether smoking causes cancer might enable consumers to make a more informed decision [17, 18]; assessing the causal effect of a vaccine is essential for effective policy-making during a pandemic [14, 44, 72, 97]; and understanding the interplay behind family background, education and income helps devise effective education policies [10, 11, 30, 73].
Our opening quote therefore mirrors the aspirations of many scientists in artificial intelligence and causal inference: to construct a machine capable of performing sound causal reasoning, and able to answer causal questions at scale and with ease. Recent advances in large language models (LLMs) have brought about a paradigm shift in natural language processing (NLP) and artificial intelligence [7, 15, 39, 56, 76, 103, inter alia]. These transformative developments raise the question of whether these machines are already capable of causal reasoning: Do LLMs understand causality?
*Main contributors.
<sup>1</sup>Our data is open-sourced at https://huggingface.co/datasets/causalNLP/cladder, and our code can be found at https://github.com/causalNLP/cladder.
Question: Imagine a self-contained, hypothetical world with only the following conditions, and without any unmentioned factors or causal relationships:
Physical vulnerability has a direct effect on the likelihood of fatality and vaccination decision. Vaccination has a direct effect on the fatality rate.
In the entire population,  $50\%$  of the people are vulnerable to a certain disease.
For vulnerable and vaccinated people, the fatality rate is  $4\%$ . For vulnerable and unvaccinated people, the fatality rate is  $7\%$ .
For strong and vaccinated people, the fatality rate is  $1\%$ . For strong and unvaccinated people, the fatality rate is  $5.8\%$ .
Overall, the fatality rate for vaccinated people is  $5\%$ , while the fatality rate for unvaccinated people is  $4.5\%$ .
Does getting vaccinated increase the likelihood of death?
Ground-Truth Answer: No
CLadder
Correct steps to lead to the ground-truth answer:
Figure 1: Example question in our CLADDER dataset featuring an instance of Simpson's paradox [63]. We generate the following (symbolic) triple: (i) the causal query; (ii) the ground-truth answer, derived through a causal inference engine [66]; and (iii) a step-by-step explanation. We then verbalize these questions by turning them into stories, inspired by examples from the causality literature, which can be expressed in natural language.
1) Parse the causal graph: Confounding Subskill: Causal Relation Extraction
2) Classify the query type: Average Treatment Effect Subskill: Causal Question Classification
3) Formulate the query to its symbolic form: E[Y | do(X=1)] - E[Y|do(X = 0)] Subskill: Formalization
4) Collect the available data:  
P(Z=1)=0.5 Subskill: Semantic Parsing  
P(Y=1|Z=1,X=1)=0.04, P(Y=1|Z=1,X=0)=0.07  
P(Y=1|Z=0,X=1)=0.01, P(Y=1|Z=0,X=0)=0.058  
P(Y=1|X=1)=0.05, P(Y=1|X=0)=0.045
5) Derive the estimand using causal inference:  
E[Y | do(X=1)] - E[Y|do(X = 0)] Subskill: Formal Causal Inference  
 $\equiv$ $\sum_{\mathrm{v}}\{Z = \mathrm{v}\} P(Z = z)^{*}[P(Y = 1|Z = z,X = 1) - P(Y = 1|Z = z,X = 0)]$  #remove "do" using do-calculus  
 $= P(Z = 0)^{*}[P(Y = 1|Z = 0,X = 1) - P(Y = 1|Z = 0,X = 0)]$ $+P(Z = 1)^{*}[P(Y = 1|Z = 1,X = 1) - P(Y = 1|Z = 1,X = 0)]$  #turn the expression into terms in the available data
6) Solve for the estimand by plugging in the relevant data in Step 4:  $= 0.5^{*}(0.01 - 0.058) + 0.5^{*}(0.04 - 0.07)\#$  plug in the numbers in the available data  $= -0.039$  Subskill:Arithmetics  $\angle 0$  #the effect size is negative, so the final answer is "No"
Many previous works addressed the above question by focusing on commonsense causality [34, 100, 101], inspired by the literature that explores LLMs as knowledge bases [40, 70, 83] (we refer to this line of work as causality as knowledge). This involves assessing the alignment between commonsense knowledge about causal relationships in humans and LLMs. This line of work generally does not focus on evaluating how well models are capable of causal reasoning. For example, it may be difficult to rule out the possibility that LLMs perform potentially unreliable amortized causal inference, answering causal questions by a simple repetition of verbal patterns present in the texts composing their training data.[2,3] in other words, LLMs may just be "causal parrots" [100].
In this work, we introduce a way to test the formal causal reasoning in LLMs. To this end, we introduce the CLADDER dataset. The specificity of CLADDER is that causal questions posed in natural language are grounded in symbolic questions and ground truth answers: the latter are derived through an oracle causal inference engine (CI engine) [66], which abides by the rules of the causal inference approach described by Pearl [61], based on graphical models and structural causal models (SCMs) [23, 59, 61, 69, 88]. We compose more than 10,000 causal questions that cover a variety of causal queries across the three rungs of the Ladder of Causation [3, 66]—i.e., associational (Rung 1), interventional (Rung 2), and counterfactual (Rung 3). We consider several causal graphs, giving rise to scenarios which require different causal inference abilities. Additionally, we generate ground-truth explanations with step-by-step reasoning for more in-depth analysis of LLM behavior. Our symbolic questions and answers are then verbalized, by turning them into stories which can be expressed in natural language. To probe whether LLMs employ amortized causal inference, we construct stories with commonsensical, as well as anti-commonsensical and with nonsensical causal relations: in these latter cases, amortized causal inference is expected to fail, whereas formal causal reasoning would still yield the correct answer. An example question from CLADDER is shown in Figure 1.
Exploiting CLADDER, we also introduce a method to elicit sound causal reasoning in LLMs and help them solve challenging causality questions. Specifically, we develop CAUSALCoT, a chain-of-thought prompting strategy [96] inspired by the CI engine, which prompts the LLM to extract the causal graph, causal query, and available "data" (e.g., conditional or interventional do-probabilities [24]) from the question, formalize them precisely, and perform correct causal inferences.
2which may itself contain instances of fallacious causal reasoning.
3The extent to which this would imply an inaptitude of LLMs for causal reasoning has been questioned [38].
Our experiments indicate that CAUSALCoT achieves an accuracy of  $70.40\%$ , which substantially improves the performance of vanilla GPT-4 by 8.37 points on CLINNER.
1. In contrast to most other works on causality in LLMs, focusing on commonsense causal knowledge, our goal is to assess the LLMs' ability to perform formal causal reasoning (briefly reviewed in Section 2).
2. We introduce CLADDER (Section 3), a dataset containing more than 10K causal questions, spanning all three rungs of the ladder of causation, several causal graphs, and various stories for verbalization.
3. We develop CAUSALCOT (Section 4), a chain-of-thought prompting strategy to elicit formal causal reasoning in LLMs, inspired by the causal inference engine.
4. We perform extensive experiments on eight LLMs (Section 5), analyze fine-grained errors to showcase the limitations of LLMs in formal causal reasoning, and suggest directions for future research.
2 Preliminaries on Causal Inference
Our dataset design takes inspiration from the Causal Inference Engine as postulated by Pearl and Mackenzie [66], see also [59]. We begin with a brief overview of the causality framework by Pearl et al. [67]. This framework was largely developed within the field of artificial intelligence, and therefore puts particular emphasis on algorithmic aspects of causal reasoning (e.g., [62])—which makes it particularly suited for our work, where we want to algorithmically generate ground truth answers to causal queries, without having to appeal to common sense to assess the correctness of an answer.
2.1 The Ladder of Causation
The Ladder of Causation, introduced by Pearl and Mackenzie [66], is a proposed taxonomy, and hierarchy, of causal inference tasks [3]. It consists of three distinct rungs.
Rung 1 ("seeing"). This describes statistical associations ("How often do I take an aspirin when I have a headache?"). Rung 1 deals with statistical dependences among random variables, and involves probabilistic reasoning about joint and conditional distributions,  $P(X = x, Y = y)$  and  $P(Y = y | X = x)$ , which can be formalised through Bayesian Networks [12, 58] representing a set of variables and their conditional dependencies via a directed acyclic graph (DAG).
Rung 2 ("doing"). This enables us to formalize the concept of actively intervening in the world, and modifying it toward some end ("If I take an aspirin now, will my headache subside?"). Interventions can be formalized using the do-operator [24] and Causal Bayesian Networks [67] to represent, for example, the distribution over  $Y$  when intervening on  $X$  to set its value to  $x$  as  $P(Y = y|\mathrm{do}(X = x))$ .
Rung 3 ("imagining"). This rung deals with counterfactual reasoning, i.e., reasoning about alternative scenarios in which the world could have been different, possibly even contradicting the factual state ("Would my headache have subsided, if I had taken an aspirin?"). Counterfactual probabilities can be written as  $P(Y_{x} = y)$ , representing the probability that "Y would be  $y$ , had  $X$  been  $x$ ". Reasoning about Rung 3 quantities requires the introduction of Structural Causal Models (SCMs) [67]. SCMs are especially powerful as they enable any quantity in Rungs 1, 2, and 3 to be formulated precisely [3].
2.2 Causal Inference
Identification. Causal inference is especially difficult since we typically only have measurements from lower rungs, but want to reason about higher ones. A crucial question is then under what conditions are such inferences possible, i.e., what assumptions and measurements are required to unambiguously answer a causal query of interest: this is the question of identification. As argued in [3], "it is generically impossible to draw higher-layer inferences using only lower-layer information". One may be able to draw inferences at a higher layer given a combination of partial knowledge of the underlying SCM, in the form of a causal graph, and data at lower layers. The graphical structure therefore plays a crucial role in bridging the rungs of the Ladder of Causation, and many prior works have been dedicated to exploiting properties of the graph to transform higher-rung queries into expressions which can be estimated based on lower-rung quantities [36, 64, 84].
<sup>4</sup>We refer to [3, 65] for a comprehensive introduction. See also Appendix C for further details.
Figure 2: The data-generating process of the CLADDER dataset. The upper part of the figure describes the formal part of the question generation, which samples inputs for the CI Engine and derives a ground truth answer. The bottom part describes the natural language part of the question generation—i.e., its verbalization, based on multiple stories and different degrees of alignment with commonsense knowledge.
Causal Inference Engine. An overarching objective of this research is the construction of a Causal Inference Engine (CI Engine) [37, 59, 66], which takes as input a query, a graph, and some available data (typically from lower rungs than the query); and outputs whether a solution exists, and, if so, an equivalent expression of the query which is estimable from the available data. While some previous works refer to the CI engine in the context of Rung 2 queries, where it corresponds to the do-calculus [36, 84], here we refer to it in a more general sense, encompassing all three rungs.
3 Composing the CLADDER Dataset
Task Formulation. Like in the example of Figure 1, our dataset  $\mathcal{D} \coloneqq \{(q_i, a_i, e_i)\}_{i=1}^N$  consists of  $N$  triples, each containing a question  $q_i$ , binary answer  $a_i \in \{\text{Yes}, \text{No}\}$ , and an explanation  $e_i$ . Our main task is to test the accuracy of the prediction function  $f: q \mapsto a$ , i.e., a LLM which maps a natural language causal question to an answer. Apart from directly evaluating the answer, we also compose the ground-truth explanations  $e$  to evaluate the reasoning steps of LLMs.
Design Principles. In the composition of our dataset, we adhere to the following design principles. First, we ensure broad coverage of all rungs of the ladder of causation. Second, we avoid settings that involve continuous variables and use binary variables instead: this is partly due to the large availability of identifiability results for binary and categorical variables, and partly because queries involving binary variables lend themselves to more natural-sounding verbalization. Moreover, since LLMs struggle with calculation-heavy tasks [32, 91], and we are chiefly interested in causal reasoning abilities, we focus on graphs with few (three to four) variables, in various common configurations, to produce questions which are identifiable from the outset. Lastly, we carefully design a rich set of templates to translate the abstract formulas into grammatically correct and natural-sounding, fluent prompts.
Overall Pipeline. The generation pipeline for CLADDER, depicted in Figure 2, consists of two parts:
1. In the Formal Part (which we illustrate in Section 3.1), we specify all the required inputs (query, model, data) and the ground truth answer generated by the CI Engine.
2. In the Natural Language Part (in Section 3.2), we verbalize the formal queries and specification of the causal model and data by associating them to a story or narrative, using a rich set of templates.
3.1 Formal Part of the Question Formulation
The first step of our data generating process is to construct a set of inputs to the CI Engine such that by design there exists a well-defined ground truth answer: i.e., we construct triples of causal queries, graphs, and data such that the query can be unambiguously answered based on the available data (ensuring identifiability by construction). The ground truth causal models, which specify all quantities which are considered measurable in our questions, are causal Bayesian networks (CBNs), where each causal mechanism (i.e., conditional probability of a variable given its parents in the factorization according to the causal graph  $G$ ) corresponds to a Bernoulli distribution. We compile a selection of graphs  $G$  based on examples drawn from multiple sources from the literature [66, 67, 69, 88], where suitable graph structures are used to illustrate toy problems in causal inference. The complete list of structures we consider can be found in Appendix A.3; the complete list of sources in Appendix A.1.
Selecting Query Types. We again draw from the causal inference literature to collect common query types in each rung. As illustrated in the "Sample a query type" box in Figure 2, for Rung 1, we can ask about probability distributions such as marginal probabilities and conditional probabilities. For Rung 2 questions, we can enquire average treatment effects (ATE) ("how will  $Y$  change if  $X$  changes from  $x$  to  $x'$ ?), or what constitutes a valid adjustment set that can block all backdoor spurious correlations between  $X$  and  $Y$ . Lastly, for Rung 3, we include counterfactuals ("what would happen to  $Y$  had  $X$  been  $x'$  instead of  $x$ "), average treatment effect on the treated (ATT) ("for the subpopulation whose  $X$  changed from  $x$  to  $x'$ , how does their  $Y$  change on average?"), natural direct effect (NDE) ("what is the direct effect of  $X$  in  $Y$ , but not through the mediator?"), and natural indirect effect (NIE) ("what is the effect from  $X$  to  $Y$  through the mediator?).
Applying the Causal Inference Engine for the Ground-truth answer. By construction, the causal processes we define encapsulates all necessary information to make the causal quantities of the query types identifiable. This allows us to apply the rules of causal inference to obtain an estimand for each causal graph and query type, and evaluate the estimand to get a ground truth answer. The Rung 2 queries simplify to Rung 1 terms using the rules of do-calculus [59], and, for the Rung 3 queries, we apply methods of counterfactual causal inference [67] (with details in Appendix C.3). The estimand also specifies exactly which terms are necessary to include in the prompt as "available data" in order to ensure that enough information is provided to answer the question correctly (i.e., for identifiability), provided the correct causal reasoning is applied. Our entire code base of the data generation process can be found at our GitHub repository, https://github.com/causalNLP/cladder.
3.2 Natural Language Part of the Question Formulation
While Section 3.1 describes a way to generate the ground-truth causal model, query and answers, computed through a causal inference engine, real-world causal reasoning problems are expressed in natural language rather than symbolic expressions. The next part of the data generation pipeline therefore focuses on the verbalization of all these components with a plausible narrative in natural language.
Generating the Stories. For each causal graph, we collect a set of two to five stories which consist of a list of variable names for each node in the graph. The stories are primarily selected from examples in commonly cited causal inference books and papers (see Appendix A.1), which ensures that the stories and corresponding causal graph structures adhere to empirical common sense (e.g., the drug-gender-recovery example of Pearl and Mackenzie [66]). However, it is very likely that at least some of the stories appear in the training data of many LLMs. Therefore, we also generate various anticommon sense and nonsensical variants of the stories, meant to isolate the effects of memorization. For the anti-commonsensical stories, we randomly do one of the actions: (1) replace the effect variable  $Y$  with an unusual attribute, that would not be an effect variable in any of the stories (e.g., "ear shape"); or (2) create an irrelevant treatment variable  $X$  that does not play a causal role in any of our commonsensical stories, such as "playing card games" (see Appendix A.7). For the nonsensical variants, we invent artificial words as variable names such as "zory" and "qixy" (see Appendix A.6).
We use the term "data" to denote numerical values of conditional or do-probabilities, and not as collections of data samples. This is in line with how the term is used in other descriptions of the CI Engine [37, 66].
Verbalizing the Prompts. The verbalization procedure applies the mapping of symbolic variables to semantic concepts to form a plausible narrative for the underlying causal process and then translates the symbolic expressions from the underlying causal process to natural language using carefully designed templates.
Specifically, we use several different grammatical forms for each semantic concept  $\pmb{t}$  in the story to make the resulting prompt sound natural and grammatically correct. We first have the overall variable name  $v_{\text{overall}}(\pmb{t})$  (e.g., the recovery status), and, then, for each binary value  $i \in \{0,1\}$ , we compose its noun  $v_{\text{noun}}(\pmb{t} = i)$  (e.g., recovery), verb (e.g., to recover), sentence  $v_{\text{sent}}(\pmb{t} = i)$  (e.g., the patients recover), noun with attributive clause  $v_{\text{attr}}(\pmb{t} = i)$  (e.g., patients who recover), and third conditional  $v_{\text{cond}}(\pmb{t} = i)$  (e.g., if the patient had recovered).
Using these elements, we first verbalize the causal graph by iterating through each node and its outgoing edges, using the template “ $t$  has a direct effect on  $\mathbf{CH}(t)$ ”, where  $\mathbf{CH}(\cdot)$  denotes the set of direct effects (children) of a variable. Then, for the available data  $d$ , we verbalize each conditional probability by “For  $v_{\mathrm{attr}}(t_m = i)$ , the probability of  $v_{\mathrm{noun}}(t_n = 1)$  is  $p$ ”, and each marginal probability by “The overall probability of  $v_{\mathrm{attr}}(t = 1)$  is  $p$ .” Note that our distributions are Bernoulli, so it is adequate to just introduce the parameter  $p$ , which is the likelihood of  $t = 1$ . For example, we generate sentences such as “The overall probability of recovery is  $60\%$ ” and “For patients who have small kidney stones, the probability of recovery is  $70\%$ ”. Finally, for the query  $q$ , we instantiate each query type in our dataset following our question templates in Appendix A.5 such that the questions can always be answered with “yes” or “no”.
Generating the Explanations. Apart from the question-answer pairs, we also generate the step-by-step explanations. Our goal is to provide all intermediate reasoning steps a student of causal inference would use to answer the questions, so that each necessary subskill necessary for causal inference can be evaluated individually. We identify the following six subskills: ① causal graph extraction; ② correct query type interpretation; ③ symbolic formalization of the query; ④ semantic parsing to compile the available data; ⑤ estimand derivation; and ⑥ arithmetic calculation to solve the estimand, as in the colored boxes in Figure 1. Our explanation  $e$  verbalizes all the elements ①-⑥ as sequential steps using our template in Appendix A.8.
3.3 Dataset Statistics
Our data-generating procedure has the potential to algorithmically generate a vast large number of questions. In practice, we pick a dataset size that is large enough to be representative, and at the same time not too large to be problematic given the expensive inference costs of LLMs. We therefore set our dataset size to be 10K, and report the statistics in Table 1.