CLadder: Assessing Causal Reasoning in Language Models
Authors: Zhijing Jin
Yuen Chen
Felix Leeb
Luigi Gresele
Ojasv Kamal
Zhiheng LYU
Kevin Blin
Fernando Gonzalez Adauto
Max Kleiman-Weiner
Mrinmaya Sachan
Bernhard Schölkopf
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Models
Causal Reasoning
Causal Inference
Benchmark Dataset
Natural Language Processing
TL;DR: We create a large-scale benchmark, CLadder, to probe causal reasoning abilities in LLMs, and propose Causal Chain-of-Thought (CoT) as a solution.

Abstract: The ability to perform causal reasoning is widely considered a core feature of intelligence. In this work, we investigate whether large language models (LLMs) can coherently reason about causality. Much of the existing work in natural language processing (NLP) focuses on evaluating _commonsense_ causal reasoning in LLMs, thus failing to assess whether a model can perform causal inference in accordance with a set of well-defined _formal rules_. To address this, we propose a new NLP task, _causal inference in natural language_, inspired by the _"causal inference engine"_ postulated by Judea Pearl et al. We compose a large dataset, CLadder, with 10K samples: based on a collection of causal graphs and queries (associational, interventional, and counterfactual), we obtain symbolic questions and ground-truth answers, through an oracle causal inference engine. These are then translated into natural language. We evaluate multiple LLMs on our dataset, and we introduce and evaluate a bespoke chain-of-thought prompting strategy, CausalCoT. We show that our task is highly challenging for LLMs, and we conduct an in-depth analysis to gain deeper insight into the causal reasoning abilities of LLMs. Our data is open-sourced at https://huggingface.co/datasets/causalNLP/cladder, and our code can be found at https://github.com/causalNLP/cladder.

CLADDER: Assessing Causal Reasoning in Language Models
Zhijing Jin Yuen ;hen', Felix Leeb';;uugi Gresele";
Ojasv Kamal Zhiheng Lyu', Kevin Blin', Fernando Gonzalez', Max Kleiman- Kleiman-Weiner'
Mrinmaya Sachan', Bernhard Schölkopf"
MPI for Intelligent Systems, Tübingen 2ETH Zürich 'IIT Kharagpur *University of Hong Kong UUniversity of Washington
j jinzhi@ethz. ch Chenyuea0103@0erzeel." edu
fleebetue mpg.
mpg. mpg.mmg...o de
Abstract
The ability perform causal reasoning is widely considered core feature of in- telligence. In this work, we investigate whether large language models (LLMs) can coherently reason about causality. Much of the existing work in natural language processing (NLP) focuses on evaluating commonsense causal reasoning in LLMs, thus failing assess whether a model can perform causal inference in accordance with set of well-defined formal rules. To address this, we propose a new NLP task, causal inference in natural language, inspired by the "causal inference engine" postulated by Judea Pearl et al. We compose large dataset, CLADDER, with 10K samples: based on collection of causal graphs and queries (associational, interventional, and counterfactual), we obtain symbolic questions and ground-truth answers, through an oracle causal inference engine. These are then translated into natural language. We evaluate multiple LLMs on our dataset, and we introduce and evaluate a bespoke chain-of-thie offthhuught prompting strategy, CAUSALCOT, We show that our task is highly challenging for LLMs, and we conduct in in-depth analysis to gain deeper insights into the causal reasoning abilities of LLMs.!
Introduction
Once we really understand the logic behind causal thinking, we could emulate it on modern computers and create an "artificial scientist".
Pearl and Mackenzie [2018]
Causal reasoning is believed be one of the hallmarks human intelligence [29, 68). The ability to draw causal inferences from available information is crucial for scientific understanding and rational docision-making for example, knowing whether smoking causes cancer might enable consumers to make more informed decision [17, 18]; assessing the causal effect of vaccine is essential for effective policy-making during a pandemic [14, 44, 72, 97]; and understanding the interplay behind family background, education and income helps devise effective education policies [10, 11, 30, 73].
Our opening quote therefore mirrors the aspirations of many scientists in artificial intelligence and causal inference: to construct a machine capable of performing sound causal reasoning, and able to answer causal questions scale and with ease. Recent advances in large language models (LLMs) have brought about a paradigm shift in natural language processing (NLP) and artificial intelligence [7, 15. 39, 56, 76, 103, inter alia]. These transformative developments raise the question of whether these machines are already capable of causal reasoning: Do L.MMs understand causality?
Main contributors.
OOur data is open-soureed at comm/causaaMMPP/llaeer. can found at pestps:/ggithin
and code
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Imagine self-contained, hypothetical world with only the following conditions, and without any unmentioned factors or causal relationships:
Question:
Physical vulnerability has a direct effect on the likellhood of fatality and vaccination decision. Vaccination has direct on the fatality rate.
the entire populstion, 50% of the people are vulnerable to certain disease. For vulnerable and vaccinated people, the fatality rate is 4%. For vulnerable and unvaccinated people, the fatality rate is 7%. For strong and vaccinated people. the fatallly rate is 1%, For strong and unvaccinaled people, the fatallty rate is 5.8%.
Overall, the fatality rate for vaccinated people is 5%, while the fatality rate for unvaccinated people is 4.5%.
Does getting vaccinated increase the likelihood of death?
CLadder
Ground-Truth Answer: No
Correct steps to lead to the ground-truth answer:
Parse the causal graph: Confounding
Derive the estimand using causal Inferance:
Subskill: Causal Relation Extraction
E| da[X=r]] E[Yida(X 0j)
laum (Zv)) P(Z=z//PPV=1]2=xxxx1) P[Y=1/Z=z, X-0) a remove "do using do-estculus
Subskill: Formal Causal Inference
2) Classify the query type: Average Treatment Effect
Subskill: Causal Question Classilication
P P(Y=1 i tum the expression into terms in
3) Formulate the query to its symbolic form: E|Y| | dopx=1]] E[Y]do[X Oy)
the available dats
Subskill: Fomalization
Solve for the astimand by plugging In the relevant data in Step 4:
4) Collect the available data:
0.55(0.01 0.058)+0.5"00 (0..00050000000..0) (0.04-0.07) plug the numbers in the available data Subskili Arthmmaties
PiZ-1)-0.5
Subakili Semantic Parsing
-0.039
P/Y=1|z=1, x==)-0211-00=00077
<0 B the offect size is negative, the final answer is "No"
P|Y=1[2-0,x-1/+000 X0=1][2000000000000 PiY-1ix--1--0005, PYY-1xX-0-00045
Example question in our CLADDER dataset featuring an instance of Simpson's 's paradox [63]. We generate the following (symbolic) triple: (i) the causal query: (ii) the ground-truth answer, derived through causal inference engine [66]; and (iii) step-by-step explanation. We then verbalize these questions by turning them into stories, inspired by examples from the causality literature, which can be expressed in natural language.
Figure I:
Many previous works addressed the above question by focusing on commonsense causality [34, 100, 101], inspired by the literature that explores LLMs as knowledge bases [40, 70, 83] (we refer to this line of work as causality as knowledge). This involves assessing the aligmment between commonsense knowledge about causal relationships in humans and LLMs. This line of work generally does not focus on evaluating how well models are capable of causal reasoning. For example, it may be difficult to rule out the possibility that LLMS perform potentially unreliable amortized causal inference, answering causal questions by simple repetition of verbal patterns present the texts composing their training data: 2.3 in other words, LLMs may just be "causal parrots" " [100].
In this work, we introduce a way to test the formal causal reasoning in LLMs. To this end, we introduce the CLADDER dataset. The specificity of CLADDER is that causal questions posed in natural language are grounded in symbolic questions and ground Iruth answers: the latter are derived through an oracle causal inference engine (CI engine) [66], which abides by the rules of the causal inference approach described by Pearl [61], based on graphical models and structural causal models (SCMs) [23, 59, 61, 69, 88]. We compose more than 10,000 causal questions that cover a variety of causal queries across the three rungs of the Ladder of Causation [3, 66]-i.e., associational (Rung 1), interventional (Rung 2), and counterfactual (Rung 3). We consider several causal graphs, giving rise to scenarios which require different causal inference abilities. Additionally, we generate ground-truth explanations with step-by-step reasoning for more in-depth analysis of LLM behavior. Our symbolic questions and answers are then verbalized. by turning them into stories which can be expressed in natural language. To probe whether LLMs employ amortized causal inference, we construct stories with commonsensical, as well as anti-commmnsensical and with nonsensical causal relations: in these latter cases, amortized causal inference expected to fail, whereas formal causal reasoning would still yield the correct answer. An example question from CLADDER is shown in Figure 1.
Exploiting CLADDER, we also introduce a method to elicit sound causal reasoning in LLMs and help them solve challenging causality questions. Specifically, we develop CAUSALCOT, thhhughh prompting strategy [96] inspired by the CI engine, which prompts the LLM to extract the causal graph, causal query, and available "data" (e.g., conditional or interventional do- probabilities [24]) from the question, formalize them precisely, and perform correct causal inferences.
wwhich may itself contain instances fallacious causal reasoning.
The extent which this would imply inaptitude of LLMs for causal reasoning has been questioned [38].
2

---

Our experiments indicate that CAU achieves an accuracy of 70.40%, which substantially improves the performance of vanilla GPT-4 by 8.37 points on CLADDER.
We summarize the main contributions of our work:
1. In contrast to most other works on causality in LLMs, focusing on commonsense causal knowledge, our goal is to assess the LLMs' ability to perform formal causal reasoning (briefly reviewed in Section 2).
2. We introduce CLADDER (Section 3), a dataset containing more than 10K causal questions, spanning all three rungs of the ladder of causation, several causal graphs, and various stories for verbalization.
3. We develop CAUSAI.COT (Section 4), a chain-of-thought prompting strategy to elicit formal causal reasoning in LLMs, inspired by the causal inference engine.
4. We perform extensive experiments on eight LLMs (Section 5), analyze fine- nne-graied errors to showcase the limitations of LLMs in formal causal reasoning, and suggest directions for future research.
2 Preliminaries on Causal Inference
Our dataset design takes inspiration from the Causal Inference Engine postulated by Pearl and Mackenzie [66], see also [59]. We begin with a brief overview of the causality framework by Pearl al. [67]. 4 This framework was largely developed within the field of artificial intelligence, and there- fore puts particular emphasis on algorithmic aspects of causal reasoning (e.g.. (62))-which makes it particularly suited for our work, where we want to algorithmically generate ground truth answers to causal queries, without having to appeal to common sense to assess the correctness of an answer.
2.1 The Ladder of Causation
The Ladder of Causation, introduced by Pearl and Mackenzie [66], a proposed taxonomy, and hierarchy, of causal inference tasks [3]. It consists of three distinct rungs.
Rung ("seeing"). This describes statistical associations ("How often do I take an aspirin when I have headache?"). Rung 1 deals with statistical dependences among random variables, and involves probabilistic reasoning about joint and conditional distributions, P(X x, Y = y) and P(Y = X = x). which can be formalised through Bayesian Networks [12. 58] representing a set of variables and their conditional dependencies via a directed acyclic graph (DAG).
Rung 2 ("doing")). This enables us to formalize the concept of actively intervening in the world, and modifying it toward some end ( ("If take an aspirin now, will my headache subside?") Interventions can be formalized using the do-operator [24] and Causal Bayesian Networks [67] to represent, for example, the distribution over Y when intervening on X to set its value to as P(Y y)do(X z)).
Rung 3 ("imagining"). This rung deals with counterfactual reasoning, i.e., reasoning about alter- native scenarios in which the world could have been different, possibly even contradicting the factual state ("Would my headache have subsided, if I had taken an aspirin?")). Counterfactual probabilities can be written as P(Y, y). representing the probability that "Y would be y. had X been x². Reason- ing about Rung 3 quantities requires the introduction of Structural Causal Models (SCMs) [67]. SCMs are especially powerful as they enable any quantity in Rungs 1, 2, and 3 to be formulated precisely [3].
2.2 Causal Inference
Identification. Causal inference is especially difficult since we typically only have measurements from lower rungs, but want to reason about higher ones. A crucial question is then under what conditions are such inferences possible, i.e., what assumptions and measurements are required to unambiguously answer a causal query of interest: this is the question of identification. As argued in [3], "it generically impossible to draw higher-layer inferences using only lower-layer information" ". One may be able to draw inferences at a higher layer given combination of partial knowledge of the underlying SCM, in the form of a causal graph, and data at lower layers. The graphical structure therefore plays a crucial role in bridging the rungs of the Ladder of Causation, and many prior works have been dedicated to exploiting properties of the graph to transform higher-rung queries into expressions which can be estimated based on lower-rung quantities [36, 64, 84].
Wee refer [3, 65] for comprehensive introduction. See also Appendix C for further details.
3

---

Formal Part of the Question Generation
Sample a causal graph
Sample query type
Generate available data
Common causal graphs with treatmentt offect (X-Y) pairs
Rung 1: Association
Obeervatioaa:: P(..)
Marginal prob.
Conditional prob.
Confounding:
Diamond:
Intervention
Intervantional: E[--joo(--]]
Valid ad adustment set
Mediation:
Chain:
Rung 3: Counterfactuals
Counteriactual prob. ATT
NDE
NIE
Collision:
Fork:
Map to the eslimand
ATE do/X-18 Haj[=D0
Apply do-caloulus gen the causal graph
data
= =Iz=, PIZ = B1 X 122,220))
the cstmaadd is denedifaale
example option to choose
Natural Language Part of the Question Generation
Sample degree alignment with common sense
Sample story variable name instantiation
Verbalize the entire question
Levels of Empirical Alignment
Stories (Veriable Name Instantiations)
For Commonsenstal Confoueding Graphs:
Ant-commonsenataal o.g. smoking affects ear shapo
Commonsensicall in smoking causes cancer
X-vaccine, Zevulnerabiiity Y-tatalitt
Noreensical g. zory allects glay
Story X-drup Z-gender, Y-reovery
Story X-treabment, Z=age, Y=reazaary
The data-generating process of the CLADDER dataset. The upper part of the figure describes the formal part of the question generation, which samples inputs for the CI Engine and derives ground truth answer. The bottom part describes the natural language part of the question generation i.c., its verbalization, based on multiple stories and different degrees of alignment with commonsense knowledge.
Figure 2:
Causal Inference Engine. An overarching objective of this research is the construction of a Causal Inference Engine (CI Engine) [37, 59, 66), which takes as input a query, graph, and some available data (typically from lower rungs than the query); and outputs whether a solution exists, and, if so, an equivalent expression of the query which is estimable from the available data. While some previous works refer to the CI engine in the context of Rung 2 queries, where it corresponds to the do-calculus [36, 84], here we refer to it in more general sense, encompassing all three rungs.
Composing the CLADDER Dataset
Task Formulation. Like in the example of Figure our dataset D == {(g;, ai, N consists of triples, each containing a question qi, binary answer as E (Yes, No), and an explanation e,- q + a, i.c., LLM which maps
Our main task is to test the accuracy of the prediction function a natural language causal question to an answer. Apart from directly evaluating the answer, we also compose the ground-truth explanations
evaluate the reasoning steps of LLMs.
Design Principles. In the composition of our dataset, we adhere the following design principles. First, we ensure broad coverage of all rungs of the ladder of causation. Second, we avoid settings that involve continuous variables and use binary variables instead: this is partly due to the large availability of identifiability results for binary and categorical variables, and partly because queries involving binary variables lend themselves to more natural-sounding sounding verbalization. Moreover, since L.MMs strug- tasks [32, 91], and we chiefly interested in causal reasoning abilities,
gle with calculation-hean we focus on graphs with few (three to four) variables, in various common configurations, to produce questions which are identifiable from the outset. Lastly, we carefully design rich set of templates to translate the abstract formulas into grammatically correct and satural-sounding. fluent prompts.
Overall Pipeline. The generation pipeline for CLADDER, depicted in Figure 2, consists two parts:
1. In the Formal Part (which we illustrate in Section 3. we specify all the required inputs (query, model, data) and the ground truth answer generated by the CI Engine.

---

2. In the Natural Language Part (in Section 3.2), we verbalize the formal queries and specification of the causal model and data by associating them to a story or narrative, using a rich set of templates.
3.1 Formal Part of the Question Formulation
The first step of our data generating process is to construct a set of inputs to the CI Engine such that by design there exists well-defined ground truth answer: i.e., we construct triples of causal queries, graphs, and data such that the query can be unambiguously answered based on the available data (ensuring identifiability by construction). The ground truth causal models, which specify all quantities which are considered measurable in our questions, are causal Bayesian networks (CBNs), where each causal mechanism (i.e.. conditional probability of variable given its parents in the factorization according to the causal graph G) corresponds to a Bernoulli distribution. We compile a selection of graphs G based on examples drawn from multiple sources from the literature [66, 67, 69, 88], where suitable graph structures are used to illustrate toy problems in causal inference. The complete list of structures we consider can be found in Appendix A.3; the complete list of sources in Appendix A.I.
Selecting Query Types. We again draw from the causal inference literature to collect common query types in each rung. As illustrated in the "Sample a query type" box in Figure 2, for Rung 1, we can ask about probability distributions such as marginal probabilities and conditional probabilities. For Rung 2 questions, we can enquire average treatment effects (ATE) ("how will Y change if X changes from x x' ?"), what constitutes valid adjustment set that can block all backdoor spurious correlations between X and Y. Lastly, for Rung 3, we include counterfactuals ("what would happen to Y had X been x' instead of x?"",, "), average treatment effect on the treated (ATT) ( ("for the subpopulation whose X changed from x to x', how does their Y change on average?") natural direct effect (NDE) ("what is the direct effect of X in Y. but not through the mediator?"). and natural indirect effect (NIE) ("what is the effect from X to Y through the mediator?").
Applying the Causal Inference Engine for the Ground-truth answer. By construction, the causal processes we define encapsulates all necessary information to make the causal quantities of the query types identifiable. This allows us to apply the rules of causal inference to obtain an estimand for each causal graph and query type, and evaluate the estimand to get a ground truth answer. The Rung 2 queries simplify to Rung terms using the rules of do-calculus [59], and, for the Rung 3 queries, we apply methods of counterfactual causal inference [67] (with details in Appendix C.3). The estimand also specifies exactly which terms are necessary to include in the prompt as "available data" " in order to ensure that enough information is provided to answer the question correctly (i.e., for identifiability), provided the correct causal reasoning is applied. Our entire code base of the data generation process can be found at our GitHub repository, https: htt///gitttt giggtu..b.... com/ www.saaall/caaaaaca
3.2 Natural Language Part of the Question Formulation
While Section 3.1 describes way to generate the ground-truth causal model, query and answers, com- puted through a causal inference engine, real-world causal reasoning problems are expressed in natural language rather than symbolic expressions. The next part of the data generation pipeline therefore focuses on the verbalization of all these components with a plausible narrative in natural language.
Generating the Stories. For each causal graph, we collect set of two to five stories which consist of list of variable names for each node in the graph. The stories are primarily selected from examples in commonly cited causal inference books and papers (see Appendix A.1), which ensures that the stories and corresponding causal graph structures adhere to empirical common sense (e.g., the drug- gender-recovery example of Pearl and Mackenzie [66]). However, it is very likely that at least some of the stories appear in the training data of many LLMs. Therefore, we also generate various anti- common sense and nonsensical variants of the stories, meant to isolate the effects of memorization. For the anti-com commoonssseeil stories, we randomly do one of the actions: (1) replace the effect variable Y with an unusual attribute, that would not be an effect variable in any of the stories (c.g.. "ear shape"); or (2) create an irrelevant treatment variable X that does not play a causal role in any of our commonsensical stories, such as "playing card games" (see Appendix A.7). For the nonsensical variants, we invent artificial words as variable names such as "zory" and "qixy" (see Appendix A.6).
'ee use the term "data" to denote numerical values of conditional or do- ho--rrbbbiliites, and not as collections of data samples. This is in line with how the term is used in other descriptions of the CI Engine [37, 66].
5

---

Verbalizing the Prompts. The verbalization procedure applies the mapping of symbolic variables semantic concepts form a plausible narrative for the underlying causal process and then translates the symbolic expressions from the underlying causal process to natural language using carefully designed templates.
Specifically, we use several different grammatical forms for each semantic concept t in the story to make the resulting prompt sound natural and grammatically correct. We first have the overall variable name Voverall (t) (e.g., the recovery status), and, then, for each binary value E {0, 1 we compose its noun Unoun (t 2) (e.g., recovery), verb (e.g., to recover), sentence Useut (t i) (e.g.. the patients recover), noun with attributive clause Pattr = i) (e.g., patients who recover), and third conditional Iconn (t (e.g.. if the patient had recovered).
Using these elements, we first verbalize the causal graph by iterating through each node and its outgoing edges, using the template "t has direct effect on (t),t),, where C denotes the set of direct effects (children) of a variable. Then, for the available data d. we verbalize each conditional probability by "For (attr(tm i), the probability of Unoun (t,, 1) is p.", and each marginal probability by "The overall probability of Vastr (t = 1) is p." Note that our distributions are Bernoulli, so it is adequate to just introduce the parameter p. which is the likelihood of t 1. For example, we generate sentences such as "The overall probability of recovery is 60%." and "For patients who have small kidney stones, the probability of recovery 70%." Finally, for the query q. we instantiate each query type in our dataset following our question templates in Appendix A.5 such that the questions can always be answered with "yes" or "no".
Generating the Explanations. Apart from the question-answer pairs, we also generate the step-by- step explanations. Our goal is to provide all intermediate reasoning steps student of causal inference would use to answer the questions, so that each necessary subskill necessary for causal inference can be evaluated individually. We identify the following six subskills: ① causal graph extraction; @ correct query type interpretation; @ symbolic formalization of the query; + semantic parsing to compile the available data; @ estimand derivation; and @ arithmetic calculation solve the estimand, as in the colored boxes in Figure 1. Our explanation e verbalizes all the elements Q-@ as sequential steps using our template in Appendix A.8.
3.3 Dataset Statistics
Our data-generating procedure has the potential algorithmically generate a vast large number of questions. In practice, we pick dataset size that is large enough to be representative, and at the same time not too large to be problematic given the expensive inference costs of LLMs. We therefore set our dataset size to be 10K, and report the statistics in Table 1.
The dataset roughly balance across the query types, graph structures. stories, and ground truth answers (as seen in Figure 3). Note that some causal queries are only compatible with a subset of the graphs, thereby resulting in slightly lower representation of those queries (such the NDE and NIE). More details on our design choices can be found in Appendix A.4.
Total
Rung
Rung 2
Rung 3
Size
# Samples
10,112
3,160
3,160
3,792
Question
NIE
ATT
# Sentences/Sample
6.01
5.88
5.37
6,65
# Words/Sample
80.9
73.43
76.95
90.42
cand Prob.
Counart
# Nodes/Grrahh
3.52
3.5
3.5
3.54
# Edges/Graph
3.38
3.3
3.3
3.5
Rung
Answer
Rung 2
Asjust, Sist
Positive Class (9)
50
50
50
50
Marg Prob
Explanations
# Sentences/Sample
9.11
8.1
9.96
ATE
# Words/Sample
47.95
49.87
32.8
58.97
Figure 3: Distributions of query types in our 10K data.
Table Statistics of CLADDER dataset v1.5.
3.4 Data Quality Check
Our dataset is generated through an algorithmic procedure, which has the following potential benefits: formal correctness; zero human annotation cost; and, most importantly, controllabilit--ee. e.g. for

---

the question distribution, as well as for making it more unlikely that the data was previously seen by the model. However, since the dataset is different from common NLP datasets collected from human natural language writing, we also need to perform additional data quality checks. We therefore checked for list of non-formal, natural language properties: grammaticality; human readability; naturalness/perplexxi and how well humans perform on this task.
For grammaticality, we ran a grammatical error check on our dataset using the Language package [51], and got on average 1.26 grammatical errors per 100 words (i.e., 98%74% correctness), which shows that most of the language our dataset follows English grammar. For human readability, we checked how comprehensible the questions are to students who have taken causality courses. We selected random subset 50 questions from the dataset, and let a graduate student annotator go through the questions judge whether they could understand them or not: 96% of the questions were deemed readable. Next, for the naturalnesspprplexity score, we used the open-sourced GPT-2 model and obtained perplexity score of 21.17 on our dataset, which is substantially lower (i.e., closer to the distribution of natural humte-written text) than the one MATH [32], a commonly used dataset of maths questions. Lastly, conducted sanity check where expert evaluator tried solve random sample of 50 questions from the dataset, and we recorded an accuracy of 82% on this task.
4 Our CAUSALCOT Model
Question
Our Causal Chain-of-Thought (CausalCoT) Model:
Guidance: Address the question by following the steps below:
Step
Extract the causal graph.
Preparation Phase
Step 2
Determine the query type.
Average Treatment Effect (ATE)
Step
Formalize the query.
E[Mdo(X=1)) - E[V|d(XX= 0)
PIZ-1)-0.50 RYY-11X-0--00045 PCY-1XX-1--0055 P|Y=|jz=0x==0==0 07, Riy=1j=-0 P|V=1j==1x=0)0000 04, P/==((2=1x==)=0000
Step
Gather all relevant data.
Deduce the estimand using causal inference: Given all the information above, deduce the estimand using skills such as do-calculus, counterfactual prediction, and the basics of probabililies.
Step
Solution Phase
Σz-z P(Z-z) PY==1122-2-11))X--112 X-0j (Apply backdoor adjustment formula)
Calculate the estimand: Insert the relevant data in Step into the estimand, perform basic arithmetic calculations, and derive the final
Step
ATE(X) -00221 <
Final Q
Based on all the reasoning above, output one word to answer the initial question with just "Yes" or "No".
Final answer: No
Figure 4: Illustration of our CAUSALCOT prompting strategy, which designs a chain of subquestions inspired by the idea a CI engine [66].
In order to guide LLMs in correctly answering the questions in CLADDER, we draw inspiration from the ideal functioning of the CI engine [66], which breaks down a causal reasoning problem into multiple coroolicalll-grooneee
simpler steps. We develop CAUSALCOT, a multi-step causal prompt in Figure 4, which combines formal causal reasoning skills with the idea of chain-of-thouge prompting [96] and the use of scratch pads for solving more complicated problems requiring a long list of steps [55] for L.MSs.
We base our prompt design on the multi-step reasoning process of causal inference as shown in Figure 4, first starting with four preparation steps: ① identifying the causal graph structure; @ determining the causal query type:" @ formulating the query symbolically precisely: and D extracting relevant data from the prompt. Then, given all the information collected the preparation stage, we introduce the formal solution: @ correctly deducing the estimand using causal inference techniques; and finally @ evaluating the estimand to answer the question. This set of steps require both natural language understanding to parse the question (as in most steps the preparation phase), well as formal causal reasoning to derive the correct estimand (as in the solution phase).
This step amounts multi-class classification problem, where each class is different causal query.
7

---

We build our CAUSALCOT prompting strategy using GPT-4 [56], recent autoregressive LLM that achieves state-of-ther of-the-art performance on many tasks. This latest model builds upon the previous series of general pretrained models (GPT) [7. 76] and adds reinforcement learning with human feedback, or instruction-tuninn
[1, 57, 104], to align the model responses to free-form questions with human preferences. It has achieved human-competitive performance over list of tasks [8, 43, 54, 56, 105), among which the more formal tasks unseen in the training data still remain elusive [42, 78, 91].
Given a causal question q, of the detailed descriptions of the six steps 81....1 ., 86 in Figure 4. As the model fLLM 8, + T, autoregressively produces responses ".. To sequentially corresponding to the six steps. we concatenate all the above before asking the final question "Based on all the reasoning above, output one word to answer the initial question with just 'Yes' or 'No'." See the complete prompt in Appendix B.1. the end, we obtain the binary answer (Yes, No} as the final result.
we provide the LLM a list of instructions e := (81...., 96) consisting
Compared with the standard strategy of directly prompting the LLMS a question, we impose an inductive bias upon LLMs by using the causal inference framework, thus incorporating some of the powerful, principled insights of the causal inference community for NLP tasks. In this way, we enhance the strong natural language ability of LLMs with formal causal reasoning skills.
Testing LLMs with CLADDER
5.1 Experimental Setup
Our empirical investigation focuses on some of the most recent language models. We include the latest GPT-4 [56] with IT parameters by the time we conduct the experiments (i.e., gpt-4- 1 106-preview) the previous ChatGPT (i.e., GPT-3.5) with 175B parameters, and then a series of earlier models with instruction-tuna -unniiiin--nnnggggggg
the 175B GPT-3 (text-davinci-001. -002, and -003) [57]. As baselines, include the non- instruction eastrucctonntuued GPT-3 (davinci). We use the OpenAl API with temperature 0 when querying these models. We also include open-source, more efficient models like LLaMa [93] and its instruction-tun -undd version Alpaca [92], both with the same number of parameters, 6.7B.
also
5.2
Main Results
Acc. by Rung
Acc. by Commonsense Alignment
Overall Acc.
3
Comm.
Nonsens.
Anti-C.
Random
49.27
50.28
48.4
49.12
49.01
49.69
49.12
LLaMa
44.03
48.23
29.46
52.66
45.14
44.22
42.67
Alpaca
44.66
52.03
29.53
51.13
44.86
44.40
44.77
GPT-3 Non-Instr. (davinci)
49.92
50.00
49.75
50.00
49.06
49.97
50.72
GPT-3 Instr. (text-davinci-WW)
51.40
51.30
52.63
50.47
54.31
50.13
50,05
GPT-3 Instr. ((ext-davinci-002)
53.15
50.85
56.96
51.90
55.33
52.47
51.81
GPT-3 Instr. (1ext-davinci-0033)
56.26
51.11
62.97
54.96
56.83
54.79
57.49
GPT-3.5
52.18
51.80
54.78
50.32
54.09
50.68
52.09
GPT-4
62.03
63.01
62.82
60.55
62.27
63.09
60.47
+ COUSALCOTT
70.40
83.35
67.47
62.05
69.25
71.58
70.12
We report the overall accuracy (Acc.), and also fine-grained accuracy by rung, and by degree of commonsense alignment, from commonsensical (Comm.), nonsensical (Nonsens.), to ootoonsmmmss (Anti-C.).
Table 2: Performance of all models our CLADDER dataset v1.5.
We compare the performance of all models in Table 2. First, we can see that the causal reasoning task in CLADDER is in general very challenging for models. Models such as the earlier, non-instruction- tuned GPT-3, and both LLaMa and Alpaca around random performance. With instruction-tuni tuning, models start to show some improvement. And amongst all, our CAUSALCOT achieves the highest performance of 70.40%, which is substantially better than the vanilla GPT-4 by 8.37 points. Morcover, CAUSAL.COT also achieve the best performance across all three rungs of causal questions, with a monotonically decreasing performance as the rungs get higher, i.e., the questions get more difficult. See Appendix for experiments on our earlier dataset v1.0.
5.3 Isolating the Effect of Data Contamination
well-known problem with evaluating LLMs on caattionnnnserinng tasks the data contamination problem, i.e., that LLMS perform well test set because the test set is (unintentionally) contained partially or even entirely in the training data [7, 56). We address this problem by creating not only the commonsensical subset of our dataset, but also anti-commonsensicall and nonsensical, both of which,

---

by construction, are very likely not in the training data of LLMs. From the accuracy by commonsense aligmment degree in Table 2, we can see the original GPT-4 model performs the worst on the anti- commonsensical subset (1.8 points lower than that on the commonsensical subset). However, our CAUSAI.COT enhances the reasoning ability across all levels, with substantial improvement on anti-commonsensica data by 9.65 points, highlighting the strength of CAUSALCOT on unseen data.
5.4 Error Analysis by Subquestions
Step ①
Step 2
Step @ & (
Step (
Step s
Node Edge Dist. (L)
Overall Rung Rung 2 Rung 3
Estimand
Fl
Arithmetic
99.34 97.01 1.69
50.65
69.99
59.14
42.12
53
47.53
99
Table Performance for each step in CAUSALCOT. For Step @, report the Fl score of node prediction, edge prediction, and also the graph edit distance (Dist) with the true graph. See more details in Appendix E.I.
We conduct a fine-grained error analysis by looking into the performance of different steps of CAUSALCOT in Table 3.7 We can see that the model is good at Step @ to extract causal graph G. achieving high Fl scores for predicting both the nodes and the edges correctly, although not perfect, still leaving a graph edit distance of 1.69 between the ground truth causal graph and the model-identified graph. The other steps are more challenging for the model. Among those, Steps @, @ and @ require careful and correct application of causal inference, where the model struggles. This reveals a notable weakness of current LLMs to perform formal causal reasoning, which is an important direction for future work on improving and enhancing LLMs. To better understand the reasoning abilities of L.LMs, we also perform an extensive analysis taking the entire reasoning chain of our CAUSAL COT and the ground-truth explanations, to produce 20 fine- fine-grained scores about the multi-step reasoning quality using the ROSCOE framework [25], and show detailed results in Appendix E.2.
5.5 Effect of In-Context Learning
Marg.
20
As an additional analysis, look into the effect of in-context learn- ing (ICL) by providing an example solution before asking the ques- tion. The interesting question us is whether models can generalize across different query types. Namely, we keep our CAUSALCOT framework, and prepend a reasoning example of query type i, and then calculate how much improvement it can bring when models answer new questions of query type j. In Figure 5, we can see that conditional probability and NIE are the questions that benefit the most from ICI., and showing examples of marginal probability and ATT are among the most helpful to all questions in general.
Cond.
ATE
10
Count.
ATT
10
NDE
-0
NIE
Marg. Cond.
ATE
Count
ArT
NDE
NIE
Figure 5: Heatmap showing the
how helpful each query type is to solving subsequent query types.
6 Related Work
part of the literature aimed at evaluating the performance of current LLMs [7, 15, 56, 76, 103, inter alia], focusing on understanding their strengths and weaknesses. Various studies into the capabilities of LLMs [8, 39, 56, 74] change people's perception of domains such as education [2, 80), medicine [54, 87], law [43], and computational
Skill evaluation for LLMs. Our work may be
social science [105]. However, most work evaluates new models on existing datasets from previously-
curated large-scale benchmarks [89, 94, 95], or human exams [41, 43, 56] which is becoming increasingly unreliable due training set contamination.
rausality-related skills for NLP. With the increasing attention on LLMs and causality [100, 101], review several formulations of causality-relater skills for NLP, which summarize into (1) causality as knowledge, (2) causality as language comprehension, and (3) causality as reasoning. In the causality-as as-knowledge line of work, many existing studies investigate how well NLP models understand commonsense causality, such as the cause and effect of an agent's 's action [81], motivation and emotional reaction in social context [82], correspondence of of steps with high-level goal [102], development of a story given a different beginning [75], and how in general L.MSs serve knowledge base of causality [100]. Concurrent work [45] focuses evaluating LLMs on various causality related tasks by leveraging the conceptual knowledge accrued from the training
We experienced some rate- lite-iimi in the fine grained analysis of LLMs that are only accessible through a web API. As result, occasionally had to evaluate subset of 2K random samples.

---

data, rather than formal causal inference, except for their causal sufficiency analysis which is close to our counterfactual questions. Importantly, most work in this line does not define explicit causal graphs, making it difficult to quantitatively define the ground-truth causal relationships in a principled way. The causality-as- suusalit-as-languuge-- comprreeenooionn line of work stems from traditional linguistic studies on causal connectives and causal language usage [9, 90, 99], to the recent causal relation extraction [4, 33, 98] to identify cause-effect pairs as a subtask of information extraction from text.
Finally, for causality as formal reasoning, our CLADDER work formulates the task of causal infer- ence for NLP, and our other work, CORR2CAUSE [42], addresses the causal discovery problem to infer causation from correlation. Together, they cover the two major branches of causal reasoning investigated in existing technical literature on causality. See a comprehensive comparison of literature in Appendix F.
7 Discussion of Limitations and Future Work
A Natural Language "Mini Turing Test" for Causality. Pearl and Mackenzie [66] describe an ideal Marini test" to assess understanding of causal inference, and argue that if a machine can answer all possible questions correctly, then it "understands" causality. According to the authors, this is because there are possible shortcuts when you consider all possible combinations of queries, graphs and data in this ideal test: due to their combinatorial explosion, the machine can only answer all questions right if it correctly applies causal reasoning. From this point of view, our work constitutes a first step towards a mini-Turing test formulated in natural language. However, we cover only some of the commonly studied causal queries spanning all three rungs. Future work may extend this to further queries, such as, e.g., path-specific effects other than NDE and NIE [52], thereby increasing the number of potential questions and moving closer to the ideal test.
LLMs and Causal Reasoning. has been claimed that LLMs understand causality well (e.g.. [45] report high performance, such 97% and 92%). In contrast, our work suggests that LLMs may still be far from reasoning reliably about causality (reaching only 60+% on CLADDER). As argued in Section 1, we believe that investigating this aspect may be of particular importance, since causal inference is crucial in many policy-relevant scenarios, where reliable AI systems could assist decision- making: from epidemiology [22, 79] economies [10,37] fairness [47, 71]. Testing the abilities of these systems in semi-realistic scenarios is therefore crucial, motivating some of the design choices in dataset: e.g., the example in Figure I was inspired by similar questions which arose in the context of the COVID-19 pandemic, where incorrect causal reasoning resulted in fallacy where vaccinations were considered to be harmful instead of beneficial [20, 49]. Further work may be dedicated to making the questions and verbalizations even closer to realistic instances of causal inference problems.
A CI Engine Plug-in for LLMs. An interesting direction for future research could be to provide the LLM access to an actual implementation of the CI engine. For example, Davis and Aaronson [13] tested the improvement of math abilities in L.MMs augmented with plug-ins (i.e., external modules that extend the model's capabilities by adding specific functionality or customizing its behaviour for particular tasks, like calculator), suggesting that they significantly enhance the model's ability to solve these problems. However, even with plug-ins, there are still often "interface" " failures: that is, "[the LLM) often has trouble formulating problems in a way that elicits useful answers from the plug-ine ins". We hypothesise that something similar would happen for causal inference: even once suitable plug-ins are built, the language-to- to-toll interface may still be non-trivial research question.
8 Conclusion
We proposed formal causal reasoning as a new task to evaluate LLMs, and created the CLADDER benchmark, covering several aspects of causal inference across all rungs of the ladder of causation and verbalizations involving semi-realistic scenarios. To address the task, we proposed prompting strategy, CAUSALCOT, inspired by the principles of formal causal inference, which introduces multistep tfain-of--tought reasoning for causal questions. Extensive experiments indicate that this dataset is highly challenging, thus offering principled tool to gain a better understanding of the reasoning abilities of L.L and to develop better models for causal reasoning in natural language.
Acknowledgment
We thank Sergio Hernan Garrido Mejia for pointing us to Python implementations of the causal inference engine. We thank Armen Aghajanyan for the idea of testing psychological bias in LLMs,
10

---

which partly contributes to the idea of exposing causal bias in LLMs. We thank András Strausz for various timely coding help, especially for our Simpson's paradox case.
The material presented in this manuscript is partly based upon works supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FK7: 011S18039B; by the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645; the Swiss National Science Foundation (Project No. 197155); a Responsible Al grant by the Haslerstiftung; an ETH Grant (ETH-19 21-1); and by the John Templeton Foundation (grant #61 156). Zhijing Jin is supported by PhD fellowships from the Future of Life Institute and Open Philanthropy, as well as the travel support from ELISE (GA no 951847) for the ELLIS program. Felix Leeb is supported by the International Max Planck Research School for Intelligent Systems (IMPRS-IS). Luigi Gresele is supported by the VideoPredict project, FKZ: 011S21088.
Author Contributions
The conceptualization and design of this project was led by Zhijing, Luigi and Felix, and supervised by Mrinmaya on the NLP part, and Bernhard on the causality part. Max provided timely iniighs from cognitive science different types of causal tasks and on the project design. In the exploration stage, Ojasv did substantial work on discovering causal fallacies in news and on Twitter, which, while not included in the current systematic way of generating causal inference questions, was significant contribution in the course of the project and in comparing various task formulations.
As for the operationalization and programming, the dataset composition was mainly led by Yuen and Felix, together with daily discussions with Zhijing, and weekly discussions with Luigi. Zhiheng supported important function of generating the backdoor adjustment set for given causal graph with the treatment and effect variables. The experiments are mainly conducted by Zhijing and Fernando, with Kevin finishing the evaluation results using the ROSCOE package.
References
[1] Yuntao Bai. Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.06 .008062. [Cited on page 8.J
Danny
[2] David Baidoo- Anu and Leticia Owusu Ansah. 2023. Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Available at SSRN 4337484. [Cited on page 9.]
[3] Elias Bareinboim, Juan D Correa, Duligur lbeling, and Thomas Icard. 2022. On Pearl's hierarchy and the foundations of causal inference. In Probabilistic and causal inference: the works of judea pearl, pages 507-556. [Cited on pages 2 and 3.]
[4] Steven Bethard, William Corvey, Sara Klingenstein, and James H. Martin. 2008. Building corpus of temporal-causal structure. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Language Resources Association (ELRA). [Cited on page 10.]
[5] Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In Sth International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [Cited page 27.1
[6] Stephan Bongers, Patrick Forré, Jonas Peters, and Joris M Mooij. 2021. Foundations of structural causal models with cycles and latent variables. The Annals of Statistics, 49(5):2885- 2915. [Cited on page 23.J
[7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh,
11

---

Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual. [Cited on pages 1, 8, and 9.]
[8] Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li. Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303.12712, [Cited pages 8 and 9.]
[9] Angela Cao, Gregor Williamson, and Jinho D. Choi. 2022. A cognitive approach to annotating causal constructions in cross-genre corpus. In Proceedings of the 16th Linguistic Annota- tion Workshop (LAW-XVI) within LREC2022, pages 151-159, Marseille, France. European Language Resources Association. [Cited on page 10.]
[10] David Card. 1999. The causal effect of education on earnings. Handbook of labor economics, 3:1 1118011163. [Cited on pages and 10.]
[1]] Raj Chetty, John Friedman, Nathaniel Hilger, Emmanuel Saez, Diane Whitmore Schanzen- bach, and Danny Yagan. 2011. How does your kindergarten classroom affect your earnings? Evidence from project star. The Quarterly journal of economics, 126(4):1593-166) [Cited on page 1.]
[12] Robert G Cowell, Philip Dawid, Steffen L Lauritzen, and David J Spiegelhalter. 2007. Proba- bilistic networks and expert systems: Exact computational methods for Bayesian nelworks. Springer Science & Business Media. [Cited on page 3.]
[13] Ernest Davis and Scott Aaronson. 2023. Testing GPT-4 with Wolfram Alpha and Code Interpreter plug-ins on math and science problems, arXiv preprint 2rXiv:2308.057133 [Cited on page 10.]
[14] Gaston De Serres, France Markowski, Eveline Toth, Monique Landry, Danielle Auger, Marlène Mercier, Philippe Bélanger, Bruno Turmel, Horacio Arruda, Nicole Boulianne, et al. 2013. Largest measles epidemic in North America in a -eueee, Canada, 2011: Contribution of susceptibility, serendipity, and superspreading events. The Journal of infectious diseases, 207(6):990-99) 998. [Cited on page 1.]
[15] Jacob Devlin, Ming-Way Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers), pages 4171- 4186, Minncapolis, Minnesota. Association for Computational Linguistics. [Cited on pages 1 and 9.]
[16] Quang Do, Yee Seng Chan, and Dan Roth. 2011. Minimally supervised event causality identification. In Proceedings of the 201 Conference Empirical Methods in Natural Lan- guage Processing, pages 294-303, Edinburgh, Scotland, UK. Association for Computational Linguistics. [Cited on page 27.]
[17] Richard Doll and A Bradford Hill. 1950. Smoking and carcinoma of the lung. British medical journal, 2(4682):739. [Cited on page 1.]
[18] Richard Doll and A Bradford Hill. 1954. The mortality of doctors in relation to their smoking habits. British medical journal, 1(4877):1451. [Cited on page 1.]
[19] Jesse Dunietz, Lori Levin, and Jaime Carbonell. 2017. The BECauSE corpus 2.0: Annotating causality and overlapping relations. In Proceedings of the 11th Linguistic Annotation Workshop, pages 95-104, Valencia, Spain. Association for Computational Linguistics. [Cited on page 27.]
[20] Jordan Fllenberg. 2021. Coronavirus vaccines work. But this statistical illusion makes people think they don't. 't. The Washington Post. [Cited on page 10.]
12

---

[21] Jörg Frohberg and Frank Binder. 2022. CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 2126-2140, Marseille, France. European Language Resources Association. [Cited on page 27.]
[22] Thomas A Glass, Steven N Goodman, Miguel A Hemánn and Jonathan M Samet. 2013. Causal inference in public health. Anmual review of public health, 34:61-75. [Cited on page 10.]
[23] Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. 2016. Causal inference in statistics: A primer. John Wiley and Sons. [Cited on pages and 19.]
[24] Moisés Goldszmidt and Judea Pearl. 1992. Rank-based systems: A simple approach to belief revision, belief update, and reasoning about evidence and actions. KR, 02:9622:6666 1676. [Cited on pages 2 and 3.]
[25] Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. [Cited on pages 9 and 25.]
[26] Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. SemEval-2012 task
7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394-398, Montréal, Canada. Association for Computational Linguistics. [Cited on page 27.]
[27] Joseph Y. Halpern and Judea Pearl. 2005. Causes and explanations: A structural-model approach. part i: Causes. The British Journal for the Philosophy of Science, 56(4):843-887. [Cited on page 19.]
[28] Joseph Halpern and Judea Pearl. 2005. Causes and explanations: structural-model approach. part ii: Explanations. The British journal for the philosophy of science, [Cited on page 19.]
[29] Yuval Noah Harari. 2014. Sapiens: A brief history of humankind. Random House. [Cited on page 1.]
[30] James Heckman, Lance J Lochner, and Petra E Todd. 2006. Earnings functions, rates of return and treatment effects: The mincer equation and beyond. Handbook of the Economics of Education, 1:307 458.. [Cited on page 1.]
[31] Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Pad6, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval- 2010 task 8: Multi-wa, way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33-38, Uppsala, Sweden. Association for Computational Linguistics. [Cited on page 27.]
[32] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. [Cited on pages 4 and 7.]
[33] Christopher Hidey and Kathy McKeown. 2016. Identifying causal relations using parallel Wikipedia articles. In Proceedings of the 54th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 1224-133,, Berlin, Germany. Association for Computational Linguistics. [Cited on page 10.]
[34] Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujic Lu, and William Yang Wang. 2022. Wikiwhy: Answering and explaining cause-and-effeed questions. arXiv preprint arXiv:2210. 12152. [Cited on page 2.]
[35] Mark Hopkins and Judea Pearl. 2007. Causality and counterfactuals in the situation calculus. Journal of Logic and Computation, 17(5):939-9533 [Cited on page 19.]
13

---

[36] Yimin Huang and Marco Valtorta. 2006. Pearl's calculus of intervention is complete. In Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence, pages 217-224. [Cited on pages 3 and 4.]
[37] Paul Hünermund and Elias Bareinboim. 2019. Causal inference and data fusion in econometrics. CoRR, abs/1912.09104 [Cited on pages 4, 5, and 10.]
[38] Ferenc Huszár. 2023. We may be surprised again: Why take Ilms seriously. [Cited on page 2.]
[39] Oana Ignat, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Gunal, Jacky He, Ashkan Kazemi, Muhammad Khalifa, Namho Koh, Andrew Lee, Siyang Liu, Do June Min, Shinka Mori, Joan Nwatu, Verónica Pérez-Rosas, Siqi Shen, Zekun Wang, Winston Wu, and Rada Mihalcea. 2023. A PhD student's perspective on research in NLP in the era of very large language models. CoRR, abs/2305.12544, [Cited on pages I and 9.]
[40] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can know what language models know? Transactions of the Association for Computational Linguistics, 8:423-435 [Cited on page 2.]
[41] Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding. Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schölkopf. 2022. Logical fallacy detection. In Findings the Association for Computational Linguisties: EMNLP 2022, pages 7180-7198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [Cited on page 9.]
[42] Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona T. Diab, and Bernhard Schölkopf. 2024. Can large language models infer causation from correlation? In The Twelfth International Conference on Learning Representations, ICLR 2024. OpenReview.net. [Cited on pages 8, 10, and 27.]
[43] Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2023. Gpt-4 passes the bar exam. Available at SSRN 4389233. [Cited on pages and 9.]
[44] Armin Kekić, Jonas Dehning, Luigi Gresele, Julius von Kügelgen, Viola Priesemann, and Bernhard Schölkopf. 2023. Evaluating vaccine allocation strategies using simulation-assisteed causal modeling. Patterns. [Cited on page 1.]
[45] Emre KıCıman, Robert Ness, Amit Sharma, and Chenhao Tan. 2023, Causal reasoning and large language models: Opening new frontier for causality. arXiv preprint arXiv:2305.00055 [Cited on pages 9 and 10.]
[46] Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2021. TellMeWhy: A dataset for answering why-questions in narratives. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 596-610, Online. Association for Computational Linguistics. [Cited on page 27.]
[47] Joshua R Loftus, Chris Russell, Matt Kusner, and Ricardo Silva. 2018. Causal reasoning for algorithmic fairness. arXiv preprint
[Cited on page 10.]
[48] Paramita Mirza, Rachele Sprugnoli, Sara Tonelli, and Manuela Speranza. 2014. Annotating causality in the TempEval-3 corpus. In Proceedings of the EACL 2014 Workshop on Computa- tional Approuches to Causality in Language (CAtoCL), pages 10-19, Gothenburg, Sweden. Association for Computational Linguistics. [Cited on page 27.]
[49] Jeffrey Morris. 2021. Israeli data: How can efficacy severe disease be strong when 60% of hospitalized are vaccinated? Accessed: 27th October 2023. [Cited on page 10.]
[50] Nasrin Mostafazadeh, Alyson Grealish, Nathanael Chambers, James Allen, and Lucy Van- derwende. 2016. CaTeRS: Causal and temporal relation scheme for semantic annotation of event structures. Proceedings of the Fourth Workshop on Events, pages 51-61, San Diego, California. Association for Computational Linguistics. [Cited on page 27.]
[51] Daniel Naber et al. 2003. A rule-based style and grammar checker. [Cited on page 7.J
14

---

[52] Razieh Nabi and Ilya Shpitser. 2018. Fair inference outcomes. In Proceedings of the Phirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1931-1940. AAAI Press. [Cited on page 10.]
[53] Brady Neal. 2020. Introduction to causal inference. [Cited on page 19.]
[54] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of GPT-4 medical challenge problems. CoRR, abs/2303.13375, [Cited on pages 8 and 9.]
[55] Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. CoRR, abs/2 12.00114. [Cited on page 7.]
[56] OpenAL. 2023. GPT-4 technical report. CoRR, abs/2303.08774 [Cited on pages 1, 8, and 9.]
[57] Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. CoRR, Abs/2203.02155. [Cited on page 8.]
[58] Judea Pearl. 1988. Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan Kaufmann. [Cited on page 3.]
[59] Judea Pearl. 1995. Causal diagrams for empirical research. Biometrika, 82(4):669 -68.. [Cited on pages 2, 3, 4, 5, and 24.]
[60] Judea Pearl. 2009. Causal inference in statistics: An overview. Statistics Surveys, 3(none):96- 146. [Cited on page 19.]
[61] Judea Pearl. 2009. Causality: Models, reasoning and inference (2nd ed.). Cambridge University Press. [Cited on pages 2 and 23.]
[62] Judea Pearl. 2011. The algorithmization of counterfactuals. Annals of Mathematics and Artificial Intelligence, 61 61:29-39. [Cited on page 3.]
[63] Judea Pearl. 2022. Comment:
understanding simpson's paradox. In Probabilistic and causal inference: The works of judea Pearl, pages 399 412. [Cited on page 2.]
[64] Judea Pearl and Elias Bareinboim. 2022. External validity: From do-calculus to transportability across populations. In Probabilistic and causal inference: The works of Judea Pearl, pages 451-482. [Cited on page 3.]
[65] Judea Pearl, Madelyn Glymour, and Nicholas Jewell. 2016. Causal inference statistics: primer. John Wiley & Sons. [Cited on page 3.]
[66] Judea Pearl and Dana Mackenzie, 2018. The book of why: The new science of cause and effect. Basic books. [Cited on pages 1, 2, 3, 7, 10, and 19.]
[67] Judea Pearl et al. 2000. Causality: Models, reasoning and inference. Cambridge University Press. [Cited on pages 3, 5, 19, and 24.]
[68] Derek Penn and Daniel Povinelli. 2007. Causal cognition in human and nonhuman animals: A comparative, critical review. Annu. Rev. Psychol., 58:97-118. [Cited on page 1.]
[69] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2017. Elements of causal inference: Foundations and learning algorithms. The MIT Press. [Cited on pages 2, 5, and 19.]
15

---

[70] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural I. Language Processing (EMNLP-IJCNLP), pages Hong Kong, China. Association for Computational Linguistics. [Cited on page 2.]
[71] Drago Plecko and Elias Bareinboim. 2022. Causal faimess analysis. arXiv:2207.11385. [Cited on page 10.]
arXiv preprint
[72] Stanley A Plotkin. 2005. Vaccines: Past, present and future. Nature medicine, 11(Suppl 4):55 S11. [Cited on page 1.]
[73] George Psacharopoulos and Harry Anthony Patrinos. 2004, Returns to investment in education:
A further update. Education economics, 12(2):111-134, [Cited on page 1.]
[74] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. chatgpt a general abs/2302.06676. [Cited on page 9.]
general-purpose purpose natural language processing task solver? CoRR,
[75] Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Flizabeth Clark, and Yejin Choi. 2019, Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 5043-5053, Hong Kong, China. Association for Computational Linguistics. [Cited on pages 9 and 27.]
[76] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models unsupervised multitask learners. OpenAl Blog, 1(8). [Cited on pages 1, 8, and 9.]
[77] Hannah Rashkin, Maarten Sap. Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2Mind: Commonsense inference events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 463-473, Melbourne, Australia. Association for Computational Linguistics. [Cited on page 27.]
[78] Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [Cited on page 8.]
[79] Kenneth J Rothman and Sander Greenland. 2005. Causation and causal inference in epidemi- ology. American journal of public health, 95(S1): S144 1114-4155000 [Cited on page 10.]
[80] Jürgen Rudolph, Samson Tan, and Shannon Tan. 2023. Chatgpt: Bullshit spewer the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1). [Cited on page 9.]
[81] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie. Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. ATOMIC: an atlas of machine commonsense for if-then reasoning. In The Thirty- Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Firs Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pages 3027-3035. AAAI Press. [Cited on pages and 27.]
[82] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Commonsense reasoning about social interactions. In EMNLP 2019. [Cited on pages and 27.]
[83] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),, pages 1222-4235 Online. Association for Computational Linguistics. [Cited page 2.]
16

---

[84] Ilya Shpitser and Judea Pearl. 2006. Identification of conditional interventional distributions. In 22nd Conference on Uncertainty in Artificial Intelligence, UAI 2006, pages 437-444. [Cited on pages 3 and 4.]
[85] Ilya Shpitser and Judea Pearl. 2006. Identification of joint interventional distributions in recur- sive semi-markon markovian causal models. In Proceedings, The Twenty- First National Conference Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, July 16-20, 2006, Boston, Massachuseits, USA, pages 1219-1226. AAAI Press. [Cited page 24.]
[86] Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormolabashi, Te-lin Wu, Xuezhe Ma, and Nanyun Peng. 2021. COM2SENSE: A commonsense reasoning benchmark with complemen- tary sentences. In Findings of the Association for Computational Linguistics: ACI-IJCNIPP 202/, pages 883-898, Online. Association for Computational Linguisties. [Cited on page 27.]
[87] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip Andrew Mansfield, Blaise Agüera y Arcas, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingan, and Vivek Natarajan. 2022. Large language models encode clinical knowledge. CoRR, abs/2212.13138, [Cited on page 9.]
[88] Peter Spirtes, Clark Glymour, and Richard Scheines. 2000. Causation, Prediction, and Search, Second Edition. Adaptive computation and machine learning. MIT Press. [Cited on pages 2 and 5.]
[89] Aarohi Srivastava, Abhinav Rastogi, Abhishck Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Ag- nieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes. Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206.046155. [Cited on page 9.]
[90] Manfred Stede. 2008. Connective- based local coherence analysis: A lexicon for recognizing causal relationships. In Semantics in Text Processing. STEP 2008 Conference Proceedings. pages 221-237. College Publications. [Cited on page 10.]
[91] Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmaya Sachan. 2023. A causal framework quantify the robustness of mathematical reasoning with language models. In Proceedings of the 6l Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Toronto, Canada, Association for Computational Linguisties. [Cited on pages 4 and 8.]
[92] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An iestruction-followwie Ilama model. sttpps//giis gighhu........ wwmmmotttsus
[Cited on page 8.]
[93] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302. [Cited on page 8.]
[94] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general- purpose language understanding systems. In Advances in Neural Information Processing
17

---

Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pages 3261-3275. [Cited on page 9.]
[95] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super- naturalinstructions: Generalization via declarative instructions Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 5085-5109, Association for Computational Linguistics, [Cited on page 9.]
1600+ NLP tasks. In
[96] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. [Cited on pages 2 and 7.]
[97] Cynthia G Whitney, Fangjun Zhou, James Singleton, and Anne Schuchat. 2014. Benefits from immunization during the vaccines for children program era- tra-united states, 1994-2013. Morbidity and Mortality Weekly Report, 63(16):352. [Cited on page 1.]
[98] Jinghang Xu, Wanli Zuo, Shining Liang, and Xianglin Zuo. 2020. A review of dataset and labeling methods for causality extraction. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1519-1531, Barcelona, Spain (Online). International Committee on Computational Linguisties. [Cited on page 10.]
[99] Bei Yu, Yingya Li. and Jun Wang. 2019. Detecting causal language use science find- ings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages Linguistics. [Cited on page 10.]
Hong Kong, China. Association for Computational
[100] Matej Zečević, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting. 2023, Causal parrots: Large language models may talk causality but are not causal. Transactions on Machine Learning Research. [Cited on pages 2 and 9.]
[101] Cheng Zhang, Stefan Bauer, Paul Bennett, Jiangfeng Gao, Wenbo Gong, Agrin Hilmkil, Joel Jennings, Chao Ma, Tom Minka, Nick Pawlowski, et al. 2023. Understanding causality with large language models: Feasibility and opportunities. arXiv preprint arXiv:2304.05554 [Cited on pages 2 and 9.]
[102] Li Zhang, Qing Lyu, and Chris Callison- 2020. Reasoning about goals, steps, and temporal ordering with WikiHow. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4630-4639, Online. Association for Computational Linguistics. [Cited on pages 9 and 27.]
[103] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov. Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open pre-trained transformer language models. CoRR, abs/2205.01063 [Cited on pages
and 9.]
[104) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019, Fine-tuning language models from human preferences. CoRR, abs/1909.0888 08593. [Cited on page 8.]
[105] Caleb Ziems. William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large language models transform computational social science? [Cited on pages 8 and 9.]
18

---

A Supplementary for Dataset Generation
A.1 List of References for Causal Inference
When collecting the causal graphs, query types, and commonsensical stories for our dataset, we took our examples from the following books (sorted by year):
1. Causality [67]
2. Causal inference statistics: A Primer [23]
3. Elements of Causal Inference [69]
4. The Book of Why [66]
5. Introduction to Causal Inference [53]
And the following papers:
Causes and Explanations: A Structural-Model Approach. Part Causes [27]
2. Causes and Explanations: A Structural-Modas Approach. Part II: Explanations [28]
3. Causality and Counterfactuals in the Situation Calculus [35]
4. Causal inference statistics: An overview [60]
Causal Graph Alias
Trealment- Pair
CGTE Alias
Confounding
Confounding
x
Mediation
Immorality
Fork
Collision
Diamond
Diamond
Chain
Arrowhead
Arrowhead
Frontdoor
Prontdoor
Figure 6: List of all ten causal graphs with treatment-effect pairs (CGTEs). We omit CGTEs that trivially resemble existing ones.
A.2 Formulation of the Query Types
Here, we introduce all the query types included in our dataset.
Rung-1 Queries: Marginal and Conditional Probabilities. For marginal probabilities, we ask questions about the overall distribution of a variable. For conditional probabilities, we ask whether conditioning on one variable increases or decreases the likelihood of another variable. For the explaining away questions, we condition on collider node and ask how that affects the correlation between the two parents.
19

---

Rung-2 Queries: ATE and Adjustment Set. For ATE questions, we ask whether the treatment (X 1) increases or decreases the likelihood of the effect variable Y = y. For adjustment set questions, we ask whether a set of variables should be adjusted for when estimating the causal effect between treatment and effect. By adjusting, we aim to blocked the non-causal paths from the treatments to effect, and hence eliminate spurious correlation. For example, to query whether the set gender is an adjustment set for the effect a treatment on recovery, we ask "To estimate the effect of the treatment recovery, should we directly look at how the treatment correlates with recovery, or should we look at gender-specific correlation?" In the collider bias questions, similarly to the collider variable and ask about how intervention
explaining away questions, condition one of the parents (treatment X) affects the other parent (outcome Y). However since by construction X and Y do not have common causes, the answer to this question is always "no".
Rung-3 Queries: Counterfactual Probability, ATT, NDE, and NIE. For counterfactual probabil- ity, we ask about what would have been the likelihood of Y y. if the treatment variable X had been r, given sufficient evidence e such that the query is identifiable. For ATT, we ask how the likelihood of Y = y would change for those who received treatment (X- 1) if there had been no treatment (X = 0). For NDE, we ask whether the X = I directly increases or decreases the likelihood of the y,, not through any mediators. For NIE, we ask whether the treatment (setting 1) increases or decreases the likelihood of Y = y through mediators, not directly.
A.3 Collection of Causal Graphs
We include all the ten causal graphs with treatment-effect pairs (CGTEs) in Figure
Note that one causal graph can have several different CGTEs, such as the confounding structure, which has three CGTEs: confounding, mediation, and collision in the triangle form. To generate all the causal graphs and CGTEs here. we iterate all commonly used ones within four nodes in the CI books, and omit CGTEs whose solution by CI methods trivially resembles existing ones.
A.4 Data Coverage
Starting from the full set of 12 distinct causal graphs and 10 query types, there are few combinations that must be omitted as the ground truth answer would be trivial or ill-defined. For example, in the "Immorality" graph, the treatment "X" and outcome "Y" are by construction statistically independent, so there correlation is necessarily 0. Similarly, there are several graphs where certain causal queries are ill-defined don't 't make sense to ask. Specifically:
1. For the Natural Direct Effect, we only include questions on the "IV", "Arrowhead", "Con- founding", "Mediation" and "DiamondCut" graphs.
2. For the Natural Indirect Effect, we only include questions on the "Mediation", "Frontdoor", "Arrowhead", "Diamond" and "Chain" graphs.
3. For the Collider Bias and Explaining Away effect, we only include questions on the "Colli- sion" graph.
4. For the Average Treatment Effect, we include questions on all graphs except "Collision".
5. For the (deterministic) Counterfactuals, we include questions all graphs except "Colli- sion".
6. For the Average Treatment Effect on the Treated (ATT), we include questions on all graphs except "Collision" and "IV".
The "balanced" benchmark (main benchmark in v1.5), containing 10, 12 questions split between all stories, graphs, query types, and commonsensicalness, is balanced such that there are roughly the same number of questions for each distinct gory-graph--qurry combination (ranging from 50-100 per combination) across the different variants: commonsense, anticommonsense, and nonsense. Furthermore, we balance the distribution of correct answers so that there are the same number of "yes"'s 's and "no"s.
The "aggregate" variant (main benchmark in v1.0) contains 10,560 questions and is primarily balanced across all stories. However since the number of stories for each variant (commonsense, anticommonsensee, and nonsense) varies significantly, the results in an unbalanced benchmark in terms of sensicalness.
20

---

A.5
Query Form and Text Templates
We provide in Table 4 the text templates we use for each query type.
Query Type
Symbolic Expres-
Natural Language Question Template
sion
Rung I: Association
Marg. Prob.
P(Y)
Is the overall likelihood of (Vnoun chance?
= 1}} greater than
Cond. Prob.
P(YIX)
Is the chance of {Vnoun (Y 1)} larger when observing Enomn (X 1)3?
Rung 2: Intervention ATE
E dX = 1)
Will {Uaoun (X 1)3 increase the chance of (Vnoun (Y = 1)}?
E Y do(X = 0))
Adjust. Set
If S opens back-
To understand how (Uoveral(()) affects Woverall should we look directly at how Woverall (X)} correlates with (Voverat (Y)) in general, or this correlation case by case accord- ing to [Voverai
1}},
door path
Rung 3: Counterfuctuals Counterf. Prob.
P(Y2 v)
Can we infer that {Usear (Y = 1}} had it been that EVcond (X 1)} instead of X=0?
ATT
E|y1 YoIX
For Wasut(
1)3, would it be more likely to see (Unoun (Y
1)} Weonnd (X = 0)}
NDE
E|Yi,Mo - Mi,
If we disregard the mediation effect through (ooverall (Y = 1)3.
would {Unoun (X 1)}} still positively affect {Unoun (Y = 1)}?
NIE
E|Y., Yo.mol
Does {Woverall (X)) affect (oovrrall (Y)} through
(Foverall (Uoversl((Ohher (Other Vars))?
Table Example natural language templates for each query type.
A.6 Nonsensical Stories
To come up with collection of nonsensical variable names, use GPT-4 generate some meaningless words. Specifically, we use the prompt: "Create 100 non-existent words that are short, i.e., within 5-characters.", with temperature=0 with the OpenAl interface. The collection of nonsensical words we later use as variable names are as follows: ziblo, truq, fyze, glimx, jory, wexi, snov, yupt, kraz, qixy, vubr, chiz, pliv, moxa, fygo, rukz, tasp, xevo, jyke, wibl, zorf, quzy, nyrp, gwex, smez, vytz, hupx, cwoj, lirf, ovka, pexu, yigz, twaz, kwox, zuph, frag, jyxo, swoy, uvzi, nekl, gyzp, rixq, vwem, xyfu, blyz, qwip, zeku, tijv, yomx, hwaz, czix, plof, muvy, fyqo, rujz, tasb, xevi, jyka, wibm, zorx, quzw, nyro, gwet, smeu, vyta, hupz, cwoi, lirg, ovki, pexy, yigw, twac, kwoz, zupj, fraq. jyxi, swoq, uvzo, nekm, gyzl, rixw, vwen, xyfo, blyx, qwiu, zeky, tijw, yomz, hwax, czir, ploz, muvq, fyqi, rujx, tasn, xevu, jyko, wibp, zory, and quzt.
A.7 Anti-Commonsensical Stories
For the anti-commonsensical stories, we randomly do one of the actions:
1. Replace the effect variable Y with an attribute that would not be an effect variable in any of the stories. Such replacement variables include: "lip thickness", ""arthquakes", "lactose intolerance", "rainfall", "is allergic to peanuts", "brown eyes", "curly hair", "black hair", "foot size". "freckles"
2. Create an irrelevant treatment variable X that does not play a causal role in any of our commonsensical stories. Such as: "can swim", "is religious"; "has a brother", "has visited England". "likes spicy food", "is vegetarian", "speaks english", "drinks coffee". "plays card games", "listens to jazz", "solar eclipse", "has a sister", "full moon"
To transform commonsensical story into cotmooommmnsl story, we apply of these replacements sampled uniformly, resulting in stories such as:
Ability to swim has a direct effect on studying habit and exam score. Studying habit has direct effect on exam score.
Gender has direct effect on department competitiveness and peanut allergy. Department competitiveness has direct effect peanut allergy.
21

---

Liking spicy food has direct effect on relationship status. Appearance has direct effect on relationship status.
Playing card games has direct effect on diabetes and lifespan. Smoking has direct effect on diabetes and lifespan. Diabetes has direct effect on lifespan. Smoking is unobserved.
For full list of the replacements and how the replacements are made, check out the code.
A.8 Explanation Template
Step @ Extract the causal graph: The causal graph expressed in the context is: "g".
Step @ Identify the query type: The query type of the above question is squery_type"
Step Formulate the query to its symbolic form: The formal form of the query is
Step ① Collect all the available data: The available data are: "d".
Step @ Derive the estimand: Based the graph structure and causal query, the question can be simplified into estimand "est".
Step @ Solve for the estimand: Plug in the available data "d" into "est". est(d) d) = float(a)
Since the estimate for the estimand is float(a), the overall answer to the question is bool(a).
B Experimental Details
B.1 CAUSALCOT Prompt
Q: lquestion from the dataset]
Guidance: Address the question by following the steps below:
Step 1) Extract the causal graph: Identify the causal graph that depicts the relationships in the scenario. The diagram should simply consist of edges denoted in "varl -> var2" format, separated by commas.
Step 2) Determine the query type: Identify the type of query implied by the main question. Choices include 'marginal probability". "conditional probability". "cxplaining away effect", "backdoor ad- justment set", "average treatment effect", "collider bias". "normal counterfactual question", "average treatment effect on treated", "natural direct effect" or "natural indirect effect", Your answer should only be term from the list above, enclosed in quotation marks.
Step 3) Formalize the query: Translate the query into its formal mathematical expression based on its type, utilizing the "do(-)" notation or counterfactual notations as needed.
Step 4) Gather all relevant data: Extract all the available data. Your answer should contain nothing marginal probabilities and conditional probabilities in the form "P((..).... or "P(......)=..."" ". each probability being separated by semicolon. Stick to the previously mentioned denotations for the variables.
Step 5) Deduce the estimand using causal inference: Given all the information above, deduce the estimand using skills such as do-calculus, counterfactual prediction, and the basics of probabilities. Answer step by step.
Step 6) Calculate the estimand: Insert the relevant data Step into the estimand, perform basic arithmetic calculations, and derive the final answer. There is an identifiable answer. Answer step by step.
A: [LLM previous response]
Q: Based on all the reasoning above, output one word to answer the initial question with just "Yes" or "No".
22

---

A: [LLM final answer]
C Additional Technical Background for Preliminaries
C.1 Graphical Models
We adopt the causal inference framework described in [61]. causal graph G == (V, E) consists of a set of k vertices V (Vi,..., V and directed edges E == {eigh J. where the existence of each eij means that there is direct causation from V; to V,, also denoted introduce some notations to describe the relative positions among the nodes. Following a standard assumption causality (but see, e.g., [6]), we will assume that G is direct acyclic graph (DAG), E}. We denote descendants V2 E E} of node V; as all the nodes that have at least one direct path leading to a node. We call node V/2 as a confounder (i.e., common cause) of the other two nodes 1/ and V, if ekis Cky E E; a collider (i.e., common effect) if eyk. Cjk E; and a mediator Cik, Ckj E E.
V; Vj. We also
where we denote the parents of a node Vú as PA(V)) ;= (Viles, DE(V)) == [VgVV,
X and Y denote two special variables, the treatment and Among all the variables in V. effect, respectively.
C.2 Illustration of the Three Rungs of the Causal Ladder
In Figure 7, we illustrate the difference among the three rungs by enumerating what actions are performed on the variables other than target variables X and Y.
Rung Association
Intervention
Rung Counterfactuals
The cometition and Y. Le. P(V[X) flows thruugh alll undinedtdd patfic
do(X)
Force to be countrttrrtall value s
N
Direct cousation path
Direct intevention on CuTS of al parents
Bodiidoor path thouugh the confouncer Z
complestty effect we the oounterlectun P(Y,: X-e) hamely we inter all non descendants of i were dki the ongannl value x.
Average over all the non-descondants of to get PLY| do]K))
Figure 7: The Causal Ladder consists three rungs: association, intervention and counterfactuals. We color in blue the treatment X and effect Y, as well as the actions on X. We color in orange words about how to get the estimand, and we use the orange circle to include all the non non-descendants of X.
C.3
Causal Inference Methods
We introduce do-calculus which can downgrade the Rung-2 queries to Rung-1 quantities when is applicable, and counterfactual predictions which downgrade the Rung-3 queries.
C.3.1 Do-Calculus
Do-Operator as Notation mentioned in Rung 2, the do-operator a convenient notation to represent intervention on a variable. For example, do X = x) sets the value of variable X to a.
Three Inference Rules for Climbing the Ladder Do-calculus is set of rules that allows to answer higher-rung questions using lower-rung quantities, such as probability distributions of Rung 1. Given causal graphical model with and four disjoint of variables X, Y. Z, and W, and a joint probability distribution that is Markov and faithful to the graph, do-calculus contains the following three rules:
Rule I (Insertion/delltion of observations):
(Y))))(X) Z,W P(Y P(Y]do(X).
Y and Z are d-separated by X UUW in G", the graph obtained from g by removing all arrows pointing into variables in X.
Rule 2 (Aciionfobservation exchange):
dl)) P(Y do()), Z, W),
(2)
23

---

if Y and Z are d-separated by X U W in G", the graph obtained from G by removing all arrows pointing into variables in X and all arrows pointing out of variables Z.
Rule 3 (Iseeriion/deletion of actions):
P(Y (oo))) W) P(Y W).
if Y and Z are d-separated by X UW in G the graph obtained from G by first removing all arrows pointing into variables in X (thus creating G*) and then removing all arrows pointing into variables in Z that are not ancestors of any variable in W in G".
These rules are sound and complete [85]. Namely, iff we have all the terms on the right hand side, then the causal term on the left hand side is identifiable.
Example Application of Do-Calculus Taking the example in Figure 2, 91 maps the query type ATE its symbolic expression ([Y]d((3 = 1)] E do = =]]-
Next, 92 further simplifies the estimand given the confounding graph, as in the flow chart in the middle of Figure 2:
ATE = E|y do(X = 1)] Y do = 0]]
P(Z = B 1, Z =) E(|X 0, Z =)]]
(5)
which which resolves all the do(-) terms probability terms. This example shows the famous backdoor adjustment do-calculus [59].
C.3.2 Three Steps for Counterfactual Prediction
Given a SCM M, distribution on the exogenous variables P(u), and evidence e from the model (M, P(u)), the probability of the counterfactual "if X had been then Y would have been given we observed e," denoted P(Y, = yle), can be evaluated using the following three steps [67]:
Abduction: Update the probability distribution P(u) by the evidence e to obtain P(ule)
Action: Modify M by the action do(X 2.), i.e. replace X with X = x in the structural equations, to obtain the modified SCM M,
Prediction: Use the modified model (M,, P(ule)), to compute the probability of y.
D Previous Results on CLADDER v1.0
D.1 Dataset Statistics for v1.0
Total
Rung
Rung
Rung
Size
# Samples
10,560
3,288
3,288
3,984
Counterf. NDF
Question
# Sentences/Sample
6.85
7.00
7.25
ATT
# Words/Sample
94.47
76.41
96.84
103.42
Rung
# Edges/Graph
# Nodes/Graph
3.54
3.44
3.41
3.54
3.43
3,55
3,54
3.46
Cond
Rung
Rung
Aglust
Answer
Positive Class (%)
50
50
Explanations
LorO Ribb
# Sentences/Sample
13.11 146.82
12.04
13.76
13.83
# Words/Sample
141.88
147.88
151.30
Figure Distributions of query types in our dataset v1.0.
Table 5: Statistics of our CLADDER data v1.0.
Our data-generating procedure has the potential to algorithmically generate very large amounts of questions. In practice, we pick dataset size that is large enough to be representative, and the same time not too large to be problematic given the expensive inference costs of LLMs. We therefore set dataset size be 10K. We report the statisties of our dataset in Table 5.
The dataset roughly balanced across the query types, graph structures, stories, and ground-truth answers (as seen in Figure 8). Note that there are some slight adjustments such as more samples for ATE because it allows us to test various techniques, including backdoor and front door adjustments. More details on design choices can be found in Appendix A.4.
24

---

D.2
Main Results on v1.0
Acc. by Rung
Acc. by Empirical Alignment
Overall Acc.
Anti-C.
Nonsens.
Comm.
Random
49.27
50.28
48.40
49.12
49.69
49.01
49.12
LLaMa
45.22
63.33
31.10
41.45
45.31
45.21
45.12
Alpaca
45.54
63.33
31.57
41.91
45.94
45.21
45.49
GPT-3 Non-Instr. (davinci)
47.42
63.88
32.99
44.89
47.0
48.28
46.97
GPT-3 Instr. (1exI-davinci0001)
57.07
63.95
63.63
48.04
59.12
57.81
54.28
GPT-3 Instr. (texI-davinci-002)
56.24
46.03
69.55
55.04
54.75
59.65
54.31
GPT-3 Instr. (lexI-davinci-003)
62.69
58.0
80.83
54.52
63.93
62.09
62.05
GPT-3.5 (queried in May 2023)
61.71
65.12
69.9
54.11
65.43
55.15
64.55
GPT-4 (queried May 2023)
64.28
53,94
81.87
63.11
65.75
60.87
66.21
+ CAAUSACOT
66.64
61.67
86.13
58.23
69.32
63.02
67.60
Table 6: Performance of all models on our CLADDER dataset v1.0. We report the overall accuracy (Ace.), and also fine-grained accuracy by rung and by empirical alignment.
We compare the performance of all models in Table 6. First, we can see that the causal reasoning task in CLADDER is in general very challenging for all models. And models such the earlier, non-instruction- tundd GPT-3 and both LLaMa and Alpaca are no better than random performance. With instruction-tunie -unnnng models start to show some improvement. And amongst all, our CAUSALCOT achieves the highest performance 66.6 66.64%, which is 2.36 points better than vanilla GPT-4.
Moreover, from the accuracy by empirical alignment level in Table 6, we can see that the original GPT-4 model performs the best commonsensical data, but 5.34 points worse nonsensical data. However, our CAUSALCOT enhances the reasoning ability across all levels, with substantial improvement cotteosemsnnna data and nonsensical data, indicating that CAUSALCOT is particularly beneficial on unseen data.
D.3 Ablation Study on v1.0
We conduct an ablation study for our multi-step CAUSALCOT. We ablate each of the four subquestions, and observe in Table 7 that classifying the query type and formalizing has the most effect on the model's ' perfor- mance, which might be because that they are the crucial formalization step in order to do the causal inference correctly. Meanwhile, removing Steps @ and @, which are mostly about parsing the prompt correctly, have the least impact on performance.
Acc.
CAUSALCoT
66.64
w/o Step i
64.54
w/o Step 2
63.74
w/o Step @
63.43
wlo Step @
64.47
Table Ablation study.
E
More Experiments
E.1 Details of Our Error Analysis
For Step about the query type prediction, we report the overall F1 classification score, and also F1 by rungs. For the rest of the steps, we manually annotate the correctness of 100 samples of CAUSALCOT. We report the correctness of est by accuracy, and the correctness of the predicted set of available data by taking the F1 with the ground-trun -tuth d. For Step 5, we report the accuracy of whether the model simplifies the estimand correctly to est' using causal inference, and also arithmetic correctness (Arith.).
E.2 ROSCOE Evaluation
We employed the ROSCOE suite of evaluation metrics on step-by-step text reasoning, as introduced by [25], to automate the evaluation of the outputs from CAUSALCOT 2,000 randomly sampled questions from our dataset. Differing from conventional metrics, ROSCOE is specifically designed scrutinize the quality of large language model outputs, focusing on aspects such as semantic consistency, logicality, informativeness, fluency, and factuality, all evaluated within the context of step-by-step reasoning, rather than solely the final response. This allows for more objective and comprehensive assessment of a model's output, greatly aiding in the verification of its interpretability. The results of this evaluation can be found in Table and Figure We consider the model's performance as unsatisfying if it falls out of the top quantile, namely receiving score E [0, 1] smaller than 0.25 when the score should be minimized, or greater than 0.75 when should be maximized.
25

---

We can see in the plot that the good-performin performing aspects are faithfulness to the original question, reasoning alignment with the ground truth, and absence of external hallucinations, which are consis- tently within the top quantile. This suggests that the model carries out accurate reasoning within the constraints of the fictitious world introduced each question.
However, there are some performance dips in redundancy, perplexity chain, and missing step metrics. The first could potentially be attributed to complex elements such graph notation, while the relatively lower "missing step" score warrants further investigation. Despite these observations, this analysis largely aligns with qualitative understanding of the models' good response ability in answering causal questions in our dataset.
Mean
Std
Min
25%
50%
75%
Max
Faithfulness
0.89
0.02
0.83
0.88
0.89
0.90
0.93
Informativeness Step
0.88
0.01
0.83
0.87
0.88
0.89
0.92
Informativeness Chain
0.88
0.03
0.76
0.87
0.89
0.90
0.96
Faithfulness Word
0.95
0.01
0.92
0.94
0.95
0.96
0.97
Repetition Word
0.02
0.02
-0.00
0.00
0.02
0.04
0.05
Repetition Step
0.02
0.01
-0.00
0.00
0.01
0.03
0.06
Reasoning Alignment
0.92
0.01
0.86
0.91
0.92
0.93
0.95
External Hallucination
0.97
0.02
0.84
0.97
0.98
0.99
Redundancy
0.80
0.05
0.56
0.77
0,80
0.83
0.92
Common Sense Error
0.95
0,01
0.86
0.94
0.95
0.96
0.98
Missing Step
0.78
0.03
0.58
0.76
0.78
0.80
0.88
Semantic Coverage Step
0.99
0.01
0.95
0.98
0.99
0.99
1.00
Semantic Coverage Chain
0.98
0.01
0.93
0.98
0.98
0.99
0.99
Discourse Representation
0.06
0.13
0.00
0.01
0.01
0.05
0.67
Coherence Step Vs Step
0.14
0.27
0.00
0.00
0.01
0.07
0.94
Perplexity Step
0.02
0.01
0.00
0.02
0.02
0.03
0.07
Perplexity Chain
0.17
0.07
0.05
0.11
0.23
0.42
Perplexity Step Max
0.00
0.00
0.00
0.00
0.00
0.01
0.02
Grammar Step
0.93
0.04
0.77
0.90
0.93
0.96
0.99
Grammar Step Max
0.53
0.35
0.02
0.12
0.65
0.85
0.99
Table 8: Statistics of ROSCOE scores evaluated answers from CAUSALCOT on 2,000 randomly sampled
questions from our dataset.
F Comparison with Existing Causality-Related Datasets
We show in Table 9 the distinction of our work from all existing causality-relatee related datasets that address either the causality-as aasalliy----koowledge task, or the causality-as sisiitt--s--anneee saingugge-copprhhet task.
26

---

Question Types
Skill Types
Formalization of Causal Queries
Qualitative Reasoning
Assoc. Interv. Counterf. CI Method
Causal RE
Datasets for Causality Knowledge (Commonsense Causality)
CОPA [2012]
X
Event2Mind [2018]
X
x
x
ATOMIC [2019]
X
X
X
x
X
SocialIQA [2019]
X
/
X
X
x
X
X
TimeTravel [2019]
X
J
X
X
X
X
Goal-Step [2020]
X
X
Abductive (ART) [2020]
X
X
X
Com2Sense [2021]
X
X
X
X
X
X
CRASS [2022]
X
X
J
X
X
Datasets for Causality Language Comprehension (Causal Relation Extraction)
SemEval2021 Task8 [2010]
X
EventCausality [2011]
X
X
X
X
Causal- TimeBank [2014]
X
X
X
X
X
CaTeRS []016]
X
X
X
X
BECauSE [2017]
x
TellMeWhy [2021]
X
Datasets for Formal Causal Reasoning
Corr2Cause [42]
X
X
X
CLADDER (Ours)
Table 9: Comparison of our dataset and existing causal or reasoning datasets. The aim of our datasct is to test the pure reasoning ability of LLMs on causal questions. For each dataset, first identify whether its question types the three rungs: association (Assoc.), intervention (Interv.), and counterfactuals (Counterf.). We also check what skill types the dataset tests: the application causal inference methods (CI Method), formalization of causal queries, causal relation extraction from the given text (Causal RE), and qualitative reasoning.
27

---

brtruhness
informatiionzaa
193
793
34
190
01
02
1.0
00
0.3
fatbf-lnoss ww
raperitan word
rspettiion stap
1230
-02
700
100
-00
130
02
-30
103
230
198
30
02
00
04
common seace error
Ticsing stop
semanbi coverage dhop
000
199
307
-00
-30
200
132
195
02
coverage
ingecsentaaton
2009
1230
000
100
302
-30
250
200
01
persheniig
perplaniiy chein
perplesily
30
102
1200
75
631
00
02
04
05
0.
02
2.5
0.
10
00
04
06
03
grammar slep
or PEATTEEETT stap
100
190
198
50
Figure 9: ROSCOE scores answers from CAUSALCOT on 2,000 randomly sampled questions from our dataset.
28


Reviews:
[{"review_id": "GjYnifHs9j", "reviewer": "Reviewer_bMQ8", "summary": "This paper evaluates the causal reasoning ability of language models and proposes a structured prompting approach to improve this ability. For this goal, the authors create a benchmark. The benchmark includes synthesized causal graphs and related queries. The answers to queries are computed based on a causal reasoning formalism given the causal graph.  The graph and queries are verbalized for creating the corpus to provide natural language context and questions.  For evaluation purposes, a variety of current large language models such as GPT 3.5 and GPT 4 are used.  The language models are given the questions that need causal reasoning to be answered.  In addition to the new benchmark and the LLM evaluations, the authors propose a structured chain of thought prompting strategy (called CausalCOT) to improve the ability of LLMs in causal reasoning. The idea is to provide step-by-step prompts based on the steps of formal reasoning. The steps include prompting for extraction of the causal graph,  classifying the query type, and several others. \nThe results show that structured prompting improves the causal reasoning ability of the large language models. Especially for the questions that are anti-commonsensical and are not contained training large language models, the models show a more considerable extent of improvements compared to the baselines. \n", "questions": "See the above weakness. ", "limitations": "there is a section on this in the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper focuses on an important problem which is causal reasoning with LLMs.\nThe created benchmark is potentially a valuable evaluation resource. \nStructured prompting is a recent research trend for improving LLMs, and the authors follow the same trend but with a novel structure that is based on causal reasoning steps. \nThe findings show the effectiveness of structured prompting that is in line with other related research. \n\n", "weaknesses": "--It is not clear to me how the LLMs follow each step of reasoning:  Is this totally zero-shot? or do you do in-context learning with a few examples? if this is zero-shot, it is a bit hard to see why the model knows about causal graphs and specific query types for example. \n--What is the lexicon used for verbalization of the causal model? How many distinct works are used? How do you evaluate if the text  that is generated based on a template, is in line with commonsense or not? \n--The data set is synthetic which is fine. However, do you have any realistic scenarios to test the model on? I mean some realistic problem setting that shows this causal reasoning is required and the structured prompting is helpful in that realistic setting. More specifically, do we have real natural language explanations that explain complex casual graphs? can you add more discussions in this direction to the paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comment that this work “focuses on an important problem”, “The created benchmark is potentially a valuable evaluation resource”, and our CausalCoT is “a novel structure that is based on causal reasoning steps”.\n\n> Is this totally zero-shot? or do you do in-context learning with a few examples?\n\nTo avoid effects of arbitrary choices of hyperparameters of a few-shot evaluation (e.g. number of samples and sampling method), we report zero-shot performance of all models in the main results (Section 5.2), since most LLMs are trained on the large internet corpus that covers basic causality knowledge. In the paper, we also did a small-scale analysis of the few-shot setting in Sec 5.6 “Effect of In-Context Learning”. \n\nDuring the rebuttal period, we extended the experiments, and obtained the performance for few-shot learning using 10 examples to instruct CausalCoT, leading to an accuracy of 70.09% (+4 point increase over the 0-shot CoT in Table 2), as the model learns more correct reasoning from the examples.\n\n> if this is zero-shot, it is a bit hard to see why the model knows about causal graphs and specific query types for example. \n\nAs shown in Figure 1, the first step is causal relation extraction (i.e., text-to-causal graph parsing). In each input prompt, all necessary the causal relations are explicitly mentioned (in natural language), so instead of inferring causal relations from training data, the model only has to parse the input and then solve the causal inference problem. This way we focus our evaluation on formal reasoning, rather than commonsense-based causal reasoning as in related work.\n\nHence, in the error analysis by subquestions (Section 5.4), the first step tests whether models can comprehend causal text expressions well enough to ground them into symbolic forms. And the second step to classify a question into specific query types is also a skill that LLMs should have knowledge of given its training data of the entire internet and many book corpora. For example, GPT-3.5 and GPT-4 can act as a causality textbook to explain the definition of each query type and give examples. And this step is to test the reverse ability. For example, the question “Is the chance of YY larger when observing XX?” is asking about correlations (Rung 1), and the question “Will XX increase the chance of YY?” is asking about average treatment effect (Rung 2). We regard such skills within the current capabilities of LLMs given its large training data.\n\n> How many distinct works are used?\n\nAs listed in Appendix A.1, we collected causal questions from a list of nine commonly-used causality books and papers (Pearl et al., 2000; Glymour et al., 2016; Peters et al., 2017; Pearl and Mackenzie, 2018; Neal, 2020; Halpern and Pearl, 2005a; Halpern and Pearl, 2005b; Hopkins and Pearl, 2007; Pearl, 2009a).\n\n> What is the lexicon used for verbalization of the causal model? \n\nWe report the question statistics in Table 1, where we have on average 94.47 words per question, and 539 vocabulary words in total. We also show data examples in Appendix Tables 5-12. \n\n> How do you evaluate if the text that is generated based on a template, is in line with commonsense or not?\n\nAs in lines 171-183, we collect commonsensical causal stories from common examples in causal inference books and papers (in Appendix A.1), which always set up the problems in a realistic way, such as the drug-gender-recovery example of Pearl and Mackenzie (2018) to illustrate Simpson’s paradox. And for the non-commonsensical stories, their detailed generation process are in line 177-183.\n\n> The data set is synthetic which is fine. However, do you have any realistic scenarios to test the model on? I mean some realistic problem setting that shows this causal reasoning is required and the structured prompting is helpful in that realistic setting. More specifically, do we have real natural language explanations that explain complex causal graphs? Can you add more discussions in this direction to the paper?\n\nYes, we can add more discussions about extending the dataset to more real-world scenarios in the camera-ready version of the paper, such as fake news debunking, and logical fallacy correction. In the meantime, when composing the dataset’s commonsensical part, we took care to set many problems more realistically. Some of the “commonsensical” questions included in our dataset were inspired by real-world examples of policy-relevant causal inference questions. For example, the dataset includes questions on vaccine efficacy (see the example in our Figure 1) inspired by similar questions which arose in the context of the COVID-19 pandemic, and where incorrect causal reasoning resulted in fallacies where vaccinations were considered be harmful instead of beneficial in preventing severe cases (see, e.g., https://www.covid-datascience.com/post/israeli-data-how-can-efficacy-vs-severe-disease-be-strong-when-60-of-hospitalized-are-vaccinated).\n\nAnother note for the general usage of our theoretical framework is that the notions of NDE and NIE are very applicable to cases like bias and discrimination. Let’s take the example of how the double-blind review process for papers work: Potentially, we have the causal graph of author identity->paper quality->acceptance and also author identity->acceptance. The unfair causal effect is the natural direct effect, i.e., NDE(author identity->acceptance), and that’s why author identities are anonymized to avoid this problem.\n"}, {"review_id": "36k87q8BqW", "reviewer": "Reviewer_5uc1", "summary": "The research question is whether large language models are capable of doing\nformal causal reasoning as formalized by the approach with SCMs popularized by Pearl.\nThe authors provide a benchmark of 10k examples to evaluate such abilities; they describes in\ndetail the design choices for this benchmark.\nThey then propose a chain-of-thought prompting strategy (CausalCoT) to enhance LLMs with causal reasoning abilities; evaluations are carried out both on OpenAI's proprietary LLMs\nand open-source LLMs.", "questions": "* L. 75; is it correct that Rung i+1 is more difficult than Rung i? Maybe it could be emphasized also in this section.\n* L. 264: please add an explanation of why temperature=0\n* Table 2: Can you explain why CausalCoT makes performance drop on Rung 3, which should be the most challenging one in the Ladder?\n* I am having trouble to understand how CausalCoT works in practice. Could you point me to where I can find an interaction with the model\nin the supplementary materials? It would be great if one such example could make it to the paper or the appendix. I am sorry if I missed it somewhere.\nMore specifically, according to this [post](https://www.lesswrong.com/posts/yZb5eFvDoaqB337X5/investigating-causal-understanding-in-llms)\nsome form of few-shot examples are needed to successfully prompt models, so I am wondering if one needs to do the same to get the causal graph representation when using CausalCoT.\n\nOverall, my main concern is a lack of discussion or argument for why evaluation on this specific formalization of causal reasoning should matter. \nI am happy to increase the score if my concerns get addressed.", "limitations": "Good. During the rebuttal period the authors agreed to discuss a couple of extra limitations of the work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The design principles behind the benchmark are well-explained. In particular since LLMs\ntend to struggle with arithmetic-heavy tasks authors took care of using binary variables and\nsimple causal graphs.\n* A variety of topologically distinct treatment-effect pairs is considered, covering thus multiple\ncausal graphs.\n* Non-sensical and anti-commonsense variants of stories are injected to reduce the effect of LLMs having\nmemorized commonsense causal knowledge. Results are reported for the different splits (commonsense, anti-commonsense, etc...).\n* Good quality figures and diagrams; sufficient background material in the appendix for those not familiar with Pearl's causal framework.\n* A new application of CoT to formal causal inferences.\n", "weaknesses": "* I have some concerns about whether the research question is appropriate for the conference audience. For example, the cited (in the paper) concurrent\nwork of (Kiciman et al 2023) considers model evaluation on several ways of formalizing causal discovery; this broadens the scope of the potentially interested audience.\n* In my opinion it would be very important to discuss why this specific approach to evaluating causality matters\nand the ML community should care. For example, one might argue that human beings have deeply advanced their causal understanding\nof the world for centuries without using Pearl's framework to draw causal conclusions, so why should we care about LLMs using this specific framework? To draw another analogy,\nmathematics keeps progressing, yet almost no professional mathematician can write a formal proof or uses a formal\nproof to discover new math; the interest in understanding if LLMs can write formal proofs or can assist in writing formal proofs should be well-motivated.\n* If formal causal reasoning abilities matter for discovering new science (e.g.~the quote opening the Introduction),\nthen it would be good to discuss why a setting in which the model uses software tools for causal inference was not considered. Do such tools exist? For example, for formal mathematical proofs this is the case and a natural approach would be to make LLMs make use of such tools.\n* CausalCoT makes Rung 3 worse (Table 2). IIUC Rung 3 is the most challenging task (counterfactuality), so it would be good to explain why CausalCoT makes the accuracy worse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing that “The design principles behind the benchmark are well-explained”, the dataset is diverse with “A variety of topologically distinct treatment-effect pairs” “covering thus multiple causal graphs”, the experiments are multi-faceted with “Non-sensical and anti-commonsense variants of stories”, and the paper has “Good quality figures and diagrams”.\n\nIn the following, we will focus on a few main points: \n1. The value of the paper, including (a) the relation with the concurrent paper Kiciman et al. (2023), and (b) the value of automating formal reasoning abilities for LLMs/society, and \n2. Several technical details to clarify.\n\n### 1. Clarification of the Value of the Paper\n\n> the cited concurrent work of (Kiciman et al 2023) considers model evaluation on several ways of formalizing causal discovery.\n\nThank you for bringing up the comparison of our paper and this concurrent work (published in May, two weeks before the NeurIPS deadline). Firstly, note that  under the [NeurIPS policy](https://nips.cc/Conferences/2023/PaperInformation/NeurIPS-FAQ#:~:text=Authors%20are%20not%20expected%20to%20compare%20to%20work%20that%20appeared%20only%20a%20month%20or%20two%20before%20the%20deadline.), “Authors are not expected to compare to work that appeared only a month or two before the deadline” including self-published work such as arXiv submissions.\n\nNonetheless, we are happy to highlight the _orthogonality_ between Kiciman et al. (2023) and our work. Their work falls under the umbrella of “causality as knowledge”, whereas our work focuses on “causality as formal reasoning”, following the categorization in our Abstract and Intro, L4-6 and L33-39.\n\nTo illustrate the difference between the two approaches, we use the formulation of “Counterfactual” (Rung 3) as an example:\n\n**Example from our data:**\n\nQ: Effort has a direct effect on college admission. [...] For nonsmokers who are lazy, the probability of college admission is 55%. For nonsmokers who are hard-working, […] If we disregard the mediation effect through effort, would smoking positively affect college admission?\n\nA: Yes. (This answer is obtained by applying the causal inference engine.)\n\n**Example from Kiciman et al (2023)’s data (Table 8):**\n\nQ: A woman does not order Chinese food. \nWhat would have happened if she had ordered Chinese food?\n\nA: The woman would have become less hungry.\n\n**Difference:** Crucially, our answer deductively follows directly by the rules of formal causal inference and information given in the prompt, whereas a commonsense question relies on a common sense understanding of actions and consequences, which is far more susceptible to implicit biases, e.g., if the woman was ordering for someone else, then her hunger level wouldn’t change at all.\n\n> it would be very important to discuss why this specific approach to evaluating causality matters and the ML community should care.\n\nWe share the reviewer’s thoughtfulness concerning the value of our work to the ML community. Here are some of the main aspects to better motivate the value of our contributions:\n\nOur causal inference dataset relies on _formal reasoning_, but not _generating proofs_. Our benchmark occupies a hitherto unaddressed niche of evaluating the formal reasoning abilities of LLMs (much like math word problems or university math exams), but for problems concerning causality. \n\nNow, one might question to what extent it is necessary to evaluate the formal causal reasoning abilities of LLMs. Aside from (1) the academic benefits of understanding the strengths and weaknesses of the current state-of-the-art for LLMs, we believe our work is also a step towards some important practical applications. (2) From analyzing medical studies to risk assessments, formal causal inference problems occur in many high value tasks. Here, as the use of LLMs becomes more common to automate certain tasks, it is important to (3) understand to what extent LLMs can reliably understand and solve such problems. It has been claimed that LLMs understand causality well (e.g., Kiciman et al report very high performance numbers such as 97% and 92%). In contrast, we are the first to show that LLMs have a far longer way to go, reaching only 60+% now on CLadder. It is scientifically worthwhile to study what LLMs can already do and what they cannot.\n\n> it would be good to discuss why a setting in which the model uses software tools for causal inference was not considered. Do such tools exist? \n\nWe concur that it could be interesting to connect LLMs to causal inference tools (e.g., TETRAD etc.). However, we argue that one important step towards augmenting LLMs is understanding to what extent they already solve certain reasoning tasks.\n\n### 2. Technical details\n\n> Table 2: Can you explain why CausalCoT makes performance drop on Rung 3\n\nIn short, Rung 3 has difficulty even with the assistance of CausalCoT because it fails severely at one of the steps. In “5.4 Error Analysis by Subquestions”, we can see that in Table 3, one of the trickest steps for LLMs is Step 2, to identify the query type (e.g., ATE, NDE, …), which only reaches 50% F1 in general, and the worst for Rung 3, with only 42% F1.\n\n> why temperature=0?\n\nFor reproducibility, otherwise LLMs' performance will fluctuate.\n\n> Could you point me to where I can find an interaction with the CausalCoT model in the supplementary materials?\n\nThank you for the good suggestion, we will add the example prompts and more experimental details in the Appendix of the improved version of the paper. We will also show here the prompt of our zero-shot and 10-shot experiments.\n\nAn example of our **zero-shot prompt** is at this [anonymous link](https://anonymous-link.notion.site/CLadder-031085fc56854955bcf3d30d499f0f42#0df909ae3e064b89afc78817332d6798).\n\nAn example of our **10-shot prompt** is at this [anonymous link](https://anonymous-link.notion.site/CLadder-031085fc56854955bcf3d30d499f0f42#bda58d0c708e43c4bb8417bb58b647b8).\n\n"}, {"review_id": "XuNZGgNpjR", "reviewer": "Reviewer_hcc4", "summary": "The author proposes a math-word-style formal causal reasoning dataset, CLADDER, which is the first one that can be used for formally evaluating the causal reasoning/inference ability of LLMs. The authors also proposed a new prompting strategy, CausalCoT, which improves the zero-shot performance of GPT-4 by 2.4% on the proposed dataset.  ", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. Illustrations are helpful.\n\n2. It is the first formal causal inference/reasoning dataset in a natural language, math-word-problem-like format, which provide a nice addition to the current logical/math reasoning datasets. \n\n3. It provides a way to formally evaluate the causal reasoning ability of LLMs for the first time.", "weaknesses": "1. The causal inference questions used in the dataset seem to be a bit simple for evaluating GP3/GPT4 scale LLMs. To my understanding, they only require a one-step application of a causal inference equation to binary variables. It would be great if hard questions with multi-step reasoning could be included.\n\n2. To my understanding, only zero-shot prompting is used to evaluate the performance of LLMs in Table 2, while the common evaluation paradigm for the reasoning abilities of LLMs is through few-shot CoT prompting. It would be helpful if the authors provided another set of results with few-shot prompting. \n\n3. It would be helpful to actually fine-tune a small LLM (e.g., 7B LLaMa) on the proposed dataset since the dataset is large enough to be trained on (10K). In this case, we can have a better understanding of the difficulty of the proposed dataset, and whether there are spurious correlations/shortcut features in the dataset that can be captured by a statistical model.\n\n4. The authors only apply the proposed CausalCoT method to GPT4, and the improvement seems to be small (2.4%). The evaluation of the proposed prompting technique seems to be insufficient. I'm wondering what is the results when CausalCoT is applied to other models and when it is applied with few-shot in-context learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback that “The paper is well-written and easy to follow”, the dataset is valuable as “It is the first formal causal inference/reasoning dataset in a natural language, math-word-problem-like format”, which constitutes “a nice addition to the current logical/math reasoning datasets”, and “It provides a way to formally evaluate the causal reasoning ability of LLMs for the first time.”\n\n\nBelow we address several comments.\n\n> To my understanding, they only require a one-step application of a causal inference equation to binary variables. It would be great if hard questions with multi-step reasoning could be included.\n\nThis comment might stem from the misunderstanding that our questions require one-step reasoning: we explain below that this is not the case, and we will clarify this in a revised version of our paper.\n\nAs shown in Figure 1, correctly solving our causal questions needs five steps. The last step (Step 5) of the solution corresponds to the idea “a one-step application of a causal inference equation[s]”, which might have led to the confusion. For the remaining four steps, they involve different subskills such as (1) parsing the causal graph by causal relation extraction, (2) classifying the query type, (3) deriving the estimand by causal inference (e.g., through backdoor adjustment), and (4) collecting available data by semantic parsing. In particular, to generate ground truth solutions, step (3) is taken care of by the causal inference engine, which in general requires non-trivial reasoning about an underlying causal graph and additional assumptions.\n\nThis 5-step reasoning process makes our CLadder dataset actually very challenging for the current LLMs, with GPT4 reaching only 64.28% accuracy. \n\nAs for the connection with other multi-step reasoning works such as HotpotQA, it can be an interesting extension to do causal inference several times to get the final answer, or do a formal causal inference + commonsense reasoning combination, by letting the LLM to use common knowledge (e.g., age affects experience but not vice versa) to build the causal graph.\n\n> It would be helpful if the authors provided another set of results with few-shot prompting.\n\nOur Section 5.6 has a small-scale experiment to analyze the effect of few-shot/in-context learning. Following your suggestions, we also dedicated a few-shot experiment and reported the performance below:\n\n||0-Shot|Few-Shot (10-Shot)|\n|----|---|---|\n|GPT-3 Non-Instr. (davinci)|47.42|49.83|\n|GPT-3 Instr. (davinci-001)|57.07|57.19|\n|GPT-3 Instr. (davinci-002)|56.24|57.36|\n|GPT-3 Instr. (davinci-003)|62.69|63.50|\n|GPT-3.5|61.71|61.85|\n|GPT-4|64.28|65.43|\n|+ CausalCoT|66.64|70.09|\n\nSeveral key takeaways are (1) few shot brings some improvement, but relatively minor for most models, maybe because the task is really complex. (2) The strongest model, few-shot CausalCoT, reaches 70.09%. The +4 point improvement could be because CausalCoT uses much richer information to complete the task, thus benefiting more learning from examples.\n\n> It would be helpful to actually fine-tune a small LLM (e.g., 7B LLaMa) on the proposed dataset since the dataset is large enough to be trained on (10K). In this case, we can have a better understanding of the difficulty of the proposed dataset, and whether there are spurious correlations/shortcut features in the dataset that can be captured by a statistical model.\n\nWe carefully designed the dataset to avoid trivial shortcut features by, for example, balancing the number of times the correct answer is “yes” vs “no” across all stories and query types. Additionally, to minimize any correlation between for every question we generate both “polarities” (such that the same question can be asked with the correct answer being “yes” or “no”). For example, for a question like “Is ringing alarm less likely than silent alarm overall?” we also generate the positive polarity version “Is ringing alarm more likely than silent alarm overall?” and then randomly select one of the verbalizations with the corresponding answer.\n\nFurthermore, note that our benchmark is a challenge set rather than a train+test set, much like the other challenge sets TruthfulQA (ACL 2022), and MoralExceptQA (NeurIPS 2022). For challenge sets such spurious correlations are generally less of a concern, because no training is involved to let the model learn/overfit to any undesirable “shortcuts.”\n\n"}, {"review_id": "7oOR6yfQlP", "reviewer": "Reviewer_rbbm", "summary": "The paper introduces a large dataset, CLADDER, designed to evaluate the ability of LLMs to perform causal inference. The dataset includes 10K binary questions about variable association and treatment efficacy along with a ground truth explanation. Questions are answered in the context of an hypothetical closed world in which the available measurements allow to unambiguously answer the questions. \n\nMost of the classical causality related NLP tasks focus on commonsense reasoning and domain knowledge, assessing the ability of the model to make deductions based on knowledge of properties of an object or human behavior in everyday life situations. The CLADDER data goes beyond domain knowledge, requiring the model to perform causal inference: inferring the causal graph, describing the treatment effect of interest related to the question (the estimand), choosing an estimator and expressing the numerical results.\n\nTo build the dataset, a causal graph derived from a list of 10 classical structures is sampled. A query type, describing the treatment effect of interest is then derived. The underlying estimand is then derived and an estimate is obtained using the rules of do-calculus and the abduction-action-prediction steps of counterfactual prediction. All those components, causal graph, query and results are then translated into natural language to be ingested by classical models. For each causal graph a set of 2 to 5 stories taken from the literature are used to ensure that the variable names and their relation are not nonsensical. A template is also used to generate step-by-step explanations. \n\nEight models are then evaluated on the task, with an overall accuracy of 66.64 being reached for GPT-4 and CausalCot, a chain-of-thought prompting strategy that relies on 4 sub questions, corresponding to natural reasoning steps, to guide the model. A nonsensical evaluation, accounting for potential data contamination is also performed. The evaluation is further refined for the CausalCot model, by including an ablation study and assessing the effect of in context learning. ", "questions": "Line 156 _\"processes we define specify all necessary information\"_  A bit heavy with two verbs following one after the other. I suggest rephrasing, maybe as: \"the causal processes defined encapsulates all necessary...\"\n\nIn Figure 4, _Translate the question to a formal estimand_, please note that the notation used with a _\\sum_ can be ambiguous as this is only valid for an infinite amount of data (ambiguous outside of the Pearl's community). The estimand doesn't depend on the data observed, the estimator does.", "limitations": "The conclusions made are only valid in a self contained, hypothetical world without any unmentioned factor or causal relationship. A potential negative impact would be for the model to wrongly infer causal effect in the presence of many unmeasured confounders where conditional and marginal are available for few variables. If widely deployed, that kind of model should be able to state the hypothesis it is relying on or state that not enough information is available to draw conclusions on efficacy.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Classical causality related NLP tasks mostly focused on commonsense reasoning and domain knowledge. The paper goes beyond that, introducing a new original task, CLADDER, which theoretically requires the model to infer the causal graph, describe the treatment effect of interest, choose an estimator and provide an estimate. The paper nicely builds on the framework proposed by Pearl with a dataset of 10K questions covering a large variety of causal graph structures and estimands. Additionally, the explanation generated together with the questions allows further assessment of the reasoning ability of the model. \n\nThe evaluation of state of the art models on the task is well conducted with a particular methodology to isolate the effect of data contamination. Additionally, a methodology to boost model's performance using chain-of-thought prompting is proposed. If the improvement is marginal, it allows further investigation on the ability of the model to perform the natural reasoning steps to complete the task. ", "weaknesses": "To better understand how the information provided is used, it could be interesting to mention a new attribute in the text, completely unrelated for which you provide the marginal probability. Given the low performance, adding noise to the available information is not the priority but it could be nice for future work. \n\nOn top of the ablation study, it would be interesting to know how the CausalCoT model performs if it was given the question for step 1 along with the answer and then only the question for step 2 to 4. Then doing the same but providing the answer for step 1 and 2. And finally the same for step 1, 2 and 3. It would provide insights on the ability of the model to perform the task if it received partial answers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing very detailed comments and an elaborate summary of the work. We appreciate the reviewer’s comments that “The paper nicely builds on the framework proposed by Pearl with a dataset of 10K questions covering a large variety of causal graph structures and estimands.” and “The evaluation of state of the art models on the task is well conducted with a particular methodology to isolate the effect of data contamination.”\n\nHere we address some remaining questions and comments of the reviewer.\n\n> To better understand how the information provided is used, it could be interesting to mention a new attribute in the text, completely unrelated for which you provide the marginal probability. Given the low performance, adding noise to the available information is not the priority but it could be nice for future work.\n\nThis is a perfect suggestion! Actually, we are working on a follow-up paper using this idea to see what perturbation in the prompt will distract the model. And we agree, these explorations (in lines of adversarial attack, or LLM performance analysis) are very interesting to work on.\n\n> On top of the ablation study, it would be interesting to know how the CausalCoT model performs if it was given the question for step 1 along with the answer and then only the question for step 2 to 4. Then doing the same but providing the answer for step 1 and 2. And finally the same for step 1, 2 and 3. It would provide insights on the ability of the model to perform the task if it received partial answers.\n\nThis is a nice thought. We conduct the aforementioned experiments, and report the results in the following:\n| Providing the answer of | Asking the question of | Performance |\n| ----------------------- | ---------------------- | ----------- |\n| None                    | Step 1, 2, 3, 4        | 66.64       |\n| Step 1                  | Step 2, 3, 4           | 68.10       |\n| Step 1, 2               | Step 3, 4              | 71.03       |\n| Step 1, 2, 3            | Step 4                 | 72.49       |\n| Step 1, 2, 3, 4         | None (Just final Q)    | 74.51       |\n\nThere is a very nice monotonically increasing trend as we provide more partial answers in the CausalCoT process. We are happy to put this result table also in the camera ready version of the paper.\n\n\n> In Figure 4, Translate the question to a formal estimand, please note that the notation used with a \\sum can be ambiguous as this is only valid for an infinite amount of data (ambiguous outside of the Pearl's community).\n\nIn our work, the word “data” is used to refer to population-level information, as opposed to finite samples: this is indeed unusual, although consistent with how it is often used by Judea Pearl et al. to stress that certain key problems of causal inference are orthogonal to estimation from finite samples. We will add a footnote and a clarification in the image caption to stress this.\n\n> Line 156 \"processes we define specify all necessary information\" A bit heavy with two verbs following one after the other. I suggest rephrasing, maybe as: \"the causal processes defined encapsulates all necessary...\"\n\nThank you for the suggestion. We will improve the presentation of the camera-ready version of our paper accordingly. Thank you for the two notes!\n"}, {"review_id": "tRnPKjcoFh", "reviewer": "Reviewer_YWrR", "summary": "The paper collects a new dataset, named Cladder, that aims at evaluating the causal reasoning abilities of language models. The datasets contain 10,000 questions asking about boolean questions that require intensive causal reasoning.\nThe dataset is constructed based on the idea of a causal inference engine, which covers three rungs of causal inference. Specifically, the authors first sample formal representations of causal inference problems by sampling causal graphs as well as query types. Next, the formal representations are verbalized into natural language problems, where the variables in the formal representations are named with concepts in the stories collected from commonly cited causal inference books. Finally, the representations are verbalized into natural language problems using fixed templates.\n\nUsing the collected datasets, the paper evaluates a line of language models’ causal reasoning abilities. Experiments results suggest that these problems are still hard for language models in the zero-shot setting (if I understand correctly). In addition, the paper proposes CausalCoT which lets LMs explicitly state causal problems and then solve the problem. The results suggest CausalCoT is better than the baseline.\n", "questions": "Are the baselines all in zero-shot setting?\n\nIn addition, it would be helpful if the authors can include some full prompt examples in the appendix.\n\n", "limitations": "The paper discusses the limitations in Section 7.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper works on an interesting problem, causal inference, and collects a useful dataset consisting of 1,0000 examples.\n\nThe data creation process is well-designed. The dataset covers multiple rungs and multiple causal graphs. In addition, the alignment also includes anti-commonsense and nonsensical settings, which is essential for controlling possible effects of memorization in LLMs.  \n\nThe paper includes additional error analysis highlighting the LM’s capabilities in performing different steps in the process of solving causal inference problems.\n\nThe paper is well-written and easy to follow.\n", "weaknesses": "1: The tested baselines (in Table 2) need clarification. Also, the baselines might be somewhat weak.\n\nAre the methods tested in Table 2 implemented in zero-shot setting? (I am guessing that based on section 5.6). I wouldn’t be surprised if these language models utterly fail in a zero-shot setting.\n\nWhile the paper tries to frame that providing in-context examples is orthogonal to the paper (section 5.6). I believe it is necessary to benchmark what much better systems can do (e.g., few-shot CoT prompting or few-shot CausalCoT prompting) to understand the capabilities of LMs as well as the potential headroom.\n\n2: The language is synthetic.\n\nAs suggested in 3.2, the verbalization of the formal causal problem is rule-based, which could result in synthetic and formulaic language. The paper does not provide much evaluation of the language side of the datasets in the main body.\n\n3: No human performance\n\nThe paper does not provide human performance for this dataset. Although many problems are adapted from books, the templated-based verbalization as well as the nonsensical setting could make some problems tricky even for humans. Thus, human evaluation is still valuable. Also, it can help verify the quality of the datasets.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the overall positive comments about the meaningfulness of the problem (“The paper works on an interesting problem”), dataset contribution (“a useful dataset consisting of 1,0000 examples”), “well-designed” “data creation process” that “covers multiple rungs and multiple causal graphs” and “includes anti-commonsense and nonsensical settings”, comprehensive experiments with “additional error analysis”, and “well-written and easy to follow” writing.\n\nIn the following, we address the three major comments raised by the reviewer.\n\n### 1. Additional Few-Shot Experiments\n\n> Are the baselines all in zero-shot setting?\n\nYes, our main results (Section 5.2) are zero-shot, and our Section 5.6 analyzes the few-shot/in-context learning. Following your suggestions, we also dedicated a few-shot experiment and reported the performance below:\n\n||0-Shot|Few-Shot (10-Shot)|\n|----|---|---|\n|GPT-3 Non-Instr. (davinci)|47.42|49.83|\n|GPT-3 Instr. (davinci-001)|57.07|57.19|\n|GPT-3 Instr. (davinci-002)|56.24|57.36|\n|GPT-3 Instr. (davinci-003)|62.69|63.50|\n|GPT-3.5|61.71|61.85|\n|GPT-4|64.28|65.43|\n|+CausalCoT|66.64|70.09|\n\nSeveral key takeaways are (1) few shot brings some improvement, but relatively minor for most models, maybe because the task is really complex. (2) The strongest model, few-shot CausalCoT, reaches 70.09%. The 4 point improvement could be because CausalCoT uses much richer information to complete the task, thus benefiting more learning from examples.\n\n> I believe it is necessary to benchmark what much better systems can do (e.g., few-shot CausalCoT) to understand the capabilities of LMs as well as the potential headroom.\n\nThank you for your advice. We implemented the suggestion, and find that few-shot CausalCoT reaches 70.09%, +4 point increase over the 0-shot CoT. We hope this result helps to better understand the capabilities of LMs as well as the potential headroom.\n\n### 2. Evaluation of the Natural Language\n\n> The paper does not provide much evaluation of the language side of the datasets\n\nIn the composition of this work, we followed the standards of dataset creation and experiments in previous work on formal tasks for LLMs, such as this recent [ICLR 2023 paper](https://openreview.net/forum?id=qFVVBzXxR2V) on formal reasoning in logic. Similarly, their language is also formulaic. Nonetheless, we are happy to add evaluations of the quality of the natural language in our data below.\n\n**Evaluation:** For our benchmark dataset, we think the set of criteria to judge it is: (1) grammaticality, (2) human readability, (3) formal correctness, and (4) naturalness, whose performance we report below.\n\n1. **Grammaticality:** We run the grammar check on our dataset by the LanguageTool package, and get on average 1.26 grammatical errors per 100 words (i.e., 98.74% correct), which shows that most of our language follows English grammar.\n2. **Human readability** (i.e., how comprehensible/intelligible are the questions to, e.g., a student who has taken a causality course): 96%. We obtain this score by sampling 50 random questions, and let a grad student annotator go through the questions to mark the questions that they cannot understand.\n3. **Formal correctness:** 100%, because of our systematic generation process together with the formal solutions.\n4. **Naturalness/Perplexity:** In addition, we also produce how natural the questions sound by the automatic metric of perplexity (since human evaluation might be subjective). We use gpt2 to calculate the perplexity scores, and obtained a score of 21.17 (the lower, the more natural the language sounds to GPT). For comparison, the perplexity of the MATH Dataset (NeurIPS 2021, by [Hendrycks et al.](https://arxiv.org/abs/2103.03874)) is 57.45, which means that the language in our data is more natural-sounding. \n\nFor a more intuitive feeling, please feel free to check the examples of our dataset in Appendix Tables 5-12.\n\n**Camera Ready:** We are happy to add the above evaluation scores to the camera-ready version of our dataset.\n\n### 3. Reporting the Human Performance\n\n> Some problems could be tricky even for humans. Thus, human evaluation is still valuable. \n\nWe agree with the reviewer’s intuition that the problems can be challenging for humans too. In addition to the readability score, we also conducted a small experiment getting our coauthors to work through 50 random samples. In these preliminary results, we obtained a correctness score of around 80% as the human performance score. We plan to extend this to a larger cohort for the camera-ready version.\n\n**Camera Ready Plan:** We plan to use the human evaluation score in the following way:\n- The score indicates the human performance, against which we can illustrate whether LLMs surpass it or not.\n- We want to emphasize that the human performance will only be used as a baseline, but not a judgment for the dataset quality. Imagine a university exam that is valid and good-quality, but challenging for the students. Hence, **a good-quality dataset** (i.e., formally correct and readable) **can also be challenging for humans**, and requiring specific skills to solve. That’s another motivation for using models to automate these challenging tasks for humans.\n- Another disclaimer is that, similar to any cognitive science studies, the human performance will be a function of the subjects’ background, including their previous education background, familiarity with causal formulations, and carefulness when solving these problems. Our reported performance is based on a cohort of researchers familiar with the study and took the questions with caution.\n\n> it would be helpful to include some full prompt examples in the appendix.\n\nSure, we can provide the full prompts in our camera ready version. We composed several examples of our prompts at this anonymous link: https://anonymous-link.notion.site/CLadder-031085fc56854955bcf3d30d499f0f42"}]

Author rebuttal:
Firstly, we would like to thank all reviewers for a wealth of feedback. Four out of five reviewers recommended acceptance, and we are encouraged by the large number of thoughtful comments and suggestions, which demonstrate not only that all reviewers largely understood our focus, but also that they expressed interest in our work.

We appreciate the acknowledgement of our work’s contributions in the following ways:
1. **Relevance:** This work “focuses on an important problem” (bMQ8) and “provides a way to formally evaluate the causal reasoning ability of LLMs for the first time” (hcc4).
1. **Dataset contribution:** The created benchmark is “the first formal causal inference/reasoning dataset” (hcc4), thus “a valuable evaluation resource” (bMQ8) and “The paper nicely builds on the framework proposed by Pearl with a dataset of 10K questions covering a large variety of causal graph structures and estimands” (rbbm).
1. **Method contribution:** Our CausalCoT is “a novel structure that is based on causal reasoning steps” (bMQ8) and “which lets LMs explicitly state causal problems and then solve [them]” (YWrR).
1. **Comprehensive evaluation results:** “The evaluation of state of the art models on the task is well conducted with a particular methodology to isolate the effect of data contamination.” (rbbm) and addressing "effects of memorization in LLMs" (YWrR).

A shared question across most reviewers is more clarification of whether we used a zero-shot or few-shot evaluation. We clarified in our response that our main experiments are zero-shot to simplify analysis, while we include some few-shot results in the paper. Upon the request of the reviewers, we have also conducted several new experimentsto report the few-shot performance of all models. Additionally, some reviewers asked about the quality of the dataset, for which we conduct new human evaluations and automatic evaluations to gain better insight into the quality of our data. Finally, we also supplemented our rebuttal with ablation studies and various experiments requested by the reviewers (discussed in the corresponding rebuttals).

We look forward to the discussions to make any further clarifications to help the reviewers reach a consensus on our submission's value to the community.


Meta-review: There is consensus between reviewers that this paper presents a solid contribution in the form of a causal reasoning benchmark that can be of large importance to the community.  There is excellent discussion between authors and reviewers regarding some subset of topics presented in the paper.  Given the excitement about the paper, I suggest that we accept it to the conference.
Decision: Accept (poster)