SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rung-2 Queries: ATE and Adjustment Set. For ATE questions, we ask whether the treatment  $(X = 1)$  increases or decreases the likelihood of the effect variable  $Y = y$ . For adjustment set questions, we ask whether a set of variables should be adjusted for when estimating the causal effect between treatment and effect. By adjusting, we aim to block the non-causal paths from the treatments to effect, and hence eliminate spurious correlation. For example, to query whether the set gender is an adjustment set for the effect of a treatment on recovery, we ask "To estimate the effect of the treatment on recovery, should we directly look at how the treatment correlates with recovery, or should we look at gender-specific correlation?" In the collider bias questions, similarly to the explaining away questions, we condition on a collider variable and ask about how an intervention on one of the parents (treatment  $X$ ) affects the other parent (outcome  $Y$ ). However since by construction  $X$  and  $Y$  do not have common causes, the answer to this question is always "no".
Rung-3 Queries: Counterfactual Probability, ATT, NDE, and NIE. For counterfactual probability, we ask about what would have been the likelihood of  $Y = y$ , if the treatment variable  $X$  had been  $x$ , given sufficient evidence  $e$  such that the query is identifiable. For ATT, we ask how the likelihood of  $Y = y$  would change for those who received treatment ( $X = 1$ ) if there had been no treatment ( $X = 0$ ). For NDE, we ask whether the  $X = 1$  directly increases or decreases the likelihood of the  $Y = y$ , not through any mediators. For NIE, we ask whether the treatment (setting  $X = 1$ ) increases or decreases the likelihood of  $Y = y$  through mediators, not directly.
A.3 Collection of Causal Graphs
We include all the ten causal graphs with treatment-effect pairs (CGTEs) in Figure 6.
Note that one causal graph can have several different CGTEs, such as the confounding structure, which has three CGTEs: confounding, mediation, and collision in the triangle form. To generate all the causal graphs and CGTEs here, we iterate all commonly used ones within four nodes in the CI books, and omit CGTEs whose solution by CI methods trivially resembles existing ones.
A.4 Data Coverage
Starting from the full set of 12 distinct causal graphs and 10 query types, there are a few combinations that must be omitted as the ground truth answer would be trivial or ill-defined. For example, in the "Immorality" graph, the treatment "X" and outcome "Y" are by construction statistically independent, so there correlation is necessarily 0. Similarly, there are several graphs where certain causal queries are ill-defined or don't make sense to ask. Specifically:
1. For the Natural Direct Effect, we only include questions on the "IV", "Arrowhead", "Confounding", "Mediation" and "DiamondCut" graphs.
2. For the Natural Indirect Effect, we only include questions on the "Mediation", "Frontdoor", "Arrowhead", "Diamond" and "Chain" graphs.
3. For the Collider Bias and Explaining Away effect, we only include questions on the "Collision" graph.
4. For the Average Treatment Effect, we include questions on all graphs except "Collision".
5. For the (deterministic) Counterfactuals, we include questions on all graphs except "Collision".
6. For the Average Treatment Effect on the Treated (ATT), we include questions on all graphs except "Collision" and "IV".
The "balanced" benchmark (main benchmark in v1.5), containing 10,112 questions split between all stories, graphs, query types, and commonsensicalness, is balanced such that there are roughly the same number of questions for each distinct story-graph-query combination (ranging from 50-100 per combination) across the different variants: commonsense, anticommonsense, and nonsense. Furthermore, we balance the distribution of correct answers so that there are the same number of "yes"s and "no"s.
The "aggregate" variant (main benchmark in v1.0) contains 10,560 questions and is primarily balanced across all stories. However since the number of stories for each variant (commonsense, anticommonsense, and nonsense) varies significantly, the results in an unbalanced benchmark in terms of sensicalness.
A.5 Query Form and Text Templates
We provide in Table 4 the text templates we use for each query type.
Query Type	Symbolic Expre-sion	Natural Language Question Template
Rung 1: Association
Marg. Prob.	P(Y)	Is the overall likelihood of {v_noun(X = 1)} greater than chance?
Cond. Prob.	P(Y|X)	Is the chance of {v_noun(Y = 1)} larger when observing {v_noun(X = 1)}?
Rung 2: Intervention
ATE	E[Y|do(X = 1)]- E[Y|do(X = 0)]	Will {v_noun(X = 1)} increase the chance of {v_noun(Y = 1)}?
Adjust. Set	If S opens a back-door path	To understand how {v_overall(X)} affects {v_overall(Y = 1)}, should we look directly at how {v_overall(X)} correlates with {v_overall(Y)} in general, or this correlation case by case accord-ing to {v_overall(S)}?
Rung 3: Counterfactuals
Counterf. Prob.	P(Yx = y)	Can we infer that {v_sent(Y = 1)} had it been that {v_cond(X = 1)} instead of X=0?
ATT	E[Y1 - Y0|X = 1]	For {v_attr(X = 1)}, would it be more likely to see {v_noun(Y = 1)} {v_cond(X = 0)}?
NDE	E[Y1,M0 - Y1,M0]	If we disregard the mediation effect through {v_overall(Y = 1)}, would {v_noun(X = 1)} still positively affect {v_noun(Y = 1)}?
NIE	E[Y0,M1 - Y0,M0]	Does {v_overall(X)} affect {v_overall(Y)} through {v_overall(OtherVars)}?
Table 4: Example natural language templates for each query type.
A.6 Nonsensical Stories
To come up with a collection of nonsensical variable names, we use GPT-4 to generate some meaningless words. Specifically, we use the prompt: "Create 100 non-existent words that are short, i.e., within 5-character." with temperature=0 with the OpenAI interface. The collection of nonsensical words we later use as variable names are as follows: ziblo, truq, fyze, glimx, jorv, wexi, snov, yupt, kraz, qixy, vubr, chiz, pliv, moxa, fygo, rukz, tasp, xevo, jyke, wbl, zorf, quzy, nyrp, gwex, smez, vytz, hupx, cwoj, lirf, ovka, pexu, yigz, twaz, kwox, zuph, fraq, jyxo, swoy, uvzi, nekl, gyzp, rixq, vwem, xyfu, blyz, qwip, zkip, tijv, yomx, hwaz, czix, plof, muvy, fyquo, rujz, tamb, xevi, jyka, wibm, zorx, quzw, nyro, gwet, smeu, vyta, hupz, cwoi, lirg, ovki, pexy, yigw, twac, kwoz, zupj, fraq, jyxi, swoq, uvzo, nekm, gyzl, rixw, vwen, xyfo, blyx, qwiu, zeky, tijw, yomz, hwax, czir, ploz, muqv, fyqi, rujx, tasn, xevu, jyko, wbp, zory, and quzt.
A.7 Anti-Commonsensical Stories
For the anti-commonsensical stories, we randomly do one of the actions:
1. Replace the effect variable  $Y$  with an attribute that would not be an effect variable in any of the stories. Such replacement variables include: "lip thickness", "earthquakes", "lactose intolerance", "rainfall", "is allergic to peanuts", "brown eyes", "curly hair", "black hair", "foot size", "freckles"
2. Create an irrelevant treatment variable  $X$  that does not play a causal role in any of our commonsensical stories. Such as: "can swim", "is religious", "has a brother", "has visited England", "likes spicy food", "is vegetarian", "speaks english", "drinks coffee", "plays card games", "listens to jazz", "solar eclipse", "has a sister", "full moon"
To transform a commonsensical story into an anti-commonsensical story, we apply one of these replacements sampled uniformly, resulting in stories such as:
- Ability to swim has a direct effect on studying habit and exam score. Studying habit has a direct effect on exam score.
- Gender has a direct effect on department competitiveness and peanut allergy. Department competitiveness has a direct effect on peanut allergy.
- Liking spicy food has a direct effect on relationship status. Appearance has a direct effect on relationship status.
- Playing card games has a direct effect on diabetes and lifespan. Smoking has a direct effect on diabetes and lifespan. Diabetes has a direct effect on lifespan. Smoking is unobserved.
For a full list of the replacements and how the replacements are made, check out the code.
A.8 Explanation Template
Step ① Extract the causal graph: The causal graph expressed in the context is: "G".
Step ② Identify the query type: The query type of the above question is "query_type".
Step ③ Formulate the query to its symbolic form: The formal form of the query is "symbolic_expression".
Step ④ Collect all the available data: The available data are: "d".
Step ⑤ Derive the estimand: Based on the graph structure and causal query, the question can be simplified into estimand "est".
Step ⑥ Solve for the estimand: Plug in the available data "d" into "est".
est  $(\pmb{d})$
$\approx$  float(a)
Since the estimate for the estimand is  $\mathrm{float}(a)$ , the overall answer to the question is  $\mathrm{bool}(a)$ .
B.1 CAUSALCOT Prompt
Q: [question from the dataset]
Guidance: Address the question by following the steps below:
Step 1) Extract the causal graph: Identify the causal graph that depicts the relationships in the scenario. The diagram should simply consist of edges denoted in "var1 -> var2" format, separated by commas.
Step 2) Determine the query type: Identify the type of query implied by the main question. Choices include "marginal probability", "conditional probability", "explaining away effect", "backdoor adjustment set", "average treatment effect", "collider bias", "normal counterfactual question", "average treatment effect on treated", "natural direct effect" or "natural indirect effect". Your answer should only be a term from the list above, enclosed in quotation marks.
Step 3) Formalize the query: Translate the query into its formal mathematical expression based on its type, utilizing the "do(\cdot)" notation or counterfactual notations as needed.
Step 4) Gather all relevant data: Extract all the available data. Your answer should contain nothing but marginal probabilities and conditional probabilities in the form "P(...) = ..." or "P(...|...) = ..., each probability being separated by a semicolon. Stick to the previously mentioned denotations for the variables.
Step 5) Deduce the estimand using causal inference: Given all the information above, deduce the estimand using skills such as do-calculus, counterfactual prediction, and the basics of probabilities. Answer step by step.
Step 6) Calculate the estimand: Insert the relevant data in Step 4 into the estimand, perform basic arithmetic calculations, and derive the final answer. There is an identifiable answer. Answer step by step.
A: [LLM previous response]
Q: Based on all the reasoning above, output one word to answer the initial question with just "Yes" or "No".
A: [LLM final answer]
C Additional Technical Background for Preliminaries
C.1 Graphical Models
We adopt the causal inference framework described in [61]. A causal graph  $G \coloneqq (V, E)$  consists of a set of  $k$  vertices  $V: \{V_1, \ldots, V_k\}$  and directed edges  $E \coloneqq \{e_{ij}\}$ , where the existence of each  $e_{ij}$  means that there is a direct causation from  $V_i$  to  $V_j$ , also denoted as  $V_i \to V_j$ . We also introduce some notations to describe the relative positions among the nodes. Following a standard assumption in causality (but see, e.g., [6]), we will assume that  $\mathcal{G}$  is a direct acyclic graph (DAG), where we denote the parents of a node  $V_i$  as  $\mathbf{PA}(V_i) \coloneqq \{V_j | e_{ij} \in E\}$ . We denote descendants  $\mathbf{DE}(V_i) \coloneqq \{V_j | V_j \to \dots \to V_i \in E\}$  of a node  $V_i$  as all the nodes that have at least one direct path leading to a node. We call a node  $V_k$  as a confounder (i.e., common cause) of the other two nodes  $V_i$  and  $V_j$  if  $e_{ki}, e_{kj} \in E$ ; a collider (i.e., common effect) if  $e_{ik}, e_{jk} \in E$ ; and a mediator if  $e_{ik}, e_{kj} \in E$ .
Among all the variables in  $V$ , we use  $X$  and  $Y$  to denote two special variables, the treatment and effect, respectively.
C.2 Illustration of the Three Rungs of the Causal Ladder
In Figure 7, we illustrate the difference among the three rungs by enumerating what actions are performed on the variables other than target variables  $X$  and  $Y$ .
Rung 1. Association
Rung 2. Intervention
Rung 3. Counterfactuals
Figure 7: The Causal Ladder consists of three rungs: association, intervention and counterfactuals. We color in blue the treatment  $X$  and effect  $Y$ , as well as the actions on  $X$ . We color in orange words about how to get the estimand, and we use the orange circle to include all the non-descendants of  $X$ .
C.3 Causal Inference Methods
We introduce do-calculus which can downgrade the Rung-2 queries to Rung-1 quantities when it is applicable, and counterfactual predictions which downgrade the Rung-3 queries.
C.3.1 Do-Calculus
Do-Operator as a Notation As mentioned in Rung 2, the do-operator is a convenient notation to represent an intervention on a variable. For example,  $\mathrm{do}(X = x)$  sets the value of variable  $X$  to  $x$ .
Three Inference Rules for Climbing the Ladder Do-calculus is a set of rules that allows us to answer higher-rung questions using lower-rung quantities, such as probability distributions of Rung 1. Given a causal graphical model with and four disjoint sets of variables  $X$ ,  $Y$ ,  $Z$ , and  $W$ , and a joint probability distribution that is Markov and faithful to the graph, do-calculus contains the following three rules:
Rule 1 (Insertion/deletion of observations):
$$
P (Y | \operatorname {d o} (X), Z, W) = P (Y | \operatorname {d o} (X), W), \tag {1}
$$
if  $Y$  and  $Z$  are d-separated by  $X \cup W$  in  $G^*$ , the graph obtained from  $\mathcal{G}$  by removing all arrows pointing into variables in  $X$ .
Rule 2 (Action/observation exchange):
$$
P (Y | \mathrm {d o} (X), \mathrm {d o} (Z), W) = P (Y | \mathrm {d o} (X), Z, W), \tag {2}
$$
if  $Y$  and  $Z$  are d-separated by  $X \cup W$  in  $G^{\dagger}$ , the graph obtained from  $\mathcal{G}$  by removing all arrows pointing into variables in  $X$  and all arrows pointing out of variables in  $Z$ .
Rule 3 (Insertion/deletion of actions):
$$
P (Y | \operatorname {d o} (X), \operatorname {d o} (Z), W) = P (Y | \operatorname {d o} (X), W), \tag {3}
$$
if  $Y$  and  $Z$  are d-separated by  $X \cup W$  in  $G^{\ddagger}$ , the graph obtained from  $\mathcal{G}$  by first removing all arrows pointing into variables in  $X$  (thus creating  $G^{*}$ ) and then removing all arrows pointing into variables in  $Z$  that are not ancestors of any variable in  $W$  in  $G^{*}$ .
These rules are sound and complete [85]. Namely, iff we have all the terms on the right hand side, then the causal term on the left hand side is identifiable.
Example Application of Do-Calculus Taking the example in Figure 2,  $g_{1}$  maps the query type ATE to its symbolic expression  $\mathbb{E}[Y|\mathrm{do}(X = 1)] - \mathbb{E}[Y|\mathrm{do}(X = 0)]$ .
Next,  $g_{2}$  further simplifies the estimand given the confounding graph, as in the flow chart in the middle of Figure 2:
$$
\begin{array}{l} \operatorname {A T E} := \mathbb {E} [ Y | \mathrm {d o} (X = 1) ] - \mathbb {E} [ Y | \mathrm {d o} (X = 0) ] (4) \\ = \sum_ {z} P (Z = z) \left[ \mathbb {E} (Y | X = 1, Z = z) - \mathbb {E} (Y | X = 0, Z = z) \right], (5) \\ \end{array}
$$
which which resolves all the  $\mathrm{do}(\cdot)$  terms to probability terms. This example shows the famous backdoor adjustment in do-calculus [59].
C.3.2 Three Steps for Counterfactual Prediction
Given a SCM  $M$ , distribution on the exogenous variables  $P(u)$ , and evidence  $e$  from the model  $\langle M, P(u) \rangle$ , the probability of the counterfactual "if  $X$  had been  $x$  then  $Y$  would have been  $y$ , given we observed  $e$ ," denoted  $P(Y_x = y|e)$ , can be evaluated using the following three steps [67]:
Abduction: Update the probability distribution  $P(u)$  by the evidence  $e$  to obtain  $P(u|e)$ .
Action: Modify  $M$  by the action  $do(X = x)$ , i.e. replace  $X$  with  $X = x$  in the structural equations, to obtain the modified SCM  $M_{x}$
Prediction: Use the modified model  $\langle M_x, P(u|e) \rangle$ , to compute the probability of  $Y = y$ .
D Previous Results on CLADDER v1.0
D.1 Dataset Statistics for v1.0
Total	Rung 1	Rung 2	Rung 3
Size
# Samples	10,560	3,288	3,288	3,984
Question
# Sentences/Sample	6.85	6.00	7.00	7.25
# Words/Sample	94.47	76.41	96.84	103.42
# Nodes/Graph	3.54	3.54	3.55	3.54
# Edges/Graph	3.44	3.41	3.43	3.46
Answer
Positive Class (%)	50	50	50	50
Explanations
# Sentences/Sample	13.11	12.04	13.76	13.83
# Words/Sample	146.82	141.88	147.88	151.30
Table 5: Statistics of our CLADDER data v1.0.
Figure 8: Distributions of query types in our dataset v1.0.
Our data-generating procedure has the potential to algorithmically generate very large amounts of questions. In practice, we pick a dataset size that is large enough to be representative, and at the same time not too large to be problematic given the expensive inference costs of LLMs. We therefore set our dataset size to be 10K. We report the statistics of our dataset in Table 5.
The dataset roughly balanced across the query types, graph structures, stories, and ground-truth answers (as seen in Figure 8). Note that there are some slight adjustments such as more samples for ATE because it allows us to test various techniques, including backdoor and front door adjustments. More details on our design choices can be found in Appendix A.4.
D.2 Main Results on v1.0
Overall Acc.	Acc. by Rung	Acc. by Empirical Alignment
1	2	3	Anti-C.	Nonsens.	Comm.
Random	49.27	50.28	48.40	49.12	49.69	49.01	49.12
LLaMa	45.22	63.33	31.10	41.45	45.31	45.21	45.12
Alpaca	45.54	63.33	31.57	41.91	45.94	45.21	45.49
GPT-3 Non-Instr. (davinci)	47.42	63.88	32.99	44.89	47.0	48.28	46.97
GPT-3 Instr. (text-davinci-001)	57.07	63.95	63.63	48.04	59.12	57.81	54.28
GPT-3 Instr. (text-davinci-002)	56.24	46.03	69.55	55.04	54.75	59.65	54.31
GPT-3 Instr. (text-davinci-003)	62.69	58.0	80.83	54.52	63.93	62.09	62.05
GPT-3.5 (queried in May 2023)	61.71	65.12	69.9	54.11	65.43	55.15	64.55
GPT-4 (queried in May 2023)	64.28	53.94	81.87	63.11	65.75	60.87	66.21
+ CAUSALCOT	66.64	61.67	86.13	58.23	69.32	63.02	67.60
Table 6: Performance of all models on our CLADDER dataset v1.0. We report the overall accuracy (Acc.), and also fine-grained accuracy by rung and by empirical alignment.
We compare the performance of all models in Table 6. First, we can see that the causal reasoning task in CLADDER is in general very challenging for all models. And models such as the earlier, non-instruction-tuned GPT-3 and both LLaMa and Alpaca are no better than random performance. With instruction-tuning, models start to show some improvement. And amongst all, our CAUSALCOT achieves the highest performance of  $66.64\%$ , which is 2.36 points better than vanilla GPT-4.
Moreover, from the accuracy by empirical alignment level in Table 6, we can see that the original GPT-4 model performs the best on commonsensical data, but 5.34 points worse on nonsensical data. However, our CAUSALCOT enhances the reasoning ability across all levels, with substantial improvement on anti-commonsensical data and nonsensical data, indicating that CAUSALCOT is particularly beneficial on unseen data.
D.3 Ablation Study on v1.0
We conduct an ablation study for our multi-step CAUSALCOT. We ablate each of the four subquestions, and observe in Table 7 that classifying the query type and formalizing it has the most effect on the model's performance, which might be because that they are the crucial formalization step in order to do the causal inference correctly. Meanwhile, removing Steps ① and ④, which are mostly about parsing the prompt correctly, have the least impact on performance.
Acc.
CAUSALCOT	66.64
w/o Step ①	64.54
w/o Step ②	63.74
w/o Step ③	63.43
w/o Step ④	64.47
Table 7: Ablation study.
E More Experiments
E.1 Details of Our Error Analysis
For Step 2 about the query type prediction, we report the overall F1 classification score, and also F1 by rungs. For the rest of the steps, we manually annotate the correctness of 100 samples of CAUSALCOT. We report the correctness of est by accuracy, and the correctness of the predicted set of available data by taking the F1 with the ground-truth  $d$ . For Step 5, we report the accuracy of whether the model simplifies the estimand correctly to  $\mathrm{est}^{\prime}$  using causal inference, and also arithmetic correctness (Arith.).
E.2 ROSCOE Evaluation
We employed the ROSCOE suite of evaluation metrics on step-by-step text reasoning, as introduced by [25], to automate the evaluation of the outputs from CAUSALCOT on 2,000 randomly sampled questions from our dataset. Differing from conventional metrics, ROSCOE is specifically designed to scrutinize the quality of large language model outputs, focusing on aspects such as semantic consistency, logicality, informativeness, fluency, and factuality, all evaluated within the context of step-by-step reasoning, rather than solely the final response. This allows for a more objective and comprehensive assessment of a model's output, greatly aiding in the verification of its interpretability. The results of this evaluation can be found in Table 8 and Figure 9. We consider the model's performance as unsatisfying if it falls out of the top quantile, namely receiving a score  $s \in [0,1]$  smaller than 0.25 when the score should be minimized, or greater than 0.75 when it should be maximized.
We can see in the plot that the good-performing aspects are faithfulness to the original question, reasoning alignment with the ground truth, and absence of external hallucinations, which are consistently within the top quantile. This suggests that the model carries out accurate reasoning within the constraints of the fictitious world introduced in each question.
However, there are some performance dips in redundancy, perplexity chain, and missing step metrics. The first two could potentially be attributed to complex elements such as graph notation, while the relatively lower "missing step" score warrants further investigation. Despite these observations, this analysis largely aligns with our qualitative understanding of the models' good response ability in answering causal questions in our dataset.
Mean	Std	Min	25%	50%	75%	Max
Faithfulness	0.89	0.02	0.83	0.88	0.89	0.90	0.93
Informativeness Step	0.88	0.01	0.83	0.87	0.88	0.89	0.92
Informativeness Chain	0.88	0.03	0.76	0.87	0.89	0.90	0.96
Faithfulness Word	0.95	0.01	0.92	0.94	0.95	0.96	0.97
Repetition Word	0.02	0.02	-0.00	0.00	0.02	0.04	0.05
Repetition Step	0.02	0.01	-0.00	0.00	0.01	0.03	0.06
Reasoning Alignment	0.92	0.01	0.86	0.91	0.92	0.93	0.95
External Hallucination	0.97	0.02	0.84	0.96	0.97	0.98	0.99
Redundancy	0.80	0.05	0.56	0.77	0.80	0.83	0.92
Common Sense Error	0.95	0.01	0.86	0.94	0.95	0.96	0.98
Missing Step	0.78	0.03	0.58	0.76	0.78	0.80	0.88
Semantic Coverage Step	0.99	0.01	0.95	0.98	0.99	0.99	1.00
Semantic Coverage Chain	0.98	0.01	0.93	0.98	0.98	0.99	0.99
Discourse Representation	0.06	0.13	0.00	0.01	0.01	0.05	0.67
Coherence Step Vs Step	0.14	0.27	0.00	0.00	0.01	0.07	0.94
Perplexity Step	0.02	0.01	0.00	0.02	0.02	0.03	0.07
Perplexity Chain	0.17	0.07	0.05	0.11	0.17	0.23	0.42
Perplexity Step Max	0.00	0.00	0.00	0.00	0.00	0.01	0.02
Grammar Step	0.93	0.04	0.77	0.90	0.93	0.96	0.99
Grammar Step Max	0.53	0.35	0.02	0.12	0.65	0.85	0.99
Table 8: Statistics of ROSCOE scores evaluated on answers from CAUSALCOT on 2,000 randomly sampled questions from our dataset.
F Comparison with Existing Causality-Related Datasets
We show in Table 9 the distinction of our work from all existing causality-related datasets that address either the causality-as-knowledge task, or the causality-as-language-comprehension task.