Causal Reasoning and Large Language Models: Opening a New Frontier for Causality
Authors: Emre Kiciman
Robert Ness
Amit Sharma
Chenhao Tan
Venue: Accepted by TMLR ()

Abstract: The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date.

That said, LLMs exhibit unpredictable failure modes and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metadata, LLMs bring capabilities so far understood to be restricted to humans, such as using collected knowledge to generate causal graphs or identifying background causal context from natural language. As a result, LLMs may be used by human domain experts to save effort in setting up a causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that LLMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at https://github.com/py-why/pywhy-llm.

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality
Emre Kıeıman
emrek@ omrek@microsoft.com
Microsoft Research
Robert Osazuwa Ness Microsoft Research
Sbberteess@mmccoooff.comm
Amit Sharma Microsoft Research
umshar@microsofff
Chenhao Tan University of Chicago
chenhao @ wwwei@@@hhhhhhhco edu
Reviewed on htapa///....aeee////o//////////a
Abstract
The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of L.LMs to benchmark their capability in generating causal arguments. Across wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date.
pairwise causal discovery
That said, LLMs exhibit unpredictable failure modes and we discuss the kinds of errors that may be improved and what are the fundamental limits of ----edsed answers. Overall, by operating on the text metadata, LLMs bring capabilities far understood to be restricted to humans, such as using collected knowledge generate causal graphs identifying background causal context from natural language. As result, LLMs may be used by human domain experts to save effort in setting up causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that L.MMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at httts:://githii comm py www.wwh.w....t http//yywyyyp/// 11m
1 Introduction
Recent advances in scaling large language models (LLMs) have led to breakthroughs in Al capabilities. As language models increase in number of parameters and are trained on larger datasets, they gain complex, emergent behaviors, such as abilities to write code in programming languages, generate stories, poems, essays, and other texts, and demonstrate
Authors listed alphabetically.

---

Real-world Causal Task
Breok (sub-jquestions Recurse, llerute, veriy
Refutanion and Volicanton
Iterate on quessioo
Ittrate on question
deenity ssing pleces
Covariance-baset Causality
Causal Question
Loglc-based Causality
Cr tiuu premises and implications
Ecimation
Iderticctionn
assumptons
(Graph)
inference
Figure 1: When tackling real-world causal tasks, people strategically alternate between logical- and covariance- iterate, and verify their premises and implications. Now, LLMs may
causal inferences as they formulate have the capability to automate or assist with every step of this process and seamlessly transition between covariance- and logie-based causality.
strong performance in certain reasoning tasks (Chen et al., 2021: Nguyen & Nadi 2022; Bubeck et al.
2023 Katz
et al.
2023 Wei et al. (202aa). Impressively, when asked to explain their outputs, update their conclusions given new evidence, and even generate counterfactuals, LLMs can create plausible responses (Nori al. 2023 Lee al.
2023a b). This apparent capacity for both implicit and explicit consideration of causal factors has created excitement
towards understanding their reasoning capabilities (Hobbhahn et al. 2022; Kosoy et al. 2022 Willig et al. 2022 Liu
et al. 2023 Zhang et al., 2023). Figure ((a) shows example of such generation of causal argument
At the same time, L.Ms are imperfect: they can make absurd claims and are often observed to make basic errors of logic and mathematics, much less complex reasoning (Bubeck et al. 2023 Zhong et al. 2023) Ghazal et al. 2017) Figure (arree(bbssoooo shows an example of LLM making basic causal mistakes. This has incited a debate about whether LLMS truly perform causal reasoning, or are simply unreliable mimics, regenerating memorized responses in the context of critical analysis and decision-making tasks across many domains, including health, medicine, legal reasoning, education, policy making and business strategy (Bender et al. 2021 Marcus 2022). Complicating the situation is the variety of distinct formal and informal methods that people ascribe to 'causal reasoning'. and the breadth of purposes for which people apply causal reasoning some form.
In short, what kinds of causal arguments can LLMs generate, how valid are these arguments, and what causal reasoning workflows can this generation support automate? This paper clarifies this debate, advancing our understanding of LLM models and their causal implications, and proposes framework for future research at the intersection of LLMs and causality.
We begin with recognition of the distinct kinds of causal knowledge and reasoning implicated in this debate, including prior knowledge of general and domain-specific causal mechanisms, intuitive and logical methods for causal and counterfactual reasoning, and covariance- causality, such as building causal DAGs and inferring causal effects (see Figure T for an illustration). We present intriguing results each of areas and then proceed with deeper analyses to better understand the drivers of LLM behavior and the reliability of their causal capabilities in two areas-causal DAG generation and token causality. Our results indicate that LLMs bring significant new capabilities which are complementary existing causal methods. They do by capturing the human domain knowledge relevant to the task, which forms an essential part of any causal analysis. As result, LLMS have the capability of transforming how causal
We use the phrase generute causal orguments distinguish LLM generation itself.
causal statement or argument causal reasoning
2

---

SYSTEM:
You are a business advisor and give clear, well- reasoned but brief (5-15 linc) advice in response to questions about what people should do.
USER: A toy shop owner in the western USA wants to decide whether the ad they bought in early December is really better than their previous ads. Here their sules data:
USER:
Thanks. In February, the toy shop owner split their audience into two groups to show them two different ads, They showed the old ad people on the newspaper; and showed the new ad to people online. When people came the owner would ask them which ad they saw. In Fehruary, they found that they sold $6000 of loys to people who saw the new online ad, and $4500 to people who saw the old newspaper ad. What should the toymaker do in March?
October: $10,200 November: $10,000; Decem- ber: $13,000; January: $10,100
They now want decide which to show February. Can you help them decide whether the increase in sales in December was due to the ads? Note that the new ad costs $1000 more to produce, so the toy maker is interested in maxi- mizing their profit.
the store,
GPT-4:
Based on the availahle data, it is difficult definitively attribute the increase in December sales to the ad, as holiday season and other extemnal factors may have influenced sales. To make more informed decision, I recommend conducting an A/B in February by dividing the targeted audience into two groups, showing one group the old ad and the other group the new ad. Compare the sales performance of both groups and caleulate the net profit (considering the $1000 additional cost for the new ad) to de- termine which ud is more effective in driving sales and maximizing profit. This will provide a clearer understanding of the ud's impact on sales.
GPT-4: Based on the February results, the new online ad generated $1 $1,500 more in sales than the old newspaper ad. Considering the new ad costs $1,000 more to produce, the net profit increase is $500. In this case, it seems that the new online ad is more effective in driving sales. For March, the toy shop owner should continue using the new online ad, but also consider monitoring the long-term performance of the ad, as well as any potential changes in the target audience's ' pref- erences external factors that may influence sales. Additionally, the owner may plore optimizing the new ad further maximize its costeeffectiveness and reach.
ex-
(a) A GPT-4 powered advisor gives sound causal advice
(b) Continuing the conversation, GPT-4 gives a wrong causal answer
Figure 2: Probing LLM-ability to build causal arguments. Two example outputs from an LLM (GPT-4). In the first dialog, the LLM discusses causal issues, such as a potential confounder and recommends an A/B experiment to correctly characterize effects and drive the requested crcision-making The second example continues the conversation and requires arguably the same kind of causal awareness of potential confounders (e.g., the population characteristics and even population sizes of the online and newspaper audience are unknown) but the LLM proceeds regardless and provides an incorrect answer.
analysis is done, with the potential to automate or assist in each step of a causal reasoning process, shown by the arrows in Figure
Summary of results. Given a set of variables, causal DAG generation is the task of specifying the causal graph characterizing their data-generating process. Starting with a pairwise causal task, we find that LLM-based methods using variable names as input substantially outperform existing algorithms for building causal DAGs from statistical evidence in data (causal discovery algorithms (Peters et al.] (017)), achieving 97% accuracy on the Tübingen benchmark compared to the previous best of 83%. The benchmark includes over hundred causal relationships from a variety of domains, including physics, biology, zoology, cognitive science, epidemiology, and soil science. We repeat the experiment on more specialized, real-world task related to medical pain diagnosis and obtain similarly high accuracy for GPT 3.5 and 4 models. On the more challenging task of discovering the full graph, we find that LLMs obtain similar accuracy to recent deep learning-bass based methods. We extend these experiments to pairwise and full graph datasets that were made public after the L.MMs training cutoff date and find similar high accuracies. The errors made by the two kinds of methods-LLMss using variable names and discovery algorithms using the observed data-are not the same, highlighting the potential of combining these methods for greater accuracy. That said, LLMs do make unexpected mistakes; the surprising part is how few such mistakes are, since L.LMs only consider the metadata (i.e., variable names) rather than analyzing the data values. To characterize our measurement constructs, we probe whether LLM has been
3

---

trained on and memorized the datasets underlying our benchmarks and lay out the implications for interpreting causal benchmarks.
The second major causal task we consider is token causality (Halpern 2016). Unlike building causal DAGs, which deals with variables and their effect on each other, token causality considers individual events and aims to find what caused them. Inferring causes of an event involves simulation of different counterfactual scenarios, but it also involves human judgment to understand the background context and determine which causes need to be considered. For counterfactual reasoning tasks, we find that LLMs like GPT-4 are capable of answering natural language questions. On benchmark of counterfactual queries spanning basic physics, logic, and common sense, gpt-4 obtains 92% accuracy, 20 points higher than previously reported accuracy. These counterfactual capabilities also help LLMs to isolate the necessary and sufficient causes given any event, one of the key building blocks of token causality methods. GPT-4 obtains over 86% accuracy in identifying the necessary and sufficient causes on a commonly used benchmark of 15 vignettes, as well as a novel benchmark that avoids any memorization concerns. That said, tasks that heavily depend on an understanding of human factors in deciding the cause of an event, e.g., assessing the normality of particular action, remain challenging for LLMs: GPT-4 obtains 70% on our benchmark task for assessing normality.
Implications for causality research. Irrespective of whether LLMs can perform true causal reasoning (as opposed to merely generate well-formed causal arguments that support such reasoning), their empirically observed ability to perform certain causal tasks is strong enough to provide a useful augmentation for aspects of causal reasoning where we currently rely on humans alone. For example, conventional causal DAG generation and causal effect inference rely strongly on prior domain knowledge of potential causal mechanisms in system. Current best practice to rely on human domain experts to provide this knowledge, yet correctly capturing domain knowledge in formal representation suitable for analysis remains challenge and is often a primary point of weakness for the validity of causal analyses yes. LLM capabilities now open the possibility of programmatic access to an array of (memorized or inferred) causal mechanisms, capturing general and domain-specific knowledge, and may augment human domain experts by aiding in bootstrapping, critiquing, etc. Other areas where LLMs provide significant benefit include the ability to understand and formalize causal scenarios, generate relevant formal premises based on background knowledge about the world; and ability to identify and correctly frame challenging causal constraints, validations and refutations, such as negative and positive controls, monotonicity relationships, etc. These are all tasks where previously we relied on human experts alone, and now can be partially or entirely automated with human supervision.
That said, LLMs do have unexpected failure modes. In each of the tasks that we studied, LLMs achieve high average accuracies but also make simple, unpredictable mistakes on certain inputs. Further, their accuracy (and consequently robustness) depends substantially on the prompt used, as observed by Long et al. (202)). We provide some basic empirical tests to probe their robustness to specific prompt language and understand the relative contribution of data memorization. However, more research is needed to understand when LLM outputs can be trusted and increase their robustness, either through external tools other instances of LLMs themselves.
We close with an updated landscape for causal analysis research and practice. Fully characterizing LLMS inherent capacity to build causal arguments and understanding its underlying mechanisms, requires significant research efforts- and until this is accomplished, it is not prudent to trust LLMs alone in critical fecision-making tasks and other causal applications. However, current capabilities are sufficiently advanced to be useful in conjunction with existing methods for formal causal reasoning, discovery, and effect inference. We discuss these opportunities, including the possibility that LLMs may provide opportunities for building tighter automated integration between logical and covariance-based approaches to causality.
Outline of the paper. Section 2 reviews background in causality and LLMs. Section 3 provides our key results of causal DAG generation on causal discovery benchmarks and the associated implications for causality effect inference research. Section 4 discusses token causality and the performance of LLMs in generating counterfactual arguments, determining necessary and sufficient causes, and causal judgment tasks. Finally, Section 5 discusses how these L.MM capabilities provide a way to augment existing causal analyses with domain knowledge and natural language- reasoning, while also making it possible combine different kinds of causal methods in a single analysis.
4

---

2
Background and Preliminaries
Working with natural language, LLMs bring capabilities to causal questions that are complementary to existing approaches. These capabilities span across hitherto disparate fields of causal enquiry, from specifying models of general causal mechanisms to token causality over events, and from statistical effect inference to logic-based reasoning. As a result, we believe that LLMs offer an opportunity to bring together different approaches to causality and build unified pipelines that seamlessly transition between language-base and data-based analyses. Below we review these different approaches or subfields under which causality is studied. We also provide introduction to LLMs and summarize methods for probing their capabilities.
2.1 Many Kinds of Causality
The science of causality is the study of cause and effect relationships, and is fundamental tool for understanding and reasoning about the world and how works. Correct causal reasoning is crucial to making correct decisions, building robust systems, and scientific discovery itself. While advances in causal modeling have formalized core concepts of causality, the varying tasks and problems across domains have led different fields to use related but distinct conceptualizations and tools.
Here we describe three orthogonal categories of causal approaches and tasks. The first axis (covariance- and logic-based causality) distinguishes methodological approaches that emphasize data analysis from those that emphasize logical reasoning. The second axis (type vs. token causality) I focuses on the setting for a causal question: Are we asking about causal relationships in general, such as for a population, or asking questions about specific causes for specific event. The third axis, presented in Section 2.2 focus on the causal question task itself: are interested in inferring new relationships, characterizing the strength of specific known or hypothesized relationships, attributing blame or reward for some outcome, etc.
Covariance- and Logic-based Causality: Many fields, including statistics, biostatistics, and econometrics, place primary emphasis on covariance-based causal analysis. This family of methods uses statistical approaches to infer and estimate the strengths of causal relationships from data (Imbens Rubin 2015 Hernán & Robins 2010). Applications include the evaluation of drug efficacy, understanding effects of novel economic policies, and optimizing business decisions. A causal analysis typically starts with question whose answer is converted to a statistical estimand and then estimated using statistical estimators that work with available data (Pearl 2009a).
Other domains such as law, forensic investigations, and fault diagnosis, often emphasize logic-based causality, which uses logical reasoning and domain knowledge to reason about causal relationships in systems (Hellner) 2000). For example, some notions of legal liability involve establishing proximate cause for an event based on reasoning about counterfactuals and plausible scenarios (Knobe & Shapiro 202T).
Type and Token Causality: Type causality encompasses inference causal relationships between variables, such as causal effect estimation (Peters et al., 2017). In contrast, token causality (also called specific causality) refers to inference of the degree to which specific events cause other events (Halpern 2016 Hausman 2005). That is, token causality is concerned with reasoning about specific events and their causes whereas type causality focuses on variables and their average effects. For example, questions concerning medical science such as "does smoking causes lung cancer" or "whaa are causes of lung cancer" or "how much does smoking increase the risk of lung cancer" are examples of type-causal inference. Scientists are often interested in type causality questions, because these questions help develop theory and make predictions. For example, average causal effect of intervention tells something about its effect general population and can be used to inform policy.
In contrast, questions under the umbrella of token causality include "Was Fred's smoking habit responsible for his lung cancer? Or was it his exposure to asbestos?", or "What was the reason this machine failed?". These are questions that concern decision-makinn in specific situations and their answers need not generalize to other situations. Still, the answers are important to inform a specific decision or conclusion that needs to be made, e.g., deciding legal case or
TType causality is sometimes referred to as general causality. Token causality is also known as specific causality or actual causality, though latter term also refers to the modeling framework for token causality primarily described (Halpem 2016).
5

---

fixing a machine in factory. In practice, beyond simple logical reasoning, to determine token causality, people use mental "forward simulations" of processes to predict the potential outcomes of an event, or attribute the underlying events led to an observed outcome (Hegarty 2004 Jeannerod (001).
2.2 Different Causal Tasks and their Connection to Kinds of Causality
Causal tasks or questions seek to reason about gruse-and-fffect relationships in system, often with the goal of understanding and improving a desired outcome, as other mechanisms and environment change. This broad umbrella encompasses a large array of distinct causal tasks, with its own developed methodologies. Notably, while some these tasks intersect-they may share common abstractions (e.g.. causal graphs) or the result of one task may be fed as an input of another-an ability to solve one task does not imply an ability to solve the others.
Specifying the causal graph is the type-causal task of specifying a graph that represents the underlying causal mech- anisms that govern a system. Causal discovery is an approach to causal graph specification that attempts to reverse engineer the causal graph from covariance in data. (Peters et al. 2017 Glymour et al. 2019).
Effect inference is the task of characterizing the strength and shape of a known or hypothesized causal relationship. While it is most commonly characterized as relying on covariance- basedd methods, effect inference does rely on logical reasoning approaches for determining validity of causal assumptions (Sharma al.) 2021) and identifying approaches to validation and sensitivity analyses (e.g.. negative controls (Lipsitch et al. 2010). Effect inference is primarily focused in type causality scenarios. However, individual treatment effect estimation and counterfactual estimation straddle the
realm of token causality.
Attribution is the task of determining the cause or causes of a change. Depending on the application domain, approaches to attribution include both covariance-basee and logical-basee approaches, and are set in both type and token causality settings. For example, determining the likely cause of performance slowdowns in large-scale Internet service is usually covariance- analysis in type causality setting (Budhathoki al., 202T]; )): determining the root cause of specific error in system execution can be a covariance-basse analysis in an token causality setting (Sharma al. 2022;; whereas determining the cause of a fire in an arson investigation may rely purely on oggical-ressonng in an token causality setting (Halpern) 2016). Judgement tasks (Gerstenberg et al. 2014) extend attribution tasks to questions of reward or blame assignment for outcomes. They usually incorporate additional considerations of morality, normality, and intent of agents (Sloman & Lagnado 2015).
These are only some of the many causal tasks. Others include policy optimization, decision makinn, explanation, scientific discovery, and others.
2.3 LLMS and Causality
A large language model is particular kind of machine learning model, built using transformers, a class of deep neural network architectures (Devlin al. 2019). The primary task of L.LMs is next-word completion. Initially, next-word prediction is based primarily on word distribution probabilities. Further training applies additional human feedback to shape the reward function to take into account factors beyond word distribution probability, such as instruction following and safety (Ouyang et al. 2022).
Recent work has explored causal capabilities of LLMs (Willig et al. 2022). For instance, Long et al. (023) consider simple graphs of 3-4 nodes and test whether LLMs can recover the correct structure. They consider each variable pair (A,B) and ask an LLM score two competing statements, one implying that A causes B and the other that B causes A. Tu et al. (2023) consider harder task using dataset on medical pain diagnosis and find that LLM-based graph inference obtains poor accuracy. On the question of inferring causality from natural language, Hobbhahn et al. (2022) study whether I.LMs can understand causal implications embedded in natural language, i.e., given two sentences, whether (the event in) one sentence is the cause of another. In this paper, we extend this line of work and make two contributions. First, we investigate causal graph generation capabilities of LLMs, as an analog to covariance based causal discovery, over a broad set of complex real-world datasets and explore the robustness of L.LM-based graph generation. As one example, in Section 3.1, we revisit the case study on LLM-based graph graph generation from

---

ct al. (2023) and show that with an appropriate prompt, LLMs can achieve substantially higher accuracy (increasing the Fl score for retrieved edges from 0.21 to 0.68). Second, we probe the ability of LLMSs to do counterfactual reasoning and infer necessary or sufficient cause based on natural language description of events. We also provide a unifying framework that shows how LLMs can be used to transfer knowledge between covariance-based and logic-based causal methods for a given real-world problem.
2.4 Probing LLM behaviors
The primary input-output mechanism exposed by an LLM is textual prompt as input and textual response as output. Given black-box access to an LLM, therefore, the main way understand its causal capabilities is to probe it with different inputs and observe how its output changes. This probing paradigm of building causal arguments, however, has the classic limitation of construct validity for the measurements (Smith 2005p. That is, we may be tempted to ascribe particular causal capability to an LLM if it answers well on a set of questions related to the capability, but the answers may not necessarily be due to the capability; they may be due to other factors such as exploiting some structure in the questions, or in the case of I.LMs, memorizing similar questions that it encountered in its web-scale training set (Carlini et al. 2022). Hence we use multiple probing strategies to understand the causal capabilities of LLMs. In addition to the standard practice of testing LLM performance on existing benchmarks, we conduct memorization tests estimate the extent to which benchmark dataset may have been memorized, construct novel datasets to avoid memorization concerns, and conduct eemantically-equivallee perturbation and other interventions on the input prompts to understand what the LLM pays attention to.
Benchmark Tests and Question-Answer Evaluation: The standard approach to evaluating LLM behavior is to ask it questions and evaluate its answers. Our question is the prompt for the LLM;; and the LLM''s completion of the text is the answer. Most large scale benchmarks and evaluations follow this approach, using purposefully created benchmarks
and/or adapting standardized exams written assess human capability (Ghazal et al. 2017 Nori al. 2023 Zhong
et al. 2023).
Our evaluation approach to characterizing the causal capabilities of LLMs begins with such question -antiion-nswerrnng evaluations. For each evaluation, we ask a series of questions or causal challenges and score the correctness of the resulting answer. Then, we probe deeper, to better understand the threats to the validity of the question-answer evaluations, and the likely robustness of LLM capabilities.
Memorization Test The primary threat to the validity of benchmark question-answer style assessments is that the LLM may have directly memorized the benchmark answers. In this case, our questions are likely not testing the LLM's inherent capability to complete task (unless memorization is the capability we are testing!)
To test whether the LLM has memorized a particular dataset or benchmark, we give the LLM a partial row of data from the dataset and ask it to autocomplete the remainder of the row. For non-tabular datasets, we present a snippet of the file and ask to auto-complete from that starting point. To encourage the LLM to succeed, we prepend details about the dataset, such as its name, URL, and description, and also provide few-shot examples. The final measurement from the memorization test is the percentage of rows the LLM was able to regenerate correctly. Supplementary E. provides additional details on the procedure.
When we see an LLM generate a correct answer to a question, we are still sometimes unsure why. Is the LLM attending to the appropriate aspects of the question, or has it learned to repeat unreliable patterns that may lead to errors in the future. To better understand what aspects of the prompt or question an LLM is attending to, we use redaction and perturbation tests, motivated from explainable Al methods for NLP[Sinha et al. (2021): DDnilevsky al., 22000). First, we redact words in the prompt, one by one, and see how the answer changes each time. Changes in the answer indicate the LLM is attending to the redacted word.
Redaction Test:
'Our approach to testing for memorization and prior exposure of the LLM follows the method as developed in Bordt et al. 2024).
7

---

3 Generating causal graphs using LLMs
The task of building a causal graph is foundational step in causal inference workflow. Typically, the goal is to obtain a directed graph where the edges denote the presence and direction of causal effect. Such a graph characterizes the underlying data-generating process (DGP) for a dataset and specifies how a change in one variable may (or may not) affect the others. This graph is then used as a base on which downstream analysis relevant to a task conducted, such as for effect inference, prediction or attribution. Having the correct graph that encodes causal assumptions is critical for ensuring the correctness of any downstream analysis.
Causal discovery attempts to address the challenge of specifying causal graph by searching for graph consistent with observed evidence of covariance between the variables in data Peters et al. (2017). The allure of causal discovery is relying on empiricism and avoiding subjective model misspecificatiin errors that could bias the downstream workflow. The challenge, however, is that generally not possible to learn the correct graph for a given dataset, given only observational data. The reason that multiple graph structures are equally likely given the same data distribution, a set of graphs known as the Markov equivalence class (Pearl) 2009c). In the last two decades, two main approaches have been proposed to overcome this limitation. The first is to restrict data-generee generating process to specific functional forms under which identification of a single graph is possible (Glymour et al. (019). In some specific settings, such as adding non-gaussian noise to linear data-generating process (Shimizu et al. 2006) or assuming that all functions are non-linear
with additive noise (Zhang & Chan 2006 Zhang & Hyvärinen 2009), identification is possible. However, there still exist simple settings that are unidentifiable, c.g., a dataset with linear equations and gaussian noise. The second approach is utilize the power of deep learning to model covariances of all variables jointly and hope that improves the quality of learnt graphs. However, the identification issue is still not resolved. As result, recent evaluations of state-of-the-ar causal discovery methods on real-world datasets present a sobering picture of their effectiveness (Kaiser & Sipos 2022
Tu et al. 2019 Huang et al. 2021).
LLMs offer a fresh perspective on the use of statistical algorithms for constructing a causal graph. This perspective focuses on the metadata associated with variables in a dataset, rather than their data values. Typically, such metadata- based reasoning is done by human domain experts when they construct causal graphs. By looking at the names of variables and the problem context, for example, people can construct causal structure based on their knowledge of physics, common sense, or specialized domain knowledge. However, this is challenging process, hindering the widespread use of such metadata- based reasoning.
We find that LLMs can bridge this gap by filling in the domain knowledge that earlier only humans could provide. In contrast to causal discovery algorithms that use data values of variables, LLMs can infer causal structure by reasoning on metadata associated with the variables, for example, the name of the variable and the problem context expressed in natural language. In other words, similar to how domain experts formalize their knowledge in a graph, LLMs can use the knowledge from their training data to infer the edges of the causal graph. To differentiate from the existing covariance- based causal discovery, call the LLMs capability as snowledge-based causal graph generation.
Remarkably, L.MSs practicing knowledge-hand based causal graph generation outperform state-of-the-ard covariance- algorithms on causal discovery benchmarks. Below we present experiments on inferring causal structure using LLMs. We first start with pairwise causal discovery: a simple task involving two variables where the goal is to decide the direction of causal effect, across a range of domains. Next, we will study the problem of full graph discovery in two datasets: one medicine and one in climate science. While the first task includes many "simple" relationships that an average person is expected to answer correctly, the other two tasks require specialized knowledge (in neuropathic pain and aretic atmosphere science respectively) in order to check whether LLM's graph generation capabilities extend to complex domains.
3.1 Pairwise causal edge inference: Inferring causal direction among variable pairs
We start with the pairwise causal edge inference task (Hoyer et al. 2008). Typically, these tasks are set up as discovering causal relations between two variables based on observed data. In this work, instead of relying on observed data, we directly ask large language models whether a variable causes another variable.
8

---

Variable A
Variable
Domain
Age Abalone
Shell weight
Zoology
Cement
Compressive strength of concrete
Engineering
Alcohol
Mean corpuscular volume
Biology
Organic carbon in soil
Clay content in soil
Pedology
PPED (Photosynthetic Photon Flux Density)
Net Ecosystem productivity
Physics
Drinking water access
Infant mortality
Epidemiology
Ozone concentration
Radiation
Atmospheric Science
Contrast of tilted Gabor patches
Accuracy of detection by participants
Cognitive Science
Time for 1/6 rotation of Stirling engine
Heat bath temperature
Engineering
Time for passing first segment of ball track
Time for passing second segment
Basic Physics
Table 1:
Example cause-effect pairs from the Tübingen benchmark. The task is to determine whether Variable A causes Variable B, or vice-versa.
3.1.1 Tübingen cause-effect pairs dataset
This dataset (Mooij et al. 2016) consists of data for 108 different cause-effect pairs selected from 37 datasets from various domains, including meteorology, biology, medicine, engineering, and economics (see examples in Table D As directionality of an edge is fundamental building block for learning the full graph, it a widely used dataset for benchmarking causal discovery algorithms. However, the Markov equivalence class of graphs admissible for a two-node graph (A.B) contains both + B and B A. Therefore, the dataset remains challenging one for causal discovery algorithms, with many recent methods achieving less than 80% accuracy, shown in Table 2 The best known accuracy is 83%, achieved by the Mosaic algorithm (Wu & Fukumizu 2020).
We now apply LLMs to the pairwise edge inference problem. We extract the names of each variable from the benchmark and use them to construct prompts for the LLMs. Example prompts are shown in Suppl. A. (Table 14). We start with a simple prompt that asks, "Does changing cause a change in B?", where and B are the variable names. For given pair (A, B), we ask the question both directions and then take the mean accuracy. We choose the mean accuracy since it maps correctly to the accuracy of any method choosing between A B or B - A. Specifically, the LLM-based method obtains accuracy of 0 if it answers both questions incorrectly. If answers one of the questions correctly (indicating that none cause each other or both cause each other), then it effectively chooses the output at random (A + B or B + A). We also report weighted accuracy, as recommended by Mooij et al. (2016) to avoid the effect of overcounting some similar pairs.
Results. The second half Table shows the performance of different LLMs on the task. Similar to prior studies on capabilities of LLMs (Wei al. (202a)), we observe the emergent behavior that only text-davinci-000 and above achieve non-trivial accuracy than random chance. With GPT3.5 class models (text-davinci-003 and gpt-3.5-turbo), accuracy of basedd method reaches 83% and is competitive to the covariance-based causal discovery algorithms. This suggests that the ability of inferring causal direction emerges the size of the model is increased. It may also depend on the specific training data and the use of reinforcement learning from human feedback (RLHF) for instruction finetuning.
Following recent work showing the value of the prompt tuning for obtaining high-quality results (Wei et al., 2022b Long et al. 2023). we employ two different kinds of prompts. In the first, we prepend the prompt with the message, "You are helpful assistant for causal reasoning". with the intention of steering the LLM's output space towards more causally consistent answers. This simple message of asking gpt-turbo-3.5 (i.e., ChatGPT) to be an agent of causal reasoning increases the (weighted) accuracy by almost 5% to 86.9%, consistent with the impact of prompt engineering observed by (Long et al. 2023). The accuracy is higher than previously reported LLM-based accuracy on the task from Choi et al. (2022) that used the davinci-instree beta model (83% unweighted accuracy). Moreover, by changing the prompt to instead ask single question that asks the L.LM to output the more likely causal direction between A or A B while explaining its reasoning in "step-by-step" manner (see Suppl. A.I the weighted accuracy improves to

---

Model
Acc.
Wt. Acc.
Slope (Marx & Vreeken
2017
0.75
0.83
bQCD
Tagasovska et al.
2002
0.68
0.75
PNL-MLP (Zhang & Hyvarinen
2012
0.75
0.73
Mosaic
WW & Fukumizu
2020
0.83
0.82
ada
0.50
0.50
text-ada-001
0.49
0,50
babbage
0.51
0.50
lext-babbage-001
0,50
0.50
curie
0.51
0.52
text-curie-001
0.50
0.50
davinci
0.48
0.47
text-davinci-001
0.50
0.50
text-davinci-002
0.79
0.79
text-davinci-003
0.82
0.83
LMPrior (Choi et al. 2022
0.83
gpt-3.5-turbo
0.81
0.83
gpt-3.5-turbo (causal agent)
0.86
0.87
gnt-3.5-turbo (single prompt)
0.89
0.92
gpt-4 (single prompt)
0.96
0.97
Table 2: Accuracy of different versions of GPT on the Tübingen cause-effect pairs dataset. The best LLM performance outperforms the current state-of-the-art covariance-based approaches that rely on observational data of the variables. Weighted accuracy weights individual pairs account for overcounting due some pairs sharing the same source dataset. The causal agent is apt-3.5-turbo with system message set as "You are helpful assistant for causal reasoning.". LMPrior uses devinci-instruut-bee
92.1%. Finally, using gpt-4 with this prompt, the weighted accuracy shoots up to 97%. This accuracy is substantially higher than that of the best-known covariance-based causal discovery method.
Understanding LLM output. The combined single prompt allows us to inspect the LLM output, since it provides an explanation addition its final answer. For the examples below, we show the output from gpt-3.5-turbo. Let us look the example, (age of abalone, length of abalone) where the LLM correctly outputs that changing the age of abalone is more likely change its length. Figure ((a) shows the LLM's output. Its explanation gives the impression that it can reason about the problem using its knowledge to arrive at the correct answer:
However, on another pair that arguably should involve similar reasoning, (age of abalone, diameter of abalone), the LLM outputs an incorrect answer and its explanation does not make much sense (Figure 3(b).
The fact that the LLM can make such errors is not surprising, since it is not explicitly optimizing for being consistent with prior knowledge causal mechanisms. However, it is remarkable that such errors occur infrequently: The LLM made only 12 errors in the dataset of 108 pairs. On some other pairs where the LLM's output did not match the ground-truth, it reveals possible ambiguity in the variable names, that can be fixed by adding context to the question. Consider the example of (ozone concentration, radiation) below where the LLM initially provides incorrect answer.
LLM:
Ozone a gas in Earth's atmosphere that absorbs most of Sun's ultraviolet (UV) radiation.
2. Chlorofluorccarbos (CFCs), pollutants that were once widely used in refrigerants and aerosol sprays, have been found to destroy ozone molecules the atmosphere.
3. The destruction of the ozone layer in the upper atmosphere causes more UV radiation to reach the surface of Earth.
10

---

SYSTEM:
You are a helpful assistant causal reasoning, Which cause-and-effect relationship is more likely? A. changing the age of an abalone causes change in its diameter.
SYSTEM: You are helpful assistant for causal reasoning.
USER:
USER: Which cause-and-effeee relationship is more likely?
B. changing the diameter of an abalone causes a change in its age.
A. changing the age of an abalone causes a change in its length.
Let's work this out in a step by step way to be sure that have the right answer. Then provide your fi- nal answer within the tags <Answer>ABB</Ansee GPT-3.5-TURBO: Step Determine which variable is the cause and which is the effect.
B. changing the length of abalone causes a change in its age. Let's work this in step by step way to be sure that we have the right answer. Then provide your final answer within the tags <Answer>A/B<AAnseer
Option ""ae" the cause and "diameter" the effect.
GPT-3.5-TURBS 1. Cause: Changing the age of an abalone
Option B, "diameter" the cause and "age" is the effect.
Effect: change in its length.
Step 2: Consider which variable more likely to cause change in the other.
2. Cause: Changing the length of
abalone
Abalone growth is primarily determined by environ- mental factors such water temperature, food avail- ability, and water quality. It is unlikely that changing the age of an abalone would cause diameter change significantly. On the other hand, changes in the environment could potentially impact the size of the abalone, and therefore, its diameter could change. Therefore, Option B is more likely: changing the diameter of an abalone causes change in its age. <Answer>B</Answer>
Effect: change in its age.
Based on the biological knowledge about abalones, it is more likely that changing the age of an abalone causes a change its length. This is because the length of an abalone is known to in- crease as it ages, and not the other way around. Therefore, the correct answer is <Answer>A</Answerr
(a) Correct
(b) Incorrect answer
Figure 3: Probing reasoning of LLMs for deciding causal direction. Two example outputs from LLM (gpt-3.5- turbo). The first gives the impression of the LLM performing causal reasoning to arrive at the correct answer-thar age may cause the length abalone to increase, The second example asks a question that requires arguably the same causal knowledge but the LLM proceeds with an incoherent argument and provides an incorrect answer.
Increased UV radiation can lead to various negative effects such as skin cancer, cataracts, and damage to crops.
From these facts, we can determine that a decrease in ozone concentration would lead to an increase radiation. Therefore, Option A is more likely:
<Answer>A</Answer>>>
The correct answer according the benchmark is that radiation causes ozone concentration. However, the LLM interpreted the question as about the ozone concentration in the stratospheric ozone layer, whereas the question about ground-level ozone concentration measured in a city. After adding this context, the LLM's answer changes,
USER: Wait, this question is not about stratospheric ozone concentration. Rather, I'm asking about the ground-level ozone concentration measured a city and the corresponding ground-level radiation due sunlight.
GPI-3.5-TURBO::
see, thank you for clarifying. In that case, the answer is likely option B: changing radiation causes a change in ground-level ozone concentration. This is because ground-level ozone is formed through a chemical reaction between nitrogen oxides (NOx) and volatile organic compounds (VOCs) in the presence of sunlight. This reaction known as photochemical smog and occurs more frequently on hot, sunny days when there is more radiation. Therefore, an increase in radiation would likely lead to an increase in ground-leve ozone concentration.
11

---

<AnswerBe/Answen
Finally, in some cases, the direction of the cause-effect relationship not absolute and depends on other supporting information. For the pair (stock return of Hang Seng bank, stock return of HSBC Hldgs.), the LLM correctly determines that Hang Seng is subsidiary of HSBC Hldgs. but its answer does not match the benchmark's s label. HSBC Hidgs. owns majority stake in Hang Seng bank, so the benchmark answer indicates that any change in Hang Seng's stock return should also affect HSBC Hldgs. stock return. At the same time, since stock markets do not follow any fixed pattern, it is possible that a major change in HSBC Hldgs, stock return causes people to invest more or less in Hang Seng Bank and hence affect its stock return, as the LLM output argues.
LLM:
To determine which cause-and-effie relationship is more likely, we need to consider the nature the two companies and their relationship with each other.
Hang Seng Bank and HSBC Holdings are two separate companies but they are both major players in Hong Kong financial market. Hang Seng Bank is subsidiary of HSBC Holdings and is listed on the Hong Kong Stock Exchange. HSBC Holdings is multinational banking and financial services company that operates in over 60 countries worldwide.
Given that relationship and the fact that Hang Seng Bank a subsidiary of HSBC Holdings, it more likely that:
CAuwwerB<</nsswerchhhnnnngg the stock return HSBC Hidgs causes change in the stock return of Hang Seng Bank.
Creating novel Tubingen dataset. It is possible that LLMs achieve high accuracy in predicting causal direction because Tubingen pairs is popular dataset which may have been included in LLMs' training data. To test generalization of LLMs to causal pairs that were not a part of LLMs' training data, we build a new dataset. Since the training cutoff date for the models we use (GPT-3.5-turbo and GPT-4) is September 2021, we aim to collect cause-effect pairs that were published after the cutoff date. To do so, we use Google Book search to search for cause-effect pairs in books published
after Oct 2021. We first categorize pairs from the original dataset into domains such biology, engineering, medicine, and so on. For each of these categories, relationship in their text. Based on this procedure, we obtain dataset with 67 variable pairs. For instance, the variable pair, "battery capacity". "ambient temperature" was derived from the book, Battery Management System and Its Applications, published in 2022. For a sample list of variables, see Appendix A.3 A.3 https://github. com/
search for appropriate books that describe causal
The full dataset is available at
ccom/py-mn py- why-why/pyae wwt.phhhyy1pyaa.
On this dataset, we applied the exact same (single) prompt from the original benchmark. GPT-3.5- and GPT-4 obtain 80.3% and 98.5% accuracy respectively, indicating that the capability identify causal direction generalizes to variable pairs outside of popular datasets.
3.1.2 Neuropathic pain dataset
We now consider ground-truth causal graph constructed by medical experts on neuropathic pain diagnosis (Tu et al. 2019). It contains relationships between different nerves and the associated symptoms that patients express. Table 3
shows example of edges in the graph. Unlike the Tübingen benchmark which contains some commonly known variables, understanding the names of variables in this graph requires specialized medical knowledge.
We evaluate LLMs on a pairwise causality task similar to the Tubingen benchmark. We do so by creating a pair for each edge in the graph and setup the task to determine the direction of the edge. That is, for all the edges present in the ground-truth graph, we ask an LLM to infer the correct direction of causality. We obtain a dataset of 475 edges.
Table 4 ssows the accuracy of different LLMs in determining the direction of edges. Similar to the Tübingen benchmark, smaller language models are unable to obtain accuracy substantially higher than random chance (50%). In addition, even 1exi- the largest text completion model in our comparison set, unable to obtain a high accuracy using the two-sided prompt. However, chat-based models, trained using human feedback, are able distinguish causal
12

---

Variable
Variable
Dir.
Right 1.1 Radiculopathy
Right adductor tendonitis
Pharyngeal discomfort
Right C3 Radiculopahty
Right L5 Radiculopathy
Lumbago
Left PTA
Left L4 Radiculopahty
Left T3 Radiculopahty
Toracal dysfunction
DLS LS-S1
Right S1 Radiculopathy
Left C3 Radiculopathy
DLS C2-C3
Left C7 Radiculopathy
Left medial elbow problem
Right Ischias
Right L5 Radiculopathy
Right Morton trouble
Right L$ Radiculopathy
Table 3: Example cause-effect pairs from the Neuropathic pain diagnosis benchmark. 'Dir." refers to the ground-trut trutt causal direction between the variables.
Model
Accuracy
ada
40.1
text-ada-001
50.0
babbage
50.0
jext-babbage-001
50.9
curie
50,0
text-curic-001
50.0
davinci
38.4
text-davinci-001
50.0
text-davinci-002
51.7
text-davinci-003
55.1
gpt-3.5-turbo
71.1
gpt-3.5-turbo (neuropathic pain expert)
75.1
gpt-4
78,4
gpt-4 (neuropathic pain expert)
84.3
text-davinci-003 (single prompt)
86.0
gpt-3.5-turbe (single prompt)
85.5
gpt-4 (single prompt)
96.2
Table 4: Accuracy of different versions of GPT on the inferring the edge directions of the Neuropathic pain diagnosis graph. As with the Tübingen dataset, LLMs like gpt-3.5-turbo obtain more than 85% accuracy on determining the direction of edges. The causal agent is gpt-3.5-turbe with system message set as "You are a helpful assistant for causal reasoning."
direction accurately. With the two-sided prompt, gpt-3.5-turbe achieves the highest accuracy of 75%. The choice of prompt also has big impact. As with the Tübingen dataset, using single prompt variable pair leads to a significant increase in accuracy: text-davinci-00s now matches gpt-3.5-turbo and both models obtain more than 85% accuracy. While the performance of these models impressive, gpt-4 obtains an even higher accuracy (96%), yielding more than 457 correct responses of 475.
Understanding LLM output. We the output of gpt-3.5-turbo (single prompt) to understand how LLMs obtain such high accuracies. On manual inspection of the ouiput by gpt-3.5-turbo, we find that it correctly understand the medical terms in almost all cases. In the example below, the L.MM provides the correct definitions of DLSS T5-T6 and Left T6 Radiculopathy (without access to any such information in the prompt) and provides the correct causal direction. Suppl. provides the LLM output for this pair. However, even as it provides the correct definitions, it can provide erroneous output in unpredictable ways. Here went from the correct definition of the same concept (DI.S) to confusing it with the "T5-T6 region" associated with DLS T5-T6 (see Suppl. A.2 for the full output).
13

---

Moreover, the example in Table Table addss evidence that LLMs may not be capable of consistent and coherent reasoning. even as they output correct answers for a majority of the causal questions. In the left panel, the LLM provides the correct reasoning for most of its answer but ultimately chooses a final answer inconsistent with its own reasoning (A instead of B). Unlike reasoning errors, fortunately self-consistency is an easier problem to verify using the LLM itself. We find that gpt3.5-turbo and earlier class of models are unable check consistency of an output, but GPT-4 able Therefore, it may be possible to pass all LLM outputs through GPT-4 verifier to ensure self-consistency of the outputs. The right panel of Table shows the verifier prompt and output from GPT-4 (the question and answer from gpt-3.5-turbo are not fully shown for brevity).
Initial response
Critiquing the response
SYSTEM:
You are an expert on neuropathic pain di- agnosis.
USER: Analyze the output from an Al assistant.
Is the final answer (A) consistent with the reasoning provided by the assistant?
USER:
You are helpful assistant to a neuropathic pain diagnosis expert. Which and-effect relationship is more likely?
Question:
A. Right C7 Radiculopathy causes DLS C6- C7.
([USIRPROMPT-RROM-BFFO
B. DLS C6-C7 causes Right C7 Radiculopa- thy.
assistant:
(LANSWER-FREC
BEFORE)]
Let's work this out step by step way to be sure that have the right answer. Then provide your final answer within the tags <An- swer>A/B</Answer.
GPT-4-WEB: The final answer (A) is not consistent with the reasoning provided by the assistant. The reasoning suggests that DLS C6-C7 could cause Right C7 Radiculopathy, which corre- sponds option B. The correct answer should be <Answer> B</Answerz.
GPT-3.5-TURRO: First, we need to understand what each term means. DLS stands for degenerative lumbar spondylosis, which is a condition where the spinal discs in the lower back degenerute and cause pain. C6-C7 refers to the sixth and seventh vertebrae the cervical (neck) spine. Radiculopathy is condition where a nerve root is compressed or damaged, causing pain, numbness, or weakness the corresponding area of the body.
Based this information, it is more likely that option A the correct cause-and and-effect relationship. This is because radiculopathy often caused by compression or damage to nerve root, which could be the result of degen- erative changes in the spine (such as DLS). In this case, the degeneration in the lower back (DLS) could be causing compression damage to the nerve root the C7 level, resulting in right C7 radiculopathy.
Therefore, the answer is
<An-
swer>A </Answer/
Table 5: Critiquing LLM output using another LLM instance. To increase robustness of an LLM's response, we can use GPT-4 as critic. The left panel shows an incorrect reply from gpt-3.5-turbo wherein the reasoning is correct but the LLM outputs the incorrect option (A). We create special "critique" prompt that asks gpt-4 to evaluate the response from an AI assistant for colf-connistency. gpt-4 finds the logical inconsistency and provides the correct answer.
14

---

In addition to celf-consisteney, when LLM outputs do not match the ground-truth, they can sometimes help critique the ground-truth itself. For instance, the ground-truth graph contains an edge from Left L.5 Radiculopathy to Obesity. However, the LLM output provides the opposite direction: obesity can cause L.5 Radiculopathy, which is corroborated by research that claims obesity as a risk factor for radiculopathy (Atchison & Vincent 2012 Tomić et al. 2009).
L.M:; Step Understand the terms
Neuropathic pain: pain caused by damage or disease affecting the nervous system
LS Radiculopathy: a condition where the nerve root that exits the spine at the LS level is compressed or irritated, causing pain, numbness, weakness in the leg
Obesity: a medical condition characterized by excessive body that increases the risk of health problems
Step 2: Analyze the cause-and-effect relationship
Option A: Left L.5 Radiculopathy causes Left Obesity. This relationship does not make sense because LS radiculopathy is nerve condition that affects the leg, while obesity is whole-body condition related to excessive body fat. is unlikely that leg nerve condition would cause obesity.
Option B: Left Obesity causes Left 1.5 Radiculopathy. This relationship is more plausible hecause obesity put extra pressure on the spine and nerves, leading to nerve compression or irritation. It is possible that obesity could cause L5 radiculopathy.
Step 3: Choose the more likely cause-and-effeee relationship
Based on the analysis, Option B is more likely: Left Ohesity causes Left L,5 Radiculopathy.
<Ansser>B</Answe>>
3.1.3 Other properties of pair-wise relationships
In addition to asking the LLM about the existence and direction of a causal relationship between a pair of edges, we can also ask many other questions well. For example, we can ask if a relationship may be confounded by other variables, or what variables may mediate or moderate the relationship. We can ask whether a causal relationship is monotonic, whether its effects are homogencous or not with respect to other properties. Other useful properties to identify include the time-lag of causal relationship, whether it is stable, and whether it is subject to spillover effects. Designing prompts for these questions should be a straightforward extension of the pairwise edge existence prompts.
3.2 Full graph generation
We now extend the pairwise analysis to generating the full graph. Given set of variables, straightforward extension is to construct a list of all possible variable pairs and repeat the pairwise test for each pair. However, full graph generation provides additional challenges. First, the pairwise analysis assumed the existence of an edge and our goal was simply to determine its direction. In the graph generation problem, there are three possible options for each variable pair: A + B, A + B, or no edge exists. Second, graph generation requires the ability to distinguish between direct and indirect causes, given the other nodes in the graph. For example, if the true relationship is - B + C, then may be correct to output both A + B and A answer. Moreover, the decision depends on which variables are present the input variable set for the graph generation task. For example, if B is not one of the variables in the input variable set, then A + C' is valid edge and should be included.
in a pairwise task, but for graph generation, outputting A - C² is an incorrect
Due to these reasons, it is unclear how well simple extension of the pairwise task using LLMs will transfer to the full graph generation problem. For simple graphs over 3-4 variables, [ong et al. (2023) show that LLMs can obtain promising accuracy in inferring the edges. Below we report the results of applying the pairwise LLM prompts on more complex, real-world datasets involving specialized knowledge: on pain diagnosis and on artic sea ice coverage.
15

---

Model
Precision
Recall
Fl
Random
0.25
0.5
0.33
chatGPT (Tu et al. 2023
1
0.12
0.21
text-davinct-003
0.59
0,68
0.63
gpt-3.5-turbo (single prompt)
0.66
0.71
0.68
gpt-4 (single prompt)
0.74
0.58
0.65
Table 6: Accuracy of different versions of GPT on the Neuropathic pain dataset.
3.2.1 Neuropathic pain dataset
We continue our exploration with the neuropathic pain dataset. It has 221 variables, so the total number of possible variable pairs are ((221, = 24310. Before conducting full graph generation, we utilize a 100-pair dataset provided by the authors of the dataset to evaluate LLM's capability to infer causal relationships (Tu et al., 2023). has 50 variable pairs that form edges in the graph and 50 pairs that do not form an edge. We notice that many of the variable names are in Swedish, so we employ LLM (gpt-4-web) to translate the names to English
preproeessing step.
Results. Table O shows the results of different LLM-based methods. As baseline, using an algorithm that return either direction at random for each pair, we would obtain 0.5 recall and 0.25 precision on recovering the true edges, leading to an F1 score of 0.33. Using a simple prompt, "A causes B. R and L refer to the right and left sides of the body respectively. Answer with true or false.", Tuu et al. (2023) use chatgpt-3.5-wy performance is worse than the random baseline which may indicate that I.LMs are not useful for full graph generation. However, with a different prompt, we find that the same model can provide over 3X increase in the Fl score. We provide the LLMs with an identical prompt to the "single" prompt from the pairwise task, with one addition: add third option, "C: No causal relationship exists". With this prompt, gpt-3.5-turbo (API analogue of chatgpt-3.5-wel obtains 0.68 Fl score. This score is double the FI score expected from a random guess baseline, indicating that the output from LLMs provide a non-trivial utility for inferring causal edges.
to obtain an Fl score 0.21. This LLM
Using a novel medical dataset. It is possible that the Neuropathic dataset may be a part of the training set for the LLMs. Therefore, we consider a causal DAG proposed by Mascaro et al.] (2023) that was published online February 2022n a few months after the training cutoff date of the LLMs considered. The DAG captures expert-provided relationships concering COVID-19 disease processes. We consider subset of the full DAG containing 11 nodes, reported by Mascaro al. in their paper. Variables correspond to concepts in respiratory medicine, such "Alveolar Epithelial infection" and "Viremia".
We apply the same prompt that we used for the Neuropathic dataset. GPT-4 obtains an F1 score of 0.73 which is comparable to the best-reported Fl on the Neuropathic dataset (F1=0.68).
3.2.2 Arctic sea ice dataset
To evaluate LLM's ability to generate causal graphs, we now consider a dataset from a different field of science: arctic sea ice and atmospheric science. This dataset is on the drivers of arctic sea ice thickness (or coverage): what causes the arctic sea coverage to increase or decrease? In a recent publication, Huang et al. (202T) provide a domain knowledge- based graph with 12 variables and 48 edges. Variables in the graph include total cloud water path, sea level pressure, geopotential height, meridional and zonal wind 10m, net shortwave and longwave flux at the surface. Importantly, the graph contains some double-sided edges.
Since the number of variables is low (12), LLMs. Huang et al. (2221) evaluate the accuracy of three recent discovery algorithms: temporal causality discovery framework (Nuuta et al. 2019), NOTEARS (Zheng et al. 2018), and DAG-GNN YYu et al. 2019). They evaluate the algorithms on the normalized Hamming distance (NHD) between the predicted graph G' and the ground-truth graph G.
able to conduct experiment to generate the full graph using
"The first article describing the dataset was published on medRxiv February 2022 the dataset was released same month (URL: bttps://csf.co 1o/bynrr./ The OSF project, itself, was created on 1st October 2021.
16

---

Algorithm
NHD
No. of predicted edges
Baseline NHD
Ratio
TCDF
0.33
9
0.39
0.84
NOTEARS (Static)
0.33
15
0.44
0.75
NOTEARS (Temporal)
0.35
7
0,38
0,92
DAG-GNN (Static)
0.32
23
0.49
0.65
DAG-GNN (Temporal)
0.34
16
0.44
0.77
gpt-3.5-
0.33
62
0.76
0,43
gpt-4
0.22
0.65
0.34
Table 7: Normalized hamming distance (NHD) for different causal discovery algorithms. Since NHD depends on the number of predicted edges, we compare the ratio of NHD and baseline NHD across algorithms. A lower NHD ratio is better. LLM-based graph generation (gpt-3.5-turbo obtains comparable NHD and the lowest NHD ratio compared to recent covariance-based discovery algorithms.
For graph with nodes, the distance is given by Σxj=1 m3 &G, the number of edges that are present in one graph but not the other, divided by the total number of possible edges. The distance zero if G' outputs exactly the same edges as G. For reference, since the number of nodes and edges in our ground-truie truth graph are 12 and 48 respectively, if G' outputs 48 completely different edges compared to the 48 edges of G, the NHD between G and G' will be 0.66. However, if we output edges at all, the NHD will be better than the algorithm, 0.33. Since the NHD depends on the number of edges returned by a discovery algorithm, we compare the ratio of NHD of the discovery algorithm and a "floor" baseline that outputs the same number of edges but all of them are incorrect. A lower ratio implies the multiple by which the graph discovery algorithm is better than the worst baseline returning the same number of edges.
Table shows the NHD metric for the three covariance- causal discovery algorithms and knowledge-based graph generation with LLMs. For each algorithm, we also show the number of predicted edges and the NHD for worst case algorithm that predicts the same number of edges. NHD for the three covariance-basse algorithms is similar. However, the ratio of NHD for the algorithm and the baseline algorithm provides the relative improvement that the algorithm provides compared to the baseline. Using the ratio metric, we see that DAG-GNN (static) performs the best and has the lowest Ratio distance.
For LLM-based graph generation, we use the same prompt as in the neuropathic pain dataset. Using gpt-3.5-turbb we obtain a normalized hamming distance of 0.33, comparable to the three recently proposed covariance-based causal discovery algorithms. However, the LLM returns 62 edges, its NHD ratio (0.43) is substantially better than novariance-based algorithms. Finally, gpt-4 obtains the lowest NHD among all algorithms and significantly outperforms state-of sere--ff-hheaaaee the-art covariance- baased discovery algorithms. It outputs almost the same edges as the ground-truth (46) with an NHD of 0.22, one-third less than other algorithms. Its NHD ratio is also the lowest. gpt-4 recovers over half (29) of the edges correctly with a precision 0.63 (FI score=0.57), indicating the competitiveness of LLM-based algorithms for causal generation.
Measuring LLM's NHD metric on a novel dataset. In addition to the Arctic sea ice dataset, we consider a dataset that was created in 2023 by consultation with medical experts on Alzheimer's disease (Abdulaal et al. 2023). Since the causal graph was created after the training cutoff date of September 2021, it cannot be present the training data for causal graph of 11 nodes. Variables in the graph include age, ventricular volume, brain MRI, etc. They the same prompt that we use for the Arctic sea ice dataset and report NHD of the obtained graphs using different LLMs. Table in Appendix A.3 reports their results on the dataset. The results show that the graph generated by gpt-4 obtains the lowest NHD (0.14) compared to existing existing data-based methods such as NOTEARS (0.22) and DAG-GNN (0.37). Its NHD ratio is also the lowest (0.28). These results, similar to those obtained for the Arctic sea dataset, indicate the generalizability of LLMs like GPT-4 to for inferring full graph over novel datasets. Note that novel datasets may still contain concepts that existed before the L.M''s training cutoff date. See Section 5.I for a discussion on utility of LLMs with respect to novel scenarios.
LLMs. Abbulaal et al. (2023) present an
17

---

Which cause-and-efiee relationship is
more likely?
A.
changing SLOTI causes change in SLOT2
B.
changing
SLOT3
causes
change SLOT4
Let's
work
this out in a step by step
way
to
be
sure
that
we
have
the right
answer
Then
provide
your
final answer within the tags
Annser A/B </nnw//AAwwr
Figure 4:
We probe the importance of individual words for getting correct answer by redacting random word from question. We highlight words based on their importance for getting a correct result. A white background indicates that redacting the word did not reduce accuracy. A dark blue highlight indicates that redacting the word reduced accuracy the most.
3.3 Probing LLM behavior further
To better characterize the validity and robustness of our experiments at the intersection of LLMs and building causal graphs, we report the results of our LLM probing experiments, described in Sec (.4
Memorization Test Results: We provide the LLM with the first 3 columns of the dataset (the row ID and 2 variable names), and ask the model to complete the remaining 2 columns of the row (the source dataset name and the ground truth causal direction. We find that GPT-3.5 is able to correctly recall 58% of the remaining cells in the dataset, and recall 19% of the rows without error. Our memorization test with GPT-4 is able recall 61% % of remaining cells in the dataset and 25% of rows without error. These results indicate that the Tübingen dataset is certainly in GPT's training set and that GPT has had an opportunity to memorize the dataset. However, the gap between percentage of inputs memorized and LLMs accuracy is still significant and more work is needed to understand the role of memorization for the inputs that are not captured by our memorization test.
As the neuropathic pain and the arctic sea ice coverage datasets are not available online in a tabular file format, we adapted our memorization test ask for completions of randomized samples of JSON and code files F We provide the LLM with contiguous section of text from randomly chosen location within a file, and ask the model to complete the file from that point. Counting the number of characters that are correctly reproduced by the LLM,, provides an indication of whether the dataset has been memorized by the model. Our results, detailed in Section E. indicate that the neuropathic dataset has been at least partially memorized by GPT-3.5 Turbo, but not GPT-4, and the arctic sea ice datasct has not been memorized by either GPT-3-5 Turbo or GPT-4.
More generally, we expect knowledge-baadd based causal graph generation be founded partly on memorized information, but also dependent on the LLM's ability to process and transform seen causal relationships for use in multiple contexts. We can conceptually model the LLM's end-to-end efficacy P(D), where P(D) represents the likelihood of causal relationship being memorized by the LLM, and P)Y the likelihood that the relationships can be correctly transformed. Knowing that our benchmark dataset is memorized implies that our experiments can only measuring P)Y but not measure PD). That is, when evaluating an L.M on an existing benchmark, its performance on the benchmark is a test of how well the LLM's ability to process and transform its knowledge into the necessary causal relationship assuming that it has been trained using some representation of the underlying information.
Redaction Test: To characterize whether or not the LLM is attending to the correct words and concepts in question when answering our pair-wise causal edge inference questions, we use redaction test. Our redaction test shows what words are important for correctly labeling a causal relationship between two features. Figure 4 shows our results for experiments in gpt-3.5-turbo, averaged across 357 random redaction probes over our Tübingen experiment. If the L.M is correctly reading the question, we expect that it will pay a lot of attention to key instructions and to the options themselves. Here we see that some of the key instruction words that indicate we are asking about causal relationships, such 'changing'. and 'causes' as well as instructions for how to format the final answer in tags are
FFor the neuropathic dutaset, use the JSON representution available the public web httt:::///baeeeeeeetee http::///bbbevbbbbbg.bbbbba com/ uturuloo the complette causal aarissal-qrae --raphhhoo of nwwoonttrtttthhccooooooo wwwpoppppattpppaa-oppoaaeaaaaaee agnosis] for the arelic sea ice dataset, we use a python file that www.c//mmtaaaaaaeaeececco //year- 3-pro projorctts////eo wec.t//ttttttteeeee NO BRRE///R/S/S./ee. compute.......oooooo.oo.c.. dist__sstt... dist_matrix
embeds 12x12 binary malrix representation of the causal graph, (tttps://gitttnn
betxeees.ypperpara.c STw.eeppppaaaaSSIICCCC TEMPORALEELLLLLLLL py We omnt tests for novel datasets created after LLM training cut dates.
18

---

critically important, each affecting the accuracy of the result in the most redactions. The SLOT phrases are important but redundant (SLOTI and SLOT44 are identical and SLOT2 and SLOT3 are identical). Therefore the L.LM seems more robust to their individual redactions in this test, each individual redaction affecting the accuracy of the result, but less often. While most unimportant words do not affect the accuracy of the result much, we do see the redaction of number of 'mnorr' words having a surprisingly strong effect, possibly indicating general sensitivity of the LLM to grammatical correctness.
4 LLMs for Token Causality and Causal Judgments
In the previous section, we saw how LLM's ability to capture common sense and domain knowledge allows for a knowledge-based approach to building a causal graph. In this section, we study how LLMs' can bring these capabilities to enable a systematized approach to token causality challenges. Token causality is motivated by problems of attribution and assigning responsibility in real world applications, such as legal reasoning. machine failure debugging, and root- cause analysis for system regressions. For example in tort law, the core problem of deciding how much blame for plaintiff'ss injury is attributed to defendant's 's action relative to other potentially mitigating events is fundamentally an token causal inference task.
Causal inference researchers have attempted to use formal causal models to define token causality in a way that is consistent with how humans naturally attribute cause and related concepts of responsibility, blame, and explanations to events and their outcomes (Halpern [0166] Kueffner (021). This task has proven exceedingly difficult, because human judgments of causality depend on elements of background context that are difficult to formalize an SCM. Types of background context that are fundamental human causal judgments but difficult formalize in an SCM include:
The set of candidate causal events that are relevant to a particular outcome event. SCMs require relevant causes be included endogenous variables, but in practice humans rely on domain knowledge and common sense to set the causal frame after the outcome occurs (Icard & Goodman 2015). For example, there are a multitude of potential causes of forest fires, but in the case of a particular forest fire, upon learning lightning struck and that a hiker was carelessly smoking cigarette during fire season, we know these are relevant causal events. We would also know whether to ignore "drought" or "lack of controlled burn" if they are not relevant in the case of this particular fire.
Causal frame:
Necessary causality: Whether a candidate cause needed happen for the outcome to occur.
Sufficient causality: Whether a candidate cause's occurance would have led the outcome event if other causal events had occurred differently.
The degree to which causal events align with statistical norms or prescriptive norms (social, moral, or legal norms) (Phillips et al. 2015)] Kominsky et al. (015). When agents violate norms, they are typically judged to be more of a cause of resulting outcomes (Phillips et al., 2015 Kominsky et al. (2015). Human causal judgments depend highly on whether candidate cause a norm violation, or whether it is more or less normal than other causal events.
Normality:
Other human factors: Other human factors include bias towards action, handling intention and epistemic state, and how bad outcomes are interpreted. When the candidate cause is an agent's behavior, humans tend to ascribe more causality actions (e.g., tossing cigarette) than to lack of actions (e.g, not doing a controlled burn) (Henne et al., 2017). In addition, when the candidate cause is an agent's 's behavior, whether the agent acted with intention and knew what they were doing (Nadelhoffer) 2006 Knobe 2003) matters. Finally, human causal judgments also depend on whether the outcome is undesirable (e.g., causing a forest fire vs. causing a reforestation initiative) (Kominsky et al. 2015).
Sidestepping the challenge of formalizing these concepts into a causal model, LLMS offer an opportunity to capture the necessary and relevant background context for an event directly from natural language description of an event.
19

---

Premise
Counterfactual Question
Mullipl-chuices answers
A woman does not order Chinese food.
What would have happesed if she had ordered Chi- nese lood?
The woman would have become less bungry. The woman would have become very hungry. That is possible.
A woman sees a fire.
What would have happened if the woman had touched the fire?
She would have been burned. ,She would not have been humed,;That (hht not possible:;;he would have seen fire.
A bird lands in a forest.
What would have happened plane had landed the torest?
The plane would have crashod. crashod.Eveeythhin. EEveyything would have been The plane would have landed safe and sound,IIn a firest you will find lots of planes. That is not possible..It would have grown quicker,The plant would have suffored. The planter would have cui tivated plant
plant grows planter.
What would have happened if the planter grows the plant?
A mortician prepares corpse.
What would have happened if the morticion had prepared a dinner?
He would have had delicious dish, Moorticians con- not prepare dinners., The dinner would have been buried. The mortician would have killed the curpse. There would have heen envirunmental pollution. That is not possible.;Ths oil tanker would have continued to carry ohe oil tanker would have been saved.;
An oil lanker sails across an ocean.
What would have happened if the oil lanker had broken up ocean?
A car crashos into tree.
What would have happened if the car had parked beneath the tree?
Nothing special would have happe ned..The car would have been hit by the The is not possi- ble.;1 think it would have crushed into the tree.
A child draws a picture.
What would have happened if the child had crased the picture?
The picture would not have heen visible. The pic- possible. That is not possible. house would have been built faster:Everythiingg would have been fine.; craftsman would have hands.
would have been bsible.,That
A craftsman builds house.
What would have happened the house had built craftsman?
A doctor washes their hands at work.
What would have happened if the doctor hadn't
The patients could get an inlection. patients could get better:/hhal not possible.
washed their hamds?
Table 8: Example scenarios from the CRASS counterfactual reasoning benchmark. The task is select the best answer choice for the counterfactual question, given premise.
Given that an LLM is trained on text narratives written by humans, the subjective elements of causal judgments may be encoded as part of the LLM's internal representations.
In this section, we first review the concept of counterfactual reasoning and show that LLMs can answer counterfactual questions about an event with accuracy close to that of humans on a benchmark task. We then present an example scenario to illustrate the challenges of modeling real-world events using token causality and introduce the two key counterfactual concepts used in formal models of token causality: necessity and sufficiency. Using 15 different vignettes commonly used in token causality literature, we provide evidence that LLM can capture the necessary background context and reason about necessity and sufficiency of token causes. Finally, we show how this capability transfers a causal judgment benchmark task from Big Bench (Suzgun et al. 2022). specifically designed to test L.LMs their ability to infer the normality of candidate causes.
4.1 Building block of token causality: Counterfactual reasoning
Counterfactual reasoning is the process of considering hypothetical situations or alternate realities by altering specific elements or conditions of an actual event or situation (Kahneman & Miller 1986; Byrne 2005). Such reasoning is a key element of token causality. To determine the causes of an event, is important to simulate alternative worlds where an event may not have happened and reason about the consequences. For example, a naive approach to token causality is to use definition based on counterfactual dependence: an event A is cause of another event B if B would not have happened without A.
Independent of token causality, counterfactual argument making is a desirable skill for a LLM, as it would assist users in decision- decision-making, making, planning, and offer insights that may not explicitly apparent from the original context. With this motivation, Frohberg&& & Binder (202)) provide a benchmark called CRASS (Counterfactual Reasoning Assessment) for evaluating the ability of LLMs to answer counterfactual conditional questions, subset of which is included in the
20

---

Model
Accuracy
GPT-3
(Frohberg & Binder 2022
58.39
TOpp
Sanh et al. 2021
72.63
text-davinci-003
83.94
gpt-3.5-turbo
87.95
gpt-4
92.44
Human annotators
98.18
Table 9: Accuracy of different LLMs on the CRASS counterfactual reasoning benchmark. gpt-4 achieves 92% accuracy. significantly higher than the previous reported accuracy on this benchmark and within six percentage points of human annotators' accuracy.
BIGBench collection of datasets Srivastava et al. 2022 6 The benchmark contains 275 instances, where each instance has the LLM select from multiple choice answers to a counterfactual conditional question such as "A woman opens treasure chest. What would have happened if the woman had not opened the treasure chest?". Table [ lists some instances from the dataset. Our memorization test indicates the CRASS dataset has not been memorized (see Section E).
The authors report GPT test-davinci-O00 Malavinci-003--e as the best performing (zero-shot) LLM this task. with top-1 accuracy of 58.39%. Another language model (TOpp) (Sanh et al. 202T), which was finetuned on multi-task setup of classification tasks, obtains 72% accuracy. This is contrasted with 98. 98.18% accuracy human baseline, indicating that counterfactual predictions remain a challenging task for LLMs.
We revisit the claim using LLMs released after the benchmark's publication. As shown in Table g each problem instance has premise, counterfactual question and list of answer options. We construct a prompt for the LLM by concatenating the premise and the counterfactual question, and then presenting the answer options as A, C, D (for example prompt, see Suppl. B.I For gpt- ept-3.5-turbo and gpt-4, we additionally provided system message, "You are helpful assistant for counterfactual reasoning."
Table 9 shows the accuracy of different LMs. Remarkably, GPT 3.5 version models show substantial improvement in accuracy over GPT 3. gpt-3,5-turbo obtains an accuracy of 87.95, gpt-4 improves it further to 92.44%, which is 20 percentage points higher than the previous best accuracy. Comparing to the human accuracy on this task, gpt-4 is within six percentage points of average human accuracy. The results indicate that large language models of GPT-3.5 and 4 series represent substantial jump in LLM ability to generate counterfactual arguments.
Understanding LLM output. We now try analyze some outputs from gpt-4 understand its counterfactual argument-making capabilities. For full LLM outputs of the prompt discussed here, see Suppl. B.I.s the high accuracy indicates, gpt-4 is able to simulate different scenarios based on the text prompt and answer what-if scenarios. In many cases, when gpt-4's answer does not match the benchmark, it reveals ambiguity in the text used to describe the scenario. In one of the cases, the LLM is asked what will happen a man catches a water balloon, given that he has caught ball. The benchmark answer that he will get wet whereas the LLM output correctly argues that he may get wet or not, depending on whether the balloon bursts as he catches it. Similarly, another question asks what will happen a man gets very nervous. The benchmark answer indicates that he will pass out, but the L.MM output correctly mentions that, "Passing out due extreme nervousness is possible, but not guaranteed. and concludes that the man would not pass out. We suspect that given enough clarifying context, the LLM should be able to correctly answer such questions.
That said, there are a few instances where the LLM does not capture context that people may easily determine through common sense. For instance, one of the questions asks what would happen if a man walks on bed instead of walking on the street. "Walking on a bed" conveys that the man likely indoors, but the LLM incorrectly imagines a scenario where the man is still walking to the same destination, but now he is walking over bed, which makes it incorrectly
'The full dataset was released Github on August 2021, so it after the training cutoff (ext-davinci-00: model. But a month before cutoff for gpt-3.5- smaller subset of inputs was released a part of BIG-Bench May 2021 with a canary string for LLMs to avoid including it their training data.
21

---

conclude that the man "will get late" (see Suppl. These results indicate the importance of minimizing ambiguity when interacting with an L.M-wherever possible, adding small additional details (e.g., "walking on his bed at home") can increase the chances correct answer.
Results on a novel counterfactual dataset. In addition to the CRASS benchmark, we consider a counterfactual dataset that was created after the LLMs training cutoff date of September 2021. et al. (2023) develop a new synthetic dataset that was first uploaded on Github in 2022. We consider only the counterfactual variant of their dataset. It includes counterfactual statements of the kind: premise, hypothesis, conclusion. For each premise, there are two statements (one correct, one incorrect). We reformat the dataset match CRASS format. That is, create multiple-choice question prompt as: premise; statement), statement2. For example, here is instance from the dataset: "If Charles Darwin had worked on curing people, A. Darwin would have written paper on Aspirin; B. Darwin would have written a paper on species." Li et al. measure whether a model ranks the correct answer (A) higher than the incorrect answer (B).
We follow the exact same prompt used for the CRASS dataset and evaluate GPT-4 (see Table 17 Appendix A.3 GPT-4 obtains an accuracy of 88.6%, which is significantly higher than the best-reported accuracy from Li et al. (71.3% for GPT-3). This indicates that models like GPT-4 can generalize counterfactual arguments to scenarios.
4.2 Inferring token causality: Necessary and sufficient causes
Given the promising performance of LLMs on a counterfactual inference task, we now investigate whether LLMs can infer whether events are necessary or sufficient for focal event.
4.2.1 Motivating Example: Necessary and sufficient causes
To motivate, present the following example story of a bottle of beer spilling party, variation on the story presented in Henne & O'Neill (2022). Since the story was published in 2022, it cannot have been in the training set of either GPT 3 or GPT 4 models.
PROMPT:
Mike, Jack, Susan, and Peter are a party. There an open bottle of beer the table. Mike is moving through the room and accidentally bumps against the table. Jack saw that the bottle was about to fall, and he reached out to eatch the bottle. But just as Jack was about to catch the bottle, Peter bumped into Jack, which caused Jack to miss the bottle, which then fell to the ground. As result, the bottle of beer fell to the floor, spilling the beer. Right after bottle hit the floor, Susan tripped and fell onto the table.
Let us unwrap the story progressively bring out the two key concepts of token causality: necessity and sufficiency. Necessary causality: The property of an event, C, that if it had not occurred, then the outcome event, E, would not have occurred.
So we start by evaluating whether the LLM can make simple statements about necessary causality in the context of the beer bottle spilling story. First start by ask the LLM to provide a one sentence counterfactual explanation as to why Mike caused the spill.
PROMPT:
Mike, Jack, Ben, Susan, and Peter are at party. There was an open bottle of beer on the table. Mike is moving through the room and accidentally bumps against the table. As a result, the bottle of beer fell to the floor, spilling the beer. Did Mike cause the beer spill? Provide one sentence counterfactual explanation to why or why not.
Limiting the response to one sentence discourages more verbose responses.
LLM: Yes, Mike caused the beer to spill; if Mike hadn't 't bumped into the table, the beer bottle would not have fallen and spilled.
22

---

The LLM zeroes in on the necessary causality as the explanation. More generally, we can prompt the LLM to invoke the necessary cause principle and output the token cause for an event.
In the above example, Mike's bumping into the table was necessary for the beer to spill. Moreover, it was also sufficient to cause the beer to spill. But there are many cases where a causal event a necessary but insufficient cause of an outcome.
Sufficient causality: The property of event, C, that if it had occurred, then the outcome event, E, would have occurred.
Formal models provide unambiguous definitions of sufficient causality; in Pearl's counterfactual notation, the probability a causal event C=1 is a sufficient cause is P(Ec=1 = 1|C = 0, E = 0) (Pearl 2009b). In natural language this reads as "in cases where was O and E was O, what is the probability that E. would have been 1 if had been 1?"
But this phrasing awkward to express in situations where Cnd E were both indeed not O. So we turn to cognitive science literature to find more natural text expression of sufficient cause. We use variant of robust sufficient causality (Hitchcock 2012; Woodward 2006
Hitchcock 2012):
Robust sufficient causality: Given that causal event C occurred, outcome event E would have occurred even if other events that contributed to B's occurrence had not occurred. To elicit a statement about robust sufficiency, we prompt the LLM to state what other causal events lead to the spill, and then predict a counterfactual about whether the spill would have occurred those events had not happened. This a more challenging task than the necessary cause prediction, because it requires the LLM to solve causal frame problem decide which other causal events to, some of which may their interactions.
PROMPT: Based on the information in the passage, what other events directly contributed to the fall of the beer and what would have happened those events hadn't 't happened?
L.M: Other than Mike bumping into the table, the presence the open beer bottle on the table directly contributed to the fall of the beer; if the bottle had not been placed on the table or had been properly closed, it would not have fallen and spilled even if Mike had bumped into the table.
This response highlights the capability of the LLM to set the causal frame, i.c., select causal events relevant to the inference problem. We did not specify which causal events were eligible for consideration. For example, the LLM could have included the presence of the table as a causal event, but did not. This is in contrast to formal models such as SCMs where all the variables must be articulated in advance of running an inference query on the model. We conjecture that, probability model, the LLM is selecting probable causal abstractions based on both the prompt and statistical regularities in the training data. This allows its causal argument generation to adapt based on the scenario; if Mike bumping the table is the only causal event in consideration, it is sufficient, but that may change if other causal events are present. This highlights the flexibility of LLM's in causal analysis relative to SCMs and other formal models.
4.2.2 Evaluating LLMs on inferring necessary and sufficient causes
When the principle of necessity and sufficiency yields the same cause, token cause of an event can be casily determined. However, in many real-world scenarios, they may output different answers. For example, in the beer spilling example above, suppose there were two beer bottles on the table. Then the presence of any beer bottle on the table is a sufficient cause for beer spilling, but not necessary. Even if one of the bottles was not placed on the table. the other one would have spilled. Therefore, inferring necessary and sufficient causes typically requires applying the definitions of necessity and sufficiency and using formal reasoning (Halpern) 2016).
Below we investigate whether I.LMs can generate causal arguments directly using the natural language description of an event. We evaluate on collection of 15 vignettes (example scenarios) from Kueffner (2021). Table I0 shows sample vignettes. These vignettes are widely used to discuss and critique token causality definitions in the literature and span challenging scenarios across seven different types: symmetric overdetermination, switch, late preemption, early preemption, double preemption, bogus preemption, short circuit, and other miscellancous examples. For example,
23

---

Vignetle Type
Inpul Context
Event
Actor
Nec.
Suff.
Overdeteriiaaionn
Alice (AF) and Bob (BF) each fire a bullet window, simultane- onsly striking the window, shaltering it (WS).
window shatter- ing
Alice
No
Yes
Switch
Alice pushes Bob. Therefore, Bob hit truck. Boh dies. Otherwise, Bob would have been hit by bus, which would have killed him well.
Bob's death
Alice
No
Yes
Late preemption
Alice (AF) and Bob (BF) each fire baller at window, Alice's bullet bits the window first (AH). The window shatters (WS). Bob's bullet arrives second and does not hit the window (BH).
window shatter- ing
Alice
No
Yes
Early preemption
Suppuse Alice reaches out and catches pussing cricket hall. The thing the hall's trajectory was solid brick wall that would have stopped the ball. Beyond that there was a window. a bullet window (AT).
window being in-
Alice
No
Yes
Double preemption
Alice intends
intends prevent Alice from hitting the window (BI). Bob tries stop Alice (BSA).
window shatter. ing
Alice
Bob is stopped by Carol (CSB). Alice fires bullet (AF), hits the window (AII) and shatters (WS). The window shallers (WS).
Bogus preemnttion
Alice intents to put lethal poison into Carol's water. However, Alice does not put lethal poison into Carol's water (-AP), Bob puts antidote into Carol's water (BA). The water is lothal (L), if the poison is addeed without addition of antidote. Carol would consumes the lothal water she would dic (CD), Carol consumes her water (CC). Carol does not die (-CD))
Carol's 's survival
Alice
No
Yes
Short circuit
Carol is alive (CA). Alice puts harmless untidote Carol's water (AA). Adding ant idote to the water, protects it against poison (WS- Carol's
Carol's survival
Alice
No
Yes
ob will poison the water (BP) Adding poisen to an unprotected water makes loxic (WT). Carol would drink luxic water she would die inhibiting CS). Carel consumes her water and survives (CS). there is hot weather, flowers will die. Watering prevents the flowers die in hot weather. The neighhor does water the flowers in her yard, The flowers die.
"water save') ), If Alice puts the antidote
Miscellaneous
flowers' death
neighbor's 's inne- tiom
Yes
Yes
Table 10: Example vignettes for evaluation of inferring necessary and sufficient causes, categorized by their type based on the different ways in which potential causes can interact to yield the final outcome. Each vignette tests two questions: "Is (Actor) a necessary cause of (Event)?" and "Is (Actor) a sufficient cause of (Event)?".
symmetric overdeterminatiinn is scenario where multiple processes (causes), all of which producing the same outcome (event), terminate at the same time. Suppl. C.I provides a description for each type.
For each vignette, our goal is check whether an LLM can correctly identify whether a cause is necessary, sufficient, or both. To do so, we use the following prompt template, where PRINCIPLE is replaced by either "minimal change" (corresponding to necessary cause) or "multiple sufficient causes" (corresponding to sufficient causes). In each example, we provide two options: one of them the necessary cause and the other the sufficient cause. Interestingly, the prompt itself was constructed using the g-t-3.5-turbo (see Suppl. C.2 for details), An example prompt for the early preemption vignette is shown below.
SYSTEM: You are an expert in counterfactual reasoning. Given an event, use the principle of [PRINCIPLE) to answer the following question.
PROMPT: Suppose Alice reaches out and catches passing cricket ball. The next thing on the ball's trajectory was a solid brick wall that would have stopped the ball. Beyond that there was window.
Is Alice a INECESSARY/SUFFICEEN
cause of window being intact?
After your reasoning, provide the final answer within the tags Mennswer>Ys/Nooo/Anne
Using this prompt, we evaluate all 15 example scenarios on two LLMs: gpt-3.5-turbo and GPT-4. For example, on this vignette, both LLMs correctly answer that Alice is not necessary cause; without her action, the brick wall would have stopped the Fall.For evaluating sufficiency, gpt-4 correctly answers that Alice is sufficient cause of the window being intact, but gpt-3. spt-3.5-turbo answers incorrectly (see responses in Suppl. C.4.I).
Tables I and 12 summarize the results over all vignettes. We see a significant difference between the accuracy of gpt3.5-turbo and gpt-4, indicating a marked change in ability between the two models, gpt-3.5-turbo fails to capture the nuances of necessity and sufficiency definitions and obtains an accuracy near random guess. However, gpt-4 is accurate
24

---

Vignette Type
Necessary
Sufficient
gpt-3.5-turbo
Overdetermination
V
X.
Switch
X.X
Late preemption
Early preemption
X, V.
X,
Double preemption
Bogus preemption
X, X
Short circuit
X
Miscellaneous
X. .V
V,V.X. X, V
Total Accuracy
46.6%
46.6%
gpt-4
Overdeterminationn
Switch
Late preemption
Early preemption
Double preemption
Bogus preemption
Short circuit
Miscellaneous
V.X. V.v
V.V. v
Total Accuracy
86.6 %
86.6 %
Table 11: Accuracy of gpt-3.5-turbe and gpt-4 on inferring necessary or sufficient cause on 15 standard vignettes. The vignettes are divided into eight types (e.g., Early Preemption type has three vignettes). Each /X) corresponds correct/incorreet answer single vignette. gpt-3.5-turbo fails at the task (worse than random chance) but gpt-4 can infer necessary and sufficient cause with high accuracy.
for most vignette types. This is remarkable because the LLM was not explicitly provided definitions of necessary and sufficient causes. As with the case of causal graph generation, however, the high accuracy also comes with unpredictable failure modes. For example, even as gpt-4 use the correct sufficiency principle for most vignettes, on the short circuit vignette (see Table TOp, gpt-4 uses only counterfactual dependence principle answer the question and fails reason about the sufficieny of cause. The input prompt and incorrect response are shown Suppl. C.4.2
Since the above vignettes or their variants have been quoted in many publications, there is concern that LLMs may have memorized them and may not be producing new inferences about the answers. We therefore report results on the "lab-vignettes" dataset, new dataset we created that adapts the base vignettes to a laboratory setting. We do so by keeping the same type of scenario (e.g., short circuit), but changing the situation to that in a laboratory with reagents, test tubes, mixtures and crystals. Suppl. C.3 provides details on this dataset. On this new, unseen dataset, we find similar pattern when comparing models. gpt-4 obtains significantly higher accuracy than gpt-3.5-turbo. While gpt-4 retains its overall accuracy, we find an interesting pattern: for both models, necessity an casier concept answer than sufficiency. 4pt-4 obtains over 92% accuracy deciding necessity of cause compared 78% deciding its sufficiency. This may be because necessity always involves comparing the output under a counterfactual world where only the cause is flipped; whereas sufficiency can be nuanced, involving counterfactual flips to all relevant other variables (and deciding what those variables should be).
Overall, our results indicate the capability of L.LMs understand a scenario natural language and output token causes according pre-specified definition, but also show the lack of robustness due unpredictable failures.
25

---

Vignette Type
Necessary
Sufficient
gpt-3.5-turbo
Overdeterminationn
X.
Switch
X,V
Late preemption
X
Early preemption
X
X,
Double preemption
Bogus preemption
X
Short circuit
X
X
Miscellancous
X, X, V
Total Accuracy
64.2%
42.8%
gpt-4
Overdeterminationn
V.
Switch
X.
Late preemption
Early preemption
X
Double preemption
Bogus preemption
Short circuit
Miscellaneous
V.X. V.V
V.V. X.V
Total Accuracy
92.8%
78.5% %
Table 12: Testing dataset memorization issues with a novel "lab-vignettes" dataset. The average accuracy of gpt-4 stays the same as in the std vignettes, indicating that gpt-4's capabilities infer necessary and sufficient cause can generalize to new data. Inferring necessary cause (93%) emerges
easier task than inferring sufficient cause (78%).
4.2.3 Assessing responsibility
Using LLMs produce causal arguments allows for more nuanced understanding of causality, taking into account factors such intention, moral obligation, and epistemic state that are difficult capture using formal SCM approaches.
We present study of these capabilities in Supplementary D
4.3 Evaluating LLMs ability to infer normality
Normality is key element of evaluating token-causality. When evaluating the role event had in causing some outcome, it is difficult to use formal model to quantify the normality of that event. LLMs enable to rely on common-sense knowledge to quantify how normal/abnormal an event was relative to an outcome. In this section, we describe an experiment designed to quantify the role normality plays in predicting the outcome to token causality problems.
For this analysis we used vignettes from the causal judgment benchmark in BIG-Bench Hard (Suzgun et 2022) The causal judgment benchmark collects examples from vignettes similar those in Kueffner (2021) but more general language and provide more background information. Each of the vignettes ends with yes/no question about whether particular event caused the outcome. The benchmark provides ground truth labels for each question, collected from human annotators. The vignettes vary factors that drive human causal judgments, such as whether the causal event was necessary, sufficient, normal, whether it was causation by omission, whether it lead to a harmful outcome, etc. It is an ideal benchmark for evaluating how well LLMSs would do in addressing practical token causality problems, opposed more contrived examples.
26

---

The benchmark has proven resilient to saturation as new generations of LLMs are released. For example, while several benchmarks in BIG-Bench Hard saw a dramatic increases in accuracy between GPT3 and GPT4, causal judgment had a small increase (61.5% in text-davinci-003 and has an accuracy of 67.6% in GPT-4-32K (AML-BABEL accessed 2023-04-25). Top human performance is at 100% accuracy.
Constructing a benchmark for normality. We filtered out the vignettes that ask whether an agent intentionally caused an outcome and only focus on examples that ask if an event agent caused outcome. We do this because assessing intention is a separate task from ascribing causality, and because many intention examples were contrived (e.g., trolley problems) such that normality does not apply. We then manually reworded each vignette in order to mitigate against the possibility of the vignettes having appeared in the training. Each vignette was specifically designed to include sets of the factors that drive human causal judgments (necessity, normality, etc.). So we reword the vignette by applying a minimalist lexical substitution of words in the vignette so that we avoid changing those factors. To illustrate, the following is vignette from the benchmark, which originally appeared in Knobe & Fraser (2008)
The receptionist in the philosophy department keeps her desk stocked with pens. The administrative assistants are allowed to take the pens, but faculty members are supposed to buy their own. The administrative assistants typically do take the pens. Unfortunately, so do the faculty members. The receptionist has repeatedly emailed them reminders that only administrative assistants are allowed to take the pens. On Monday morning, one of the administrative assistants encounters Professor Smith walking past the receptionist's desk. Both take pens. Later that day, the receptionist needs to take an important message... but she has a problem. There are no pens left on her desk. Did Professor Smith cause the problem?
We reworded this vignette follows, with the main changes in bold:
The executive assistant in the marketing department keeps notepads in his cabinet. The office assistants are allowed to take the notepads, but managers are supposed to buy their own. The office assistants typically do take the notepads. Unfortunately, so do the managers. The executive assistant has repeatedly emailed them reminders that only office assistants are allowed to take the notepads. On Tuesday morning, one of the office assistants encounters Manager Johnson walking past the executive assistant's cabinet. Both take notepads. Later that day, the executive assistant needs to take an important note... but he has a problem. There no notepads left in his cabinet. Did Manager Johnson cause the problem?
We then manually label each example as "normal" or "abnormal". In most cases these labels were not subjective assessments on our part. Each element in the causal judgment data set includes citation to the source of the passage. In many of these studies, normality was explicitly varied across sections of a vignette, and thus we could confirm our labels from their source papers. However, in some cases, we had to make subjective best-guess assessments.
Prompting the LLM. We follow a two-step process. We first prompt the LLM extract the causal event in question.
PROMPT:
The last sentence of the following passages is question. The question either ask if an outcome event happened because a certain causal event occurred, When the question asks specifically about "intention." the causal event is not the outcome, it is the action that lead the outcome. State the causal event being asked about. Do not include the outcome event in the statement. Do not include other causal events in the passage.
The executive assistant in the marketing department keeps notepads in his cabinet. The office assistants are allowed to take the notepads, but managers are supposed to buy their own. The office assistants typically do take the notepads. Unfortunatcly, so do the managers. The executive assistant has repeatedly emailed them reminders that only office assistants are allowed to take the notepads. On Tuesday morning, one of the office assistants encounters Manager Johnson walking past the executive assistant's cabinet. Both take notepads. Later that day, the executive assistant needs to take an important note... but he has a problem. There are no notepads left in his cabinet. Did Manager Johnson cause the problem?
Passage:
27

---

LLM: Manager Johnson took notepad from the executive assistant's desk.
We then ask the LLM to evaluate the normality of the causal event in question. Inspired by both model-based and subjective notions of normality, we construct the following prompt.
PROMPT: Given a causal event that causes an outcome event, the causal event is "abuormal" if any of the following is true:
The causal event's occurrence was unexpected. unlikely, surprising, rare, or improbable
The causal event was an agent's action or lack of action that non-accidentally, knowingly, or negligenty violated social, legal, ethical norms.
The opposite of "abnormal" is "normal". A causal event is 'normal" if its occurrence was expected, likely, probable, and not surprising. cases where the causal event is an agent's action or lack of action, the action lack of action can also be considered "normal" if it intentionally compliant with social, legal and ethical norms. The normality causal event independent of whether outcome event was and whether the outcome event normal.
Please read following passage and statement about a causal event from the passage. Provide your step-by-step rationale to why the causal event is abnormal normal. Then, return the word "normal" if you think the causal event is normal or "abnormal" if you think the causal event abnormal.
Passage: [PASSAGE]
Event: Manager Johnson took notepad from the exccutive assistant's cabinet.
LLM: Manager Johnson's action of taking notepad from the executive assistant's cabinet abnormal because violates the established norm that managers are not allowed to take notepads from the exccutive assistant's cabinet. Despite executive assistant sending reminders about this rule, Manager Johnson took notepad, making his action unexpected and unethical.
In defining normality, we attempted to capture both statistical norms (with keywords "expected," "likely," "probable," "frequent," "unsurprising," and their opposites) and prescriptive norms. Violation of prescriptive norms can depend on intention, so we use keywords "nonaccidentally" "knowingly," "negligently." and "intentionally." We also ask the model to provide a chain-of-then of-ouhuhhtthhtttth sstep-by-step rationale for its answer.
Results. Initial results show accuracy in inferring normality was 69.2% on gpt-3.5-turbo and 71.1 1% on gpt-4, which are comparable to performance on the core benchmark task. Table 13 shows three examples of correct and incorrect answers across answers by GPT-3.5-turbe and GPT-4.
Table demonstrates the capability of LLMs parse normality but also provides intuition for how the LLMs can get normality wrong. While results did improve between versions of gpt-3.5-turbo and gpt-4, there were some were gpt-4 was wrong when gpt-3.5-turbo right, although its rationale when it wrong was typically more persuasive.
4.4 Discussion
Formal approaches to token causality have struggled to represent background common elements of causal judgment, such as necessity, sufficiency, normality and responsibility. We show through analysis of the CRASS (Frohberg & Binder 2022) and causal judgment (Ghazal et al.) 2017) benchmarks and related vignettes from Kueffner 2021) that LLMS perform well in generating causal arguments with these elements, Relatively lower performance for direct and chain-of of-thought answering on the causal judgment benchmark highlight an opportunity to research and engineer systems that guide I.LMs assembling these core elements into solutions for practical problems token causality questions.
28

---

Passuge and Question
Correct Answer
gpt-3.5-turbo rationale
GPT-4 rationale
Mogan and Sarah work for the same organization. They euch need to use printer for lasks sometimes. Unfortu- nately, the printer isn't very reliahle. If two people print the same time, it usually jams, So the organization docided to institute official rule. declared that Megan would be the only one permitted to use the printer in the morn- ings and that Sarah would the only one permitted to use the printer the afternoons. As expected, Megan used the printer the next day at 9:00 am. Bui Sarah decided to ignore the official rule. She also used the printer at 9:00 am. The printer jammed immodiately. Did Sarah cause printer to jam?
Sarah using the printer at 9:00 am is ah- normal event be- cause she fol- lowed policy.
Using the printer 9:00 um is consistent with the official policy, making this causal event normal. (incorrect)
Sarah's decision to the printer al 9:00 am despile the official policy is abnornal. as it intentionally violates company' 's policy put place to prevent the printer from jamming. This action is unexpected and goes against the established social and workplace rules. (correct)
The executive assistant in the marketing department keeps notepads his cabinet. The office assistants allowed take the notepads, but managers are supposed to buy their own. The oflice assistants typically do the notepuds. Unfortunately, so do the munagers. The executive assistant has repeatedly emailed them reminders that only oflice assistants are allowed lake the notepals. On Tuesday morning, one of the office assistants encrunters Manager Johnson walking past the exccutive assistant's is cabinet. Both take notepads. Later that day, the executive assistant needs to an important note... There are no notepads left in his cabinet. Did Manuger Johnson cause the problem??
The office ax- sistant taking the notepad is a normal event because they followed policy.
Taking notepads is ex- pected behavior among both office assistants and munagers according to the rules set by the exec- usive assistant. 'Therefore, the cauxal event of both the office assistant and Manager Johnson taking notepads is normal. (in- correct for managers, this is a norm violation)
Taking a notepad from the ex- ecutive assistant's cabinet is abnormal for the office assis- tant in this situation because they contributed to the prob. lcm of the executive assistant having notepads left, even though it is typically allowed for them take notepads, However. it is important to note that managers action of taking notepad is even more abnormal and against the established nules. (incor- rect- the normality of the office assistant taking the pen is independent of the outcome)
has problem.
Mark was about go pick up his daughter from daycure in the city. In the parking lot, be stopped to chat with friends. Because of this delay picking up daughter. Mark's friend, who waited 15 min for Mark to arrive, took child home his was hit by reekless driver. Mark's daughter, the only scriously injured victim, suffered severe arm injuries. Did Mark cause his daughter's 's injury?
While stopping to talk to friends normal,
Stepping talk with friends in a parking lot ing so to the point of heing late and causing others to is a social norm vie- lation. The cause Mark's 's being late thereffoee ahnormal.
Marking stopping to talk with friends in the parking lot normal event peo- ple often engage in casual conversations when they en- counter acquaintances. It is not unexpected. unlikely, or violation of social norms. (incorrect)
a commen and ex- pected behuvior, making this causal event quite normal. (incorrect)
way home, friend's 's
Table 13: Comparative assessments of normality between gpt-3.5-turbo and gpt-4. Stories taken from the BIG-Bench causal judgments task.
5 A New Frontier for Causality
Our results on evaluating large language models for causal DAG generation, counterfactual reasoning, and token causality demonstrates that they bring significant new capabilities across wide range of causal tasks. We believe that these capabilities signal the beginnings of a new frontier and will transform the practice and research of causality. We start by summarizing key takeaways on how to interpret the new capabilities that LLM bring to causality. We then provide implications for practitioners and future research directions at the intersection of LLMs and causality.
What is new with causality and LLMs?
LLMs provide access to domain knowledge that was, heretofore, only available via human domain experts. LLMs can provide this domain knowledge when explicitly asked, and also implicitly, e.g., as we see in background knowledge necessary for token causality vignettes.
L.LMs provide flexible, natural language-based interaction mechanism for conducting causal analysis that can work alongside existing tools, thus democratizing access to causal tools and knowledge.
29

---

LLMs offer a new capability to extract the key primitives of an token causality question (necessity, sufficiency, normality, etc.). This opens up the possibility of building systems based on formal token causality research for practical scenarios for perhaps the first time.
It is important to recognize that answering causal questions require iterative steps in practice. LLMs can potentially en- able a wide and effective adoption of causality in practice and break the barriers between different research communities related to causality.
What not changing with causality and LLMs?
Especially for high-risk and high-value tasks, people have long relied on rigorous analyses in various formal frameworks to ensure correctness of their decision-making. The need for rigorous, well-documented, and verifiable analyses has not changed. This implies that LLM-driven applications must have tight integration with more formal approaches to causal reasoning. LLMs should be able to fluidly connect their language generation with the use of formal tools, including re-framing or translating natural language scenarios to formal descriptions, and ensuring their natural language responses are in accordance with the conclusions of formal analyses.
5.1 Implications for Causality Practitioners
Augmenting human expertise with LLMs. LLMs can enable effective human-Al interaction that reduces human burden during causal analyses, while ensuring trustworthiness and human verifiability. One of the key reasons causal methods are not widely deployed is because using them requires expertise in expressing the causal assumptions formally through a graph and then verifying them through robustness checks. LLMs can act as assistant in both these processes, augmenting the capability of a practitioner and helping reduce friction in starting causal analysis.
Given the challenging task of algorithmic causal discovery, most practitioners start their causal analysis by constructing graph manually based on their domain knowledge. Rather than creating a graph from scratch, we recommend that practitioners input their dataset metadata to an LLM and obtain an initial version of the graph that they can iterate on. At least for medium sized graphs, our results indicate that LLMs can match sate-of-theearr accuracy in learning real-world graphs, as long as there is descriptive metadata available for each variable. Doing so can save time since editing a graph can be faster than coming up with all edges. Alternatively, given that LLMs obtain their best accuracy in pairwise causal discovery tasks, another way is use LLMs to critique human- generated graphs. LLMs can be used to iteratively critique the presence or absence of edges and help the analyst discover any missing edges.
Apart from graph creation, the other challenging task to validate the output of any causal analysis, since the output depends on the initial causal assumptions (e.g., the conditional independence assumptions implied by the graph). Here LLMs can act as useful assistants and help the analyst through chat conversation towards finding any flaws in the research and/or planning robustness test. LLMs can also help in suggesting specific robustness checks for causal analysis, such as the variables to use as negative controls. In Suppl. F2 we present case study of a conversation with an LLM to identify negative controls for an observational study of vaccine efficacy.
Applying LLMs to novel scenarios. We evaluated LLMs on a combination of known, popular datasets (that are likely to be in LLMs' training data) and datasets developed after LLM's training cutoff date (that are not present in the LLMs' training data). On both kinds of datasets, we obtain similarly high accuracies across different causal tasks, indicating that the presence of the specific dataset in the LLMs' training data is not a prerequisite for doing well on a causal task. In our experiments with datasets outside the LLMs training data, we expect that the LLM's capabilities are result of transforming relevant domain knowledge extracted from other, older texts in the training data.
We do not expect an L.M to construct causal assumptions or causal graphs on new concepts beyond what is included in its training corpus. That said, even analyses of novel causal relationships generally occur in the context of other, better understood causal mechanisms. Therefore we expect practitioners to benefit from using LLMs for inferring at least a subset of cause-effect relationships real-world datasets. For scenarios entirely outside an LLM''s training corpus, the use of retrieval augmented generation presents an opportunity for future research.
30

---

The correctness of causal assumptions is critical for the validity of causal analyses, e.g., of numerical data for scientific discovery or other novel scenarios. Until now, the sole advice for ensuring the correctness of causal assumptions was to advise practitioners to take care and consult with domain experts. The construction of causal assumptions or causal graphs and the criticism of practitioner's assumptions based on LLM's output presents significant advance in technological support for this critical task.
LLM + Causal tools: A new pipeline for causal analysis While LLMs can help with framing and validating causal assumptions-thhe steps in a causal analysis where human expertise is central-they are not capable of replacing the tools that do statistical analysis for estimating causal quantities. These tools (e.g., DoWhy, EconML, Ananke) implement principled algorithms from graphical causal models and the potential outcomes literature and are workhorse for a causal analyst. We envision that LLMs can enable an end-to-end pipeline for causal inference where the code control seamlessly transfers between an LLM and causal tools. Practitioners can construct pipelines where graph creation is done in LLM-sssisted interfaces while the final output graph is passed in statistical inference tool. Given a graph, LLM may also be used to generate the code for conducting downstream analysis with specified causal tool. Based recent research (Schick et al. 2023), it may also be possible to augment LLMs and enable them to call out relevant causal tools and return the answer in a conversational chat interface. Suppl. F.T presents a case study conversation with GPT-4 asking it to create sample code for synthetic data generation and causal analysis using the PyWhy open-source libraries.
LLMs as a fluid conversational interface, merging covariance- and logic-based reasoning. In practice when tackling complex causal challenges, people strategically alternate between different modes and methods for causal reasoning. People may begin with logical reasoning approach frame high-level problem, then toggle to covariance based approach to solve subtask. They may switch to mental simulation to consider counterfactuals, or validate consequences against logical assumptions. And they may continue interchanging complementary analytical methods repeatedly before honing in on their final answer. This kind of fluid iteration between modes of thinking to date required humans to translate their premises and knowledge across frameworks. Now, LLMS may be able to do more of that.
If so, this may allow merging of the many kinds of causality into a single higher-level abstraction, allowing people to ask and have answered much broader and higher-level set of causal questions than could have been addressed through computational methods previously. A first step towards this goal can be to include the information about variables into the problem of generating causal DAGs.
5.2 Ethical Implications and Societal Impacts
Causal modeling is integral to Al interpretability and fairness. It provides explanations behind an Al's predictions individual cases, enhancing interpretability and faireess in AI meaking processes. However, without stringent understanding and control over how L.MSs formulate causal arguments, there's risk of these models being manipulated to produce seemingly valid yet misleading explanations for its explanations. Such mistepresentations could falsely attest to the fairness of a decision or rationalize its basis, potentially leading to misplaced trust and unjust outcomes under the guise of algorithmic neutrality and causal rigor.
The field of causal inference has focused on theoretical and empirical analysis of real-world causal mechanisms, including in high-stakes areas like policy-making and healthcare. Despite its numerical basis, this field is not immune to subjective influences. For instance. causal analysis of how 'race" affects 'wealth' relying on rigorous causal identification and high quality data would still depend on definition of race randomize in experiment and which varies across cultures and history. One of the strengths of traditional causal inference analysis is that the subjective elements of the analysis are explicit in the specification of the model, such as the choice of variable definition and the edges in DAG. When these assumptions and biases are explicit, they can be scrutinized. But with LLMs, these assumptions and biases, shaped by diverse and sometimes highly problematic internet data, are hidden the black box of learned weights in a transformer architecture. Even these weights are often concealed behind proprietary boundaries. This lack of transparency could lead the misconception that LLMs' causal
nebulous concept impossible to
31

---

claims are unbiased due to their algorithmic nature. This is concerning given claims that LLMs learn world models (i.e., causal models). Indeed, an L.M can be fine-tuned to reflect reflect any viewpoint (e.g., vaccine skepticism). It is imperative that in researching the causal capabilitiis of LLMs, we avoid creating a veneer of causal infallibility.
5.3 New research questions on causality and LLMs
Our work a proof of concept that LLMs hold the potential to provide domain knowledge and enable natural and flexible interactions for causal tasks. It will require further advances enable effective human-Al interaction realize this potential.
Knowledge-basee causal DAG generation. We acknowledge that our formulation of knowledge-basse based pairwise edge inference is different from the standard formulation. However, given that LLMs demonstrate non-trivial performance of inferring causal relationships based on variable names alone, we believe that this can have substantial implications research on algorithmic causal discovery and, more generally, causal model specification. Specifically, these results beg the question of how we can reformulate the problem of specifying causal DAG that best leverages large language models and existing knowledge in large amounts of texts.
Graphical causal models require the modeler to specify a causal graph. They do this using domain knowledge about which variables in the domain cause witch i.e., the modeler leverages meta-information about the variables to construct the graph. In contrast, the canonical problem formulation of causal discovery aims to address this task with algorithms that attempt to reverse engineer the causal graph using covariance in observational or experimental data. The attractiveness of this approach is that relying statistical evidence in the data helps avoid subjective model misspecification errors. However, this approach completely ignores meta-information about variables. Our results indicate that this may be a mistake since many datasets provide both data values and meta-information that can be useful for inferring at least a part of the full graph. One promising direction to consider the output of an LLM as prior for a causal discovery algorithm. The relevant question is to what extent having an LLM-based prior would increase the accuracy of causal discovery algorithms, where (Choi et al. 2022) present some promising results. Another, more provocative direction is to recognize that current discovery algorithms struggle real- world datasets and we can aim for redefining the causal discovery problem such that it includes the meta-information about variables and the existing knowledge encoded through LLMs (Zhiheng et al. 2022). Developing new discovery algorithms that make use of both meta-information and covariance in the data can be a fruitful research direction, wherein the LLM may be used as a prior, as a critic during learning, and/or as a post-processon
In addition to building new algorithms, LLMs can also facilitate building benchmarks for causal discovery algorithms. As illustrated from our results, causal discovery benchmarks are usually created by close collaboration with domain experts, with a strong assumption that domain experts know the groundtruth causal graphs. However, this assumption does not hold because 1) Possible edges grow superlinearly with the number of variables, so it is unlikely for human expert to go over all possible edges reliably; 2) More importantly, domain experts may not be able to precisely provide the causal link between two variables given their knowledge. LLMs can help identify potentially missing edge and correct mislabeled edge, or at least point out relations that require additional checking.
LLM-guided effect inference. Identifying the correct adjustment set (c.g., using the backdoor criterion) is one of the most important problems for causal effect inference. However, the validity of the backdoor criterion depends on access to the true graph. LLMs' capability for generating causal DAG can be useful for effect inference and opens up a research question on how to use LLMs to infer valid backdoor sets. In addition to discovering the full graph, techniques that utilize partial graph strueture (Entner et al., 20133] Cheng al., 2022) to identify backdoor sets may also benefit from the ability of knowledge-be -bsed DAG generation with LLMs to infer specific edges. Similarly, it may be interesting to see if LLMs can be used to suggest potential instrumental variables for a causal task, given metadata about observed variables.
A second promising direction is in utilizing LLM's domain knowledge to help build robustness and validation checks for a given causal analysis. Typically, this involves an analyst selecting appropriate robustness checks given a problem and then reasoning over the background context to identify the correct variables use for the check. Can LLMs be
32

---

used for determining which robustness tests may be applicable for a given problem and which specific variables are most suited to be used in those tests?
Systematizing token causality and attribution. The ability to support token causal inference is one of the most potentially disruptive capabilities of LLMs. In domains ranging from law to intelligence analysis, analysts have to first summarize key elements of text data from multiple sources, then synthesize explanations about the degree to which events contributed to other events, i.e., explanations of why and how things happened. While it is well known L.L.s excel the former task, using LLMs for token causality presents a major opportunity to impact the latter. Using LLMs for token causality also has potential impact to root cause analysis and credit assignment in domains ranging from engineering to reinforcement learning.
Formal causal models for inferring token causality questions have struggled formalizing the many elements of common sense background knowledge that rely on when judging token causality (Halpern) 2016: Icard et al. 2017; Knobe 2003 Henne et al., 2017). We'v ve argued that LLMs can work with these background concepts directly in natural language, sidestepping the challenge of shoehorning them into formal model.
The BIG-Bench causal judgment dataset (Suzgun et al., 2022) is ideal for testing LLMs ability answer practical token causality questions. LLMs have performed less well on this benchmark relative to best human performance compared to the rapid saturation of other benchmarks. However, answering these types of questions involves reasoning over commonsense background knowledge concepts, such as necessity, sufficiency, and normality. Our analysis shows that LLMS do better at answering questions about these basic elements than answering high-level token causal judgment questions directly. A promising direction of research is developing methods that guide LLMs to use these token causality primitives to answer higher level token causal judgment questions, perhaps using formal token causality theory as a guide.
Human-LLM collaboration. A recurring theme throughout the paper is the promise of human AI collaboration in causal problems. There are interesting research questions how best facilitate such human-LLLM collaboration for building graphs. The interaction will likely be iterative for the maximum impact: LLMs may suggest graph edges, take feedback from human, and also can give feedback on a manually generated graph. Given that building a graph is common impediment to doing causal analyses, an improvement in this task will have important consequences for the widespread adoption of causal data analysis.
Understanding and improving how LLMs generate causal arguments. Why LLMs demonstrate such causal capabilities, as well as the limits of these capabilities is not yet well understood. The LLMs certainly do well in interpolating between causal language in its training data to answer interventional and causal language. The question is whether they are limited to such interpolation Willig et al. (2023) or if they can learn "world model" (i.e., a catholic causal model) Hao al. (023). According to the causal hierarchy theorem (Bareinboim al. 2022), an LLM can only make interventional and counterfactual inferences outside of its training data with specific interventional and counterfactual inductive biases. Questions of whether LLM entails such causal inductive biases, what exactly they might be, and for what types of queries can they be relied upon will require further study. At the same time, we see unpredictable failure modes, even in tasks where LLMs obtain high accuracy. In summary, understanding the nature of causal reasoning in LLMs and how to improve their robustness is a key future question.
6 Conclusion
Human domain knowledge has always been central piece in causal analysis. In this paper, we studied the capabilities of large language models and found that LLMs can provide value by mimicking that domain knowledge, as a result of being trained on vast amounts of human-generated text. This mimicking is a result of a complex training process so it is not explainable and neither predictable: an LLM can fail on some queries while successfully building a causal arguments in others. What is remarkable is how few times that such errors happen: our evaluation finds that on average, L.LSs can outperform state-of-the-art causal algorithms graph discovery and counterfactual inference, and can systematize nebulous concepts like necessity and sufficiency of cause by operating solely on natural language input,
33

---

From research perspective, these results open up more questions than they answer, and we provide list of research questions at the intersection of LLMS and causality. At the same time, due to the demonstrated capabilities of LLMs, we foresee a marked impact of LLMs on the practice of causal analysis. We outlined how L.Mss can help reduce the burden of human expertise tasks like building the causal graph, effect inference, and attribution. Another contribution of LLMs is in bridging the divide between covariance- and logic-based causal analysis. By providing flexible natural language interface for answering causal queries, LLMs can serve to unify these two branches of causal analysis and allow analyses that seamlessly straddle both to answer real-world causal questions.
Acknowledgments
We thank all the people who gave us feedback on earlier versions of this work, as well as the thought-provoking questions and comments during seminars that helped refine our ideas. We also thank the anonymous reviewers whose comments helped improve the paper's contributions.
References
Ahmed Abdulaal, Nina Montana-Brown, Tiantian He, Ayodeji Ijishakin, Ivana Drobnjak, Daniel C Castro, Daniel C Alexander, et al. Causal modelling agents: Causal graph discovery through synergising metadata-and data-driven reasoning. In The Twelfth International Conference on Learning Representations, 2023.
AML-BABEL.
https: / /aml wttllbbaell..... caabel..com/benchmar.. dateaeets/biippenchhhaa/ce//ee/e jutigeeettnedeeeeennt accessed 2023-04-25.
Aml-babel benchmark: Causal judgement.
James W Atchison and Heather K. Vincent. Obesity and low back pain: relationships and treatment. Pain Management, 2(1):79-86, 2012.
Elias Bareinboim, Juan D Correa, Duligur lbeling, and Thomas Icard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and causal inference: the works of judea pearl, pp. 507-556. 2022.
Emily M Bender, Timnit Gebru, Angelina McMillan- Majaor, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability; and transparency, pp. 610-623, 2021.
Sebastian Bordt, Harsha Nori, Vanessa Rodrigues, Besmira Nushi, and Rich Caruana. Elephants never forget: Memorization and learning of tabular data in large language models, 2024. URL http:ttp://axxi.oo//////// /bs// 2440.06200
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
Kailash Budhathoki, Dominik Janzing, Patrick Bloebaum, and Hoiyi Ng. Why did the distribution change? In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 1666-1674. PMLR, 13-15 Apr 2021.
Ruth M. Byrne The Rational Imagination: How People Create Alternatives to Reality. MIT Press, Cambridge, MA. 2005.
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 2022.
Winston Chang.
Connecting counterfactual and physical causation. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 31, 2009.
34

---

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Verbert-Vosss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
Debo Cheng, Jiuyong Li, Lin Liu, Kui Yu, Thuc Duy Le, and Jixue Liu. Toward unique and unbiased causal effect estimation from data with hidden variables. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Hana Chockler and Joseph Y Halpern. Responsibility and blame: A structural-model approach. Journal of Artificial Intelligence Research, 22:93-115, 2004.
Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. Lmpriors: Pre-trained language models as task- specific priors. arXiv preprint arXiv:2210.12552 2022.
Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state of explainable ai for natural language processing, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers),
pp. 4171 2019.
Doris Entner, Patrik Hoyer, and Peter Spirtes. effects. In Artificial intelligence and statistics, pp. 256-264. PMLR, 2013.
Data-driven covariate selection for nonparametric estimation of causal
Jörg Frohberg and Frank Binder. CRASS: A novel data set and benchmark test counterfactual reasoning of large language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2126-2140, Marseille, France, June 2022. European Language Resources Association. URL httts:://cclatthol.o
org/2022, 11e.22.reec1.22
Tobias Gerstenberg, Noah Goodman, David Lagnado, and Josh Tenenbaum. From counterfactual simulation to causal judgment. Proceedings of the annual meeting of the cognilive science society, volume 36, 2014.
Ahmad Ghazal, Todor Ivanov, Pekka Kostamaa, Alain Crolotte, Ryan Voong, Mohammed Al-Kateb, Waleed Ghazal, and Roberto V Zicari. Bighench v2: the new and improved bigbench. In 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pp. 1225-1236. IEEE, 2017.
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019.
Joseph Y Halpem. Actual causality. MiT Press, 2016.
Joseph Y Halpem and Christopher Hitchcock. Graded causation and defaults. The British Journal for the Philosophy of Science, 2015.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023.
Daniel Murray Hausman. Causal relata: Tokens, types, or variables? Erkenntnis, 63(1):33-54, 2005.
Mary Hegarty. Mechanical reasoning by mental simulation. Trends in cognitive sciences, 866):280-285, 2004.
35

---

Jan Hellner. Causality and causation in law. Scandinavian studies in law, 40:111-134, 2000.
Paul Henne and Kevin O'Neill, Double prevention, causal judgments, and counterfactuals. Cognitive science, 46(5): e13127, 2022.
Paul Henne, Ángel Pinillos, and Felipe De Brigard. Cause by omission and norm: Not watering plants. Australasian Journal of Philosophy, 95(2):270 2017.
Miguel A Hernán and James M Robins. Causal inference, 2010.
Christopher Hitchcock, Portable causal dependence: tale of consilience. Philosophy of Science, 79(5)942-951, 2012.
Marius Hobbhahn, Tom Licberum, and David Seiler. Investigating causal understanding in 1lms. In NeurlPS ML Safety Workshop, 2022.
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems, 21, 2008.
Yiyi Huang, Matthäus Kleindessner, Alexey Munishkin, Debvrat Varshney, Pei Guo, and Jianwu Wang. Benchmarking of data-driven causality discovery approaches in the interactions of arctic sea ice and atmosphere. Frontiers in Big Data, 4. 2021. ISSN 2624-909X.
doi: 10.33389/cdaaaa220114442213 URL httpt::t////// www.ffrontiersi....
art wwciicees//1 Fdatt//////aaa/e 2021 642182
Thomas Icard and Noah D Goodman. resource-rational approach to the causal frame problem. In CogSci, 2015,
Thomas F Icard, Jonathan F Kominsky, and Joshua Knobe. Normality and actual causal strength. Cognition, 161:80-93, 2017.
Guido Imbens and Donald Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
Marc Jeannerod. Neural simulation of action: a unifying mechanism for motor cognition. Neuroimage, 14(1): $103-S109, 2001.
Daniel Kahneman and Dale T. Miller. Norm theory: Comparing reality to its alternatives. Psychological Review, 93(2): 136-153, 1986.
Marcus Kaiser and Maksim Sipos. Unsuitability of notears for causal graph discovery when dealing with dimensional quantities. Neural Processing Letters, 5483)15577 2022.
Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Available at SSRN 4389233, 2023.
Joshua Knobe. Intentional action and side effects in ordinary language. Analysis, 63(3): 2003.
Joshua Knobe and Ben Fraser. Causal judgment and moral judgment: Two experiments. Moral psychology 2:441-447, 2008
Joshua Knobe and Scott Shapiro. Proximate cause explained. The University of Chicago Law Review, 88(1):165-25) 2021.
Jonathan F Kominsky, Jonathan Phillips, Tobias Gerstenberg, David Lagnado, and Joshua Knobe. Causal superseding. Cognition, 137:196-209 2015.
Eliza Kosoy, David M Chan, Adrian Liu, Jasmine Collins, Bryanna Kaufmann, Sandy Han Huang. Jessica Hamrick, John Canny, Nan Rosemary Ke, and Alison Gopnik. Towards understanding how machines can learn causal overhypotheses. arXiv preprint arXiv:2206.0823 2022.
36

---

Konstantin Raphael Kueffner. A comprehensive survey of the actual causality literature. 2021.
Peter Lee, Sebastien Bubeck, and Joseph Petro. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New
England Journal of Medicine, (8) 12399, 2023a.
Peter Lee, Carey Goldberg, and Isaac Kohane. The AI Revolution in Medicine: GPT-4 and Beyond. Pearson, 1 edition, 2023b. ISBN ISBN-13: 38201182
Jiaxuan Li, Lang Yu, and Allyson Ettinger. Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pp. 804-815, 2023.
Mare Lipsitch, Eric Tchetgen Tchetgen, and Ted Cohen. Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology (Cambridge, Mass.), 21(3):383, 2010.
Hammeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4, 2023.
Tania Lombrozo. Causal-cxplanatton pluralism: How intentions, functions, and mechanisms influence causal ascriptions. Cognitive psychology, 61(4):303-333 -300003-333333000 2010.
Stephanie Long, Tibor Schuster, Alexandre Piché, ServiceNow Research, al. Can large language models build causal graphs? arXiv preprint arXiv:2303.052799 2023.
Gary Mareus. How come gpt can seem so brilliant one minute and so breathtakingly dumb the next?, 2022. URL attps://garymarcus.
www.aarrusuuuuussaa.ee.e suwst com/
Alexander Marx and Jilles Vreeken. Telling cause from effect using mdl-based local and global regression. In 2017 IEEE international conference on data mining (ICDM), pp. 307-316. IEFE, 2017.
Steven Mascaro, Yue Wu, Owen Woodberry, Erik P Nyberg, Ross Pearson. Jessica A Ramsay, Ariel O Mace, David A Foley, Thomas I. Snelling, Ann E Nicholson, et al. Modeling covid-19 disease processes by remote elicitation of causal bayesian networks from medical experts. BMC Medical Research Methodology, 23(1):76, 2023.
Joris M. Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distinguishing cause from effect using observational data: Methods and benchmarks. Journal of Machine Learning Research, 17(32): 1-102, 2016. URL http::///mm.rorrrr.ooo.oo /ttp../ogg/ppppss///////////////
htmll
Michael S Moore. Causation and responsibility: An essay in law, morals, and metaphysics. Oxford University Press on Demand, 2009.
Thomas Nadelhoffer. Desire, foresight, intentions, and intentional actions: Probing folk intuitions. Journal of Cognition and Culture, 6(1-2):133-157, 2006.
Meike Nauta, Doina Bucur, and Christin Seifert. Causal discovery with attention- based convolutional neural networks. Machine Learning and Knowledge Extraction, 1(1):312-300, 2019.
Nhan Nguyen and Sarah Nadi. An empirical evaluation of github copilot's code suggestions. Proceedings of the 19th International Conference on Mining Software Repositories, MSR '22, pp- 1-5, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393034, doi: 10. 10.114573524822.35247777 URL
https:///oo..oxg/10.. 1tt.://011111111/5544452222 3528470)
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXhv:2303.13375, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
37

---

Judea Pearl. Causal inference in statistics: An overview. 2009a.
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, edition, 2009b.
Judea Pearl. Causality. Cambridge university press, 2009c.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Jonathan Phillips, Jamie B Luguri, and Joshua Knobe. Unifying morality's influence on non-moral judgments: The relevance of alternative possibilities. Cognition, 145:30-42, 2015.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Can- cedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint WwXiv:2302.04711, 2023.
Amit Sharma, Vasilis Syrgkanis, Cheng Zhang, and Emre Kıeıman. Dowhy: Addressing challenges in expressing and validating causal assumptions. arXiv preprint arXiv:2108. 13518, 2021.
Amit Sharma, Hua Li, and Jian Jiao. The counterfactuullshapley value: Attributing change in system metrics, arXiv preprint 2tt.::::22000899999 2022.
Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
Sanchit Sinha, Hanjie Chen, Arshdeep Sekhon, Yangfeng Ji, and Yanjun Qi. Perturbing inputs for fragile interpretations in deep natural language processing, 2021.
Steven A Sloman and David Lagnado. Causality in thought. Annual review of psychology, 66:223 227,2227,122 2015.
Gregory T Smith. On construct validity: issues of method and measurement. Psychological assessment, 17(4):396, 2005.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 0tfi::200600461555 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Natasa Tagasovska, Valérie Chavez-Demoulin and Thibault Vatter. Distinguishing cause from effect using quantiles: Bivariate quantile causal discovery. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9311-9323. PMLR, 13-18 18 Jul 2020. Utt/tttts///prroeeddi ottpeeeeeeeeeaoooo..ooooeer///// 1ttps////111/taassssssa200000 html
ml www.s//ssvvvvv.
Svetlana Tomić, Silva Soldo-Butković, Biserka Kovač, Dario Faj. Stjepan Jurié, Sanja Mišević, Lidija Knežević, and Darko Vukašinović. Lumbosacral radiculopathy-fant factors effects on it's severity. Collegium antropologicum, 33(1): 175-178, 2009.
Ruibo Tu, Kun Zhang, Bertilson, Hedvig Kjellstrom, and Cheng Zhang. Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation. Advances in Neural Information Processing Systems, 32, 2019.
38

---

Ruibo Tu, Chao Ma, and Cheng Zhang. Causal- performance of chatgpt in the context of neuropathic pain diagnosis. arXiv preprint arXiv:2301. 13819, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 2022b.
Moritz Willig, Matej Zečević, Devendra Singh Dhami, and Kristian Kersting. Can foundation models talk causality?
arXiv preprint arXiv:2206, 10591, 2022.
MORITZ Willig, MATEJ ZEČEVIĆ, DEVENDRA SINGH DHAMI, and KRISTIAN KERSTING, Causal parrots: Large language models may talk causality but are not causal. 2023.
James Woodward. Sensitive and insensitive causation. The Philosophical Review, 155(1)1-50, 2006.
Pengzhou Wu and Kenji Fukumizu. Causal mosaic: Cause-effect inference via nonlinear ica and ensemble method. In International Conference on Artificial Intelligence and Statistics, pp. 1157-1167. PMLR, 2020.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning, pp. 7154-7163, PMLR, 2019.
Cheng Zhang. Stefan Bauer, Paul Bennett, Jiangfeng Gao, Wenbo Gong, Agrin Hilmkil, Joel Jennings, Chao Ma, Tom Minka, Nick Pawlowski, and James Vaughan. Understanding causality with large language models: Feasibility and opportunities, 2023.
Kun Zhang and Lai- wan Chan. Extensions of ica for causality discovery in the hong kong stock market. In Neural Information Processing: 13th International Conference, ICONIP 2006, Hong Kong, China, October 3-6, 2006. Proceedings, Part III 13, pp. 400-409. Springer, 2006.
Kun Zhang and Aapo Hyvärinen. Causality discovery with additive disturbances: An ceabooommation-hhhorttte perspective. In Machine Learning and Knowledge Discovery Databases: European Conference, ECML PKDD 2009, Bled, Slovenia, September 7-11, 2009, Proceedings. Part 20, pp. 570-585. Springer, 2009.
Kun Zhang and Aapo Hyvarinen. On the identifiability of the post-nonlinear causal model. arXiv preprint arXiv:1205.2543 2012.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018.
LYU Zhiheng, Zhijing Jin, Rada Mihalcea, Mrinmaya Sachan, and Bernhard Schölkopf. Can large language models distinguish cause from effect? In UAI 2022 Workshop on Causal Representation Learning, 2022.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:221.001910, 2022.
39

---

ball track cause A Causal graph generation: Additional details
Model version. For all experiments, we use gpt-3.5-turbo-06f and gpt-4-0613 model versions.
A.1 Example prompts for the Tubingen benchmark
Two prompts per pair
Template:
Does changing (A) cause a change in (B)? Please answer a single word: no.
Does changing (B) cause change [A]? Please answer in a single word: yes no.
Examples:
Does changing the altitude cause change in temperature?
Please answer in a single word: yes or no.
Does changing the temperature cause change in altitude? no.
Please answer in a single word:
Does changing ball's speed at the beginning of ball track cause change its speed at end of the track? Please answer in single word:
no.
Does changing ball's speed at the a change in its speed at the heginning of the track? Please answer a single word: yes or no.
Single prompt
Template:
Which cause-and-effect relationship is more likely? changing [A] causes change in (B).
B. changing (B) causes change in (A).
Let's work this in step by step way be that we have the right answer. Then provide your final answer within the tags <Answer> </Answer>. Example:
Which cause-and-effect relationship is more likely?
A. changing the altitude causes change in temperature. changing the temperature causes a change altitude.
Let's work out a step by step way to sure that we have the right answer. Then provide your final answer within the tags <Answer>>/PP </Awwerr>.
Table 14: Two kinds of prompt templates for the Tübingen benchmark. The first asks two questions per variable pair whereas the second ("Single prompt") asks single question to orient each pairwise edge.
A.2 Example LLM outputs for the Neuropathic pain dataset
An example correct output from gpt-3.5- 3pt-3.5-
LLM:
First, we need understand terms involved in cause-and aouee--nd--ffeee effect relationship.
Left T6 Radiculopathy refers to a condition where there is damage irritation to the nerve root that exits the spinal cord at level T6 on the left side the body.
40

---

DLS T5-T6 refers to degenerative changes in the discs, ligaments, and/or bones at the level of T5-T6 in the spine.
Now, consider the two options:
A. Left T6 Radiculopathy causes DLS T5-TO, This would mean that the damage or irritation to the nerve root at T6 on the left side the body somehow leads to degenerative changes the discs, ligaments, and/or bones the level of TS-T6. While it is possible that nerve damage could contribute to degenerative changes in the spine, it is more likely that the opposite is truc.
B. DLS T5-T6 causes Left T6 Radiculopathy. This option suggests that the degenerative changes in the dises, ligaments, and/or bones the level T5-T6 are causing the Left Radiculopathy. This is a more likely cause-and- and-effect relationship. degenerative changes in the spine can lead to nerve root irritation compression.
Therefore, the more likely cause-and-effect relationship is: <Answer-B</Asweer-DDb T5-T6 causes Left T6 Radiculopathy.
An example of an incorrect output.
LLM:
First, we need to understand what DLS T5-T6 and Right T6 Radiculopathy are. DLS stands for degenerative lumbar spondylosis, which is condition where the spinal discs in the lumbar region of the spine degenerate over time. T5-T6 refers the thoracic vertebrae there is compression irritation of a nerve root, which can cause pain, numbness, or weakness in the area of the body that the nerve supplies.
middle of spine. Radiculopathy is condition where
Based on information, is more likely that option is the correct cause-and-effect relationship. This because radiculopathy is typically caused by compression or irritation of nerve root, which can then lead to pain or other sym ptoms in the area of the body that nerve supplics. In this case, the Right T6 Radiculopathy causing symptoms in the area of the body that corresponds to the DLS T5-T6 region of the spine.
Therefore, the correct answer is <AnswerBB<Answeer
A.3 Novel datasets created after LLMs training cutoff date
Novel Tubingen dataset.
Table 15 shows the novel pairs constructed from recently published books.
Novel Alzheimer's dataset.
Table 16 shows graph discovery accuracy for gpt-3.5-turbo and gpt-4 compared to data-based discovery algorithms such as DAG-GNN and NOTEARS.
Novel Counterfactual dataset. Table 17
shows results of using LLMs like GPT-4 on the novel counterfactual dataset
from et al. (2023). We compare GPT-4 to accuracy obtained by previous models evaluated by Li et al. (2023).
B Evaluation: LLMs and counterfactual reasoning
B.1 Details on CRASS counterfactual evaluation
For computing accuracy, the LLM obtains of 1 if it outputs the correct answer option and 0 if outputs incorrect option. If the LLM output does contain an answer among the options A/B/C, we consider as if the LLM has skipped the question and assign score according to the probability of obtaining the correct answer under random chance (0.33).
B.1.1 Example prompt
Here is an example prompt.
41

---

Category
Varl
Var2
Causal Direction
Book Title
Year
Biology
high lemperature and high shear
protein denaturation
Right
Sustainable Protein
2023
Sources: Advances
for Healthier Tomor-
Biology
amount algae
color of coral
Right
your brains with 900+ Cool
2023
ence
Finance
credit card score
on-time payments
Left
Bounce Back: The Ul- timate Guide Finan- cial Resilience
2023
Medicine
Kidney damage
Cirrhosis
Left
Introduction to Global
2022
Hcalth
Science
grain size reduction building flcxibility
metal strength
Right
Materials Chemistry
2023
Civil Enginocring
bearing displacement
right
Advances in Struc-
2023
tural Dynamics
Medicine
inflammatory bowel discuses
Mesenteric venous thrombosis
Right
Understanding Crime
2023
Through Forensic Sci-
Heealhh
Oral contraceptive pills
weight gain
Right
Balanced Life: Your Ultimate Weight Man- agement Guide Soils for nutrition:
2023
Soil Science
Usage of Nitrogen fertilizer
Soil pH
Right
2022
state of the Battery Management
Technology
battery capacity
ambient temperalure
Left
2022
System and Its Appli- cations
Table 15: A sample of 10 pairs from the Novel Tubingen dataset.
Algorithm NHD
No. of Predicted Edges
Baseline NHD
Ratio
NOTEARS
0.22
10
0.32
0.69
DAG-GNN
0.37
20
0.44
0.83
gpt-3.5-turbo
0.21
21
0.38
0.55
gpt-4
0.14
25
0.48
0.28
Table 16: Results for the novel Alzheimer's s dataset from Abdulall et al. (203)). For data-based algorithms, we report the results for the best configuration. For LLMs, we report results with temperature=0.
Algorithm
Accuracy
BERT
34.2
RoBERTa
61.4
MPNET
66.9
GPT-2
53.7
GPT-3
71.3
gpt-4
88.6
Table 17: Accuracy results for the novel counterfactual dataset. Results for models other than gpt-4 are from Table 2 in Li et al. 22023).
42

---

PROMPT: A woman sees a fire. What would have happened if the woman had touched the fire?
A: She would have seen fire.
B: That is not possible.
C: She would not have been burned.
D: She would have been burned.
Provide your answer within the tags, <AIIser-AAB/CCI//Answww
B.1.2 Example responses from gpt-4
Here list three example responses from gpt-4 that do not match the benchmark answer. In the first examples, the answer from gpt-4 can be considered correct.
SYSTEM: You are helpful assistant for counterfactual reasoning.
USER: A catches ball. What would have happened if the man had caught water balloon?
A: He would have caught the ball and not the balloon.
B: He would have got wet.
C: He would have died.
D: He would have remained dry.
Let's work this out a step by step way to be sure that we have the right answer. Then provide your final answer within the tags, <Answer>A/B/CCD//Anwwer
GPT-4: Step I:
Identify the difference between the two scenarios.
first scenario, the man catches a ball. second scenario, man catches water balloon.
Step Determine the consequences the difference. Catching ball typically does not result in getting wet any other significant consequences. However, catching a water balloon may cause to burst, potentially getting the man wet.
Step 3: Evaluate the possible outcomes.
A: He would have caught the ball and not the balloon. (This outcome does address the counterfactual scenario where the man catches water balloon.)
B: He would have got wet. (This outcome is possible if the water balloon bursts upon being caught.)
C: He would have died. (This outcome is highly unlikely and not reasonable consequence of catching water balloon.)
D: He would have remained dry. (This outcome is possible if the man catches the water balloon without it bursting.)
Step Choose most likely outcome. The most likely outcome is that the man would have either gotten wet or remained dry, depending whether the water balloon burst or not. Since we cannot determine with certainty whether the water balloon would burst, cannot definitively choose between options B and D.
<Answer-Uncertain between B and D</Answer>
SYSTEM: You helpful assistant for counterfactual reasoning.
USER:
does get very nervous. What would have happened if he had gotten very nervous? A: That is not possible.
The man would have passed out.
C: The man would not have passed out.
43

---

Let's work this out in a step by step way to be sure that we have the right answer. Then provide your final answer within the tags, <Anwwer>A/BiCC//nsseerr.
GPT-4:
Step I:
Identify the counterfactual situation The man gets very nervous.
Step 2: Consider the possible outcomes of the counterfactual situation.
The man passes due to extreme nervousness.
b) The man does not pass out but experiences other symptoms of nervousness.
Step Evaluate the likelihood each outcome.
Passing out due to extreme nervousness is possible, but not guaranteed.
b) more likely that the man would experience other symptoms nervousness without necessarily passing out.
Final Answer: <AnswersC</Aswwer>
Below we show an example where people may easily understand that the man walking on his bed indoors, but the LLM incorrectly assumes that he continues walking outdoors on bed towards the same destination.
SYSTEM: You are a helpful assistant for counterfactual reasoning.
USER:
man walks on street. What would have happened man had walked on a bed?
A: He would have been late.
B: Nothing special would have happened.
C: He would have arrived on time.
Let's work this out in step by step way to be sure that we have the right answer. Then provide your final answer within the tags, <Answer-A/BIC</Anwee.
GPT-4: Step Consider the difference between walking on and walking bed. Walking street is a normal activity and allows for efficient movement from one place to other.
Walking bed unusual and would likely slow down person's 's movement the soft and unstable surface.
Step 2: Evaluate cach option based on this difference.
A: He would have been late. This option suggests that walking on bed would have slowed the man down, making him late for his destination.
B: Nothing special would have happened.
This option implies that walking on a bed would not have had any signi impact on the man 's journey.
C: He would have arrived on time.
This option suggests that walking on bed would not have affected the man's arrival time his destination.
Step Determine the most likely outcome. Given that walking a bed is an unusual and inefficient way to move, it is most likely that the man would have been slowed down and potentially arrived late at his destination.
AAnwerrA</Answerr
C Evaluation: Necessary and Sufficient Causes
C.1 Classes of token causality vignette scenarios
For discussion of the different scenario classes, see Chapter 3 in Kueffinr (2021). Below we provide brief summary.
Symmetric Overdetermination: Multiple processes that lead the same outcome, complete at the same time.
44

---

2. Switch: An action chooses between multiple processes, each of which lead to the same outcome. Thus, the action becomes immaterial for the outcome.
Late preemption: Two processes are running in parallel. Both would produce the same outcome, but one completes before the other.
Early preemption: Two processes would produce the same outcome, but one completes before the other has started.
5. Double preemption: A focal process leads to event. An action that would have stopped the focal process, is itself stopped by another process.
6. Bogus preemption: An action stops a focal process, but the focal process itself is inactive wrt its effect on the outcome.
Short circuit: original action makes the focal process inactive. An action is taken to prevent this inactive process; triggers the process but the process is unable to complete because of the original action.
C.2 Prompt generation for necessary and sufficient causes
Given the complicated definitions of token causality (see, for example, definitions of "actual causality" in Chapter 2 Halpeeem (2 it is non-trivial express them in natural language. We thus turned to the L.M itself to suggest prompts to use.
Below we describe a method to generate an appropriate prompt for given task, based on Zhou et al. 2022). For generating a prompt that instructs an L.M to identify the necessary cause, we provide a meta-prompt that gives a few demonstrations of the task (vignettes as input and the necessary cause as output) and then asks the LLM to generate the instruction for that task. We then use paraphrased version of the LLM's instruction create our prompt.
Specifically, provide three example vignettes with the scenario and correct token cause and ask the LLM to output the instructions that must have led to the output in each example. After few executions, the LLM provided reasonable principle called "minimal rewriting". We tweaked the wording to the more common "minimal change" principle and use it in our prompt.
Here is the meta-prompt and its response from gpt-3.5-turbo.
PROM-P:
taught my friend some concepts in counterfactual reasoning. Based on my teaching, they provided the following output, given an input scenario and candidate outputs.
Input: A window shatters when it is hit by a bullet. Alice and Bob each fire bullet at window, simultaneously striking the window, shattering it. Candidate outputs: Both Alice and Boh, Any of Alice Bob
Output: Both Alice and Bob
Input: Alice, Bob, Carol, Dave and Eve all fire at window. The window shatters after three hits.
Candidate outputs: All five individuals, any individual, the first three individuals who fired
Output: The first three individuals who fired
Input: Alice fills sink with water. each time interval Alice adds another drop of water. At one point the sink overtlows.
Candidate outputs: All droplets added to sink, Only the last droplet added to sink
Output: Only the last droplet added sink
What logic was the friend using to select the correct output?
G-T-3.5-TURBO: seems that your friend was applying the principle of minimal rewriting to select the correct output. The principle of minimal rewriting is criterion used in counterfactual reasoning to evaluate which changes to the initial situation are necessary to obtain given outcome.
45

---

Vignette Type
Inpui Context
Event
Actor
Nec.
Suft.
Overdesermination
There is a fire the chemistry lab, can of water would douse the fire. Agents X and both spray a can of water each, dousing the fire.
lire being doused
Agent X
No
Yes
Switch
Reagent X is added to a mixture, which leads to an explosion and Sam. Otherwise, Reagent in Sam's pocket would have in- fected him and killed him as well.
Sam's is death
Reagent X
No
Yes
Late preemption
Any of Reagent X or Reagent can be added to a mixture to convert into crystal. Reagent X added lirst and the mixture crysial, Reagent Y is added later and but does not mix since the crystal is already formed.
orystal formation
Reagent X
No
Yos
Early preemption
Sam reaches out and catches the falling test tube containing con- taminating reagent.
floor being con-
Sam
No
Yes
next thing on the soft foam mattress which would have prevented the test tube from reaching the floor. Beyond that there was the floor. Sam intends to add reagent crystal. Riya intends to prevent Sam from adding the reagent. Riya stop Sam from adding the reagent, but Riya stopped by Frank. Sam adds the reagent and the crystal is formed.
trajectory was
Double preemption
mixture that would convert
crystal formation
Sam
Yes
No
Bogus preemption
explode. How- ever, Sam does not put the reagent in the mixture. Bob puts an anti-explosion Reagent Y into the mixture. The mixture would explode if Reugent X is adkeed without the addition of Reagent Y. Mixture dres not explode.
intents put Reagent
mixture
mixture not ex-- ploding
Sam
Yes
Short circuit
tube contains mixture. Sam puts sand in mixture. Adding sand to the mixture protects it against explosion. Only Sam puts mixture. Adding mixture makes it explode, unless the mixture already
avoiding explo- sion
Sam
Yes
the mixture, Riya will add Reagent Reagent
contains sand. There is no explosion.
Miscellanesuus
Sam is heating Reagent X test tube. If heat not stopped the right time, the test tube would explode. Sam does not stop the beat. The tube explodes.
explosion
Sam's inaction
Yes
Yes
Table 18: "lab tab-vignettes"; Examples of novel vignettes for evaluation of inferring necessary and sufficient causes. Each vignette is associated with two questions: "Iy (Actor) a necessary cause of [Event]?" and "Is (Actor) a sufficient cause of [Event)?"
In this case, your friend was likely evaluating which minimal changes the input scenario would lead to the candidate outputs. By comparing the input scenario to the candidate outputs, your friend identified which changes were necessary obtain the desired outcome, and selected the output that required the least amount of change.
For example, in the first scenario, your friend identified that both Alice and Bob firing their bullets simalaneously was the minimal change needed to shatter the window. Therefore, they selected the output "Alice and Bob" as the correct answer, required the least amount of rewriting.
Similarly, in the second scenario, your friend identified that the minimal change to shatter the window was the first three individuals firing. Therefore, they selected "The first three individuals who fired" as the correct output.
Overall, your friends approach demonstrates good understanding of counterfactual reasoning and the principle of minimal rewriting.
We follow the same procedure for creating a prompt for identifying the sufficient cause. One of the LLM outputs mentions the "multiple sufficient causes" principle, which we use in our prompt.
C.3 "Lab-vignettes": novel token causality benchmark
To avoid LLM memorization issues, we construct a new dataset for inferring necessary and sufficient causes, based on the available dataset. Specifically, we perturb the details in the original dataset and adapt it for scenarios in a chemistry lab, involving reagents, mixtures and crystals. Table [8 shows example vignettes from this dataset. We apply our memorization test to validate that our newly created dataset cannot be recreated by the LLM.
46

---

C.4 Example responses from LLMs
C.4.1 Correct response on an Early Preemption vignette
We show the answers from gpt-3.5-turbo and GPT-4 on the example prompt main text.
For necessary cause, both models yield the correct answer.
GPT-3.5-TURBOL If Alice had not caught the passing cricket ball, the ball would have hit the solid brick wall and stopped there. Therefore, the window would not have been hit by the ball and would have remained intact. Hence, Alice is not a necessary cause for the window being intact.
<Anwwr>Noo//Aswwer>>>
GPT-4:
To determine whether Alice is necessary cause for the window heing intact, we can apply the principle of minimal change. Let's consider the counterfactual situation where Alice does catch the hall.
In the counterfactual situation, the ball would continue on its trajectory and hit the solid brick wall. As the brick wall would have stopped the ball, it would not have continued towards the window. Therefore, the window would still be intact even if Alice had not caught the ball.
<AnswerrNNo/Answerr>>
For sufficiency, however, only GPT-4 provides the correct answer (Alice's action is sufficient).
GPT-3.5-TURBO:
No, Alice is not a sufficient cause for the window being intact. While Alice catching the ball prevented it from hitting the brick wall, there could have been other factors that prevented the ball from breaking the window, such as the ball losing momentum or the window being made of sturdy material. Therefore, there multiple sufficient causes for the window being intact. <cnnwerrNoc//Asswer..
GPT-4:
Using the principle of multiple sufficient causes, we can reason that there are multiple potential causes that could result in the window remaining intact. In this scenario, Alice catching the ball is one sufficient cause, it prevents the ball from hitting the window. Another sufficient cause would be the solid brick wall, which would have stopped the ball if Alice hadn't caught it.
Given that there are multiple sufficient causes, Alice's 's action of catching the ball is not the sole determinant of the window remaining intact. However, her action is one of the sufficient causes that contributed the outcome.
<Answer> <Answer>Yes</AAswwe
C.4.2 Unpredictable failure mode: An example
Using the short circuit vignette, we show an example of gpt-4's unpredicable failure modes. It applies the correct reasoning for most prompts, but applies an incorrect reasoming (assessing necessity only instead of sufficiency) for the following prompt. We show the input prompt and gpt-4's answer below.
PROMPT: You are an expert in counterfactual reasoning. Given an event, use the principle of minimal change to answer the following question.
Carol alive (CA). Alice puts a harmless antidote in Carol's water (AA). Adding antidote to the water, protects it against poison (WS "water save"). Alice puts the antidote into Carol's water, Bob poison the water (BP) Adding poison to an unprotected water makes it toxic (WT). If Carol would drink toxic water she would die (i.e. inhibiting CS). Carol consumes her water and survives (CS).
Input:
Is Alice necessary cause for Carol's survival?
After your reasoning, provide the final answer within tags Ye/NsweeYe//oo/AAsssee
GPT-4:
We can set up the causal chain follows:
47

---

1. Alice puts the antidote in Carol's water (AA). 2. Antidote in the water protects it against poison (WS). 3. If Alice puts the antidote into Carol's water, Bob will poison the water (BP). 4. Adding poison to unprotected water makes it toxic (WT). 5. If Carol drinks toxic water, she would die (not CS).
To determine Alice is a sufficient cause for Carol's 's survival, let's consider counterfactual scenario:
If Alice had not put the antidote in Carol's water, would Carol have survived?
this scenario, we have:
1. Alice does not put the antidote in Carol's water (not AA).
Antidote the water does not protect against poison (not WS).
3. Bob does not poison the water (not BP).
4. The water remains nontoxic (not WT).
5. Carol consumes her water and survives (CS).
In this counterfactual scenario, Carol still survives even if Alice does not put the antidote in her water. This indicates that Alice's 's action was not a sufficient cause for Carol's 's survival, as Carol would have survived even without Alice's intervention.
<Answer>Noc/Answerr
D Assessing Responsibility
So far we looked at necessary or sufficient causes, which are binary: an event is either cause or not a cause another event. In natural language, the causal judgments we make are relative, e.g., we might ascribe more causality to "smoking in the woods" than to "presence of dry leaves Summer." Formal token causality models capture this with graded causation, nonbinary measures of token causation that enable comparisons between causal events HHalpern & Hitchoock (203)). In natural language we might use the simple concept of "responsibility." Responsibility is natural way to rank causal events in everyday language, e.g., "smoking in the woods was more responsible for the forest fire than the presence of dry leaves in Summer."
Halpern 2016); Chockler & Halper 2004 provide an SCM-based definition of responsibility; in simple terms, they define responsibility of event C for event E is I/(+1),, where is the minimal number other causal events contributing to E that have had not happen to make C necessary cause of But mapping the SCM definition of responsibility back to plain language is nontrivial, just as it was with model-based P(Ec- = 1 0, E 0)-definition of sufficiency back to plain language was challenge. This SCM definition is purely measure of causal contribution. But in everyday language, "responsibility" combines degree of causal contribution with the concepts of intention, epistemic state, moral obligation, etc. mentioned earlier. So for this case study, we attempted to scope the definition of responsibility to focus on causal contribution.
Responsibility: The relative degree to which an event causally contributes to a particular outcome event, relative to other events that caused that outcome. Given two causal events that cause the same outcome event, one causal event can be more, less, equally responsible relative to the other. This definition of responsibility does not depend whether the causal event is person's 's action.
D.0.1 Inferring responsibility: Motivating Example
Taking our cue from the LLM's ability to define relevant variables on the fly, our definition allows the LLM to provide relative ranking of causes, rather than quantify an absolute responsiblity value. Further, we make it clear that responsibility is the same regardless of whether the causal event was person's 's action; we hope that if the definition of responsibility is the same for both inanimate object and people, then we cue the LLM to focus on causal contribution. Moreover, we cue the definition to ignore intention, morality, epistemic state, etc., or at the very least to limit those factors levels that people usually ascribe inanimate objects when they are anthropormorphized in everyday language
48

---

(e.g. "cheated by fate", "the universe is telling you something," "my phone is acting up," etc.) We then prompted the LLM to use this definition for the beer spilling example.
PROMPT: In the context of this dialog, define "responsibility" as the relative degree to which an event causally contributes to particular outcome event, relative other events that caused that outcome. Given two causal events that cause the same outcome event, one causal event can be more, less, or equally responsible relative to the other. This definition of responsibility does not depend whether the causal event is a person's action.
Based on this definition of responsibility, make a one sentence statement about how responsible is Mike's bumping of the table is for the spilling of the heer.
LLM: Based on this definition of responsibility, Mike's bumping of the table is highly responsible for the spilling of the beer, as it was the immediate and direct cause that led to the beer bottle falling and spilling.
We now introduce two challenging scenarios. These are based on overdeteemiiaaiion and double preemption, two of the scenario types we saw Section 4.2 for deciding necessary and sufficient causes.
Overdetermination. Overdetermination is one example where a simple counterfactual dependence definition of actuality falls short relative to human judgments. To illustrate we introduce a new information to the beer spilling story, that Susan bumped into the table shortly after Mike and with enough force to spill the beer had still been there. With this additional information, Mike's bump no longer a necessary cause of the spill; if he hadn't 't bumped the table, the beer still would have spilled. Despite the new lack of counterfactual dependence, humans still judge that Mike caused the spill. Analogously, if two children throw rock at window, most people judge that they both children are window breakers, regardless of whose rock struck first and broke the glass.
First we make sure the LLM can assess that counterfactual dependence is no longer present.
PROMPT:
You said "if Mike hadn't humped into the table, the heer bottle would not have fallen and spilled." Call this statement A. Suppose that Susan tripped and fell onto the table in the instant after Mike bumped into the table and the bottle started falling. If the bottle had not already fallen, Susan's 's fall would have caused the bottle to fall. Assuming scenario A was true before, can statement A still be true? Provide one sentence explanation why why not.
L.M: No, scenario A can no longer be considered true because if Susan fell onto the table in the instant after Mike bumped it, Susan's 's fall would have been the determining factor causing the beer bottle to fall and spill, regardless of Mike's ' initial bump.
This result aligns with our intuition. Now, we expect that, given that there are two causes of the beer spilling, the LLM should be able downgrade Mike's level of responsibility relative to before.
PROMPT:
How, at all, does this new information change your assessment of the responsibility of Mike's table bump?
LLM: With this new information, the responsibility of Mike's table bump for the spilling of the beer is reduced, as Susan's 's fall would have also led to the same outcome, making both events contributing causal factors to the beer spill.
Double prevention. Double prevention describes another case where humans ascribe causality to events that are counterfactually independent of the outcome. In double prevention, have initial causal event C that causes the outcome. As before C is an event that causes outcome E. Now we introduce new causal event chain A AB BBBB... If B occurs E does not occur. If A occurs B does not occur. Thus A prevents B from preventing E.
In our beer spilling example, Mike bumps the table (event C), which causes the beer bottle to fall (event E). Jack sees the bottle falling and tries to catch it (event B) and thus prevent event E, but Peter bumps into Jack (event A), which causes Jack to miss the bottle.
49

---

In this case, both A and E have counterfactual dependence; the outcome would not have occurred if A had not occurred. But humans tend to judge that Mike's 's bumping the table caused the beer to spill and that Peter's bumping into Jack, the double preventer, did not Chang 2009) Lombrozo 2 Moore 2009). the following prompt, we extend the beer spilling scene to include a double prevention scenario. this case, Mike bumps the table (event A), Jack tried to prevent (event B), but Peter interferes with Jack's attempt (event C). We expect that the LLM should assign more responsibility Mike than Peter.
PROMPT: Suppose that Jack saw that the bottle about to fall, and he reached out to catch the bottle. But just as Jack was about to catch the bottle, Peter bumped into Jack, which caused him to miss the bottle, which then fell the ground. Who is
responsible for the bottle falling to the ground? Mike Peter?
LLM: this scenario, Peter is more responsible for the bottle falling to the ground, as his bumping into Jack directly interfered with Jack's attempt catch the bottle and prevent it from falling. While Mike's bumping the table initially set the events in motion, Peter's action had the most immediate impact on the outcome.
Here, we observe a failure of the ChatGPT4 to capture the nuance of double prevention at least by human standards.
E Probing LLM Behaviors Further
E.1
Memorization Tests
The system message used for a memorization test of row based datasets, including the Tubingen and CRASS datasets are:
SYSTEM: You are an Al assistant that has read many sources of text from the internet. I am looking at text from the <DATASET NAME AND DESCRIPTIONS<datase, published by <AUTHORS>as <PUBLICATION INFO>. Here is the README for the dataset:
<EEDDEE>
I am going to list some sample rows of data, and want you complete each row best possible. I am testing your memory.
USER: pair0005 Age
ASSISTANT: pair0005 Age Length Abalone >&nbsp:
To test for memorization of datasets where the original file is not split into rows or lines, we sample a random N-character long section from the original dataset, and ask the LLM to complete the original file from there. The system message used for a memorization test of JSON based dataset, where the original file is not split into rows or lines, are:
SYSTEM: You are an Al assistant that has read many sources of text from the internet. am looking at text from the <DATASET NAME AND DESCRIPTIONddatase, published by <AUTHORS >as <PUBLICATION INFO >, Here is the README for the dataset:
<README>
I am going to provide a random 500 character subsection from the formatttt data, and I want you to complete the next 500 characters of the data. am testing your memory.
USER: <500 characters from data
ASSISTANT:
The results of our memorization tests are presented table 19 Our memorization test shows the Tübingen dataset in the training dataset and has been least partially memorized by GPT-3.5 and GPT-4. The neuropathic pain dataset
50

---

% Columns Memorized
Dataset Tübingen
GPT-3.5
GPT-4
Cells
58.9%
61%
Rows
19.8%
25%
Dataset
Avg number of characters recovered
Sample section length (num chars)
GPT-3.5 Turbo
GPT-4
Neuropathic
100
25
200
19
34
400
136
22
Arctic Sea Ice
100
2
200
10
400
3
CRASS
100
14
2
200
10
7
400
10
7
Table 19: Our memorization tests results show the Tubingen dataset in the training dataset for GPT-3.5 and GPT-4, the neuropathic dataset maybe partially memorized, but the other datasets seem to be memorized. We run cell and row based memorization test for the tabular Tubingen dataset and unstructured memorization test for the other datasets. The unstructured test counts the average number of characters the LLM is able reproduce following randomly sampled section of the given length.
seems be partially memorized by GPT-3.5 Turbo, but results with GPT-4 are unclear. The Arctic Sea Ice and CRASS datasets do not show signs of having been memorized, as the small number of recovered tokens are related to language or formatting that can be predicted from the sample text.
E.2 Perturbation Test Example
We give brief illustration of a perturbation experiment in Figure which shows 3.5 model giving an erroneous answer to an arithmetic problem, even when correctly solving subparts of the problem when prompted. We might wonder whether the LLM has learned pattern that the final answer must match its initial answer, instead of a more correct pattern that the final answer must match the preceding arithmetic expression. With LLMs, we can run a perturbation experiment test this hypothesis, intervening on the first answer to study its influence the final answer. By redacting then perturbing each of the tokens in the prompt, one by one, the error and better understand reliabilities. Once have target token word, can perturb its value systematically to identify how it influences the final answer. Table 20 shows that the perturbation to the first answer of Figure 5 is often directly reflected in the final result. Interestingly, as the perturbation value increases, the final answer is more likely to snap to the correct result instead.
determine which tokens are leading
F Supplementary Discussion
F.1 Example of LLM-assisted Causal Analysis Code Generation
Here, we present, without comment, a brief case study asking GPT-4 to generate python code for a demonstration causal analysis.
51

---

Session 1:
Q: What is B
* 12?
A: 104
Q: Please show your work
64
12 50
64 60 = 104
Session 2:
Q: What is 8 * 8
12?
110
Please show your work
64
5 12 = 60
64 + 60 = 110
In multi-turn interaction, GPT3.5 first gives erroneous response to a math problem. Prompted to show its work, it correctly solves the sub-parts of the problem, but again gives the wrong final answer. Session 2: we probe for the influence of the first wrong answer on the final wrong answer by replaying an interventionallcooveeatttionn and asking GPT3.5 to complete only the final answer.
Figure 5: LLM completions are highlighted in yellow. Session 1:
USER: could you give jupyter notebook that creates synthetic data for causal problem (please make simple DAG and data generator); then calls CMU's causal-learn package to learn dag: then calls Do and EconML. estimate specific treatment effect using the learned graph, and validates with sensitivity analyses
I cannot directly provide you with a Jupyter Notebook file, but can give you a step-by-step guide on how to create one yourself. In this example, we'll create simple synthetic dataset with variables, where X1 X2 Y and X1 -> Y, and use causal-learn, DoWhy, and EconMI. to estimate the treatment effect of X1 on Y.
1. Install the required packages:
!pppp install numpy pandas causal-learo dowby econml
2. Import the necessary libraries:
import numpy as np import pancas pd
from causal_learn import causal_graph
from dowhy import CausalMode1
from maaumlmmtalcarnee import TLearner
from sklearn. Lincarrniicca__odee import LinearRegression
3. Generate synthetic data:
np. www.raandd.cc..o. sced (42)
n_samples 1000
# Generating X1 (treatment variable)
X1 np. random. normal m (0, 1, n_samples)
Generating (mediator variable)
X2 0.5 X1 ap. random.noomal.. (0, 1, n_samples)
52

---

Perturbation
Result
100
100
101
101
102
124
103
124
104
104
105
104
106
104
107
107
108
124
109
109
110
110
111
111
112
112
113
124
114
114
115
115
116
124
117
124
118
124
119
119
120
120
121
124
122
122
123
124
124
124
125
124
126
124
127
127
128
124
129
124
130
130
131
124
132
124
133
124
134
124
135
124 +
135
136
124
137
124
138
124
139
124
Table 20: Perturbing the first answer from Figure 5 and observing the result shows that the first answer strongly influences the final answer
# Generating Y (outcome variable)
Y 2
0.8 x2 np. caaddom.ncrraaa (0, 1, n_samples)
1 Create a catairane
data = Da. (1 XI'; X1, 'X2': X2, "Y" Y)
53

---

4. Learn the causal graph using the causal-learn package:
graph causal_graph, caapaaaagrrppccuuaaaaaaaaaa (df=data)
print (graph.graph_)
Use DoWhy to create causal model:
Create causal model using the learned graph
mausal_moced = CausalModel
data=data,
treatment='X1',
outcome='Y',
graph-graph.grapee torp:/ppph..apphh..ttssstro ()
Identify causal effect identified_est = ie ()
Estimate the treatment effect using EconML:
Train T-learner using linear regression
t learner TLearner (Linarfegressicnt) LinearRegression()
c_learner. fit (Y-data['y) T-data['XI'), X=datal['x2(11)
Estimate the treatment effect
treatment_effect = t_learner. cow.annrraootttmmmggaaaaeeeeee (datal [ (2X22
Validate results using sensitivity analyses:
Perform sensitivity analysis using Dekhy
refutation cecuurrrsuttssseeeee = causal hacelllmomeee_11 wwwel..rfuteeeeeeemmmmmmo refute estimate
feentiffed_estimeee
treatment_effert
method_ candon_common_canu
print (refutation__reaulte
This example demonstrates how create syuthetic data for a causal problem, learn DAG using the causal-learn package, estimate treatment effects using DoWhy and EconML, and perform sensitivity analyses. Note that the actual treatment effect learned treatment effect may differ noise in the assumptions made by the methods used.
F.2 Example of LLM-assisted Identification of Negative Controls
Here, we present, without comment, a brief case study asking GPT-4 to generate negative and positive controls. We add list formatting for readability.
USER: In causal observational study of vaccine efficacy, we are matching people who have voluntarily gotten vaceine with individuals who have not yet done so. We are matching on attributes intended capture all confounding effects. To help validate our analysis. what are some examples of negative controls where might expect to see zero treatment effect?
GPT-4: In a causal observational study, negative controls are variables or outcomes for which you do not expect to see a treatment effect. By including negative controls in your analysis, you can help validate your matching process
54

---

and assess the presence of unmeasured confounding. Here are some examples of negative controls where you might expect to see zero treatment effect in a study vaccine efficacy:
Hair color: The vaccine should have no impact on the natural hair color of individuals, so you should expect no difference in hair color distribution between the vaccinated unvaccinated groups.
Blood type: There should be no relationship between receiving a vaccine and an individual's blood type. So, the distribution blood types should similar both groups.
Handedness: The vaceine should not affect whether person is right-handed, left-handed, or ambidextrous, so you should expect no difference in handedness between the vaccinated and unvaccinated groups.
Taste preferences: Vaccination should not influence individual's 's taste preferences for certain foods flavors, so no difference should be observed in this regard between the two groups.
Birth month: There should be no relationship between receiving vaccine and the month person was born. Therefore, the distribution of birth months should he similar vaccinated and unvaccinated individuals.
Past unrelated medical procedures: Previous medical procedures unrelated to vaccination, such as dental work or orthopedic surgeries, should not be affected by vaccination status. As such, there should be no difference in the prevalence of these procedures between the groups.
Favorite color: An individual's 's favorite color should not be influenced by vaccination status. Consequently, the distribution of favorite colors should be similar for both groups.
Including these negative controls in your analysis can help to evaluate whether your matching process is successful in capturing all confounding effects. If you observe treatment effect in these negative controls, it may indicate that there are unmeasured confounding factors or issues with your matching process.
USER: In what subpopulations would expect to see zero treatment effect on disease prevention? (for the disease being vaceinated against)
GPT-4: There are certain subpopulations where you might expect to see zero or limited treatment effect on disease prevention for the discase being vaceinated against:
Individuals with pre-existing immunity: People who have already been infected with the discase and developed immunity might not show additional benefits from vaceination. However, the vaccine may still help boost their immunity protect against different strains of the disease.
Individuals with specific immune deficiencies: Some people have immune system disorders are taking immunosuppressive medications that may prevent them from mounting an effective immune response to the vaccine. In such cases, the vaccine may not provide any additional protection against the disease.
Non-responders to the vaccine: A small proportion of individuals might not develop an adequate immune respouse to the vaccine due to genetic factors or other unknown reasons. These non-responders may not experience any protection from the vaccine.
Individuals vaccinated after infection: someone receives the vaccine after already being infected with the disease, the vaccine may not have chance to provide any additional protection. This is because the immune response generated by the vaccine might not have enough time to develop before disease progresses.
Individuals vaccinated with expired or improperly stored vaccines: If vaccine has expired or has been stored improperly, may lose its potency and not induce the desired immune response. such cases, individuals receiving these vaccines may not experience any protection against the discase.
It's important to note that even if a subpopulation does not experience treatment effect from the vaccine, the vaccine can still be beneficial for the overall population. Widespread vaccination can lead to herd immunity, which can indirectly protect individuals in these subpopulations by reducing the overall prevalence of the discase.
USER:
Are there subpopulations in which we would expect a known but non-zero effect? to help validate our observational causal analysis?
55

---

GPT-4: Yes, there are subpopulations where you might expect known but non-zero effect of vaccination on disease prevention. These groups can serve as positive controls to help validate your observational causal analysis. Some examples include:
Age groups: Vaccine efficacy can vary by age group. For instance, older adults or very young children may have less robust immune response to vaccination compared to young adults. You may observe reduced but non-zcro effect in these age groups compared to others.
Individuals with certain chronic conditions: People with specific chronic medical conditions like diabetes, cardiovascular diseases, or respiratory illnesses might have a weaker immune response to vaccines. As a result, you may see lower but non-zero vaccine effect in these subpopulations.
Pregnant individuals: Pregnancy can cause temporary changes in the immune system, which might lead to a reduced response vaccines. Consequently, you may observe a smaller but non-zero effect in pregnant individuals compared to non-pregnant individuals.
Smokers: Smoking known to have adverse effects the immune system, potentially leading reduced immune response to vaccines. In this subpopulation, you may find a lower but non-zero vaccine effect compared non-smokers.
Individuals receiving a lower vaccine dose or a different vaccination schedule: Some subpopulations might receive lower vaccine dose different vaccination schedule, which can result in varying levels of protection. You may see reduced but non-zero effect in these cases.
By comparing the treatment effect in these subpopulations the overall population other subpopulations with expected higher vaccine efficacy, you can help validate your causal analysis. If your analysis can correctly capture these known differences in treatment effects among various subpopulations, provides evidence that your methodology is robust and can potentially account for confounding factors.
USER:
if have multiple ohservations per person over time. I.c., know whether they infected with the given disease on any given day after vaccination, then does that suggest any time-bound negative or positive controls?
GPT-4:
Yes, when you have multiplc observations per person over time (Iongitudinal data), you use time-bound negative or positive controls to help validate your causal analysis. Here are some examples:
Time-bound Negative Controls:
Compare the infection rates before vaccination between the future vaccinated and unvaccinated groups. Since neither group has received the vaccine at this stage, you should expect no difference in infection rates due to vaccination.
Pre-vaccination period:
Short time window post-vaccination: Analyze the infection rates in very short time window (e.g.. within few days) after vaccination. The vaccine's 's protective effect usually takes some time to develop, so you should expect minimal treatment effect in this short period.
Time-bound Positive Controls:
Peak immune response period: Identify the time window in which the vaccine's immune response is expected to be at its peak based on clinical trial data or previous research. Compare the infection rates between the vaccinated and unvaccinated groups during this period, and you should expect see substantial treatment effect.
Waning immunity period: Investigate the infection rates during a period when the vaccine's 's protective effect known to wane or decrease. During this time, you might still expect to see a treatment effect, but it could be smaller compared to the peak immune response period.
Post-booster vaccination: If booster shots are administered to maintain or enhance immunity, you can compare infection rates between vaceinated individuals who have received boosters and those who have not (or unvac- cinated individuals). You should expect to see positive treatment effect in those who received the booster shots.
56

---

Using these bound controls can help you assess the validity of your causal analysis, expect specific patterns the treatment effect over time. If your analysis can capture these expected temporal patterns, provides evidence that your methodology is reliable and able to account for time-varying confounding factors.
57


Reviews:
[{"review_id": "Bn1DT9M9Ak", "reviewer": "Reviewer_6N5S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The topic of causal reasoning and large language models is interesting.\nWeaknesses: The work lacks theory and methodological work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper performs robustness checks across tasks and shows that large language models might be used by human domain experts to save effort in setting up a causal analysis.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors need to add more theory and methodological work to the paper.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OihS9608u4", "reviewer": "Reviewer_ePfk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Background literature and concepts in causal reasoning are clearly explained\n- Experiments are performed across a range of LLMs\n- Quantitative results are accompanied with qualitative analyses providing a richer picture of the way in which these models are solving causal reasoning tasks\n- There is a thoughtful discussion of the ways in which LLMs might be integrated with more traditional causal reasoning techniques\n\n## Weaknesses\n- The primary weakness is that many of the datasets are likely present in the training data, as demonstrated by the memorization analyses. It is good to include these analyses, but the results do raise the possibility that the LLMs are perhaps not able to perform causal reasoning in a generalizable way. It would be ideal to combine these results with results from a newly developed dataset, or one that we can be reasonably confident isn't in the training data. This was done for one of the experiments, but the possibility still remains that the performance observed in the other experiments is dependent on similarity to the training data. It would also be good to discuss the ways in which the presence of these datasets in the training data may or may not explain the performance of the models on these tasks (i.e., the mere presence of this dataset in the training data doesn't automatically explain the ability of the LLM to perform the task, especially if the specific task and formatting differs from the standard format of the dataset, but it is important to discuss these issues).\n- The paper presents a thorough background on the relevant literature, but it is sometimes less clear how the present work is related to previous work. It would be helpful to explicitly identify the novel contributions of this work relative to previous work.\n- I don't know if the tasks in the first part of the paper would really be considered causal *reasoning*. They seem more like tests of general causal knowledge. It would be good to have some discussion of how this relates to causal reasoning.\n- In some of the tests on necessity and sufficiency, the model is prompted to produce a chain-of-thought explanation *after* first providing an answer (e.g., as seen in the example in section 4.2.1). This likely prevents the model from using this chain-of-thought output to reason about the task (and potentially arrive at a better answer). It would be informative to prompt the model to reverse the order of these.\n\n## Minor notes\n- The ability of models beyond davinci-002 to solve the cause-effect task is attributed to model size, but there are other factors that differ between these models, including the training data (e.g., some are trained on code) and training objective (e.g., RLHF).\n- For the cause-effect dataset, weighted accuracy is reported. How are the weights determined?\n- There is a typo at the top of page 16 ('geneation').", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents an investigation of causal reasoning in large language models. The results suggest that the best performing models (GPT-3.5 and GPT-4) are able to answer general causal knowledge questions (i.e., stating which of two real-world variables is likely to have a causal effect on the other), demonstrate some capacity for counterfactual reasoning, and are able to assess causal necessity and sufficiency. The implications of these results for causal modeling approaches is discussed.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Some of the experiments (e.g., the cause-effect task) need to performed with datasets that are either novel, or that we can be reasonably confident aren't in the training data.\n- The potential causal link between the presence of the datasets in the training data and the ability of the models to solve the tasks presented in this paper should be discussed.\n- Chain-of-thought should be performed before a final answer is provided, rather than as a justification of an answer already provided.\n- The relationship of the present experiments to previous work should be more clearly explained.", "broader_impact_concerns": "Broader impacts and ethical implications are adequately discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lKQsp3wyI7", "reviewer": "Reviewer_i2Bg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Thanks to the authors for the obvious very hard work on this paper. I think it's a great contribution to the field, but I have some substantial concerns that I outline below. But make no mistake - I believe this paper is very worthwhile!\n\nStrengths\n------------\n- A ton of empirical work and careful thinking\n- Lots of experiments\n- Clear writing\n\nWeaknesses\n----------------\n- The biggest concern is that LLMs have seen the various datasets in this paper. If they have, the results are far from conclusive. As you have made your own \"lab-vignettes\" dataset, I strongly recommend (a) testing whether multiple LLMs have memorized each dataset you use (not just Tubingen) and (b) making a new dataset for each of those that have been memorized.\n\n- 3.1.1 - There are only 108 questions so overfitting is a serious concern. Did you choose your prompts once without any tuning? Please detail how the process of prompt tuning was structured, and discuss overfiting risk. Similarly for for Table 4: there are multiple GPT-4 entries in the table (minor: one is incorrectly spelled), with widely different scores. If you (hypothetically) kept modifying the prompts/approaches while looking at the test set accuracy, then leakage is likely. Please clarify how prompt tuning was done. \n\n- 3.2.1 - The same questions apply. How many prompts did you try before you found the one where you add \"C: No causal relationship exists”?\n\n- 3.3 - These results are both *very* concerning and confusingly presented. Since the LLM has *seen* the labels (as per your test), then I don't see how the basic claim of your paper (that LLMs are good at causality) is being truly tested. This is just plain overfitting to the test set, and the comparisons to other non-memorizing algorithms are unfair. Please repeat the memorization test for all the datasets tested (Neuropathic pain, NHD, CRASS, Kueffner 2021, BIG-Bench Hard causal judgment), and report the results.\n\n- 4.1 - Related to the previous point, there are a number of claims about LLM capability in this paper that resemble \"The results indicate that large language models of GPT-3.5 and 4 series represent a substantial jump in LLM ability to generate counterfactual arguments\". Before making these claims, you should clearly establish that LLMs have not memorized the dataset. \n\n- 4.2.1 - \"Since the story was published in 2022, it cannot have been in the training set of either GPT 3 or GPT 4 models.\" Which exact LLM version is being used here?\n\n- Page 23 - \"This is remarkable because the LLM was not explicitly provided definitions of necessary and sufficient causes\" - why not provide these as well? \n\n- 5.1 - Generally speaking, benchmarks that have been created for purely numerical methods for causal discovery are \"valid\" (colloquial meaning) because the numerical methods don't have any prior information about the meaning of the variables. So they have to \"figure out\" causality from scratch (with some model structure priors built in by expert humans). This is important because we generally care about inferring causality in novel situations. From the perspective of the numerical causal discovery method, then, each benchmark dataset *is* a novel situation.  I don't fully believe this because of a field's tendency to overfit to its benchmarks, but that is the ideal. The fact that LLMs learned causal structure from their extensive exposure to terabytes of human text is a good finding, but that does *not* suggest they are useful for realistic causal discovery (which is what you propose in 5.1). This is because the causal questions of practical interest are new ones. I.e., not ones that appear somewhere in the pre-training corpus of a modern LLM. You have shown (subject to the concerns above) that LLMs can perform \"pairwise causal discovery\", \"counterfactual reasoning task\", and \"event causality\". If solving these tasks requires extant human knowledge, then it makes sense that powerful LLMs can perform these tasks to some degree. However, can LLMs perform these tasks in *novel* situations? Numerical algorithms can to some degree - we have evidence of this because they are able to perform well without strong knowledge-based priors in existing benchmarks. But what can we say about LLM causal inference for extra-training corpus scenarios?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A lengthy and extensive study of LLMs for various causal inference tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- A memorization test for every dataset in the paper: Neuropathic pain, NHD, CRASS, Kueffner 2021, BIG-Bench Hard and any other I may have missed - critical\n- Discussion of overfitting, prompt selection on the test set, etc - critical\n- Update discussion in 5.1 pending response to this review - critical", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The decision is mainly based on the consensus that the paper provides a thorough empirical evaluation of the causal reasoning capabilities of LLMs, which is quite novel and interesting. The reviewers appreciated the authors' thorough response to concerns about overfitting and dataset memorization. 

The paper is recommended for featured certification due to its significant empirical contributions and the novelty of integrating LLMs with causal reasoning tasks. The work stands out for its potential impact on both the academic community and practical applications in various domains where causal analysis is critical.

In summary, the decision to accept the paper is supported by the reviewers' positive feedback and the authors' effective responses to concerns, demonstrating the robustness and relevance of their findings in advancing the understanding of LLMs in causal reasoning.
Decision: Accept (poster)