Causal Order: The Key to Leveraging Imperfect Experts in Causal Inference

## Abstract

Abstract At the core of causal inference lies the challenge of determining reliable causal graphs solely based on observational data. Since the well-known backdoor criterion depends on the graph, any errors in the graph can propagate downstream to effect inference. In this work, we initially show that complete graph information is not necessary for causal effect inference; the topological order over graph variables ( causal order ) alone suffices. Further, given a node pair, causal order is easier to elicit from domain experts compared to graph edges since determining the existence of an edge can depend extensively on other variables. Interestingly, we find that the same principle holds for Large Language Models (LLMs) such as GPT-3.5-turbo and GPT-4, motivating an automated method to obtain causal order (and hence causal effect) with LLMs acting as virtual domain experts. To this end, we employ different prompting strategies and contextual cues to propose a robust technique of obtaining causal order from LLMs. Acknowledging LLMs’ limitations, we also study possible techniques to integrate LLMs with established causal discovery algorithms, including constraint-based and score-based methods, to enhance their performance. Extensive experiments demonstrate that our approach significantly improves causal ordering accuracy as compared to discovery algorithms, highlighting the potential of LLMs to enhance causal inference across diverse fields.

## 1 Introduction

Causal inference plays a pivotal role across scientific disciplines, aiding researchers in uncovering fundamental causal relationships and how they affect observed phenomena.
For example, causal inference is used to discern the causes of diseases and design effective interventions for diagnosis and treatment in epidemiology (Mahmood et al., ), to evaluate policy impact based on observational studies in economics (Imbens & Rubin, ), and to understand the effects of pollution on ecosystems in environmental science (Boslaugh, ). A key technical question for these studies is estimating the causal effect of variables on a specific outcome variable.

Inferring causal effect from observational data, however, is a challenging task because the effect estimate depends critically on the causal graph considered in the analysis. While there has been progress in graph discovery algorithms, especially for specific parametric settings (Shimizu et al., ; Hoyer et al., ; Hyvärinen et al., ; Rolland et al., ), studies on real-world datasets such as from atmospheric science (Huang et al., ) and healthcare (Tu et al., ) show that inferring the causal graph from data remains a challenging problem in practice (Reisach et al., ). As a result, causal inference studies often rely on a human expert to provide the causal graph.

In this paper, we show that obtaining the full causal graph is not necessary. Instead, a simpler structure, the topological ordering over graph variables (causal order) is sufficient for causal effect inference. Moreover, providing the order between variables is the right question to ask experts because order depends only on the variables under question, unlike existence of a graph edge that depends on which other variables are present (to account for direct and indirect effects). For example, consider the data-generating process, *lung cancer* $\rightarrow$ *doctor visit* $\rightarrow$ *positive Xray*. If an expert is asked whether there should be a causal edge from *lung cancer* to *positive Xray*, they would answer “Yes” (indeed, such an edge exists in the BNLearn Cancer dataset (Scutari & Denis, )). However, if they are told that the set of observed variables additionally includes doctor visit, then the correct answer would be to not create a direct edge between lung cancer and positive Xray, but rather create edges mediated through *doctor visit*. However, note that the causal order, lung cancer $\prec$ positive Xray remains the same in both settings ($a\prec b$ indicates that $a$ occurs before $b$ in a casual process).

Figure: Figure 1: The LLM-augmented causal inference process based on inferring causal order. We propose a triplet-based prompting technique to infer all three-variable subgraphs and aggregate them using majority voting to produce a causal order. The causal order can then be used to identify a valid backdoor adjustment set. Ties in causal order are broken using another LLM (e.g., GPT-4). For robustness, LLM-generated causal order may be used in conjunction with discovery algorithms like PC or CaMML.
Refer to caption: /html/2310.15117/assets/Tripletflowchartfinal1final.png

Our second contribution is to show that large language models (LLMs) like GPT-3.5 (Hagendorff et al., ) and GPT-4 (Peng et al., ) can be used to approximate experts’ capability to provide causal order, thereby automating the process of causal inference. Extending results using LLMs for pairwise causal discovery (Kıcıman et al., ), we find that LLMs can provide accurate causal order for a diverse set of benchmark graphs. To do so, we propose a novel triplet-based prompting strategy that asks LLM to consider three variables at once, compared to the pairwise prompts employed in past work (Kıcıman et al., ; Willig et al., ; Long et al., ; ). Causal order outputted using the triplet-based prompt outperforms pairwise prompts across all benchmark datasets we considered. Importantly, it avoids cycles in the predicted order whereas LLM outputs from pairwise prompts often yield cycles.

Still, LLMs can exhibit unknown failure modes. Therefore, a more principled way is to adapt existing graph discovery algorithms to utilize LLM output. To this end, we present two algorithms based on constraint-based and score-based discovery algorithms respectively. The first uses causal order from an LLM to orient the undirected edges outputted by a constraint-based algorithm such as PC (Spirtes et al., ). The second algorithm utilizes the LLM causal order as a prior to a score-based algorithm like CaMML (Wallace et al., ). Results show that LLM-augmented algorithms outperform the base causal discovery algorithms in determining the causal order. The overall methodology is depicted in Figure 1.
Our contributions can be summarized as follows.

- •
Based on the backdoor criterion, we prove that causal graph is not required for effect estimation; only the causal order suffices. Empirically, we show that causal order correlates strongly with the accuracy of effect inference.
- •
We provide a novel prompting strategy based on triplets and show that LLMs like GPT-3.5 can be used to obtain causal order for a diverse range of datasets.
- •
We propose two algorithms combining causal discovery algorithms with LLM output and show that the final causal order is substantially more accurate than the discovery algorithms alone.

## 2 Related Work

Combining graph discovery and causal inference. Historically, causal discovery and causal effect inference have been studied separately. Graph discovery algorithms can broadly be divided into (i) algorithms using conditional independence tests (constraint-based) (Glymour et al., ); (ii) algorithms using a score function to evaluate predicted graph (score-based) (Glymour et al., ); (iii) algorithms that determine a causal order and then infer edges (order-based) (Rolland et al., ; Teyssier & Koller, ); and (iv) deep learning-based methods that formulate an optimization problem based on acyclicity and sparsity constraints (Zheng et al., ; Lachapelle et al., ). Causal discovery methods are evaluated on error wrt. the true graph, e.g., using the structural hamming distance (SHD) (Acid & de Campos, ; Tsamardinos et al., ). In contrast, causal inference methods focus on the estimation of causal effect given a causal graph (Pearl, ); the graphs are assumed to be known. A natural way to combine these approaches to use the graph outputted by discovery algorithms in inference methods, as in  (Hoyer et al., ; Mooij et al., ; Maathuis et al., ; Gupta et al., ). In this paper, we show that there exists a simpler way to combine the two approaches: only a causal order is needed instead of the full graph.

Knowledge-driven Causal Discovery: Prior knowledge has been used in causal discovery literature (Hasan & Gani, ; Constantinou et al., ; Heckerman & Geiger, ; Teshima & Sugiyama, ; O’Donnell et al., ; Wallace et al., ). These methods rely on prior knowledge such as domain expert opinions and documented knowledge from randomized controlled trials (RCT). Various priors have been studies in literature, including the priors of the form edge existence, forbidden edge, ancestral constraints (Constantinou et al., ; Ban et al., ). Prior knowledge significantly reduces the search space over all possible causal graphs.

Recent advancements in LLMs has led to more attention towards knowledge-driven causal discovery (Kıcıman et al., ; Ban et al., ; Long et al., ; Willig et al., ). Unlike causal discovery algorithms that use statistical patterns in the data, LLM-based algorithms use metadata such as variable names. Most of these methods use only LLMs to predict the causal relationships among a set of variables (Kıcıman et al., ; Long et al., ; Willig et al., ; Long et al., ). Recent work also shows how LLMs can be used as priors for existing score-based causal discovery methods (Ban et al., ). However, the focus of these works has been on minimizing graph error metrics such as SHD. Instead, we focus on the downstream causal inference task and choose causal order as the metric since it directly correlates with accuracy in effect estimation whereas SHD does not.

LLM Prompting Strategies for Causal Discovery: Existing LLM-based algorithms for graph discovery (Kıcıman et al., ; Long et al., ; Ban et al., ) use a pairwise prompt, essentially asking “does A cause B” with varying levels of prompt complexity. Extending this line of work, we propose a triplet-based prompt that provides more accurate answers and avoids cycles when querying relationships between variables. As a result, our triplet-based prompt may be of independent interest to improve LLM-based graph discovery. We also explore the chain-of-thought prompting strategy (Wei et al., ) in our experiments.

## 3 Background and Problem Formulation

Let $\mathcal{G}(\textbf{X},\textbf{E})$ be a causal directed acyclic graph (DAG) consisting of a set of variables $\textbf{X}=\{X_{1},\dots,X_{n}\}$ and a set of directed edges E among the variables in $\mathbf{X}$. A directed edge $X_{i}\rightarrow X_{j}\in\mathbf{E}$ denotes the direct causal influence of the variable $X_{i}$ on the variable $X_{j}$.
Let $pa(X_{i})=\{X_{k}|X_{k}\rightarrow X_{i}\}$,
$de(X_{i})=\{X_{k}|X_{k}\leftarrow\dots\leftarrow X_{i}\}$ denote the set of parents and descendants of $X_{i}$ respectively.
We focus on a downstream application of causal graph discovery called causal effect inference, defined as follows.

###### Definition 3.1 .

(Average Causal Effect (Pearl, )) The average causal effect (ACE) of a variable $X_{i}$ on a variable $X_{j}$ is defined as

$$ $ACE_{X_{i}}^{X_{j}}=\mathbb{E}[X_{j}|do(X_{i}=x_{i})]-\mathbb{E}[X_{j}|do(X_{i}=x_{i}^{*})]$ (1) $$

In Defn [3.1](#S3.Thmtheorem1), $X_{i}$ is called the treatment variable and $X_{j}$ is called the target variable. $do(X_{i}=x_{i})$ denotes an external intervention to the variable $X_{i}$ with the value $x_{i}$. The interventional quantity $\mathbb{E}[X_{j}|do(X_{i}=x_{i})]$ is different from conditional $\mathbb{E}[X_{j}|X_{i}=x_{i}]$ since it involves setting the value of $X_{i}$ rather than conditioning on it. To estimate the quantity $\mathbb{E}[X_{j}|do(X_{i}=x_{i})]$ from observational data, the backdoor adjustment formula is used.

###### Definition 3.2 .

(Back-door Adjustment (Pearl, )) Given a DAG $\mathcal{G}$, a set of variables $\mathbf{Z}$ satisfies back-door criterion relative to a pair of treatment and target variables ($X_{i},X_{j}$) if

- (i)
no variable in $\mathbf{Z}$ is a descendant of $X_{i}$; and
- (ii)
$\mathbf{Z}$ blocks every path between $X_{i}$ and $X_{j}$ that contains an arrow into $X_{i}$.

where a path in a causal DAG is a sequence of unique vertices $X_{i},X_{i+1},\dots,X_{j}$ with a directed edge between each consecutive vertices $X_{k}$ and $X_{k+1}$ (either $X_{k}\rightarrow X_{k+1}$ or $X_{k+1}\rightarrow X_{k}$).
If a set of variables $\mathbf{Z}$ satisfies the back-door criterion relative to $(X_{i},X_{j})$, $\mathbb{E}[X_{j}|do(X_{i}=x_{i})]$ can be computed using the formula: $\mathbb{E}[X_{j}|do(X_{i}=x_{i})]=\mathbb{E}_{\mathbf{z}\sim\mathbf{Z}}\mathbb{E}[X_{j}|X_{i}=x_{i},\mathbf{Z}=\mathbf{z}]$ (Thm. 3.3.2 of (Pearl, )). To ensure that all variables in $\mathbf{Z}$ are observed, we make the following assumption.

###### Assumption 3.3 .

(No-latent confounding.) There are no unobserved variables causing both treatment and target variables.

###### Definition 3.1 .

###### Definition 3.2 .

###### Assumption 3.3 .

## 4 Causal Order is Sufficient for Effect Estimation

Although backdoor adjustment is defined wrt a DAG $\mathcal{G}$, we now show that the causal order is sufficient to find a valid backdoor set. We also discuss why providing the causal order is a better task for experts than providing the graph.

### 4.1 Causal (Topological) Order Yields a Valid Backdoor Set

###### Definition 4.1 .

(Topological Order.) Given a causal graph $\mathcal{G}(\mathbf{X},\mathbf{E})$, a sequence $\pi$ of variables $\mathbf{X}$ is said to be a topological order iff for each edge $X_{i}\rightarrow X_{j}\in\mathbf{E}$, $\pi_{i}<\pi_{j}$.

###### Proposition 4.2 .

Under assumption [3.3](#S3.Thmtheorem3), given an pair of treatment and target variables $(X_{i},X_{j})$ in $\mathcal{G}$, $\mathbf{Z}=\{X_{k}|\pi_{k}<\pi_{i}\}$ is a valid adjustment set relative to $(X_{i},X_{j})$ for any topological ordering $\pi$ of $\mathcal{G}$.

Proof of all Propositions are in Appendix § [A](#A1). Propn [4.2](#S4.Thmtheorem2) states that all the variables that precede the treatment variable in a topological order $\pi$ of $\mathcal{G}$ constitute a valid adjustment set. Note that the set $\mathbf{Z}$ may contain variables that are not necessary to adjust for, e.g., ancestors of only treatment or only target variables. For statistical estimation, ancestors of target variable are beneficial for precision whereas ancestors of treatment can be harmful (Cinelli et al., ). On balance though, causal effect practitioners tend to include all confounders that do not violate the backdoor criterion; we are following the same principle.

In practice, however, we may not know the true order. To evaluate the goodness of a given causal order, we use the topological divergence metric from (Rolland et al., ) (for an example, see Figure [3](#S4.F3)). The topological divergence of an estimated topological order $\hat{\pi}$ with ground truth adjacency matrix $A$, denoted by $D_{top}(\hat{\pi},A)$, is defined as $D_{top}(\hat{\pi},A)=\displaystyle\sum_{i=1}^{n}\sum_{j:\hat{\pi}_{i}>\hat{\pi}_{j}}A_{ij}$.
Where $A_{ij}=1$ if there is a directed arrow from node $i$ to $j$ else $A_{ij}=0$. $D_{top}(\hat{\pi},A)$ counts the number of edges that cannot be recovered due to estimated topological order $\hat{\pi}$.

###### Definition 4.1 .

###### Proposition 4.2 .

### 4.2 Topological divergence is the correct metric for effect estimation

Below we show that $D_{top}$ is the correct metric to optimize for effect estimation: $D_{top}=0$ for a topological order is equivalent to obtaining the correct backdoor adjustment set using Proposition [4.2](#S4.Thmtheorem2).

###### Proposition 4.3 .

For an estimated topological order $\hat{\pi}$ and a true topological order $\pi$ of a causal DAG $\mathcal{G}$ with the corresponding adjacency matrix $A$, $D_{top}(\hat{\pi},A)=0$ iff $\mathbf{Z}=\{X_{k}|\hat{\pi}_{k}<\hat{\pi}_{i}\}$ is a valid adjustment set relative to $(X_{i},X_{j}),\ \forall i,j$.

We now compare $D_{top}$ to structural hamming distance (SHD), a common metric used to evaluate graph discovery algorithms. Given a true causal DAG $\mathcal{G}$ and an estimated causal DAG $\hat{\mathcal{G}}$, SHD counts the number of missing, falsely detected, and falsely directed edges in $\hat{\mathcal{G}}$: $SHD=E_{m}+E_{f}+E_{d}$ where $E_{m}=\{X_{i}\rightarrow X_{j}\not\in\hat{\mathcal{G}}|X_{i}\rightarrow X_{j}\in\mathcal{G}\}$, $E_{f}=\{X_{i}\rightarrow X_{j}\in\hat{\mathcal{G}}|X_{i}\rightarrow X_{j}\not\in\mathcal{G}\}$, $E_{d}=\{X_{i}\rightarrow X_{j}\in\hat{\mathcal{G}}|X_{i}\leftarrow X_{j}\in\mathcal{G}\}$. Formally, $D_{top}$ acts as a lower-bound on the structural hamming distance (SHD) (Rolland et al., ). However, as we show below,
SHD is not a good metric for evaluating downstream effect estimation accuracy. Specifically, we show that SHD can be very high even when $D_{top}=0$ and a valid backdoor set can be inferred. This result is of significance since most estimated graphs (included those that are LLM-generated (Ban et al., ; Long et al., )) are evaluated on SHD.

###### Definition 4.4 .

(Level Order.) Given a causal graph $\mathcal{G}(\mathbf{X},\mathbf{E})$, the level order refers to a systematic assignment of levels to variables. This assignment begins with the set of variables $\{X_{i}|\text{pa}(X_{i})=\emptyset\}$ at level 0. Subsequently, each of the remaining variables is assigned a level $i$ such that all nodes within a given level $i$ has a directed path of length $i$ from one/more nodes in level 0.

###### Proposition 4.5 .

In a causal DAG $\mathcal{G}$ with $N$ levels in the level-ordering of variables where the level $i$ contains $n_{i}$ variables,$\exists\ \hat{\mathcal{G}}$ s.t. $SHD(\hat{\mathcal{G}},\mathcal{G})\geq\sum\limits_{i=1}^{N-1}(n_{i}\times\sum\limits_{j=i+1}^{N}n_{j})-|\mathbf{E}|$ and $D_{top}(\hat{\pi},A)=0\ \forall\hat{\pi}\text{ of }\hat{\mathcal{G}}$.

Figure: Figure 2: Variability of SHD for various graph sizes with consistent $D_{top}=0$ within each graph.
Refer to caption: /html/2310.15117/assets/x1.png

Figure [2](#S4.F2) shows the unsuitability of SHD for our work empirically. Given a fixed number of nodes, we sample a graph at random as the “ground-truth” and then consider all graph orientations of the same size (number of nodes) such that $D_{top}=0$ w.r.t to ground-truth graph. For these set of graphs, we compute the SHD w.r.t the ground-truth graph.
Notice that SHD exhibits high variance. For graphs with six nodes, SHD can vary from 0 to 14 even as $D_{top}=0$ and backdoor set validity stays the same.
Figure [3](#S4.F3) shows this phenomenon on a real-world BNLearn dataset, *Cancer*. The candidate graph (right panel) has $D_{top}=0$ wrt. the true graph (left) and yields valid backdoor identification sets. However, its SHD is high (6), showing the disconnect between SHD and causal effect identification.

###### Proposition 4.3 .

###### Definition 4.4 .

###### Proposition 4.5 .

### 4.3 Causal order is more suitable to elicit from experts than graph edges

Figure: Figure 3: Left: Causal graph of Cancer dataset. Right: GPT-3.5’s estimated causal graph of Cancer dataset. GPT-3.5 gets causal order correct at the cost of higher SHD score, which is not a relevant metric for causal inference. Here $D_{top}=0$ and $SHD=6$, showing the importance of $D_{top}$ in our study as compared to SHD.

In addition to the favorable theoretical properties, causal order is easier to elicit from experts and can be objectively evaluated. This is because given two variables, their relative causal order does not depend on other variables whereas existence of an edge between them depends on which other variables are considered. To see this, let us continue the example from Figure [3](#S4.F3) (left)
where pollution causes dyspnoea (breathing difficulty) through the intermediary node cancer. Whether an edge exists between pollution and dyspnoea depends on whether *Cancer* variable is part of the study. In case an expert is only provided *pollution* and dyspnoea, they may add an edge between the two, but if *Cancer* node is also provided, they may not. Apriori, it is difficult to know which nodes may be relevant for a pair; hence experts’ answers may not be consistent for questions about edges, but will always be consistent on causal order.

## 5 Large Language Models as Virtual Experts for Causal Order

While causal order is a simpler construct than the graph, it still needs to be provided by a domain expert. We now study whether LLMs can used to obtain causal order, thereby making automating the process. We propose two kinds of prompting strategies; 1) Pairwise; and 2) Triplet-based. All these techniques use variable names or additional metadata but do not use the data associated with variables.

### 5.1 Prompt techniques based on a pair of variables

A natural way to elicit causal order from LLMs is to ask about each pair of variables and aggregate the result. A similar pairwise strategy has been adopted by past work on inferring graph edges (Kıcıman et al., ; Ban et al., ; Long et al., ). Here we augment such strategies with additional contextual information. Our hypothesis is that adding context relevant to the pair of variables may help increase the accuracy of the LLM answers.

We propose four types of pairwise prompts (see Appendix § [D](#A4) for full prompts).

**Table 1: Example LLM (GPT-3.5-turbo) outputs for estimating the causal direction between a given pair of nodes (with their descriptions) using the CoT Prompt (refer Table [A4](#A4.T4) in Appendix for the prompt). Given a pair (X,Y), Option $A$ refers to X causing Y, option $B$ refers to Y causing X, and option $C$ refers to no causal relationship between the variables. The pairs shown are from the Child graph from BNLearn repository (Scutari & Denis, ), that is used for modeling Congenital Heart Disease in infants.**
| Input: (‘HypDistrib’, ‘LowerBodyO2’) |
| --- |
| Answer: Low oxygen areas equally distributed around the body can affect the level of oxygen in the lower body by reducing the amount of oxygen available for circulation. Therefore, the answer is <Answer>A</Answer>. |
| Input: (‘Disease’, ‘LungFlow’) |
| Answer: Infant methemoglobinemia is a condition where the blood is unable to carry enough oxygen, which can lead to low blood flow in the lungs. Therefore, the answer is <Answer>A</Answer>. |
| Input: (‘CardiacMixing’, ‘HypDistrib’) |
| Answer: Mixing of oxygenated and deoxygenated blood is a common cause of congenital heart disease in babies. This mixing can occur due to structural defects in the heart, such as a hole between the heart chambers or abnormal connections between the blood vessels. This mixing can lead to low oxygen areas equally distributed around the body. Therefore, the answer is <Answer>B</Answer>. |

- •
Basic prompt. This is the simplest technique. We directly ask LLM to find the causal direction between a given pair of variables.
- •
Iterative Context. Here we provide the previously oriented pairs as context in the prompt. Since the LLM has access to its previous decisions, we expect that it may avoid creating cycles through its predictions.
- •
Markov Blanket Context. Providing previously oriented pairs may become prohibitive for large graphs. Using the fact that a variable is independent of all other nodes given the Markov Blanket (Pearl, ), here we provide the Markov Blanket of the given node pairs as additional context in the prompt.
- •
Chain-of-Thought (+In-context learning). Based on encouraging results of providing in-context examples in the prompt for various tasks (Brown et al., ), here we include 3 examples of the ordering task that we expect the LLM to perform. Effectively, we provide example node pairs and their correct causal ordering before asking the question about the given nodes. Each example answer also contains an explanation of the answer, generated using Bing GPT-4. Adding the explanation encourages LLM to employ chain-of-thought reasoning (Wei et al., ) when deciding the causal order. To avoid overfitting, we select node pairs from graphs that are not evaluated in our study. Node pairs with and without direct edges were equally chosen. Examples of LLM’s answers (and their explanations) using the CoT prompt can be found in Table [1](#S5.T1) and Table [A12](#A4.T12) in Appendix.

**Table 2: The triplet prompt template, which includes a concise context of the graph, the input triplet nodes and their respective descriptions. As an example, for the Child graph, the context is "to model congenital heart disease in babies", the three nodes may be (‘HypoxiaInO2’, ‘Grunting’, ‘GruntingReport’); and their node descriptions are ["hypoxia when breathing oxygen", "grunting in infants", "report of infant grunting"] respectively.**
| Identify the causal relationships between the given variables and create a directed acyclic graph to {context}.<br>Make sure to give a reasoning for your answer and then output the directed graph in the form of a list of tuples, where each tuple is a directed edge. The desired output should be in the following form:<br>[(‘A’,‘B’), (‘B’,‘C’)] where first tuple represents a directed edge from Node ‘A’ to Node ‘B’, second tuple represents a directed edge from Node ‘B’ to Node ‘C’and so on. |
| --- |
| If a node should not form any causal relationship with other nodes, then you can add it as an isolated node of the graph by adding it seperately. For example, if ‘C’ should be an isolated node in a graph with nodes ‘A’, ‘B’, ‘C’, then the final DAG representation should be like [(‘A’,‘B’), (‘C’)]. |
| Use the description about the node provided with the nodes in brackets to form a better decision about the causal direction orientation between the nodes. |
| It is very important that you output the final Causal graph within the tags <Answer></Answer>otherwise your answer will not be processed. |
| Example: |
| Input: Nodes: [‘A’, ‘B’, ‘C’, ‘D’]; |
| Description of Nodes: [(description of Node A), (description of Node B), (description of Node C), (description of Node D)] |
| Output: <Answer>[(‘A’,‘B’),(‘C’,‘D’)]</Answer> |
| Question: |
| Input: Nodes: {Triplet Nodes Input} |
| Description of Nodes:<br>{Description of Each Node from the Triplet} |
| Output: |

### 5.2 Prompt technique based on triplets

As we shall see, while pairwise prompts are conceptually simple, they are prone to yielding cycles in the graph since they decide about each edge separately. Taking inspiration from the PC algorithm that employs constraints over three variables , we now describe a prompting technique based on iterating over all possible triplets given a set of nodes. Once the LLM has provided subgraphs for each triplet, we determine causal order between a pair by aggregating over all triplet LLM answers where the pair was included. To resolve ties, we use another LLM. The algorithm has the following steps.

- •
From a given set of nodes in the graph, we generate all possible triplets, each triplet treated as independently from the others.
- •
We prompt the LLM to orient nodes of each triplet group to form a Directed Acyclic Graph representing the causal relationship between the nodes of the triplet. This will result in multiple mini graphs representing causal relationship for each triplet group.
- •
LLMs will be prompted to directly orient the three nodes for each triplet, hence identifying causal relationship based on the third node, similar to how PC functions.
- •
Once we have DAGs representing each triplet, we focus on merging them. Our Merging process can be broken down into two major steps:
–
We iterate over all node pairs, and for each combination we check what is the majority orientation between them over all the triplets containing the node pair.
–
In case there occurs a clash (same number of votes) between any of the two or all three possible edge orientation (A $\rightarrow$ B; B $\rightarrow$ A; No connection between A and B nodes), we resort to GPT-4 to resolve the clash by taking the final decision via CoT prompt.

In addition, the triplet prompt uses the techniques of in-context examples and chain-of-thought from the pairwise setup. An example prompt is shown in Table [2](#S5.T2).

## 6 Algorithms for LLM-Guided Causal Discovery

LLMs using the above prompts may yield accurate causal order predictions, but may also exhibit some unknown failure modes (Kıcıman et al., ). To increase robustness of the final result, we now provide algorithms for combining LLM-outputted causal order with existing causal discovery paradigms: (i) constraint-based and (ii) score-based methods.

### 6.1 Constraint-based methods using post-hoc edge orientation by LLMs

Constraint-based algorithms return a graph where some edges may not be oriented. Given a graph from constraint-based algorithm like PC, we use the causal order $\hat{\pi}$ from LLM to orient the undirected edges. Iterating over the undirected edges, we first check if the nodes of that edge are occurring in $\hat{\pi}$. If yes, we orient the edge according to the causal order.
Since there is a possibility that LLM’s final graph might have some isolated nodes which won’t be in $\hat{\pi}$, therefore if either (or both) nodes of the undirected edge are not included in $\hat{\pi}$, we query GPT-4 using pairwise CoT prompt (from Section [5.1](#S5.SS1)) to finalise a direction between the pair.

Figure: Algorithm 1 Combining constraint based methods and experts to get $\hat{\pi}$ for a given set of variables.

### 6.2 Score-based methods using expert priors

We utilize the output of LLM as a prior in the score-based algorithms. We provide the level order of the causal graph returned by LLM as a prior for a score-based algorithm. Unlike a similar LLM-prior approach by Ban et al. (), where they combine the output of LLM and a score based method using an ancestral constraint as a prior, ours is a sequential approach where a score based algorithm starts with the order based constraint, aligning with our goal of recovering causal order among variables. Optionally, we can provide prior probability to control the influence of prior on the algorithm. Algorithm [2](#alg2) outlines the steps to combine score based method and expert knowledge in terms of level order of variables.

Figure: Algorithm 2 Combining score based methods and experts to get $\hat{\pi}$ for a given set of variables.

## 7 Experiments and Results

To evaluate the accuracy of LLM-based algorithms on inferring causal order, we perform experiments on the benchmark datasets from Bayesian network repository (Scutari & Denis, ): Earthquake, Cancer, Survey, Asia, Asia modified (Asia-M), and Child. Asia-M is derived from Asia by removing the node either since it is not a node with a semantic meaning (see Appendix§ [C](#A3) for details). We also used a medium sized subset graph (refer Figure [A6](#A3.F6) in Appendix) from the Neuropathic dataset (Tu et al., ) used for pain diagnosis.
Except Child (with 20 nodes) and Neuropathic subgraph (with 22 nodes), all other graphs are small-scale graphs with <10 nodes.

### 7.1 D t ​ o ​ p subscript 𝐷 𝑡 𝑜 𝑝 D_{top} correlates with effect estimation error

**Table 3: $\epsilon_{ACE}$ vs. $SHD$ ($D_{top}$) given $D_{top}$ ($SHD$)**
| Cancer |  |  |  |
| --- | --- | --- | --- |
| $SHD$ vs. $\epsilon_{ACE}\mid D_{top}=0$ | $D_{top}$ vs. $\epsilon_{ACE}\mid SHD=2$ |  |  |
| $SHD$ | $\epsilon_{ACE}$ | $D_{top}$ | $\epsilon_{ACE}$ |
| 0 | 0.00 | 0 | 0.00 |
| 2 | 0.00 | 1 | 0.25 |
| 4 | 0.00 | 2 | 0.50 |
| Asia |  |  |  |
| $SHD$ vs. $\epsilon_{ACE}\mid D_{top}=0$ | $D_{top}$ vs. $\epsilon_{ACE}\mid SHD=3$ |  |  |
| $SHD$ | $\epsilon_{ACE}$ | $D_{top}$ | $\epsilon_{ACE}$ |
| 0 | 0.00 | 1 | 0.14 |
| 6 | 0.00 | 2 | 0.22 |
| 10 | 0.00 | 3 | 0.57 |
| Survey |  |  |  |
| $SHD$ vs. $\epsilon_{ACE}\mid D_{top}=0$ | $D_{top}$ vs. $\epsilon_{ACE}\mid SHD=2$ |  |  |
| $SHD$ | $\epsilon_{ACE}$ | $D_{top}$ | $\epsilon_{ACE}$ |
| 0 | 0.00 | 0 | 0.00 |
| 2 | 0.00 | 1 | 0.25 |
| 4 | 0.03 | 2 | 0.50 |

Before comparing methods on the $D_{top}$ metric, we first show that $D_{top}$ has a strong correlation with effect estimation error and hence is the correct metric for effect inference. Specifically, we study how the error in causal effect, $\epsilon_{ACE}$, changes as values of the metrics $SHD,D_{top}$ change. For the datasets Cancer, Asia and Survey, we consider $dyspnoea$, $dyspnoea$, and $Travel$ respectively as the target variables. In each graph, we evaluate causal effects of each variable on the target variable. We iterate through estimated causal graphs with different values of SHD and $D_{top}$ and report the mean absolute difference between estimated and true causal effects. As Table [3](#S7.T3) shows, when $D_{top}$ is zero, effect error $\epsilon_{ACE}$ is also zero. And as $D_{top}$ increases (right panel), effect error increases. In contrast, SHD has no correlation with the $\epsilon_{ACE}$.

### 7.2 Triplet prompting technique is most accurate for causal order

Tables [7.1](#S7.SS1) and [7.1](#S7.SS1) compare the different prompting techniques. As the graph size increases, we observe limitations with pairwise prompts. In many cases, pairwise prompts yield cycles in many cases due to which $D_{top}$ cannot be computed. In particular, for Child dataset with 20 nodes, pairwise prompts yield anywhere from 13-79 cycles. LLM output tends to connect more edges than needed, which explains why SHD is high. Overall, among the pairwise prompts, the chain of thought prompt performs the best: it has the lowest $D_{top}$ on the four small graphs and lowest number of cycles for Child and Neuropathic datasets. This indicates that in-context examples and chain-of-thought reasoning helps to increase accuracy of causal order output, but other contextual cues do not matter.

Finally, the triplet prompt provides the most accurate causal order. Even for medium-size graphs like Child and Neuropathic, the LLM output includes no cycles and SHD is fairly low betwen 4-29. Moreover ,$D_{top}$ is zero for all datasets, except for Asia and Neuropathic where it is 1 and 2 respectively. That said, we do see that isolated nodes in the output increase compared to the pairwise prompts (all graphs are connected, so outputting an isolated node is an error). Considering LLMs as virtual experts, this indicates that there are some nodes on which the LLM expert cannot determine the causal order. This is still a better tradeoff than outputting the wrong causal order, which can confuse downstream algorithms. Overall, therefore, we conclude that the triplet prompt provides the most robust causal order predictions.

**Table 4: Comparison with existing discovery methods. Mean and std dev of $D_{top}$ over 5 runs. (For Neuropathic subgraph (1k samples), PC Algorithm returns cyclic graphs in the MEC)**
|  | Dataset | PC | SCORE | ICA | Direct | NOTEARS | CaMML | Ours | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | LiNGAM | LiNGAM |  |  | (PC+LLM) | (CaMML+LLM) |
|  | Earthquake | 0.30$\pm$0.44 | 4.00$\pm$0.00 | 3.20$\pm$0.39 | 3.00$\pm$0.00 | 1.80$\pm$0.74 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 4.00$\pm$0.00 | 3.60$\pm$0.48 | 2.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Survey | 0.50$\pm$0.00 | 3.00$\pm$0.00 | 6.00$\pm$0.00 | 6.00$\pm$0.00 | 3.20$\pm$0.39 | 3.33$\pm$0.94 | 0.00$\pm$0.00 | 3.33$\pm$0.94 |
|  | Asia | 2.33$\pm$0.59 | 5.00$\pm$0.00 | 6.20$\pm$0.74 | 7.00$\pm$0.00 | 4.00$\pm$0.00 | 1.85$\pm$0.58 | 0.00$\pm$0.00 | 0.97$\pm$0.62 |
|  | Asia-M | 2.00$\pm$0.00 | 5.00$\pm$0.00 | 7.60$\pm$0.48 | 6.20$\pm$1.16 | 3.40$\pm$0.48 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 1.71$\pm$0.45 |
| $N=250$ | Child | 8.16$\pm$1.58 | 8.80$\pm$2.70 | 12.8$\pm$0.97 | 13.0$\pm$0.63 | 15.0$\pm$1.09 | 3.00$\pm$0.00 | 4.00$\pm$0.00 | 3.53$\pm$0.45 |
|  | Neuropathic | 3.25$\pm$0.00 | 6.00$\pm$0.00 | 13.0$\pm$6.16 | 10.0$\pm$0.00 | 9.00$\pm$0.00 | 10.4$\pm$1.95 | 1.00$\pm$0.00 | 5.00$\pm$0.00 |
|  | Earthquake | 0.85$\pm$0.65 | 4.00$\pm$0.00 | 3.20$\pm$0.39 | 3.40$\pm$0.48 | 1.20$\pm$0.40 | 0.00$\pm$0.00 | 0.4$\pm$0.89 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 3.40$\pm$0.48 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 1.00$\pm$0.00 |
|  | Survey | 1.75$\pm$0.00 | 4.00$\pm$0.00 | 6.00$\pm$0.0 | 6.00$\pm$0.00 | 3.40$\pm$0.48 | 3.39$\pm$0.08 | 1.00$\pm$0.00 | 3.33$\pm$0.94 |
|  | Asia | 3.00$\pm$0.00 | 5.00$\pm$0.00 | 5.60$\pm$0.48 | 7.00$\pm$0.00 | 3.20$\pm$0.39 | 3.81$\pm$0.39 | 1.00$\pm$0.00 | 0.97$\pm$0.62 |
|  | Asia-M | 2.00$\pm$0.00 | 6.00$\pm$0.00 | 7.60$\pm$0.48 | 5.00$\pm$0.00 | 3.80$\pm$0.39 | 2.00$\pm$0.00 | 1.00$\pm$0.00 | 0.17$\pm$0.45 |
| $N=500$ | Child | 9.79$\pm$1.17 | 6.20$\pm$1.32 | 12.2$\pm$0.74 | 10.6$\pm$1.35 | 15.4$\pm$0.48 | 2.00$\pm$0.00 | 4.6$\pm$1.34 | 2.00$\pm$0.00 |
|  | Neuropathic | 7.50$\pm$0.00 | 6.00$\pm$0.00 | 9.00$\pm$1.41 | 13.0$\pm$0.00 | 11.0$\pm$0.00 | 5.32$\pm$0.57 | 8.00$\pm$0.00 | 7.49$\pm$0.64 |
|  | Earthquake | 0.80$\pm$0.57 | 4.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 0.20$\pm$0.44 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 1.60$\pm$0.48 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Survey | 1.00$\pm$0.00 | 4.00$\pm$0.00 | 5.80$\pm$0.39 | 5.40$\pm$0.48 | 3.20$\pm$0.39 | 2.71$\pm$0.27 | 1.00$\pm$0.00 | 2.83$\pm$0.00 |
|  | Asia | 3.09$\pm$1.05 | 4.00$\pm$0.00 | 6.20$\pm$0.74 | 6.60$\pm$0.48 | 3.40$\pm$0.48 | 1.75$\pm$0.43 | 1.75$\pm$0.95 | 0.97$\pm$0.62 |
|  | Asia-M | 2.50$\pm$0.00 | 4.00$\pm$0.00 | 8.00$\pm$0.00 | 5.20$\pm$0.39 | 3.40$\pm$0.48 | 2.04$\pm$0.51 | 2.00$\pm$0.00 | 0.65$\pm$0.47 |
| $N=1000$ | Child | 9.61$\pm$1.07 | 3.80$\pm$0.74 | 12.2$\pm$1.72 | 11.8$\pm$0.74 | 15.2$\pm$0.97 | 2.00$\pm$0.00 | 8.0$\pm$0.00 | 2.00$\pm$0.40 |
|  | Neuropathic | - | 6.00$\pm$0.00 | 4.00$\pm$0.81 | 12.0$\pm$0.00 | 12.0$\pm$0.00 | 5.54$\pm$0.75 | - | 10.1$\pm$2.12 |
|  | Earthquake | 0.30$\pm$0.67 | 4.00$\pm$0.00 | 2.80$\pm$0.39 | 3.00$\pm$0.00 | 1.00$\pm$0.00 | 0.80$\pm$0.97 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Survey | 2.00$\pm$0.00 | 4.00$\pm$0.00 | 5.00$\pm$0.00 | 5.00$\pm$0.00 | 3.00$\pm$0.00 | 3.33$\pm$0.69 | 2.00$\pm$0.00 | 2.60$\pm$0.00 |
|  | Asia | 3.05$\pm$0.94 | 4.00$\pm$0.00 | 6.60$\pm$0.79 | 4.40$\pm$1.35 | 3.40$\pm$0.48 | 1.75$\pm$0.43 | 2.80$\pm$1.30 | 0.97$\pm$0.62 |
|  | Asia-M | 1.00$\pm$0.00 | 4.00$\pm$0.00 | 7.60$\pm$0.48 | 4.60$\pm$0.48 | 3.20$\pm$0.39 | 1.68$\pm$0.46 | 0.20$\pm$0.44 | 0.00$\pm$0.00 |
| $N=5000$ | Child | 8.42$\pm$0.75 | 3.00$\pm$0.00 | 12.6$\pm$0.79 | 10.8$\pm$1.72 | 14.2$\pm$0.40 | 3.00$\pm$0.00 | 7.00$\pm$0.00 | 3.00$\pm$0.00 |
|  | Neuropathic | 9.00$\pm$0.00 | 6.00$\pm$0.00 | 9.33$\pm$0.94 | 10.0$\pm$0.00 | 10.0$\pm$0.00 | 4.20$\pm$0.96 | 9.00$\pm$0.00 | 1.23$\pm$0.42 |
|  | Earthquake | 0.00$\pm$0.00 | 4.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 1.00$\pm$0.00 | 0.40$\pm$0.48 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 0.60$\pm$0.80 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Survey | 2.00$\pm$0.00 | 4.00$\pm$0.00 | 5.00$\pm$0.00 | 5.00$\pm$0.00 | 3.00$\pm$0.00 | 3.60$\pm$1.35 | 2.00$\pm$0.00 | 1.83$\pm$0.00 |
|  | Asia | 1.95$\pm$0.41 | 4.00$\pm$0.00 | 6.00$\pm$0.00 | 4.40$\pm$1.35 | 3.00$\pm$0.00 | 1.40$\pm$0.48 | 1.20$\pm$0.83 | 0.34$\pm$0.47 |
|  | Asia-M | 1.75$\pm$0.00 | 4.00$\pm$0.00 | 8.00$\pm$0.00 | 4.80$\pm$0.39 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
| $N=10000$ | Child | 7.67$\pm$0.65 | 3.00$\pm$0.00 | 12.2$\pm$1.46 | 11.6$\pm$0.48 | 14.4$\pm$0.48 | 2.80$\pm$0.84 | 7.00$\pm$0.00 | 1.00$\pm$0.00 |
|  | Neuropathic | 10.00$\pm$0.00 | 6.00$\pm$0.00 | 1.00$\pm$0.00 | 10.0$\pm$0.00 | 10.0$\pm$0.00 | 3.00$\pm$0.00 | 10.00$\pm$0.00 | 1.00$\pm$0.00 |

### 7.3 LLMs improve causal order accuracy of existing discovery algorithms

We now study whether LLM output can be used to increase accuracy of discovery algorithms in inferring causal order.
We compare with popular causal discovery methods: PC (Spirtes et al., ), SCORE (Rolland et al., ), ICA-LiNGAM (Shimizu et al., ), Direct-LiNGAM (Shimizu et al., ), NOTEARS (Zheng et al., ), and Causal discovery via minimum message length (CaMML) (Wallace et al., ); across five different sample sizes: $250,500,1000,5000,10000$. For LLM, we use the triplet prompt. Table [4](#S7.T4) shows the $D_{top}$ metric for different algorithms and compares it to the $D_{top}$ of our combined algorithms: PC+LLM and CaMML+LLM. Among the discovery algorithms, we find that PC and CaMML perform the best, with the lowest $D_{top}$ across the five datasets. For Neuropathic dataset, ICA LiNGAM is also competitive.

For the BNLearn datasets, adding LLM output decreases the $D_{top}$ of both algorithms further. Specifically, PC+LLM leads to a significant reduction in $D_{top}$ and the gains are larger at lower sample sizes. This indicates that obtaining causal order from LLMs may matter more in limited sample settings. At sample size of 500, $D_{top}$ of PC is nearly double that of PC+LLM for most datasets. Going from CaMML to CaMML+LLM, we also see significant reductions in $D_{top}$. Interestingly, CaMML+LLM yields benefits even at higher sample sizes. At sample size of 10000, for example, $D_{top}$ of CaMML for Child and Asia is three and five times more than that of CaMML+LLM respectively.

For the Neuropathic dataset, we see a similar pattern: adding LLM to existing algorithms improves $D_{top}$ or keeps it constant, except at sample sizes 500 and 1000 where CaMML+LLM yields a worse $D_{top}$ than CaMML alone. However, as sample size increases to 5000 and 10000, we do see that CaMML+LLM improves the $D_{top}$ substantially compared to CaMML. Overall, these results show that LLM output can signficantly improve the accuracy of existing causal discovery algorithms.

## 8 Limitations and Conclusions

We presented causal order as a suitable metric for evaluating quality of causal graphs for downstream effect inference tasks. Using a novel formulation of LLM prompts based on triplets, we showed that LLMs can be useful in the generating accurate causal order, both individually and in combination with existing discovery algorithms. Our results point to techniques that can automate the causal inference process.

That said, our work has limitations. We studied LLMs utility on popular benchmarks which may have been partially memorized. It will be interesting to extend our experiments to diverse and bigger datasets. In addition, we studied only one downstream task (effect inference). Identifying the necessary graph metrics for tasks such as causal prediction and counterfactual inference will be useful future work.

## Appendix

## Appendix A Proofs of Propositions

See [4.2](#S4.Thmtheorem2)

###### Proof.

We need to show that the set $\mathbf{Z}=\{X_{k}|\pi_{k}<\pi_{i}\}$ satisfies the conditions (i) and (ii) in Defn [3.2](#S3.Thmtheorem2). For any variable $X_{k}$ such that $\pi_{k}<\pi_{i}$, we have $X_{k}\not\in de(X_{i})$ and hence the condition (i) is satisfied. Additionally, for each $X_{k}\in pa(X_{i})$ we have $\pi_{k}<\pi_{i}$ and hence $pa(X_{i})\subseteq\mathbf{Z}$. Since $pa(X_{i})$ blocks all paths from $X_{i}$ to $X_{j}$ that contains an arrow into $X_{i}$ (Peters & Bühlmann, ), $\mathbf{Z}$ satisfies condition (ii).
∎

See [4.3](#S4.Thmtheorem3)

###### Proof.

The statement of proposition is of the form $A\iff B$ with $A$ being “$D_{top}(\hat{\pi},A)=0$” and $B$ being “$\mathbf{Z}=\{X_{k}|\hat{\pi}_{k}<\hat{\pi}_{i}\}$ is a valid adjustment set relative to $(X_{i},X_{j}),\ \forall i,j$”. We prove $A\iff B$ by proving (i) $A\implies B$ and (ii) $B\implies A$.

(i) Proof of $A\implies B$: If $D_{top}(\hat{\pi},A)=0$, for all pairs of nodes $(X_{i},X_{j})$, we have $\hat{\pi}_{i}<\hat{\pi}_{j}$ whenever $\pi_{i}<\pi_{j}$. That is, causal order in estimated graph is same that of the causal order in true graph. Hence, from Propn [4.2](#S4.Thmtheorem2), $\mathbf{Z}=\{X_{k}|\hat{\pi}_{k}<\hat{\pi}_{i}\}$ is a valid adjustment set relative to $(X_{i},X_{j}),\ \forall i,j$.

(ii) Proof of $B\implies A$: we prove the logical equivalent form of $B\implies A$ i.e., $\neg A\implies\neg B$, the contrapositive of $B\implies A$. To this end, assume $D_{top}(\hat{\pi},A)\neq 0$, then there will be at least one edge $X_{i}\rightarrow X_{j}$ that cannot be oriented correctly due to the estimated topological order $\hat{\pi}$. i.e., $\hat{\pi}_{j}<\hat{\pi}_{i}$ but $\pi_{j}>\pi_{i}$. Hence, to find the causal effect of $X_{i}$ on $X_{l};\ l\neq j$, $X_{j}$ is included in the back-door adjustment set $\mathbf{Z}$ relative to $(X_{i},X_{l})$. Adding $X_{j}$ to $\mathbf{Z}$ renders $\mathbf{Z}$ an invalid adjustment set because it violates the condition (i) of Defn [3.2](#S3.Thmtheorem2).
∎

See [4.5](#S4.Thmtheorem5)

###### Proof.

Recall that SHD counts the number of missing, falsely detected, and falsely directed edges in the estimated causal graph as compared to the ground truth graph. Since we want $D_{top}(\hat{\pi},A)=0;\ \forall\hat{\pi}\text{ of }\hat{\mathcal{G}}$, there cannot be an edge $X_{i}\rightarrow X_{j}$ in $\hat{\mathcal{G}}$ such that $X_{i}\leftarrow X_{j}$ is in $\mathcal{G}$. This constraint avoids the possibility of having falsely directed edges in $\hat{\mathcal{G}}$. Consider a $\hat{\mathcal{G}}$ with all the edges in $\mathcal{G}$ and in addition, each variable in level $i$ having a directed edge to each variable in all levels below level $i$. All such edges contribute to the SHD score while still obeying the causal ordering in $\mathcal{G}$. This number will be equal to $\sum\limits_{i=1}^{N-1}(n_{i}\times\sum\limits_{j=i+1}^{N}n_{j})-|\mathbf{E}|$. The quantity $\sum\limits_{i=1}^{N-1}(n_{i}\times\sum\limits_{j=i+1}^{N}n_{j})$ is the number of edges possible from each node to the every other node in the levels below it. We need to subtract the number of existing edges in $\mathbf{E}$ to count the newly added edges that contribute to the SHD score. Now, we can remove some of the edges $X_{i}\rightarrow X_{j}$ from $\hat{\mathcal{G}}$ such that $X_{i}\rightarrow X_{j}$ is in $\mathcal{G}$ while still leading to same causal ordering of variables. This leads to increased SHD score due to missing edges in $\hat{\mathcal{G}}$. Since it will only increase the SHD score, we ignore such corner cases.
∎

###### Proof.

###### Proof.

###### Proof.

## Appendix B Additional Results

Table [A1](#A2.T1) shows the results of various prompt strategies and their improvements over no-prior methods.

**Table A1: $D_{top}$ metric results. Comparison with various prompting strategies. Neuropathic subgraph for 1k samples return cyclic graphs in the MEC. Using LLM CoT prior with Neuropathic 10k samples, orients the undirected edges to create cyclic graphs**
|  |  |  |  | Ours | Ours | Ours | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Dataset | PC | CaMML | (CoT) | (CoT) | (Triplet Pairwise) | (Triplet Pairwise) |
|  |  |  |  | (PC+LLM) | (CaMML+LLM) | (PC+LLM) | (CaMML+LLM) |
|  | Earthquake | 0.30$\pm$0.44 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 2.75$\pm$0.43 | 0.2$\pm$0.44 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Survey | 0.50$\pm$0.00 | 3.33$\pm$0.94 | 0.00$\pm$0.00 | 3.33$\pm$0.94 | 0.00$\pm$0.00 | 2.66$\pm$0.94 |
|  | Asia | 2.33$\pm$0.59 | 1.85$\pm$0.58 | 0.00$\pm$0.00 | 0.97$\pm$0.62 | 1.00$\pm$0.00 | 3.39$\pm$0.62 |
|  | Asia-M | 2.00$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 1.71$\pm$0.45 | 0.00$\pm$0.00 | 1.71$\pm$0.45 |
| $N=250$ | Child | 8.16$\pm$ 1.58 | 3.00$\pm$0.00 | 4.70$\pm$1.34 | 3.53$\pm$0.45 | 8.00$\pm$3.17 | 4.00$\pm$0.00 |
|  | Neuropathic | 3.25$\pm$0.00 | 10.4$\pm$1.95 | 1.00$\pm$0.00 | 8.00$\pm$0.00 | 0.00$\pm$0.00 | 12.63$\pm$1.92 |
|  | Earthquake | 0.85$\pm$0.65 | 0.00$\pm$0.00 | 0.4$\pm$0.89 | 2.60$\pm$0.48 | 1.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 0.00$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Survey | 1.75$\pm$0.00 | 3.39$\pm$0.08 | 1.00$\pm$0.00 | 3.33$\pm$0.94 | 0.60$\pm$0.00 | 2.66$\pm$0.94 |
|  | Asia | 3.00$\pm$0.00 | 3.81$\pm$0.39 | 1.00$\pm$0.00 | 0.97$\pm$0.62 | 0.00$\pm$0.00 | 3.28$\pm$0.64 |
|  | Asia-M | 2.00$\pm$0.00 | 2.00$\pm$0.00 | 1.00$\pm$0.00 | 0.17$\pm$0.45 | 1.00$\pm$0.00 | 1.04$\pm$0.20 |
| $N=500$ | Child | 9.79$\pm$1.17 | 2.00$\pm$0.00 | 4.60$\pm$1.34 | 2.00$\pm$0.00 | 9.20$\pm$2.16 | 3.00$\pm$0.00 |
|  | Neuropathic | 7.50$\pm$0.00 | 5.32$\pm$0.57 | 9.00$\pm$0.00 | 8.90$\pm$0.00 | 9.00$\pm$0.00 | 12.1$\pm$1.56 |
|  | Earthquake | 0.80$\pm$0.57 | 0.00$\pm$0.00 | 0.20$\pm$0.44 | 2.00$\pm$0.00 | 1.40$\pm$1.31 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Survey | 1.00$\pm$0.00 | 2.71$\pm$0.27 | 1.00$\pm$0.00 | 2.83$\pm$0.00 | 1.00$\pm$0.00 | 2.16$\pm$0.74 |
|  | Asia | 3.09$\pm$1.05 | 1.75$\pm$0.43 | 1.75$\pm$0.95 | 0.97$\pm$0.62 | 3.00$\pm$0.00 | 1.78$\pm$0.41 |
|  | Asia-M | 2.50$\pm$0.00 | 2.04$\pm$0.51 | 2.00$\pm$0.00 | 0.65$\pm$0.47 | 1.00$\pm$0.00 | 0.65$\pm$0.47 |
| $N=1000$ | Child | 9.61$\pm$1.07 | 2.00$\pm$0.00 | 8.00$\pm$0.00 | 2.00$\pm$0.40 | 6.6$\pm$1.14 | 2.83$\pm$0.00 |
|  | Neuropathic | - | 5.54$\pm$0.75 | - | 6.00$\pm$0.00 | - | 4.00$\pm$0.00 |
|  | Earthquake | 0.30$\pm$0.67 | 0.80$\pm$0.97 | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Survey | 2.00$\pm$0.00 | 3.33$\pm$0.69 | 2.00$\pm$0.00 | 2.60$\pm$0.00 | 2.00$\pm$0.00 | 1.80$\pm$0.83 |
|  | Asia | 3.05$\pm$0.94 | 1.75$\pm$0.43 | 2.80$\pm$1.30 | 0.97$\pm$0.62 | 0.00$\pm$0.00 | 1.78$\pm$0.41 |
|  | Asia-M | 1.00$\pm$0.00 | 1.68$\pm$0.46 | 0.20$\pm$0.44 | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 1.00$\pm$0.00 |
| $N=5000$ | Child | 8.42$\pm$0.75 | 3.00$\pm$0.00 | 7.00$\pm$0.00 | 3.00$\pm$0.00 | 7.4$\pm$1.51 | 1.00$\pm$0.00 |
|  | Neuropathic | 9.00$\pm$0.00 | 4.20$\pm$0.96 | 9.00$\pm$0.00 | 3.00$\pm$0.00 | 9.00$\pm$0.00 | 3.31$\pm$0.00 |
|  | Earthquake | 0.00$\pm$0.00 | 0.40$\pm$0.48 | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Cancer | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.60$\pm$0.80 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 |
|  | Survey | 2.00$\pm$0.00 | 3.60$\pm$1.35 | 2.00$\pm$0.00 | 1.83$\pm$0.00 | 2.00$\pm$0.00 | 1.08$\pm$0.64 |
|  | Asia | 1.95$\pm$0.41 | 1.40$\pm$0.48 | 1.20$\pm$0.83 | 0.34$\pm$0.47 | 0.00$\pm$0.00 | 1.78$\pm$0.41 |
|  | Asia-M | 1.75$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 1.00$\pm$0.00 |
| $N=10000$ | Child | 7.67$\pm$0.65 | 2.80$\pm$0.84 | 4.66$\pm$3.05 | 1.00$\pm$0.00 | 5.80$\pm$1.48 | 1.00$\pm$0.00 |
|  | Neuropathic | 10.00$\pm$0.00 | 3.00$\pm$0.00 | - | 2.39$\pm$0.48 | 10.00$\pm$0.00 | 1.00$\pm$0.00 |

### B.1 LLMs used in post processing for graph discovery

We conducted some experiments where we utilised discovery algorithms like PC for creating skeletons of the graph and employed LLMs for orienting the undirected edges. The idea was to utilise LLMs ability to correctly estimate the causal direction while leveraging PC algorithm’s ability to give a skeleton which could be oriented in a post processing setup. We saw that LLM ended up giving improved results as compared to PC alone.

**Table A2: PC + LLM results where LLM is used to orient the undirected edges of the skeleton PC returns over different data sample sizes. We show how LLMs can be used in a post processing setup for edge orientation besides having the capability of acting as a strong prior for different discovery algorithms.**
| 1000 samples |  |  |  |  |
| --- | --- | --- | --- | --- |
| Context | Base prompt | Past iteration | Markov Blanket | PC |
|  |  | orientations |  | (Avg. over MEC) |
| $D_{top}$ | $8.0$ | $5.3$ | $6.6$ | $9.61$ |
| SHD | $14.33$ | $12.66$ | $14.0$ | $17.0$ |
| 10000 samples |  |  |  |  |
| $D_{top}$ | $6.33$ | $9.66$ | $6.0$ | $7.67$ |
| SHD | $9.0$ | $13.33$ | $8.33$ | $12.0$ |

**Table A3: Overview of the datasets used.**
| Dataset | Number of | Number of | Description |
| --- | --- | --- | --- |
|  | Nodes | Edges | (used as a context) |
| Asia | 8 | 8 | Model the possible respiratory problems |
|  |  |  | someone can have who has recently visited |
|  |  |  | Asia and is experiencing shortness of breath |
| Cancer | 5 | 4 | Model the relation between various variables |
|  |  |  | responsible for causing Cancer and its possible |
|  |  |  | outcomes |
| Earthquake | 5 | 5 | Model factors influencing the probability of a burglary |
| Survey | 6 | 6 | Model a hypothetical survey whose aim is to investigate |
|  |  |  | the usage patterns of different means of transport |
| Child | 20 | 25 | Model congenital heart disease in babies |
| Neuropathic Pain |  |  |  |
| Diagnosis (subgraph) | 22 | 25 | For neuropathic pain diagnosis |

## Appendix C Causal Graphs used in Experiments

Figures [A1](#A3.F1)-[A5](#A3.F5) show the causal graphs and details we considered from BNLearn repository (Scutari & Denis, ).

Figure: Figure A1: Earthquake Bayesian network. Abbreviations/Descriptions: Burglary: burglar entering, Earthquake: earthquake hitting, Alarm: home alarm going off in a house, JohnCalls: first neighbor to call to inform the alarm sound, Marycalls: second neighbor to call to inform the alarm sound.
Refer to caption: /html/2310.15117/assets/x2.png

Figure: Figure A2: Cancer Bayesian network. Abbreviations/Descriptions: Pollution: exposure to pollutants, Smoker: smoking habit, Cancer: Cancer, Dyspnoea: Dyspnoea, Xray: getting positive xray result.
Refer to caption: /html/2310.15117/assets/x3.png

Figure: Figure A3: Survey Bayesian network. Abbreviations: A=Age/Age of people using transport, S=Sex/male or female, E=Education/up to high school or university degree, O=Occupation/employee or self-employed, R=Residence/the size of the city the individual lives in, recorded as either small or big, T=Travel/the means of transport favoured by the individual.
Refer to caption: /html/2310.15117/assets/x4.png

Figure: Figure A4: Asia Bayesian network. Abbreviations/Descriptions: asia=visit to Asia/visiting Asian countries with high exposure to pollutants, smoke=smoking habit, tub=tuberculosis, lung=lung cancer, either=either tuberculosis or lung cancer, bronc=bronchitis, dysp=dyspnoea, xray=getting positve xray result.
Refer to caption: /html/2310.15117/assets/x5.png

Figure: Figure A5: Child Bayesian network. Abbreviations: BirthAsphyxia: Lack of oxygen to the blood during the infant’s birth, HypDistrib: Low oxygen areas equally distributed around the body, HypoxiaInO2: Hypoxia when breathing oxygen, CO2: Level of carbon dioxide in the body, ChestXray: Having a chest x-ray, Grunting: Grunting in infants, LVHreport: Report of having left ventricular hypertrophy, LowerBodyO2: Level of oxygen in the lower body, RUQO2: Level of oxygen in the right upper quadricep muscle, CO2Report: A document reporting high levels of CO2 levels in blood, XrayReport: Report of having a chest x-ray, Disease: Presence of an illness, GruntingReport: Report of infant grunting, Age: Age of infant at disease presentation, LVH: Thickening of the left ventricle, DuctFlow: Blood flow across the ductus arteriosus, CardiacMixing: Mixing of oxygenated and deoxygenated blood, LungParench: The state of the blood vessels in the lungs, LungFlow: Low blood flow in the lungs, Sick: Presence of an illness
Refer to caption: /html/2310.15117/assets/x6.png

Figure: Figure A6: For Neuropathic dataset, we consider a sub-graph induced by one of the root nodes, containing the following 22 nodes and corresponding edges taken from [https://observablehq.com/@turuibo/the-complete-causal-graph-of-neuropathic-pain-diagnosis](https://observablehq.com/@turuibo/the-complete-causal-graph-of-neuropathic-pain-diagnosis): ‘right C7’, ‘right elbow trouble’, ‘left shoulder trouble’, ‘left bend of arm problem’, ’right shoulder trouble’, ‘right hand problem’, ‘left medival elbow problems’, ‘right finger trouble’, ‘left neck problems’, ‘left wrist problems’, ’left shoulder problem’, ‘right neck’, ‘right wrist problem’, ‘right shoulder problem’, ‘discoligment injury C6 C7’, ‘left hand problem’, ‘left C7’, ‘right arm band’, ‘left lower arm disorders’, ‘neck pain’, ‘left finger trouble’, ‘left arm’. We did not use descriptions for the nodes of Neuropathic graph.
Refer to caption: /html/2310.15117/assets/NeuroSubgraphPlot.png

## Appendix D Various Prompt Strategies Studied

**Table A4: Chain of Thought Prompt**
| Question: For a causal graph used to model relationship of various factors and outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’, ’Smoker’, ’Xray’, ’Dyspnoea’], |
| --- |
| Which cause-and-effect relationship is more likely between nodes ’smoker’ and ’cancer’? |
| A. changing the state of node ’smoker’ causally effects a change in another node ’cancer’. |
| B. changing the state of node ’cancer’ causally effects a change in another node ’smoker’. |
| C. There is no causal relation between the nodes ’cancer’ and ’smoker’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer: The causal effect of "smoker" directing to "cancer" is based on the strong evidence from epidemiological studies linking smoking to an increased risk of developing cancer. Smoking introduces harmful substances into the respiratory system, leading to cellular damage and mutation, which significantly raises the likelihood of cancer development in the lungs or respiratory tract, subsequently impacting the occurrence of respiratory problems like shortness of breath. Therefore answer is <Answer>A</Answer> |
| Question: For a causal graph used to model relationship of various factors and outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’, ’Smoker’, ’Xray’, ’Dyspnoea’],<br>Which cause-and-effect relationship is more likely between nodes ’xray’ and ’dyspnoea’? |
| A. changing the state of node ’xray’ causally effects a change in another node ’dyspnoea’. |
| B. changing the state of node ’dyspnoea’ causally effects a change in another node ’xray’. |
| C. There is no causal relation between the nodes ’xray’ and ’dyspnoea’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer: Reasoning behind the lack of causal relation between X-ray and dyspnoea is that X-ray and dyspnoea are both effects of having cancer, but they do not directly cause or affect each other. X-ray is a diagnostic test that can help detect cancer in the lungs or other organs, while dyspnoea is a symptom of cancer that involves feeling short of breath. Therefore, X-ray and dyspnoea are not causally related, but they are both associated with cancer. Therefore answer is<br><Answer>C</Answer> |
| Question: For a causal graph used to model relationship of various factors and outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’, ’Smoker’, ’Xray’, ’Dyspnoea’],<br>Which cause-and-effect relationship is more likely between nodes ’xray’ and ’cancer’? |
| A. changing the state of node ’xray’ causally effects a change in another node ’cancer’. |
| B. changing the state of node ’cancer’ causally effects a change in another node ’xray’. |
| C. There is no causal relation between the nodes ’xray’ and ’cancer’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |

**Table A5: Chain of Thought Prompt (continued..)**
| Answer:  The causal effect of cancer on X-ray is that X-rays are often used to diagnose or detect cancer in different parts of the body, such as the bones, lungs, breasts, or kidneys123. X-rays use low doses of radiation to create pictures of the inside of the body and show the presence, size, and location of tumors. X-rays can also help monitor the response to treatment or check for signs of recurrence. Therefore, having cancer may increase the likelihood of getting an X-ray as part of the diagnostic process or follow-up care. Therefore answer is <Answer>B</Answer> |
| --- |
| Question: For a causal graph used to model relationship of various factors and outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’, ’Smoker’, ’Xray’, ’Dyspnoea’],<br>Which cause-and-effect relationship is more likely between nodes ’pollution’ and ’cancer’? |
| A. changing the state of node ’pollution’ causally effects a change in another node ’cancer’. |
| B. changing the state of node ’cancer’ causally effects a change in another node ’pollution’. |
| C. There is no causal relation between the nodes ’pollution’ and ’cancer’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer:  The causal effect of pollution on cancer is that air pollution contains carcinogens (cancer-causing substances) that may be absorbed into the body when inhaled and damage the DNA of cells. Another possible reasoning is that particulate matter (tiny dust-like particles) in air pollution may cause physical damage to the cells in the lungs, leading to inflammation and oxidative stress and eventually cell mutations. A third possible reasoning is that air pollution may create an inflamed environment in the lungs that encourages the proliferation of cells with existing cancer-driving mutations. These are some of the hypotheses that researchers have proposed to explain how air pollution may cause cancer, but more studies are needed to confirm them. Therefore answer is <Answer>A</Answer> |
| Question: For a causal graph used to model relationship of various factors and outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’, ’Smoker’, ’Xray’, ’Dyspnoea’],<br>Which cause-and-effect relationship is more likely between nodes ’pollution’ and ’smoker’? |
| A. changing the state of node ’pollution’ causally effects a change in another node ’smoker’. |
| B. changing the state of node ’smoker’ causally effects a change in another node ’pollution’. |
| C. There is no causal relation between the nodes ’pollution’ and ’smoker’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer: Reason behind the lack of causal relation between pollution and smoker is that pollution and smoking are both independent risk factors for respiratory problems, but they do not directly cause or affect each other. Pollution and smoking both contribute to air pollution, which can harm the health of people and the environment. However, pollution is mainly caused by human activities such as burning fossil fuels, deforestation, or industrial processes, while smoking is a personal choice that involves inhaling tobacco smoke. Therefore, pollution and smoker are not causally related, but they are both associated with respiratory problems. Therefore answer is <Answer>C</Answer>. |

**Table A6: Chain of Thought Prompt (continued..)**
| Question: For a causal graph used for modeling factors causing Coronary Heart Diseases with the following nodes: [’Family Disease’, ’Gene’, ’Smoking’, ’Blood Pressure’, ’Coronary Heart Disease’, ’Headache’],<br>Which cause-and-effect relationship is more likely between nodes ’Family Disease’ and ’Gene’? |
| --- |
| A. changing the state of node ’Family Disease’ causally effects a change in another node ’Gene’. |
| B. changing the state of node ’Gene’ causally effects a change in another node ’Family Disease’. |
| C. There is no causal relation between the nodes ’Family Disease’ and ’Gene’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer: Reason behind the causal effect of family disease on gene is that family disease is a term that refers to diseases or health conditions that run in the family, meaning that they are influenced by genetic factors. Gene is a term that refers to the basic unit of heredity that carries information for a specific trait or function. Family disease can affect gene by altering the type or frequency of genes that are inherited by the offspring from their parents. For example, some family diseases are caused by deterministic genes, which are genes that guarantee the development of a disease if they are present in a person’s genome. Other family diseases are influenced by risk genes, which are genes that increase the likelihood of developing a disease but do not guarantee it. Therefore, family disease can causally direct to gene by changing the genetic makeup of a person or a population. Therefore answer is <Answer>A</Answer>. |
| Question: For a causal graph used for modeling factors causing Coronary Heart Diseases with the following nodes: [’Family Disease’, ’Gene’, ’Smoking’, ’Blood Pressure’, ’Coronary Heart Disease’, ’Headache’],<br>Which cause-and-effect relationship is more likely between nodes ’Coronary Heart Disease’ and ’Gene’? |
| A. changing the state of node ’Coronary Heart Disease’ causally effects a change in another node ’Gene’. |
| B. changing the state of node ’Gene’ causally effects a change in another node ’Coronary Heart Disease’. |
| C. There is no causal relation between the nodes ’Coronary Heart Disease’ and ’Gene’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer: Possible reasoning behind the causal effect of gene on coronary heart disease is that gene is a term that refers to the basic unit of heredity that carries information for a specific trait or function. Gene can affect coronary heart disease by influencing the structure and function of the blood vessels, the metabolism and transport of lipids (fats) in the blood, the inflammation and clotting processes, or the response to environmental factors such as smoking or diet. For example, some genes code for proteins that regulate the cell cycle and growth of the cells that line the arteries, which can affect their susceptibility to damage or plaque formation. Other genes code for proteins that control the synthesis and clearance of cholesterol or other lipids, which can affect their levels and deposition in the arteries. Therefore, gene can causally direct to coronary heart disease by modifying the biological pathways that contribute to the development or progression of the disease. Therefore answer is <Answer>B</Answer> |

**Table A7: Chain of Thought Prompt (continued..)**
| Question: For a causal graph used for modeling factors causing Coronary Heart Diseases with the following nodes: [’Family Disease’, ’Gene’, ’Smoking’, ’Blood Pressure’, ’Coronary Heart Disease’, ’Headache’],<br>Which cause-and-effect relationship is more likely between nodes ’Blood Pressure’ and ’Smoking’? |
| --- |
| A. changing the state of node ’Blood Pressure’ causally effects a change in another node ’Smoking’. |
| B. changing the state of node ’Smoking’ causally effects a change in another node ’Blood Pressure’. |
| C. There is no causal relation between the nodes ’Blood Pressure’ and ’Smoking’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer: Possible reasoning behind the causal effect of smoking on blood pressure is that smoking is a habit that involves inhaling tobacco smoke, which contains nicotine and other harmful chemicals. Smoking can affect blood pressure by activating the sympathetic nervous system (SNS), which is the part of the nervous system that controls the body’s response to stress or danger. When the SNS is activated, it releases hormones such as adrenaline and noradrenaline, which cause the heart to beat faster and harder, and the blood vessels to constrict. This results in a temporary increase in blood pressure, which can last for 15 to 20 minutes after each cigarette. Therefore, smoking can causally direct to blood pressure by stimulating the SNS and increasing the cardiac output and vascular resistance. Therefore answer is <Answer>B</Answer>. |
| Question: For a causal graph used for modeling factors causing Coronary Heart Diseases with the following nodes: [’Family Disease’, ’Gene’, ’Smoking’, ’Blood Pressure’, ’Coronary Heart Disease’, ’Headache’],<br>Which cause-and-effect relationship is more likely between nodes ’Headache’ and ’Smoking’? |
| A. changing the state of node ’Headache’ causally effects a change in another node ’Smoking’. |
| B. changing the state of node ’Smoking’ causally effects a change in another node ’Headache’. |
| C. There is no causal relation between the nodes ’Headache’ and ’Smoking’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer: One possible reasoning behind the lack of causal relation between headache and smoking is that headache and smoking are both associated with various health conditions, but they do not directly cause or affect each other12. Headache is a term that refers to pain or discomfort in the head, scalp, or neck, which can have many possible causes, such as stress, dehydration, infection, injury, or medication. Smoking is a habit that involves inhaling tobacco smoke, which contains nicotine and other harmful chemicals, which can increase the risk of diseases such as cancer, heart disease, stroke, and lung disease. Therefore, headache and smoking are not causally related, but they are both linked to different health problems. Therefore the answer is <Answer>C</Answer> |

**Table A8: Chain of Thought Prompt (continued..)**
| Question: For a causal graph used for modeling factors causing Coronary Heart Diseases with the following nodes: [’Family Disease’, ’Gene’, ’Smoking’, ’Blood Pressure’, ’Coronary Heart Disease’, ’Headache’],<br>Which cause-and-effect relationship is more likely between nodes ’Headache’ and ’Smoking’? |
| --- |
| A. changing the state of node ’Headache’ causally effects a change in another node ’Smoking’. |
| B. changing the state of node ’Smoking’ causally effects a change in another node ’Headache’. |
| C. There is no causal relation between the nodes ’Headache’ and ’Smoking’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer: One possible reasoning behind the lack of causal relation between headache and smoking is that headache and smoking are both associated with various health conditions, but they do not directly cause or affect each other. Headache is a term that refers to pain or discomfort in the head, scalp, or neck, which can have many possible causes, such as stress, dehydration, infection, injury, or medication. Smoking is a habit that involves inhaling tobacco smoke, which contains nicotine and other harmful chemicals, which can increase the risk of diseases such as cancer, heart disease, stroke, and lung disease. Therefore, headache and smoking are not causally related, but they are both linked to different health problems. Therefore the answer is <Answer>C</Answer> |
| Question: For a causal graph used for modeling factors causing Coronary Heart Diseases with the following nodes: [’Family Disease’, ’Gene’, ’Smoking’, ’Blood Pressure’, ’Coronary Heart Disease’, ’Headache’],<br>Which cause-and-effect relationship is more likely between nodes ’Coronary Heart Disease’ and ’Smoking’? |
| A. changing the state of node ’Smoking’ causally effects a change in another node ’Coronary Heart Disease’. |
| B. changing the state of node ’Coronary Heart Disease’ causally effects a change in another node ’Smoking’. |
| C. There is no causal relation between the nodes ’Coronary Heart Disease’ and ’Smoking’. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |
| Answer: Possible reasoning behind the causal effect of smoking on coronary heart disease is smoking damages the heart and blood vessels by raising triglycerides, lowering HDL, increasing blood clotting, and impairing blood flow to the heart. This can lead to plaque buildup, heart attacks, and death. Therefore answer is<br><Answer>A</Answer>. |
| Question: For a causal graph used for context with the following nodes: nodes,<br>Which cause-and-effect relationship is more likely between nodes X and Y? |
| A. changing the state of node X causally effects a change in another node Y. |
| B. changing the state of node Y causally effects a change in another node X. |
| C. There is no causal relation between the nodes X and Y. |
| Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: <Answer>A/B/C</Answer>.<br>It is very important that you output the final Causal graph within the tags like <Answer>A/B/C</Answer> otherwise your answer will not be processed. |

**Table A9: Base prompt**
| Which cause-and-effect relationship is more likely? |
| --- |
| A. changing the state of node which says X causally effects a change in another node which says Y. |
| B. changing the state of node which says Y causally effects a change in another node which says X. |
| C. There is no causal relationship between node X and Y. |
| Make sure to first output a factually grounded reasoning for your answer. X and Y are nodes of a Causal Graph.<br>The causal graph is sparse and acyclic in nature. So option C could be chosen if there is some uncertainity about causal relationship between X and Y. |
| First give your reasoning and after that please make sure to provide your final answer within the tags <Answer>A/B/C</Answer>. |
| It is very important that you output your final answer between the tags like <Answer>A/B/C</Answer> otherwise your response will not be processed. |

**Table A10: Iterative orientation prompt**
| For the nodes X and Y which form an edge in a Causal Graph, you have to identify<br>which cause-and-effect relationship is more likely between the nodes of the edge. This will be used to rearrange the nodes in the edge<br>to create a directed edge which accounts for causal relation from one node to another in the edge. |
| --- |
| A. changing the state of node X causally affects a change in another node Y. |
| B. changing the state of node Y causally affects a change in another node X. |
| C. There is no causal relation between the nodes X and Y. |
| You can also take the edges from the skeleton which have been rearranged to create a directed edge to account for causal relationship between the nodes: directed_edges. |
| Make sure to first output a factually grounded reasoning for your answer. First give your reasoning and after that please make sure to<br>provide your final answer within the tags <Answer>A/B/C</Answer>. |
| It is very important that you output your final answer between the tags like <Answer>A/B/C</Answer> otherwise your response will not be processed. |

**Table A11: Markov Blanket prompt**
| For the following undirected edge in a Causal Graph made of nodes X and Y, you have to identify<br>which cause-and-effect relationship is more likely between the nodes of the edge. This will be used to rearrange the nodes in the edge<br>to create a directed edge which accounts for causal relation from one node to another in the edge. |
| --- |
| A. changing the state of node X causally effects a change in another node Y. |
| B. changing the state of node Y causally effects a change in another node X. |
| C. There is no causal relation between the nodes X and Y. |
| You can also take the other directed edges of nodes X: X_edges and Y: Y_edges of the Causal graph as context to redirect the edge to account for causal effect. |
| Make sure to first output a factually grounded reasoning for your answer. First give your reasoning and after that please make sure to<br>provide your final answer within the tags <Answer>A/B/C</Answer>. |
| It is very important that you output your final answer between the tags like <Answer>A/B/C</Answer> otherwise your response will not be processed. |

**Table A12: Example LLM (GPT-3.5-turbo) reasoning outputs for estimating causal directionality between different pairs of nodes using CoT Prompt (refer Table [A4](#A4.T4) for the prompt) for Neuropathic subgraph (used for pain diagnosis).**
| Input: (‘Right C7’,‘Discoligment injury C6-C7’) |
| --- |
| Answer: Discoligment injury C6-C7 can cause compression of the nerve roots that exit the spinal cord at the C7 level, which can lead to symptoms such as pain, numbness, and weakness in the right C7 dermatome. Therefore, the answer is <Answer>B</Answer>. |
| Input: (‘Right C7’, ‘Left C7’) |
| Answer: Right C7 and left C7 are both parts of the cervical spine and are not known to directly influence each other. Therefore, the answer is <Answer>C</Answer>. |
| Input: (‘Right elbow trouble’, ‘Left shoulder trouble’) |
| Answer: There is no direct causal relationship between right elbow trouble and left shoulder trouble. They may both be symptoms of a larger underlying condition, but they do not directly cause or affect each other. Therefore the answer is <Answer>C</Answer>. |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: + Why do you think the premise of LLMs as $\epsilon$-imperfect experts makes sense more generally in causal discovery, given that $\epsilon$ *should* large if we're solving a truly interesting causal discovery problem?

+ Can you the details about the empirical studies that I mention above?


Edit: The authors satisfactorily addressed my questions and concerns in their response; I'm happy to thus raise my score and support their work.
soundness: 4
strengths: + The technical as well as written clarity of the paper is excellent. Technically, the notation is clear, all preliminaries and results are well-defined and well-explained. The paper is also written in a way that is easy to read and follow.

+ The technical result about the optimality of the triplet querying strategy over the pairwise querying strategy is novel, to the best of my knowledge, and could have an impact on causal discovery algorithms more broadly.

+ The empirical studies are comprehensive, and the initiative to contrast both humans and LLM imperfect experts is impressive since studies with humans are time-consuming and costly.
summary: This paper considers optimal ways of querying imperfect experts (e.g., LLMs) when the aim is to discover a causal DAG over a set of variables. The key proposed idea is to query imperfect experts about the causal ordering over variables instead of graphical relationships like edges. The paper shows that certain ways of orienting edges based on queries to experts can lead to errors in the overall DAG, while the causal ordering from a perfect expert will never be incorrect. The paper proposes ways of integrating causal ordering into known causal discovery algorithms, and then focuses on extracting orderings from imperfect experts. To minimize errors by imperfect experts, the paper proposes querying for the order over triplets of variables while enforcing acyclicity, proving that this makes fewer errors than querying about pairs of variables (under some assumptions). The work then studies this querying strategy empirically using a variety of hand-curated known causal DAGs. They study both human and LLM imperfect experts, and find evidence that the triplet querying strategy improves causal discovery and effect estimation performance over both the pairwise querying strategy and using no experts at all.
weaknesses: + In the broader context of causal discovery, I question the idea that LLMs are $\epsilon$-imperfect experts. Existing results including those in this work consistently consider causal discovery problems that involve standard medical knowledge, or general knowledge about the world that's fairly well-established by now. However, causal discovery as a field ought to be about discovering new knowledge, in domains where we have very only faint hypotheses about variables relate, e.g., consider abstract variables that represent aspects of human behavior like "trust" or "aggression." In these settings, we should expect $\epsilon$ to be so large as to render LLM experts unreliable. I'm curious to see the authors better justify and contextualize the role of LLMs in genuinely challenging causal discovery problems.

+ The premise in this paper is that querying experts with questions of the form "does A cause B" and directing the edge A->B if the expert says yes leads to incorrect graphs. However, the query "does A cause B" isn't by itself inherently flawed -- it's already a question about causal ordering -- the policy to orient edges based on ancestral causal relations is what's flawed. Nevertheless, the introduction and parts of section 3 (e.g., definition 3.1) seem to critique the query itself, when really, the issue is that responses to this query should not be used to directly orient edges (e.g., the perfect expert in 3.1 is defined based on a flawed policy). Therefore, the argumentation strategy in the paper feels like setting up a strawman argument. Note, though, that this is just a critique about the storytelling; the technical results are valid and make sense regardless. One could pivot the story slightly to argue that the flaw isn't with the "does A cause B" style of query, but in ensuring that this is interpreted only as a causal ordering.

+ There is a bit of extra context that I would have liked in the empirical studies: 1) can you contextualize the datasets and graphs that are studied further and discuss whether they are semi-synthetic, purely realistic, etc.? 2) Despite the results in Figure 3 on synthetic data, it would be useful in the broader empirical studies to compare the causal ordering queries to the flawed policy of orienting edges based no responses to "does A cause B"-style questions. (I apologize if this comparison is indeed there and I missed it -- if that ended up being the case, please point me to the correct figure or table in the empirical studies.)

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - While domain expert knowledge is independent of the number of observations, it significantly affects the causal discovery algorithm. Therefore, it raises curiosity about how much performance improvement the proposed method can achieve in situations of data scarcity. Could you conduct an experiment with a very small dataset as a showcase for a data scarcity regime? It would be very interesting.
soundness: 3
strengths: - The paper is well-written. Motivation, method, and experiments are logically straightforward.
- The attempts to utilize LLM for causal discovery have identified certain problems, such as the difficulty in distinguishing between direct and indirect causal relationships, which consequently lead to cycles in the causal graph. The authors convincingly argue that these issues are inherent to the pairwise prompting approach.
- The authors propose using causal order to leverage domain expert knowledge in causal discovery without encountering the issues associated with pairwise prompting. Additionally, it presents a novel prompting strategy to support this approach.
- When the proposed method was utilized, performance improvement was consistently observed across experiments. Additionally, the experimental design accommodates node size scaling from small to large datasets, allowing for a comprehensive understanding of how the method is affected by node size.
summary: The authors aimed to find a better interface to use domain knowledge for causal discovery, including LLM, but not restricted. For LLM, this is done via a pairwise prompting strategy. The authors found that expert knowledge often falls short in distinguishing direct and indirect causal relations between pairwise variables, correspondingly many cycles in causal graphs, but their causal order is relatively precise. Given this finding, the authors suggest using expert knowledge of causal order instead of causal relation. While integrating the strategy of using causal order and LLM prompting, the authors propose to introduce an auxiliary variable to improve the conventional pairwise prompting strategy by avoiding cycles within the triplet. Experimental results support that the proposed prompting methods increase the robustness and performance.
weaknesses: Overall, I found this work very interesting, though a few questions remain unanswered. Addressing these issues could potentially lead to a higher score.

- It is hard to determine whether triplet prompting is effective for high-performance LLMs (such as GPT-4, Claude3-opus). It appears that the prompting strategy and LLM can be applied orthogonally, so it is weird the corresponding results are omitted. Could you add results for Triplet prompting + high-performance LLMs in Table 2, to clarify this matter?
- In the experimental design related to the downstream causal discovery algorithm, it is difficult to determine how the proposed method's advantages change with varying observations. Could you add corresponding experiments with varying sample sizes to clarify this matter?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Q1: Can you clarify the experimental setup in Table 4? The title shows ‘ours - < causal discovery > + LMM’. But in line 279 it is described that “causal order is used to reduce search space of causal discovery methods”. In that sense, should it be LLM + < causal discovery > instead? As the LLM with triplet is used to reduce the search space inside the causal discovery method. 

Q2: Can you describe in more details the causal effect component? How is the causal order used in that context? Is it used to drop variables (aka, no informative features or down-stream features)? How are the counterfactuals estimated? 

Q3: Another ablation study that would be helpful is to see the robustness to the number of variables. Most causal discovery methods struggle to scale for larger graphs / number of features. How LLMs + triplet could be adopted to reduce search space and improve this current causal discovery bottleneck?
soundness: 3
strengths: Well-written motivation and discussion of limitations; 

Good discussion on the utility of the causal order, and related works. 

Sound experiments with empirical evidence of the effectiveness of the proposed method.
summary: LLMs are often used as an expert for finding causal graphs, but they can’t distinguish between direct and indirect edges. Authors proposed to use causal order as a more stable version of it, along with a triplet prompt approach. 
- Their proposed approach facilitates the recovery of causal order despite imperfect experts. 
- Prompt strategy that introduces auxiliary variables that introduces an auxiliary variable for every variable pair and instructs the LLM to avoid cycles within this triplet. 
- Triplet prompt leads to fewer cycles than pairwise	
- Causal order is a simpler structure that can still encode helpful information for down-stream tasks;
weaknesses: Despite a sound experimental analysis, there a few details missing, i.e., how many repetitions were done in Tables 1 and 2, 

One potential area not explored/mentioned is what happens with a large number of features. As the proposed method decreases the feature space, would the proposed method also facilitate the adoption of causal discovery areas that contains larger graphs?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The current method produces a causal order rather than a full causal graph. While the authors suggest that, in the absence of confounders, this causal order can offer a superset of backdoor adjustment sets, the presence of “imperfect experts” in real-world applications may lead to additional error accumulation in the causal order. This raises uncertainty about whether the derived causal order is sufficiently reliable for downstream tasks, such as causal effect estimation, and how significantly these biases could impact the results. Experimental validation would be valuable to assess the practicality and risks of this trade-off.
2. In the triplet method, the authors employ a four-step process to determine causal order. Unlike the pairwise approach, each iteration uses a third auxiliary variable \(C\) to determine the causal edge direction between \(A\) and \(B\), followed by aggregation across all variables. My understanding is that the pairwise approach is simply a special case of this method where the number of auxiliary variables equals zero. The authors could further clarify the distinctions and differences between the pairwise approach under this four-step method and the current approach.
soundness: 2
strengths: The approach introduces the use of causal order as the output for causal discovery algorithms employed by LLMs, with D_{top} topological divergence as a metric. This novel approach brings a fresh perspective and insight into the fields of LLMs and causal discovery.
summary: This paper proposes a new technique for causal discovery algorithms by leveraging triplets. The authors argue that traditional causal discovery algorithms based on pairwise variable relationships cannot handle situations involving mediating variables and can lead to circular structures in causal graphs. Thus, they emphasize the importance of utilizing causal order, meaning modeling based on the topological order among variables. Causal order is effective for downstream tasks, and the triplet approach can efficiently identify causal order. The authors discuss the error of estimated causal order from both empirical and theoretical perspectives, especially in the presence of imperfect experts.
weaknesses: 1. Typo error: Lines 333-334 seem to have missed a closing parenthesis in O(V^3).
2. Complexity considerations: According to the paper, the triplet-based method has a complexity of O(V^3), whereas the traditional pairwise method has a complexity of O(V^2). For larger-scale graphs, the time complexity of these methods poses significant limitations for causal discovery.
3. The approach of using LLMs to simply determine causal relationships between variables is challenging to implement effectively in real-world applications. It requires the large model itself to possess a high level of expert knowledge, which is often difficult to achieve, as real causal inference typically demands deep domain-specific expertise and nuanced understanding beyond general-purpose language models.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Experiment - Missing reference to Table 2 in the main paper.
2. Line 114 - Could the authors provide some justification on why the redundancy leads to a more reliable order?
3. Line 52 - Would using triplet prompt lead to a more efficient method in this sense?
soundness: 2
strengths: 1. Regarding the first key contribution, I find the proposal of identifying causal orders rather than full causal graphs to be both useful and insightful.
2. Overall, the paper is easy to follow. While there are some typos, the writing remains generally clear.
summary: This paper explores the use of large language models (LLMs) for causal inference tasks, particularly causal discovery. The authors present two key arguments: (1) For both LLMs (termed as imperfect experts) or perfect experts, it is more effective to identify the topological order of causal variables rather than directly attempting to discover the full causal graph. They demonstrate that this approach yields more robust results and remains valuable for downstream tasks such as causal discovery or treatment effect estimation. (2) Since LLMs are imperfect experts, relying on existing pairwise prompts to determine causal order may still result in cycles. To address this, the authors propose a triplet-based prompt design. They empirically validate their claims on several datasets.

**Claim**: I have research experience in causality. However, I have limited experience in LLMs or LLMs for causality. Hence, I might miss something especially when it comes to evaluation of novelty or comparison with existing works.
weaknesses: I will present my main concerns here. Minor concerns or questions can be found in the Question section.

**Significance of contribution**

I have several concerns regarding the methodology section:
1. 3.1 - While I agree with the intuition that identifying causal order is easier and leads to fewer errors, I’m not sure the result in this subsection is particularly interesting by itself. It seems almost obvious, as knowing the correct causal order is a prerequisite for identifying the causal graph. What matters more is whether the reduced chance of errors still preserves the usefulness of the method.
2. 3.2 - This section should have provided evidence addressing the question above. However, I find it difficult to assess the significance of the method presented. Proposition 3.2 offers a sufficient condition for backdoor adjustment, but applying this condition broadly to include everything that satisfies it seems impractical.
3. 4.1 - Should these be considered as contributions of the paper?
4. 4.2 - The authors try to justify why triplet prompt is better than pairwise prompt here. One thing unclear to me is that, as more variables are provided in the context (as the authors also mentioned), there is a higher chance of LLMs making mistakes. In practice, does that mean an $\epsilon$-expert LLM would become an $\epsilon’$-expert where $\epsilon’>\epsilon$? If so, would this become a trade-off and how should we interpret this tradeoff?
5. 4.2 - Another question regarding this theorem: It seems that the major problem the authors try to resolve is to avoid cycles. However, Proposition 4.1 focuses on the error of edge prediction given the assumption of acyclicity. Hence, I am not sure if this theorem provides insight on how to resolve the key problem of this paper.

**Experiment**

My main concern is the lack of comparison with existing LLM-based methods. For example, while it’s valuable to verify that providing causal order helps with downstream causal tasks, do we know if integrating causal order into causal discovery outperforms directly using LLMs to identify the causal graph or providing other types of inputs to a causal discovery algorithm? Could the authors justify their choice not to compare against existing baselines, such as those mentioned in Section 2?

Meta-review: The paper introduces a strategy for querying imperfect experts (such as LLMs) for causal discovery, focusing on causal ordering over graphical relationships.


Strengths:

+ The paper proposed triplet querying strategy for causal discovery.


+ The inclusion of both human and LLM experts in experiments provides valuable insights, especially given the cost and time involved in human studies.

Weaknesses:

+ The paper lacks justification of the use of LLMs as imperfect experts, especially in challenging causal discovery tasks where LLMs might be unreliable.

+ The critique of the "does A cause B" question feels misplaced, as the issue lies more in how edge orientation is determined, not the query itself.

+ Some key details in the empirical studies, such as the dataset context and variations in experimental setups, are missing or unclear, limiting the robustness of the findings.

Decision: Accept (Poster)