Causal Order: The Key to Leveraging Imperfect Experts in Causal Inference
Authors: Aniket Vashishtha
Abbavaram Gowtham Reddy
Abhinav Kumar
Saketh Bachu
Vineeth N. Balasubramanian
Amit Sharma
Venue: ICLR 2025 Poster (2025)
Keywords: Causal Order
Imperfect Experts
Causal Inference
LLMs
Primary area: causal reasoning

Abstract: Large Language Models (LLMs) have recently been used as experts to infer causal graphs, often by repeatedly applying a pairwise prompt that asks about the causal relationship of each variable pair. However, such experts, including human domain experts, cannot distinguish between direct and indirect effects given a pairwise prompt. Therefore, instead of the graph, we propose that causal order be used as a more stable output interface for utilizing expert knowledge. When querying a perfect expert with a pairwise prompt, we show that the inferred graph can have significant errors whereas the causal order is always correct. In practice, however, LLMs are imperfect experts and we find that pairwise prompts lead to multiple cycles and do not yield a valid order. Hence, we propose a prompting strategy that introduces an auxiliary variable for every variable pair and instructs the LLM to avoid cycles within this triplet. We show, both theoretically and empirically, that such a triplet prompt leads to fewer cycles than the pairwise prompt. Across multiple real-world graphs, the triplet prompt yields a more accurate order using both LLMs and human annotators as experts. By querying the expert with different auxiliary variables for the same variable pair, it also increases robustness---triplet method with much smaller models such as Phi-3 and Llama-3 8B outperforms a pairwise prompt with GPT-4. For practical usage, we show how the estimated causal order from the triplet method  can be used to reduce error in downstream discovery and effect inference tasks.

Published as a conference paper at ICLR 2025
CAUSAL ORDER: THE KEY TO LEVERAGING IMPERFECT EXPERTS IN CAUSAL INFERENCE
Aniket Vashishtha' Abbavarrmm Gowtham Reddy?; "eeddy?AAbiinn Abbinay Kumar), Saketh Bachu", Vineeth N Balasubramanian Amit Sharma² 'UIUC PCISPA Helmholtz Center for Information Security, Germany, 'IT, 'IIT Hyderabad, India, 'Microsoft Research, India aniketv28illinet edu, gowtham. bobbavaramecisee de, akumar03@mit. edu, sa wakethwnit@@mee 1. com, vineeth, nb@mi crosoft. www.orrosootttttooommmmmmmmocc.. com, amshar r@mi crosoft.com
ABSTRACT
Large Language Models (LLMs) have been used as experts to infer causal graphs, often by repeatedly applying a pairwise prompt that asks about the causal rela- tionship of each variable pair. However, such experts, including human domain experts, cannot distinguish between direct and indirect effects given a pairwise prompt. Therefore, instead of the graph, we propose that causal order be used as a more stable output interface for utilizing expert knowledge. Even when querying a perfect expert with a pairwise prompt, we show that the inferred graph can have significant errors whereas the causal order is always correct. In practice, however, LLMs are imperfect experts and we find that pairwise prompts lead to multiple cycles. Hence, propose the triplet method, novel querying strategy that intro- duces an auxiliary variable for every variable pair and instructs the LLM to avoid cycles within this triplet. It then uses voting-based ensemble method that results in higher accuracy and fewer cycles while ensuring cost efficiency. Across multiple real-world graphs, such a tripler-based method yields a more accurate order than the pairwise prompt, using both LLMs and human annotators. The triplet method enhances robustness by repeatedly querying an expert with different auxiliary variables, enabling smaller models like Phi-3 and Llama-3 8B to surpass GPT-4 with pairwise prompting. For practical usage, we show how the causal order from the triplet method can be used to reduce error in downstream graph discovery and effect inference tasks.
INTRODUCTION
Based on evidence that LLMs' domain knowledge, even if imperfect, can be used to decide the direction of causal relationship between a pair of variables (Kıeıman al., 2023; Willig et al., 2022), recent years have seen the use of LLMs for inferring the entire causal graph for given problem domain. This done by typically invoking a pairwise prompt-of the form: "does variable A cause variable B?" m?"-multiple times for different pairs of variables (Long et al., 2022; Antonucci et al., 2023; Kıeıman al., 2023; Cohrs al., 2023). other related efforts, causal graphs or edges obtained from LLMs are used Khatibi et al., 2024; Ban et al., 2023a) for causal discovery algorithms, showing that LLM-derived graphs enhance downstream graph discovery accuracy.
prior (Takayama et al., 2024) or constraint (Long et al., 2023;
However, we highlight a key limitation of using graphs as the output interface for such domain knowledge inferred from LLMs, for that matter, even other imperfect experts (c.g., humans). Obtaining the complete graph requires distinguishing between direct and indirect effects among variables. Given only pair of variables, it is not possible to decide whether edge exists or is
1 Work primarily done as Research Fellow at MSR India, with additional contributions made at UTUC. 6 Work primarily done Hydeeadraba,,, with additional contributions made CISPA Work primarily done at UIT Hyderabad, with additional contributions made at MSR India.
Code: https:/ //ss://g/ //gi iigtthe thub. com/ An ket Vashi MatthaaCussa __rrder___ Imper fect, Feec__expeettteeeeeeeeeee

---

Published as a conference paper at ICLR 2025
Pollution
Smaking
Pairwise Prompt
Triplet Prompt
cause
cause BP 8 cause AT Or
Provice sousal rdsllonships betweon
True Graph
Cancer
PERFECT EXPERT (DOMAIN EXPERT)
X-ray
SHID O, Diog
A>
9HD
C+AR
Causal Order: A
Pollution
Smoking
Erro to avold cydle
Cancer
IMPERFICT EXPERT (LLMa)
CYCLE!!
X-ray
CPAXB
SHD Nonc, Caueal Order: None Gausal Order: C A
BHD +1, Dop O
Figure 1: Cancer dataset (Scu- tari Denis, 2014): Top: True causal graph. Bottom: Expert- estimated causal graph. Note that the latter, while not correct wrt. the true graph, yields the correct causal order.
Using the pairwise prompt, even under perfect expert (e.g., domain expert), the estimated graph may not be correct (SHD = 1). Causal order, however, is correct (Diop 0) and hence better metric. Bottom: under imperfect experts such as LLMss, pairwise prompts may not lead to valid order, creating cycles. The proposed triplet prompting strategy alleviates this issue to provide better estimates of causal order (Dtop = 0).
Figure 2: Top:
mediated by another variable, even for a perfect human expert-the existence of an edge depends on which other variables considered to be a part of the node set in the query. For example. consider the true data- gentratnnng process, Smoking + Lung Damage + Respiratory Diseases. If an expert is asked whether there should be direct causal edge from Smoking to Respiratory Diseases, they would answer "Yes", which may not capture the true process. However, if they are told that the set of observed variables additionally includes Lung Damage, then the correct answer would be to not create a direct edge between Smoking and Respiratory Diseases, but rather create edges mediated through Lung Damage. In large graphs, keeping track of the different variables that can affect a given pairwise decision can be cumbersome.
As another example showing the subjectivity of deciding direct or indirect edges, consider the scenario in Fig 1 with the variables: Pollution Exposure, Cancer, Dyspnoea, Smoking History and Positive X-ray. When queried only for the presence of a causal edge from Pollution to Dyspnoea (shortness of breath), an expert may answer "Yes". However, if one has to provide complete graph, it may be non-trivial for an expert to decisively agree on adding direct edge from Pollution to Dyspnoea, creating edges mediating through Cancer, or both.
Causal Order: Significance and Utility. We instead propose causal order as a more stable approach to obtain experts' domain knowledge. Causal order is defined as the topological ordering over graph variables. Since the causal order does not distinguish between direct and indirect effects, in both examples above, the causal order is unique and unambiguous. In the first example, Smoking < Respiratory Diseases is a valid causal order (a < b indicates that a occurs before b in a casual process). Similarly, in the second example, the causal order, Pollution Dyspnoea holds true in all three cases considered above by the expert. Formally, we show that for an (optimal) perfect expert that is given only a pair of variables at time, the predicted causal graph can be incorrect but the predicted causal order is always correct. As a result, the standard practice of obtaining a causal graph from LLMs and growd-sourced annotators (using pairwise questions) may introduce errors in inferred edges, which can be mitigated by using causal order in downstream causal algorithms. Order a stable causal construct, independent of other variables present in the query, making more generalizable. Though simpler than full graph, it aids tasks like effect inference and graph discovery. We show that the correct causal order is sufficient for identifying a valid backdoor set for any pair of treatment and outcome variables. Moreover, causal order-based metric, topological divergence (Drop). correlates better with effect estimation accuracy than commonly used graph metrics such as structural hamming distance (SHD). Specifically, Drop 0 if and only if the causal order provides valid backdoor adjustment set. Causal order enhances effect inference and improves graph discovery. We provide simple algorithms to integrate it into existing causal discovery methods.
2

---

Published as a conference paper at ICLR 2025
Causal Order: Eliciting from Experts. In practice, obtaining causal order from experts is still challenge because we need to account for imperfect experts such as human annotators and LLMs. Using the standard method (Kıcıman et al., 2023; Long al., 2022) of iterating with pairwise prompt/question over a set of variables, while a perfect expert would always predict the correct causal order, find that using LLMS as experts leads many cycles. To reduce the number of cycles from LLM output, we propose a novel tripler method for obtaining causal order (see Figure 2). Rather than asking questions about a pair of variables, the triplet prompt asks about the causal relationship between the pair and an auxiliary variable, and instructs the LLM to obey acyclicity for the triplet. We theoretically show that given an imperfect expert with an error e on each prediction, using the triplet-based prompt results in an error less than e, which is less than the error of the pairwise prompt. Moreover, since each variable pair occurs in more than one triplet, the repetition allows for ensembling strategies for more reliable order. Using human annotators and L.Mss as imperfect experts, the triplct method provides more accurate causal order than the pairwise prompt, especially in large graphs. It also enables small models like Phi-3 and Llama-3 8B to outperform GPT-4's pairwise prompt.
RELATED WORK
Domain Expertise-aided Causal Discovery. Prior knowledge has been used in causal discovery literature (Hasan & Gani, 2022; Constantinou et al., 2023; Heckerman & Geiger, 2013; Teshima & Sugiyama, 2021; O' Donnell et al., 2006; Wallace et al., 1996). These methods rely on prior knowledge such as domain experts' opinions and documented knowledge from randomized controlled trials. Various priors have been studied in literature, such as edge existence. forbbidee_orrbidenn edge (Meek, 1995), and ancestral constraints (Constantinou et al., 2023; Ban et al., 2023b). Recent advances in LLMS have led to more attention how LLMs may act as imperfect experts and provide causal knowledge based on metadata such as variable names (Kıcıman et al., 2023; Ban et al., 2023b; Long et al., 2023; Willig al., 2022). Early methods (Kıcıman et al., 2023; Willig et al., 2022; Long al., 2022) rely on LLMs to predict the complete causal structure, which is evaluated using metrics for full graph structure such as Structural Hamming Distance (SHD). Recent methods however use LLM's output to improve accuracy of graph discovery algorithms. The key idea is that LLM can provide information about edges in the graph, which can then be added a prior or constraint (Long et al., 2023; ?) to improve the accuracy for a causal discovery algorithm. For example, (Long et al., 2023) use LLMs to improve output of a constraint- based algorithm for full graph discovery by orienting undirected edges in the CPDAG. Most of these works, however, depend on obtaining correct edge information from L.MSs and evaluate L.MMs' quality by full graph metrics (Naik al., 2023; Zhang et al., 2024) such SHD (Kıсıman et al., 2023; Long al., 2023). We observe that imperfect experts (LLMs or humans) cannot reliably provide edge information given a pair (or subset) of variables. Causal order may be more appropriate causal structure to elicit from experts. For the same reason, the quality of an imperfect expert's 'ss output for such tasks is better evaluated on the accuracy of causal order, rather than the full graph structure.
LLM-based Prompting Strategies. Existing LLM-based algorithms for graph discovery (Kıcıman et al., 2023; Long al., 2022; Ban al., 2023b; Antonucci et al., 2023) use pairwise prompt, essentially asking "does A cause B?" with varying levels of prompt complexity. Going beyond this line of work, we propose triplet-based prompt that provides more accurate answers through aggregation and provides an uncertainty score for each edge to aid in cycle removal. As a result, our triplet-based prompt may be of independent interest for causal tasks.
3 CAUSAL ORDER: A STABLE INTERFACE FOR EXPERTS' KNOWLEDGE
Preliminaries. Let G(X, E) be a causal directed acyclic graph (DAG) consisting of a set of vari- ables X (XI,..., X,,) and a set of directed edges among variables in X. directed edge X, + X, E E denotes the direct causal influence of X; on X,. Let pa(X,) = (Xxl X₂ + X;). de(X,) = {x f ... + X.). ch(X.) = [Xx[X, + Xx) denote the set of par- ents, descendants and children of X; respectively. If variable Xx is a descendant of X; (but they are not connected by a direct edge), then X, is said to have an indirect effect on variable X, is defined X; = E[x, [do = æ;)] E[X, do(X, = x;)), where X, is called the treatment, X; is called the target, and do(X, = 22) denotes an external intervention to the variable X, with the value T.. If set of variables Z satisfies the backdoor criterion (Defn. B. relative to (X,, X,).
Xt. Average causal effect (Pearl, 2009) (ACE) of a variable X,
as: A
3

---

Published as a conference paper at ICLR 2025
= z4)] Exx 2,, Z z] (Thm. 3.3.2 of Pearl (2009)); and Z is called a valid adjustment set. We now define the causal (topological) order and the topological divergence metric (Rolland et al., 2022) that measures the goodness of given causal order wrt. the ground-truth graph.
B[X, |do[X; = =x]]] can be computed as:
Definition 3.1. Topological Order. Given causal graph G E), a sequence (or ordered permula- tion) x of variables is a lopological order iff for each edge X; + X; E E, Te < <----
Definition 3.2. The topological divergence of an estimated order â with ground truth adjacency matrix A, denoted by Drop (ū, A), is defined as: Drop ( A) = -Ei= Eyitis Asg where Aij 1 if there is a divected edge from node to j else A = 0. Drop ( A) counts the number of ground-truth edges that cannot be recovered due to the estimated topological order ft.
Structural Hamming Distance (SHD) is also a popular metric for assessing the goodness of a predicted DAG, Given a true DAG 9 and an estimated DAG g. SHD counts the number of missing, falsely detected, and falsely directed edges in G. Dtop acts as a lower-bound on SHD (Rolland et al., 2022).
3.1 CAUSAL ORDER FROM A PERFECT EXPERT IS ALWAYS ACCURATE, BUT GRAPH IS NOT
The predominant approach to extract causal knowledge from LLMs is to use pairwise prompt (KICI- man et al., 2023; Long et al., 2022: Choi et al., 2022) determine the existence of an edge and then aggregate to build a causal graph. We highlight key limitation of pairwise prompts for inferring edges and causal graphs, even with a hypothetical perfect expert, as LLMs are imperfect.
Revisiting the two graphs in Fig. 1, the second graph is estimated by asking pairwise questions to a perfect expert that (hypothetically) knows about all cause-effect relationships in a domain (see Defn. 3.3 for a formal definition). The difference in edge predictions is introduced due to the existence of direct and indirect effects. For example, when asked about the relationship between Pollution and Dyspnoea, it may be valid to draw direct edge if the expert is not aware of the Cancer node. As a result, if we compare the estimated graph in Fig. I using standard graph comparison metrics such as SHD, we may find that that the estimated graph is significantly different from the true graph and (incorrectly) conclude that the expert's knowledge was insufficient. Instead, if we compute the causal order using Def. 3.1 for the predicted graph (Fig, I right), we obtain (Smoking, Pollution} < Cancer < (Dyspnoea, X-ray). This order is fully consistent with the true graph (Fig. 1 left), and thus is a valid causal order. We could thus correctly validate the expert's s knowledge as perfect. In particular, using causal order as the output interface of the expert- esstrt-estimated graph ensures that no incorrect constraints are added. If the expert was asked output the entire graph, erroneous edge constraints such as Pollution - Dyspmoea may be added to a downstream discovery algorithm. However, causal order only constrains that some path exists from Pollution to Dyspneoa, and allows the downstream algorithm to learn the correct edges from data.
Note that the limitation is not about using a pairwise prompt, but using its output to infer edges in graph. As stated earlier, given pair of variables, it is not possible to determine whether an edge exists between them, without knowing whether potential mediators between the two variables exist. By not explicitly inferring edges, causal order instead corresponds to an ancestor-descendann relationship between a pair of variables which can be objectively decided given only the two variables. One can view our approach formalizing the intuition in Ban al. (2023b) who consider LLM's pairwise answer to represent ancestor relationship between a pair of variables. We now formally show that causal order is a more accurate measure of an expert's 's knowledge. All proofs are in Appendix B. Definition 3.3. Perfect Expert. A perfect expert is an entity with access to the full moound-rruth DAG G(X,E), Given two variables two variables, X,,X, E X. and (optionally) an auxiliary set of nodes Oi C X (note that rest of the variables in set U = X Oiş U(X,,X,) need not be known), the expert can provide information on the existence of a causal edge between X, and X3 ("does X; cause X;,) as follows:
X,, - ~xx: If there is directed edge from X, to X, (X; + X, E E), or if directed path exists from X, to X, such that it does not contain any node Z E Oij-
Xxj- + +X,; If there is directed edge from X, to X, (X + X; E), or if a directed path exists
from X, to X; such that it does not contain any node Z E Ozg
Otherwise, output no edge.
Definition 3.4. Level Order. Given a causal DAG G(X, E), its level order is the systematic as- signment of levels to variables, beginning with level O to the set of variables (Xx) 0).
4

---

Published as a conference paper at ICLR 2025
Subsequently, each remaining variable is assigned a level such that for each variable given level i, the length of the longest directed path from one/more variables in level 0 is i.
Proposition 3.1. Let the true causal DAG be ( E) with ground-truth adjacency matrix A. Consider a procedure to estimate a graph G by querying a Perfect Expert (as in Def. 3.3) with pairwise queries X Xj with auxiliary set followed by subsequent aggregation of predicted edges from each query (i.e. from total of X C2 queries). The causal order of the graph G thus estimated correct, i.e. (T) A) for all values of the sets Oij. As a corollary, the causal graph thus estimated can however have errors. In other words, when Oij = o VVi Drop (T (T()), A) 0 whereas Structural Hamming Distance (SHD) between G and 9 X [de(X;)] (ch(X;))
Figure 3 illustrates the result of the proposition using empirical simu- lation. Given fixed number of nodes, we sample graph at random as the 'ground truth' and then consider all graph orientations of the same size (number of nodes) such that Dtop = w.r.t. the ground truth graph. These are potentially the graphs outputted by Perfect Expert with different values of the auxiliary set O. For this set of graphs, we compute SHD w.r.t the ground truth graph. Notice the variance in SHD, despite Drop being O. For graphs with six nodes, SHD can vary from O to 14 even as Diop= 0.
SMD score when D
12-
10.
Figure 3:
Variability of SHID for various graph sizes with Deop = 0 within each graph.
The above observations indicate that SHD can be high even when we obtain information from a Perfect Expert, but Drop is always O. This result is of significance since most estimated graphs (including those that are L.M--genertee (Ban et al., 2023b; Long et al., 2023)) are evaluated using graph metrics such SHI. Rather than the graph, it motivates us posit the use of causal order as more accurate output interface for experts' domain knowledge, since it allows objective evaluation of the expert's output using the topological divergence metric (Defn 3.2).
3.2 DOWNSTREAM UTILITY OF CAUSAL ORDER: DISCOVERY AND EFFECT INFERENCE
While the causal order is a more stable measure of experts' knowledge than the full graph, a natural question is whether it is useful measure by itself. We now show the utility of causal order for effect estimation and causal discovery, which is also demonstrated by our experimental results in Sec 5, Specifically, we show that causal order is sufficient to find a valid backdoor set and Dtop is an ideal metric to minimize for effect estimation, assuming no latent confounders. Effect estimation error correlates more with topological divergence than does with SHD. Causal order is also useful as a prior or constraint to increase accuracy of graph discovery algorithms.
Correct topological order is necessary and sufficient for finding valid backdoor set. We first present the (known) result that a correct causal order is sufficient for identifying backdoor set. We assume there are no unobserved variables in the underlying causal graph.
Proposition 3.2. (Pearl, 2009; Cinelli al., 2022) Under the no latent confounding assumption, for a pair of treatment and target variables (x, in a DAG G. Z = {Xxl | < T is a valid adjustment set relative to (x), for any topological order of G.
Proofs of all propositions are provided in App. $ B. Propn 3.2 states, in simple words, that all variables that precede the treatment variable in a topological order x of G constitute a valid adjustment set. Note that the set Z may contain variables that are not necessary to adjust for (e.g., ancestors of only the treatment or target variables). For statistical efficiency purposes, ancestors of the target variable are helpful for precise effect estimation, whereas ancestors of treatment variable can be harmful (Cinelli et al., 2022). In practical scenarios, however, it is recommended adjust for all available adjustment variables since one cannot possibly rule out unknown confounding factors (Sauer et al., 2013; VanderWer Weele & Shpitser, 2011), 1), which aligns with the set obtained using the causal order in Propn 3.2 (see App. C for a discussion).
We now show that Dzop is an optimal metric to minimize for effect estimation. That is, Dtop being for topological order is equivalent to obtaining the correct backdoor adjustment set using Propn. 3.2. And if Drop # O, there exists some treatment- -taget pair whose backdoor set is not correctly identified. Proposition 3.3. For an estimated topological order î and true topological order of a causal DAG G with the corresponding adjacency matrix A, Drop (t, A) = 0 iff Z = {Xal ft < #.) is a valid adjustment relative to (X,, Xz), VT, < T.j-
5

---

Published as a conference paper at ICLR 2025
Empirically, the correlation of Dtop with effect estimation is shown in App. E.2 for common BNLearn datasets. As long as Dtop is zero, changing the graph has no impact on effect estimation error.
Topological order can improve accuracy of graph discovery algorithms. Constraints implied by the topological order can be used to reduce the search space for discovery algorithms. For instance, X; X, in the order, then X, cannot be descendant of X, in the corresponding causal graph.
Using causal order with Constraint- Based Discovery Methods: Constraint based causal discovery algorithms usually return a Completed Partially Directed Acyclic Graph (CPDAG), from which Markov equivalence class of graphs can be obtained. However, not all edges in a CPDAG are oriented. Given CPDAG from constraint-basee algorithm like PC (Spirtes et al., 2000), we use the causal order ft obtained from experts to orient the undirected edges, similar to the algorithm from Meek (1995). Iterating over undirected edges, we first check if the nodes of that edge occur in म.. If yes. we orient the edge according to f.. Since it is possible that the causal order obtained from querying experts may not include some nodes (Isolated Nodes (IN)). if either (or both) nodes of the undirected edge are not in â, we query a superior expert (e.g. oracle) (see Sec 4) to finalize a direction between the pair. Algorithm (Appendix D) outlines the specific steps for this integration.
Using causal order with score-based discovery methods: Score-based methods like CaMML (AMMM..Walaa et al., 1996) allow the specification of prior constraints which are respected while obtaining the complete graph. We hence utilize the causal order R obtained from experts as level order prior (Defn 3.4) to such methods. We handle any cycles in the expert's s output by assigning all nodes in a cycle to the same level. The approach is similar an LLM- -pior approach by Ban al. (2023b) where the output of LLM and a score- score-based method are combined using an ancestral constraint. This approach also allows us to provide prior probability control the influence of prior on the discovery method. Algorithm 2 (Appendix D) outlines the specific steps for this integration.
4 OBTAINING A CAUSAL ORDER FROM IMPERFECT EXPERTS
If we assume a Perfect Expert, then aggregating edge responses from the standard pairwise prompt (Kıemmn et al., 2023) can yield an accurate order. However, in practice, LLMs are im- perfect experts and their answers can contain unpredictable errors. As a result, aggregating responses from the pairwise prompt leads to many cycles in the final graph (see Sec. 5, Table 2), which in turn implies that the causal order is undefined. In this section, we propose two ways to reduce the errors made by an imperfect expert such as LLM, motivated by Prop. 3.1 that showed that adding additional context may help an expert avoid creating unnecessary edges. First, we consider strategies to add auxiliary context in the pairwise prompt. Second, we propose strategy that adds dynamic context to each variable pair by iterating over all triplets of variables.
4.1 ENHANCING ACCURACY OF PAIRWISE PROMPT
One way to avoid cycles is to make the pairwise prompt more robust. Beyond the standard pairwise prompt that asks the expert identify the causal relationship between a pair of variables (Kıcıman et al., 2023), we consider the following strategies to add contextual information (see Appendix G).
Iterative Context. Here we provide the previously oriented pairs as context in the query while iteratively prompting for next pair.
One-hop Iterative Context. Providing all previously oriented pairs can become prohibitive for large graphs. Therefore, in this setting, we limit the provided information to the already oriented edges connecting the node pair under inspection with their adjacent neighbors. Specifically, we only supply the orientation details for the current node pair and their neighboring nodes.
Ohain-of-Thoughee (+In-context Learning). Here we include names of all variables in the graph as additional context. Based recent results on providing in-context examples in LLM prompts for various tasks (Brown et al., 2020), we include examples of the ordering task (viz. node pairs and their correct causal ordering), before asking the question about the given nodes.
4.2 THE TRIPLET METHOD FOR PROMPTING LLMS
Rather than providing a pair of variables, another way is to provide a larger set of nodes in prompt and ask LLM to obey the acyclicity constraint while providing the edges among them. The number of total prompts used for graph with size [V] would be O(VV)*) where &s is the size of the subset included in each prompt. In addition, LLM's accuracy is known to reduce as the query prompt becomes more complex (Levy et al., 2024). Therefore, while the set of nodes can be of any size, we

---

Published as a conference paper at ICLR 2025
decide to go with saplet-based prompts they allow for adding more context with minimal increase in prompt complexity and the total number of LLM calls. Moreover, empirically, we did not see a noticeable improvement in accuracy when moving from triplet quadruplet prompt (see Table A9).
In effect, we move from O(IV)) calls to ( LLM calls. A key benefit is that for each pair of nodes, we have 1 responses from the LLM,, each considering different auxiliary node as context. For large graphs. we can use a variant that considers a constant l responses for each pair of nodes, leading to O (2)) complexity. To aggregate the final graph, we take a majority vote on the answers from each edge, further leading to robustness.
1. From a given set of graph nodes, generate all possible triplets of nodes. 2. We query the expert to orient nodes of each triplet group to form DAG representing the causal relationship between the triplet's 's nodes. This results in multiple acyelic mini-graphs representing causal relationships for each triplet group.
3. Once we have DAGs for each triplet, we focus on merging them. This is done in two steps: (i) We iterate over all node pairs, and for each combination we obtain majority vote on the orientation between them across all triplets containing the node pair; (ii) In case of a conflict (or tie in the majority vote) among the three possible edge orientations (A + B; B + A; No edge between A and B), we resort to high-cost expert for tie-breaking 4. Finally, a causal order is extracted from the merged graph.
Our triplet prompt additionally use in-context examples and the chain-of-thought strategy from the pairwise setup. An example prompt is shown in Table A28.
Theoretical Analysis. Next, we analyze the triplet strategy for its impact on predicting incorrect edges. We begin by defining (imperfect) e-experts as in Long et al. (2023). For case of cxposition. we define the e-expert have error probability exactly equal to e; this could however be generalized to have error probability at most c. By enforcing the acyclicity constraint for each triplet, the triplet prompt avoids errors that a pairwise prompt may make. Below assume that the c-expert's predictions satisfy acyclicity for subgraphs having 3 nodes."
Definition 4.1 (c-Experts). Given two nodes A and B of a graph and three options of the causal relationship between them: (i) A > B, (ii) A t- B, and (iii) no edge between A and B (denoted as (e1,02, cal), an expert E queried for the causal relationship between A and B is said to be an e-expert (denoted as E.) if the probability of making an error in the prediction of the causal relationship between A and B is e, where E E 10,
Proposition 4.1. Given two nodes A and B of underlying causal graph, access an c-expert & that doesn't I produce any cycles in the predicted causal graph (see Assm B. for formal statement) and and renormalizes the probability in case an option is not available (see Assm B.2 for formal statement), let C + A + B be any other node in the graph. & predicts causal relationship between all pairs of nodes sequentially, the marginalized probability that &, makes an error in predicting the causal relationship between A and B, after it has already predicted the causal relationships between (C, A) and (C, B), is less than e, where marginalization is over all possible causal graphs that can be formed between A, B and C, with each of such graphs being equally likely.
Thus, given two nodes A and B, querying strategy using triplets will have error probability < on determining the causal relationship between A and B than a pairwise strategy (proof in Appendix B). Still, some cycles may be produced in the aggregated global graph, hence we use cycle removal algorithm from Zheng et al. (2018) in the third step of our triplet method. For every edge, we leverage the votes from the triplet prompts to establish a probability distribution over edge orientations. We use this to compute entropy for each edge, removing those with higher entropy (lower confidence). To minimize Drop, we prune edges with entropy below the mean of all entropies.
5 EXPERIMENTS AND RESULTS
Datasets, We evaluate the triplet method using benchmark datasets from the BNLearn repository (Scu- tari & Denis, 2014): Earthquake, Cancer, Survey, Asia, Asia modified (Asia-M), and Child. Asia-M is derived from Asia by removing the node either since it is not node with a semantic meaning (see
FFor imperfect experts, it is possihle to enforce acyclicity by removing any cycles from their triplet output. However, this step is not needed for GPT-3.5 and GPT-4 as they follow acyclicity constraint with high accuracy.
7

---

Published as a conference paper at ICLR 2025
App.$ G for details). To address memorization concerns with the BNLearn datasets, we also use recently proposed datasets that require nuanced medical domain understanding. (i) Neuropathic: A medium-sized subset graph from relatively less popular Neuropathic dataset (Tu et al., 2019) (see Appendix Fig A8). (ii) Alzheimers: This graph (refer Figure A9) provides features (such as ventricular volume, brain volume, APOE4, etc) to study the clinical and phenotype of Alzheimer's disease (Abdulaal et al., 2024). It was created by a consensus of human experts. (ii) Covid-19: This graph, curated by medical experts, models the pathophysiologica process of SARS-CoV-2 in the respiratory system which involves outlining the various pathways from viral infection to key complications (refer Figure A10). Orienting this graph requires understanding of how nodes like Pulmonary capillary leakage, systemic inflammatory response, Virema and more influence each other (Mascaro et al., 2022). All graphs are real-worle graphs constructed by human experts. We provide more details on the datasets in Table A 188
Imperfect Experts. We consider two types of imperfect experts: LLMS and human annotators.
Human Annotation. We considered 15 human annotators, each with undergrad-leve training in STEM but no formal experience in causality. Each annotator was randomly assigned graphs for pairwise and triplet query strategies while ensuring no annotator got the same graph to query with both strategies. To get an estimate of the upper bound of human performance. for resolving tie-breaking conflicts in the triplet method, dataset, three human annotators were asked to annotate the final graph and the aggregate of that was reported. For feasibility reasons, human annotations were done only for the BNLearn graphs.
used ground truth-based oracle (proxy for human domain expert). For each
We consider two main LLM-based experts, GPT-3.5-turbo and GPT-4. For the triplet method, GPT-4 was used for tie-breaking. To understand the effect of model size, we also evaluate the pairwise and triplet methods on Phi-3 (3.8B parameters) (Abdin, 2024) and Llama3 (8B parameters) (Dubey, 2024) which are significantly smaller models than GPT-3.5-turbo and GPT-4.
Baselines. In addition to the pairwise prompt and its extensions from Sec. 4.1, we consider two methods based on breadth-first search from Jiralerspong al. (2024). The first method, BFS, iterates over nodes and uses an LLM to query children of each node. The second method, BFS+Sttats, uses correlation coefficient between nodes as additional context in the LLM's prompt.
5.1 ACCURACY OF CAUSAL ORDER WITH TRIPLET VS. PAIRWISE METHODS
We first present the accuracy of obtaining causal order using our triplet method over other pairwise query strategies. Subsequently, we present the results of using the causal order obtained from imperfect experts to downstream tasks such causal discovery and effect inference.
Human Experts. With human annotators, Table shows that graphs like Survey and Asia-M result in cycles when queried pairwise. However, no cycle formations were observed across annotators when they were queried to orient causal graphs using the triplet method. Moreover, the triplet method shows consistently low Drop and SH D across all human outputs, highlighting its effectiveness.
GPT-3.5-turbo as Expert. Tables A5 and A6 present the performance of various pairwise optimization strategies from Sec. 4.1. While strategies like CoT offer some gains over the base pairwise method, they often produce cycles, especially in larger graphs like Child. These findings show that our pairwise variations improve graph discovery but still fall short. Table 3 compares the base pairwise method, best pairwise variation (CoT), and the triplet method across benchmark datasets using metrics like Drop. SHD, Cycles, IN, and Total Nodes (TN). Triplet method consistently outperforms the best pairwise CoT approach, showing significant performance gap over the base pairwise method. For larger graphs like Child, the pairwise base approach shows more pronounced difference, with higher cycle counts and SHD. Results on the Neuropathic dataset further confirm that the triplet method yields low Drop and significantly lower SHD than pairwise methods.
GPT-4 as Expert. Table A7 shows the impact of using more advanced model like GPT-4 for the pairwise method. Despite superior model capabilities, we observe a consistently high number of cycles in bigger, complex graphs such as Child, Neuropathic, Covid-19 19 and Alzheimers, indicating that simply upgrading the model is not sufficient. In comparison, upgrading to GPT-4 for orienting subgraphs for the triplet method leads to further performance improvements as shown in Table A10.
Results with Small LMs. To assess the robustness of the triplet method, we use it with small LMs such as Phi-3 and LLama3-8B experts and GPT-4 for tie-breaker. Remarkably, as shown in Table 2, the
8

---

Published as a conference paper at ICLR 2025
Dataset
Metric
Pairwise
Triplet
Table 3: Results using GPT-3.5-Turbo. Performance of triplet method, best performing pairwise query strat- egy (Chain of Thought), standard pairwise technique (Base) multiple benchmark datasets acToss diff metrics: Drop. SHD, (Num of) Cycles, IN, TN. When number eycles>00, f cannot be computed, hence Dtop is given by *-' While CoT method shows im- provement over base pairwise. triplet method outper- forms pairwise methods across all datasets and met- rics, with significant improvements on larger graphs such as Child and Neuropathic.
Using Human Annotators
Drop SHD
1.67
Farthquake
Cycles
IN
0.33
Dzop
0
SHD
6.33
3.67
Survey
Cycles
0.67
IN
Dim
SHD
4.33
3.67
Cancer
Cycles
IN
0.67
0
Pairwise (Base)
Pairwise (CoT) Triplet
Drop
1,33
Using
SIID
11.67
11.33
Asia-M
Drop
0
Cycles
IN
Eartiquake
Cycles
SHD
4
0
4
Dign
0
Table Experiments with non-expert human annota- tors show that the triplet method consistently produces
Survey
Cycles
SHD
12
9
0
9
0
IN/TN
lower SHD and Drop values.
Dage
Metric
Pairwise GPT-4
Triplet Phi-3
Triplet
Cancer
Cycles
IN/TN
SHD
0)
6
Dase
Dup SHD
13
Asia-M
Cycles
SHD
15
13
1
11
11
Asia
Cycles
IN/TN
0/7
0/7
Dise
Dusp
Child
Cycles
SHD
<3k
177
+500
138
28
0
Cycles
IN/TN
0/20
0/29
0/20
Alzheimers
IN/TN
1111
0/11
1/11
Diar
SHD
Deap
148
69
17
129
12
Covid
Cycles
IN/TN
SHD
1100
0/20
41
0/29
27
0
1/20
30
0
Cycles
10kk
IN/TN
0/20
0/20
0/20
Alzheimers
Cycles
SHD
684
26
28
Table 2: Comparison of triplet method using Phi- I/L.ama3 against pairwise (hase) using GPT-4. Triplet method with significantly smaller models obtains lower SHD and Dtop values while avoiding cycles.
IN/TN
0/20
0/20
1/20
D
Neuropalhic
Cycles
w5k
IN/TN
0/22
0/22
13/22
triplet method using smaller LMs outperforms the base pairwise method using GPT-4, particularly for complex networks. Results with small LMs are shown Table A8. The triplet method outperforms the pairwise method consistently, yielding low Dtop values for both small and large graphs.
Comparison with BFS and Sta+S We also compare the triplet method to recently proposed BFS-based methods on subset of the datasets in Table A17. BFS and BFS+Stats methods obtain lower accuracy than the triplet method. Across datasets, SHD and Dtop for BFS and BFS+Stats methods (especially with GPT-3.5-turbo are higher than the triplet method. Among the Child and Covid-19 datasets, all configurations lead to cycles in atleast one of them, except BFS with GPT-4.
Cost Estimation Analysis: Pairwise Triplet for LLMs. The triplet method optimizes cost by using smaller models efficiently, reserving larger models for clash resolution, reducing inference costs while improving accuracy over pairwise methods. See Appendix for detailed comparison.
5.2 USING CAUSAL ORDER FOR DOWNSTREAM APPLICATIONS
Causal Discovery. Table presents the Dtop results of using the causal order obtained from the triplet method (both using LLMs and humans) to assist causal discovery methods. We compare our approach using the triplet method with well-known causal discovery methods: PC (Spirtes al., 2000), SCORE (Rolland et al., 2022), ICA-LiNGAM (Shimizu et al., 2006), Direct-LiNGAM (Shimizu et al., 2011 10. NOTEARS (Zheng et al., 2018), and CaMML (Wallace et al., 1996) across five different sample sizes: 250, 500, 1000, 5000, 10000 (complete results in Table A3). Among the discovery algorithms, we find that PC and CaMML perform the best, with the lowest Drop across all datasets. We hence studied 4 variants of using the causal order with discovery algorithms: PC+Human,

---

Published as a conference paper at ICLR 2025
Dutuset
SCORE
ICA
Direct
NOTEARS
C&MML
Ours
Ours
Ours
Ours
LINGAM
LINGAM
(LLM)
(LLM)
iBuman PCC
CuMMILi
Fanhqgake
0.
4000:000
100:000
1.000000
10000000
2,00:0.00
0.80:0.00
0U00 0J00
10.00
1 100 0.00
Cenerr
0000-1000
5000000
2,000000
2,00-0.00
2.0000.00
20000000
0.80-0.00
0.00-UU0
0.60-0.00
Sarvey
0.50=000
402+0000
2,0+0000
4.0000.00
4.0000.00
3.35-0.44
0.80:0.00
3.350094
0.00:0.00
0U00-1U00
Asia
2.0
7000:000
3.33
100=0000
3.00:9.00
1.85:0.5
1.00:0.00
0U7:0262
N/A
N/A
Asa M
1502:000
6002000
10000000
5.00:000
5.0000000
1.00 0U00
1,00:00.00
1:21:0045
1,0000,00
2 20000.00
Child
5 5.7580000
12.0:000
14, 33.3330
16.0:0.00
14:0:0000
3,00 0000
4 0.00:0::0
5:5::0445
N/A
N/A
Neumgoohic
40000000
600+000
13.0
10.0-0.00
9.00-0.00
10:41:155
10.00
5:0000000
N/A
NIA
Eathequke
0000:0000
400000000
5,00:00000
5,00:0000
1.0000000
0.404:0445
0000:00.00
0.00::0.00
10300
Clsicet
2,00:00000
3002:000
3,0000000
3,00::000
20000000
0.00
2,00::0:000
0.00:0000
2000::0:00
Survey
200000000
40000000
5.00
5.00:0000
5.00
1.60
2.00:00.00
1:33::000
2,00::0,00
0,00100000
Asia
1,330.00
4000-0000
50000000
1,302:1335
3.00000000
14
0.00:00000
0.34
N/A
R/A
Asaa
1001:000
4.0010000
K0000000
4.8020039
5,0000000
20000000
0,00 0.0000.00
0.0000000
0,00,10.00
5,000.0000
Child
5.
30010000
12 16
11.-0088 88
14.4:048
880-084
5,00:264
1,00-0.00
N/A
N/A
Neuropatic
1000000.00
600000000
10:0:0000
10:0=0000
3.00=0000
10000:00.00
1,00:00000
N/A
NIA
Table 4: Comparison with causal discovery methods, showing mean and std dev of Dtop over 3 runs. (For the Neuropathic subgraph (1k samples), PC Algorithm returns eyclic graphs in the MEC).. Human experiments not conducted for Neuropathic, Child (due to feasibility issues) and Asia; hence rows marked as N/A.
CaMML+Human, PC+LLM, and CaMML+LLM. The results show that using expert- ppport-provide causal order improves Dtop across our experiments consistently. Specifically, the improvement (reduction) in Dtop when using our approach is larger at lower sample sizes. This indicates that obtaining causal order from imperfect experts like humans and LLMs can help with causal discovery in limited sample settings. While the results on the BNLearn datasets may be impacted by memorization, we obtain consistent results on the less popular Neuropathic dataset that requires nuanced medical knowledge.
Causal Effect Inference. Table 5 presents the results of using the causal order obtained from the triplet method to compute average causal effect (ACE). We report the error in ACE CACE e across the same set of methods and datasets as above. The obtained causal order shows unanimous improvement in performance across the studies, especially when using the causal order from CaMML+LLM. Following Proposition 3.2, we use all variables that precede the treatment variable in estimated topological order as the adjustment set. Once the adjustment set identified, the causal effect is estimated using the Dohhy library (Sharma & Kiciman, 2020) and linear regression as the estimator. Table A16 compares the causal effects estimated using this approach versus minimal backdoor set adjustment in the Asia dataset, showing minimal differences.
Dataset
Metrie: sack (Treamment, Target)
SCORE
ICA LINGAM
NOTEARS
CaMML
Ours (LLM+C&MML)
LiNGAM
(LLM-PC)
Eantquuak
(Jhhnalls alarm)
0.00 0:00
0.85 0.00
0.63 1.000
0.63
0.21 1.012
0.8 1.00
0.00 0.000
0.00 0.00
Cancer
(dyspooescmmerrr
0.20 0.01
0.30-1:00 t:000
0.30 0:001:00:0
0.30:20001
0.18::00:2
0.06 0:00
0.30:20.00
0.00= 0.00
Survey
(LE)
0.02 0.02::0.000 11,00
0.04 1 0.00
0.6 0.00
0.01
0.0 0.00
000
0.2 0.01
0.01 = 0.01
Asia
(smoke.yy
0.10 0.00
0.00 0:00
0.27 +.00
0.270000
0.14
0.0 00000
0.2 0:00
0.00= 0.00
Child
(Lung Farench.
0.22.1 0.01
0.00 1:0000000
0.52 10.00
0.52
0.52 0.07
0.00 1:000
0.22 0:0000000
0.00 0.00
Lowerbody
Table 5: Comparison of causal effect inference with existing methods, showing mean and std dev of error in Average Causal Effect (c)) of a variable on another, over runs.
PC
SCORE
ICA LINGAM
Direct LINGAM
NOTEARS
CaMML
Ours (LLM+PCC)
Ours (LLM+CaMMIL)
NN 250
4.000
6.00
63.016.16
10.0 0.00
9.0010.00
10.4 1.95
5.00 10.00:10.00
3,00 0.00
N = 10000
10.00-0.00
6.00=0.00
1.00=0.00
10.0=0.00
10.010.00
3.00=0.00
1.00:10.00
Table 6: Performance on causal discovery for the Neuropathic dataset subgraph (1k samples), showing mean and std dev of Dtop over runs.
6 CONCLUDING DISCUSSION
Obtaining reliable knowledge from imperfect experts is challenging. We presented causal order as a suitable output interface to elicit causal knowledge from imperfect experts like LLMs and human annotators. Compared to the full graph, we showed that causal order is a more stable quantity to elicit from imperfect experts since avoids making a distinction between direct and indirect effects. We also proposed novel triplet-baset method to query experts for obtaining the causal order.
Limitations. While LLMs can provide causal order for relationships over known variables, we do not expect them to be useful for completely novel nodes and causal relationships. Separately, causal order may not be sufficient for tasks such as counterfactual estimation (Janzing et al., 2019) that require the graph structure for estimating functional equations. For both these cases, viable method may be to obtain causal order over known relationships from LLMs and use it as a prior /constraint for existing discovery methods to obtain the full causal graph.
10

---

Published as a conference paper at ICLR 2025
ETHICAL IMPACT AND REPRODUCIBILITY
Ethical Statement. All datasets used in our work are publicly available and are accurate to the best of our knowledge. We made best efforts to compare against contemporary benchmarks in a fair manner. There may be direct harmful impact, especially considering our causal order is only pre pre-processing steps for downstream algorithms. However, since LLMs may be used in our approach, suitable prudence may be necessary to avoid ill-effects in applications.
Reproducibility. Our methods are fairly straightforward, and implementation details are already included in paper descriptions. Our code is publicly available at https:/ /att:://ggthbb com/ Crade Imperfect, Eetrr__xpertss
REFERENCES
Abdin. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL httpg//bb/24444 14219.
Ahmed Abdulaal, Nina Montana Noontn Tiantian He, Ayodeji ljishakin, Ivana Drobnjak, Daniel Castro, Daniel C Alexander, et al. Causal modelling agents: Causal graph discovery through synergising metadata-and data-driven reasoning. The Twelfth International Conference on Learning Representations, 2023.
Ahmed Abdulaal, adamos hadjivasiliou. Nina Montana-Browe Tiantian He, Ayodeji Ijishakin, Ivana Drobnjak, Daniel C. Castro, and Daniel C. Alexander. Causal modelling agents: Causal graph discovery through synergising metadata- and data-driven reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL https: htttps://openrevieee wtt.ee/wwwtt//oottttet forum? id=pAogRITBtY.
Alessandro Antonucci, Gregorio Piqué, and Marco Zaffalon. Zero-shot causal graph extrapolation from text via llms. arXiv preprint arXiv:2312. 14670, 2023.
Taiyu Ban, Lyuzhou Chen, Derui Lyu, Xiangyu Wang, and Huanhuan Chen. Causal structure learning supervised by large language model. arXiv preprint arXiv:2311.1168) 2023a.
Taiyu Ban, Lyvzhou Chen, Xiangyu Wang. and Huanhuan Chen. From query tools to causal architects: Harnessing large language models for advanced causal discovery from data. arXiv preprint Xr3v:22306. 16902 2023b.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel MerberttVosss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR. abs/2005.14165. 2020.
Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. Lmpriors: Pre-trained language models as task-specific priors, 2022. URL https: / //tp/aaavvvv//////////.ee/.awav. wwt.//rg///a/oo.ooo..//o.. abss/2100 12530.
Carlos Cinelli, Andrew Forney, and Judea Pearl. crash course in good and bad controls. Sociological Methods & Research, pp. 004912412210999
2022.
Kai-Hendrik Cohrs, Emiliano Diaz, Vasileios Sitokonstantinou, Gherardo Varando, and Gustau Camps-Valle Valls, Large language models for constrained based causal discovery. In AAAI 2024 Workshop on" Are Large Language Models Simply Causal Parrots?", 2023.
Anthony C. Constantinou, Zhigao Guo, and Neville K. Kitson. The impact of prior knowledge on causal structure learning. Knowledge and Information Systems, 65(8):3385-34344 2023.
Dubey. The laaa 3 herd of models, 2024. URL Utt.https:://arxx..oorr /24p:///as//2//..ooo 217883.
11

---

Published as a conference paper at ICLR 2025
Uzma Hasan and Md Osman Gani. Kerl: A prior knowledge based causal discovery framework with reinforcement learning. In Proceedings of the 7th Machine Learning for Healthcare Conference, 2022.
David Heckerman and Dan Geiger. Learning bayesian networks: unification for discrete and gaussian domains. arXiv preprini
2013.
Dominik Janzing, Kailash Budhathoki, Lenon Minorics, and Patrick Blöbaum. Causal structure based root cause analysis of outliers, 2019. URL /attts://azxxer org/ abs /tfss/1912 1912. 02724.
Thomas Jiralerspong, Xiaoyin Chen, Yash More, Vedant Shah, and Yoshua Bengio. Efficient causal graph discovery using large language models, 2024. URL httpt: /ttp://aaiivvv..iia/.ai http///wwwi..rgaaaaaa/aaaaaaaa/a 01207.
Elahe Khatibi, Mahyar Abbasian, Zhongqi Yang, Iman Azimi, and Amir M Rahmani. Alem: Autonomous llm-augmented causal discovery framework. arXiv preprint arXiv:2405.01744 2024.
Emre KIcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. arXiv preprint arXiv:2305.0000 2005000000 2023.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models, 2024. URL httpppp::///xxi..ooo org/ abss / 00.42.1.44444
Stephanie Long, Tibor Schuster, and Alexandre Piché. Can large language models build causal graphs? In NeurlPS 2022 Workshop on Causality for Real-world Impact, 2022.
Stephanie Long, Alexandre Piché, Valentina Zantedeschi, Tibor Schuster, and Alexandre Drouin. Causal discovery with language models as imperfect experts. In ICML 2023 Workshop on Struc- tured Probabilistic Inference & Generative Modeling, 2023.
Steven Mascaro, Yue Wu, Owen Woodberry, Erik P. Nyberg, Ross Pearson, Jessica Ramsay, Ariel Mace, David Foley, Tom Snelling, and Ann E. Nicholson. Modeling covid- 19 disease processes by remote elicitation of causal bayesian networks from medical experts. medRxiv, 2022. doi: 10.1101/ URL http::ttt/... //t////www///.../ medrxiv v. org/ /ttp///etet/tall///ee/eeeeeee/ee 17/2022. 02. 4. 22270925.
Steven Mascaro, Yue Wu, Owen Woodberry, Erik P Nyberg, Ross Pearson, Jessica A Ramsay, Ariel Mace, David A Foley, Thomas L Snelling, Ann E Nicholson, et al. Modeling covid-19 disease processes by remote elicitation causal bayesian networks from medical experts. BMC Medical Research Methodology, 23(1):76, 2023.
Christopher Meek. Causal inference and causal explanation with background knowledge. In Proceedings of the Eleventh conference on Uncertainty in artificial intelligence, pp. 403-410, 1995.
Narmada Naik, Ayush Khandelwal, Mohit Joshi, Madhusudan Atre, Hollis Wright, Kavya Kannan, Scott Hill, Giridhar Mamidipudi, Ganapati Srinivasa, Carlo Bifulco, et al. Applying large language models for causal structure learning in non small cell lung cancer. arXiv preprint arXiv:2311.0719. 2023.
Rodney O' O'Donelll, Ann E Nicholson, Bin Han, Kevin B Korb, Md Jahangir Alam, and Lucas R Hope. Causal discovery with prior information. Al 2006: Advances in Artificial Intelligence: 19th Australian Joint Conference Artificial Intelligence, Hobari, Australia, December 4-8, 2006. Proceedings 19, pp. 1162-1167. Springer, 2006.
Judea Pearl. Causality. Cambridge university press, 2009.
Jonas Peters and Peter Bühlmann. Structural intervention distance for evaluating causal graphs. Neural computation, 27 2015.
Paul Rolland, Volkan Cevher, Matthäus Kleindessner, Chris Russell, Dominik Janzing, Bernhard Schölkopf, and Francesco Locatello, Score matching enables causal discovery of nonlinear additive noise models. In ICML., 2022.
12

---

Published as a conference paper at ICLR 2025
Brian C Sauer, M Alan Brookhart, Jason Roy. and Tyler VanderWeele. A review of covariate selection for non-experimental comparative effectiveness research. Pharmacoepidemiolog and drug safery, 22(11):1139- 1145, 2013.
M. Scutari and J.B. Denis. Bayesian Networks: With Examples in R. Chapman & Hall/CCC Texts in Statistical Science. Taylor & Francis, 2014.
Amit Sharma and Emre Kiciman. Dowhy: An end-to-end library for causal inference. arXiv preprint arXiv:2011.04216 2020.
Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non-gaussian acyclic model for causal discovery. JMLR, 7(10), 2006.
Shohei Shimizu, Takanori Inazumi, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik O Hoyer, Kenneth Bollen, and Patrik Hoyer. Directingam: A direct method for learning linear con-gaussian structural equation model. JMLR, 12(Apr))1225-1245 2011.
Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2000.
Masayuki Takayama, Tadahisa Okuda, Thong Pham, Tatsuyoshi Ikenoue, Shingo Fukuma, Shohei Shimizu, and Akiyoshi Sannai. Integrating large language models in causal discovery: statistical causal approach. arXiv preprint arXiv:2402.01454. 2024.
Takeshi Teshima and Masashi Sugiyama. Incorporating causal graphical prior knowledge into predictive modeling via simple data augmentation. In UAI, pp. 86-96. PMLR, 2021.
Ruibo Tu, Kun Zhang, Bo Bertilson, Hedvig Kjellstrom, and Cheng Zhang. Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation. NeurIPS, 32, 2019.
Tyler J. VanderWeele and Ilya Shpitser. new criterion for confounder selection. Biometrics, 67, 2011. URL https: / //ppi..... semant: cscholas .rr.rrr../..ooo................a hhtpg//orpussii ID: :23400719
Chris Wallace, Kevin B Korb, and Honghua Dai. Causal discovery via mml. In ICML, volume 96, pp. 516-524, 1996.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et Chain-of-tie of-th-of-thought prompting elicits reasoning large language models. NeurIPS, 35: 24824-24837, 2022.
Moritz Willig, Matej Zečević, Devendra Singh Dhami, and Kristian Kersting. Probing for correlations of causal facts: Large language models and causality. 2022.
Yuzhe Zhang, Yipeng Zhang, Yidong Gan, Lina Yao, and Chen Wang. Causal graph discovery with retrieval-angmeented generation based large language models, arXiv preprint arXiv:2402.15301, 2024.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. NeurIPS, 31, 2018.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
In this appendix, we include the following additional information, which we could not include in the main paper due to space constraints:
Appendix A: Illustration of our triplet query strategy
Appendix B: Proofs of propositions
Appendix C: Practicality of the adjustment set obtained using the Causal Order
Appendix D: Algorithms to integrate causal order into existing discovery methods
Appendix E: Additional results, including L.LMs used post-processing for graph discovery and discussion of triplet vs pairwise query strategies
Appendix F: More details and examples of our query strategies
Appendix G: Causal graphs used in our experiments of the datasets
A ILLUSTRATION OF OUR TRIPLET QUERY STRATEGY
present an intuitive illustration of our overall triplet querying framework obtain causal order from imperfect experts in Fig Al below.
1. Nodes Triplets
Noiey Exports
2. OOddringgSabgraapp
Query Experts
Causal Order
J.C.D.D..I
[AR.C.C...DIJJ
LLMs
4. Getting Final Causal Order
3. Majority Voting Edge Orientation
Assimilate & infegrate obtained order
Final DAG
Deciding edge direction
Deciding edge direction
A B:
Causal Order
A
No Connection
B:
2
final!
No Connection
2
Tie!
Standard Causal Discovery & effect estimation
Causul Order
Final
PC/ CAMML
Outcome
Observational
Treatment
5. Using Causal Order Prior for Discovery Algorithms
6. Using Causal Order from graph for Causal Effect Inference
Figure A1: Leveraging Causal Order from Imperfect Experts. Our triplet-based querying method infers all three-variable subgraphs from imperfect experts and aggregates them (using majority voting) to produce a causal order. Ties causal order are broken using high-cost expert. Expert-generatee causal order integrated with discovery algorithms, before estimating causal effect.
B PROOFS OF PROPOSITIONS
To estimate doox =I:)] from observational data, the backdoor adjustment formula is used.
14

---

Published as a conference paper at ICLR 2025
Definition B.1. Backdoor Adjustment (Pearl, 2009). Given DAG G, a of variables Z satisfies the backdoor criterion relative to a pair of treatment and target variables (X,, X₃) (f) no variable in Z is descendant of X;; and (ii) Z blocks every path between X, and X; that contains arrow into X
Proposition 3.1. Let the true causal DAG be 9(X, E) with ground-truth adjacency matrix A. Consider procedure to estimate a graph G by querying Perfect Expert (as in Def. 3.3) with pairwise queries X,, X3 with auxiliary set followed by subsequent aggregation of predicted edges from each query (i.e. from total of X C2 queries). The causal order of the graph G thus estimated is correct, i.e. (T)) = 0 for all values of the sets Oij. As a corollary. the causal graph thus estimated can however have errors. In other words, when Oi & Vi, Deop (xop(=(()) A) O whereas Structural Hamming Distance (SHD) between G and 9 = Σ X Ide{ (X,) - c
Proof. First claim (Diop( ( (9). 0): By definition, the Perfect Expert adds new edges that are not present in the true G, but cannot miss predicting ground truth edge. This implies that all edges between any two level j where i < j that are present in the ground truth graph G are also present in the estimated graph G. Given any two nodes X and X with the same level order "I" in the true causal graph. Since there is no directed path between X and X2 the perfect expert will never predict any edge between them (using Def. 3.3). Combining these two observations, the level order of both the graphs 9 and G remains the same. Next, will use the following lemma that states that if the level order of two graphs remains the same then the topological order remains the same thus completing the proof of the first claim.
Lemma B.1. Given two DAG G1 and G2 have same level order (see Def. 3.4) then there exist two topological order TI (91) and π (G2) corresponding to the two DAG s.t. the ordered set (()1) = (G2).
Proof. Since the level order the same for both the graphs, all the nodes on given level "I" for both graphs G1 and G2 are the same. Now, any two nodes on the same level don't 't have any edges between them. Thus add all the nodes on the level in the same order to both (91) = TG2)). Thus when we are done adding the nodes from all the levels in the topological order we get the (G)) = x (G2). D
Second claim (SHD> > 0): Recall that SHD counts the number of missing, falsely detected, and falsely directed edges in the estimated causal graph as compared to the ground truth graph. Since the perfect expert correctly predicts all the ground truth edges, there are no falsely directed or missing edges in the predicted graph. From Def. 3.3, when queried over all [X C22 pairs of nodes the perfect expert will add additional (falsely directed) edges between a node and all its descendants. Thus total
| number of falsely directed edges = | de((x)) [ch(X;)] = SHD.
Proposition 3.2. (Pearl, 2009; Cinelli et al., 2022) Under the no latent confounding assumption, for a pair of treatment and target variables (X,, X,) in DAG G. Z = {Xx T < Ti is a valid adjusstmntt set relative to (Xi, X,) for any topological order π of G.
Proof. Before starting the proof, we define a confounding variable. A confounder is a variable that should be casually associated with both the treatment and the target variables and not on the causal pathway between treatment and target. An unmeasured common cause can also be source of confounding the treatment- + target relationship. Coming to the proof, we need to show that the set Z {Xx IT& } satisfies the conditions (i) and (ii) in Defn B. For any variable X such that тæ < Te, we have X & de(X,) and hence the condition (i) is satisfied. Additionally, for each X1 pX() we have Th n. and hence pa(X,) C Z. Since pa(X,) blocks all paths from X, to X, that contains an arrow into X; (Peters & Bühlmann, 2015), Z satisfies condition (ii).
Proposition 3.3. For an estimated topological order ã and a true topological order т of a causal DAG G with the corresponding udjacency matrix A, Drop (ū, A) 0 iff Z (Xel < ft) is valid adjustment set relative to Xi), VT, < T..
15

---

Published as a conference paper at ICLR 2025
Proof. The statement of proposition is of the form A = B with A being "Drop (ft A) 0" and B being "Z = {Xag fTk < } is a valid adjustment set relative to (X,, X,), Vi We prove B by proving (i) A = B and (ii) B = A.
A =
(i) Proof of A = B: If Deop [ft A) = O, for all pairs of nodes (X,, we have R; < ū whenever T, Ty. That is, causal order in estimated graph is same that of the causal order in true graph. Hence, from Propn 3.2, Z = {Xa f < #;} is a valid adjustment set relative to (X;, X2), Vi
(ii) Proof of B the contrupositive of B =
A A.... -A A. To this end, assume Deop (ft, A) * O, then there will be at least one edge X X, that cannot be oriented correctly due the estimated topological order म.. i.e., T < n. but Ty > Ty. Hence. to find the causal effect of X, on Xr; I # j. X, is included in the back-door adjustment set Z relative (X,, X1). Adding X3 to Z renders Z invalid adjustment set because it violates the condition (i) of Defn B.
A: we prove the logical equivalent form of B
-B,
Assumption B.1 (DAG Acyclicity). Given that e-expert & is used to predict a causal graph between a set of nodes, the predicted causal graph is acyclic.
Assumption B.2 (Error Distribution and Probability Renormalization). Let [01, C22 c3) be the three choices for a causal relationship between node A and B (see Def4.1). Let P((z) and P(ca) be the probability of selecting the corresponding three choices by the e-expert E.. We assume that the probability for the two wrong options are equally likely, i.e., equal to e/2. If any constraint T renders some of the choices not possible i.e. P(c,IT) = 0 for some E 2,, then & renormalizes the posterior probability over the other choices,i.e. P(c|T) = Ey.Meyyli P(aj
P(e.)
where the denominator is summed over s.t. P(cşIT) #0.
Proposition 4.1. Given two nodes A and B of an underlying causal graph, access an c-expert & that doesn't ' produce any cycles in the predicted causal graph (see Assm B. for formal statement) and and renormalizes the probability in case an option is not available (see Assm B.2 for formal statement), lel C + A + B be any other node in the graph. If & predicts causal relationship between all pairs of nodes sequentially, the marginalized probability that & makes an error in predicting the causal relationship between A and B, after it has already predicted the causal relationships between (C, A) and (C, B), is less than e, where marginalization is over all possible causal graphs that can be formed between A, B and C, with each of such graphs being equally likely.
Proof. Without any additional constraint, e-expert (E₂) has "c" probability of making incorrect prediction. But in presence of additional constraint. e.g. DAG constraint (see Assm B.1), the probability of error changes and is given by the following lemma:
Lemma B.2. Suppose we have two nodes A and B and three possible choices (c1,02, C22 C3) for causal relationship between them ie A - B, B - A or no edge between them (not in any particular order). Without loss of generality, let c3 be the ground truth causal relationship between node A and B. Thus, without any additional constraint, let the probability assigned to each of the three choices by e-expert (E.) ) is P(ct) = 61. P(cz) ) = 62 and P(cs) = 1 - 61 - E2 respectively where e = 61 + 62. If due to additional constraint (e.g. acyclicity Assm B. ). one of the incorrect choice gets discarded, say et then the new probability of selecting the wrong choice (C2 given by é' is always less than e. However if the correct/ground truth choice is discarded due to this additional constraint the new probability of selecting the wrong choice (cr or c2) is 1. In case, no options are discarded the new probability of choosing the wrong choice remains same i.e as before.
Proof. For the case when the correct/ground truth choice i.e C3 is discarded due some constraint, the only left out choices are wrong choices i.e. C1 and c2. Thus the probability of making error in selecting the correct choice is 1. Next, for the case when one of the incorrect choice (here ei w.l.o.g) is discarded, we are left with one incorrect (c2) and one correct choice (ca). From Assm B.2 once a particular option is discarded, the e-expert renormalizes the probability proportional to their initial probability. Thus the new probability (c2)) of choosing wrong option 02 is:
P(c2) = 1 1- E1 62
62
62
c/2
E
(I)
+
1- E1
1-e/2
2 -
16

---

Published as a conference paper at ICLR 2025
where 61 = 62 e/2 from Assm B.2. Next, we can show that P(cz)
completing our proof. To
have P((2) <. need:
P((z) = --6 9
62 <<=a
7 62 <1 + 62 - 1 116
(2)
0 + 62 1--11 101 <0
which is always true since from Assm B.2 we have c > 0, Eg > 0 and 1 - 61 - 62 > 0.
Now, give any three nodes A, B and C, Table Al summarizes all possible partially completed graph (henceforth partial graph) possible between those nodes. Each partially-complet generated more DAG based the orientation of the node A and B. Specifically, each of the partial graph 1, 2, 3, 4, 5, 7 and 9 generated three graphs (A - B. B - A or no edge between A and B) and partial graph 6 and 8 will give two DAG (one option not possible constraint). Thus overall we have 25 possible graphs. Our next goal is to show that the marginal probability of choosing the wrong causal relationship for node (A, B) when oriented last among is less than e, where marginalization is over all the causal graph depicted in Table A1 (assuming all graphs are equally likely). The expert &, finds the causal relationship sequentially for all the pairs in {(C, A), (C. B), (A, Bj}. We are interested in the case when & finds the causal relationship for pair (A, B) in the end. Let F, S. T (called first,second and third) be three binary random variable and the value 0 represent whether the causal relationship discovered by &, for first, second or last/third pair respectively is incorrect and I represent it is correct. So the probability of error when finding the causal relationship between node A and B when oriented last/third (denoted by P(T)) is given by: Σ
DAG in Table Al
maintain acyclicity
P(T = Σ
P(G))(F, (F)
GEG
(3)
Σ
E
-
25
GEG (.TE(011xx(0.))
where G denotes the set of graphs generated by orienting the causal relationship between A and B for all partial graphs in Table A1, 9 25 and all the graphs are equally likely, different configuration of (F, S) shows whether the causal relationship between first two pairs (C, A) and (C. B) are correct or not. When orienting the first two pair of nodes i.c (C, A) and (C,, B) there is no DAG constraint thus we have:
2
when S T
c(1 - E)
when 5= 0, 0,T=1 =1 when S S= =,1,7' 1,7'=
P(F,S)
=
e(1 E)
(1 e)² when $ = 1,7
Now based on the graph G E 9 and the setting of S.T, P(T = =. G) takes different values. Suppose that the causal relationship between the first two pairs (C, and (C, are already predicted by the expert. We observe that the DAG acyclcity constraint (Assm B.2) will only change the probability of error for orienting nodes (A, B) (P(T = F, G) given by Lemma B.2) when the predicted causal graphs is either B + C C + A or A - C + B after orienting (C, A) and (C, B). For all the other predictions of (C. and (C, B). they don't Ion't enforce any acyclicity constant for finding the causal relationship between (A, B). thus, P(T = O| S, G)
(from Lemma B.2). Table A2 summarizes of error probability for all the partial graphs in Table A1 (P(F, SIG) and P(T F, S, G)). The first column shows different partial graphs from Table A1. The second column then shows different causal relationships that are possible between the nodes A and B for a particular partial graph. Given one true orientation between node A and B we get final ground truth graph. Thus the third column shows the probability of prediction of structure A + C + B for a particular true graph and the fourth column shows the probability of making an error in predicting the third causal relationship i. between (A, B) given the first and second pair (C, A) and ((C is already predicted. Similarly, the fifth and sixth columns show the same thing for the predicted structure A + C + B for each of the ground truth graphs. The partial-grap number 4, 7,8 is not depicted in the table but the entries for 4 graph is the same as 2nd, 7th is the same as 3rd and 8th is same as partial-strucctre.
6th due to symmetry in the partial- structure. The value of -ー in 4th and 6th column is given
by renormalized probability Eq. in Lemma B.2. Substituting the values from Table A2 in Eq.
17

---

Published as a conference paper at ICLR 2025
1.
2.
3.
5.
6.
B
8.
9.
B
Table Al: All possible causal graph between three variables A, B and C. The dashed arrow represented undecided causal relationship between node A and B. So, the dashed arrow can take one of three choices edge between A and B. To ensure that the graph is acyclic, some of the graphs above
A + B, A < B might allow all three choice for causal relationship between node A and B. Hence the causal-graph and 7 each have three possible graphs and 5, 6, and each have two possible graphs hased on the valid choice of causal relationship between A and that preserves acyclicity constraint. So overall there are 25 possible different causal graph between three variables A, B and C.
3,
and using the value P(T|F,S, G) = e for the rest of the predicted structure not mentioned in the Table A2 we get:
2
+ [ - 2 é²
P(T
2
2
2e + 1
25
4
2
611 $)
"[[2
12 + +1] 1]+
e²
<(1
6)
+
+
x #4
4
4
(1 c)
c(1 - c)
+
2 *
2c + 1
+ - 2#
x2
(5)
2
2
+
(1 - +)2 2
+
2
+1
[ 1- (1 e)-
2
*
4
<(3e² 30E + 52)
25
4 - 2c
Now we want to show that the error probability for the third pair (A, B) given by the above equation less than For that, we need:
c(3e² - 330e + 52)
25
4 - 2E
(6)
3ea + 20e 48 <
The above inequality is always satisfied since E (0, 1) and 3e² 20c 48 is always less than in
the allowed range of since the roots of the quadratic equation are -10/3 2//1/33 = -85 and
-10/3 2/61/2 /3 1.87. Thus P(T 0) < for all values of E (0, 1) completing our proof.
18

---

Published as a conference paper at ICLR 2025
Predicted Orientation in first two steps (F.S) S) A + C B
True Orien- tation (A, B)
A C B
Partial True Graph
P(F,SIG) G) P(F, P(T)E,S,G)
1.
C
no edge
A + B
A
B
A + B
1
2.
C
no edge
(5) 2
A - B
3 (1 c)
A
B
A - B
1
3.
C
no edge A + B
(L) c)
A
B
A + B
é
1
5.
no edge
é'
(3)(1-)) c)
5 (1) (1-d) c)
A 7 B
A
B
A + B
1
6.
no edge A < B
é'
(5) 2
61-
é'
1
A
B
9.
edge
(2)(1-)
à (1-1 (
é
A B
A
B
A A-D B
Table A2: Summary of Error Probability for all the partial graphs in Table Al (P(F, S)G) and P(T S, G)): The first column shows different partial graphs from Table AL. The second column then shows different causal relationships that are possible between the nodes and B a particular partial graph. Given one true orientation between node A and we get final ground truth graph. Now we observed the proof of Proposition 4.1 (sce Proof B), that the error probability for the prediction of causal relationship for the pair (A, B) will only change when the e-expert predicts the the structure A <- C + B or A C + B for the pair of nodes (C, A) and (C, B) for any ground truth graph. For the rest of the possible predictions of pair of nodes (C, and (C, B) in any ground truth graph, the error probability for (A, B) remains (see Lemma B.2). Thus the third column shows the probability of prediction of structure A <- C - B for a particular true graph and the fourth column shows the probability of making an error in predicting the third causal relationship i. between (A, B) given the first and second pair (C, A) and (CC, B)) is already predicted. Similarly, the fifth and sixth columns show the same thing for the predicted structure A C for each of the ground truth graphs The partial-graph number 4, 7, is not depicted in the table but the entries for 4h graph is the same as 2d 7th is the same as 3rd and 8t is same as 6th due to symmetry in partial-structure. The value of = z'c th and 6th column is given by renormalized probability Eq. in Lemma B.2.
C PRACTICALITY OF THE ADJUSTMENT SET OBTAINED USING THE CAUSAL ORDER
Including variables appearing before treatment (in the causal order) is actually a widespread practice in biomedical and social science empirical studies. In these studies, such variables are called "pre- treatment variables" and a common practice is to condition on all of them. For this reason, we do not think that our proposal is impractical. The importance of Prop. 3.2 is to show the utility of the causal order to identify such a commonly used adjustment set.
For example, refer the Covariate selection chapter (Sauer et al., 2013) by Sauer, Brookhart, Roy and Vanderwheele in a User Guide ("Developing a Protocol for Observational Comparative Effectiveness Research"). In the section "Adjustment for all observed pre-treatment covariates", they mention
19

---

Published as a conference paper at ICLR 2025
the widely used propensity score adjustment and write, "The greatest importance is often placed on balancing all pretreatment covariates." " Thhyy also add that while theoretically colliders can bias the result, "in practice, pretreatment colliders are likely rarer than ordinary confounding variables.".
Further, when unobserved confounding cannot be ruled out (as is the case with most observational studies), evidence is not clear whether should include all treateettmett covariates select a few, especially because the true graph may be unknown. "Strong arguments exist for error on the side of overadjustment (adjusting for instruments and colliders) rather than failing to adjust for measured confounders (underadjustment). Nevertheless, adjustments for instrumental variables have been found to amplify bias in practice". As the last sentence suggests, note that we are not claiming that adjusting for all pre- tre-treatment variables (variables before treatment in causal order) is always the correct approach; but rather showing that it can practical in many situations.
Theoretically, of course, improvements to this causal order criterion are possible. Vanderweele and Shpitser (2011) 1) (VanderWeele & Shpitser, 2011) cite the popular practice of using "all pre-treatment variables" and propose the Disjunctive Cause criterion an improvement. This criterion states that if a pre-treatment variable causes the treatment, outcome, or both; then it should be included the adjustment set. Note that this and/or outcome- is quite close to the causal order-based criterion in our paper. Except for possibly conditioning collider in cases where there are unobserved variables in the graph (see Fig. I from (Vander Weele & Shpitser, 2011)), additional variables in the causal order adjustment superset will not have significant effect the estimate.
including all pre-treatment ancestors of treatment
D ALGORITTMM FOR INTEGRATING CAUSAL ORDER IN EXISTING DISCOVERY METHODS
In continuation to the discussion in Sec 3.2, the algorithms for integrating causal order into exist- ing constraint- based and score-based discovery methods are summarized in Algorithms 1 and respectively.
Algorithm 2 Integrating á in score-based methods
Algorithm 1 Integrating â in constraint-
based methods
1: Input: Dataset D) Variables {.i,..., Xn Ex- pert &, Score-hased method S, Prior probability p.
1: Input:
Noisy expert topological order-
2: Output: Estimated topological order Nimal of
ing ft, Expert E, CPDAG
{Xi...,,X,, X.).
2: Output: Estimated topological order
Ammal of (X......XX.
3: G = -(,,,,..,,XX X,,)
3: for (i (j) mndirected-edges(S do
4: level order of 9
5: for cycle C E 9 do
4: both nodes and j are in स and if
R; < <Rt. orient (i - jj as (i + j) in
6: for node E C do
C.
7: L(node) min(leve((e) Ve E C)
8: end for
5: Otherwise, use expert & to orient the edge.
9: end for
10: G = S(D,L.)) //L is provided as
6: end for
7: innal topological ordering of
prior
11: Fimal topological ordering of 9
8: return ftml
12: return mamal
E ADDITIONAL RESULTS
E. STUDY ON DOWNSTREAM TASKS: CAUSAL DISCOVERY
In continuation to the results presented in Sec 5 of the main paper, we present the performance on the causal discovery task across all sample sizes Table A3. Evidently, as stated in the main paper, the results show that using expert-provided causal order improves Dtop across our experiments consistently. CaMML+Human/LLMM yields benefits even at higher sample sizes. At a sample size of 10000, CaMML's Dtop for Child and Asia surpasses CaMML+LLM by three and fourfold respectively. In specific datasets like Survey where the variables are better understood by humans, incorporating human priors to CaMML. leads consistently zero Dtops outperforming L.LM output.
20

---

Published as a conference paper at ICLR 2025
SCORE
ICA
Direct
NOTEARS
CaMML
Ours
Ours
LINGAM
LINGAM
(PCILEND
(COMNNL-LLMO
(CaMML
Fanhquako
0.
4.000000
10010000
100100000
100000000
2,00:0000
0 1.00-000
CL0STBUM
0,00 00.00
1.0
Cancer
0000-1000
1,000000
2,020000
2,000000
2,000000
2,00+0000
0
0L00-0U00
0.00-0.00
1U90-1000
Sarvey
0.50:000
4000:000
2,020000
4,0000000
4.00+0.00
3.33=0.94
0
535400944
0.00:0.00
0,90-1000
Adiz
20010059
7.001000
3.3310.47
100000000
3,0020.000
1.8510055
10005000
0.7180.22
N/A
NIA
Asa M
1500000
6,000000
10010.00
3,0010000
3,0010000
1.00::0000
1.00.0000
17711045
1,0000.00
2 00000.00
Child
5.753000
12.01000
14, 4,3330047
16.00.000
14.010.00
3.00:0.00
4.0010000
45530045
N/A
NIA
Neumpothic
40010000
6,000000
13016.16 16
10010:00
9,0010.00
10.4 195
3 1000-800 8000
5,0000000
N/A
NIA
Eatliguake
0.75:025
4020000
1000000
1,5000
1,:50.0
0 0.00
0.00:80110
0.00:0..00
Cuncer
0.168826
3,0000000
4.4000.55
3,0000000
2.00:00.00
100010000
0.334:3557
1001::000
0.00100.00
0,00
Survey
1254000
4031000
6.00-0:0
6,0H000
10000.00
1,0:10.00
0,00
5,6+0000
503H000
7,00000
3.81
100-0.00
0L9TH
N/A
AxisM
200+000
6000000
5,0+0000
1.0000.000
2.0000000
0.17::0.55
1.3300.57
3
Chila
80090000
62061133
12.24074
10,441.3
2.00:0000
N/A
NIA
Neumpathic
6:0000.000
9.00
13,050.00
11.030.000
9.32
5,0000000
Banhquake
3,0050000
000300000
0.
0.00:00.00
200:0.00
4.0000000
5.0210.00
5.0010.00
2.00
2,0033000
2.250.00
2.00:0000
10010.00
4.0010.00
6.0010.79
4.4011.35
3.0010.45
1.75
0.72:0622
NIA
20010000
4,0010000
7.6010.45
4.5010.45
3.2010.39
1.64
2,0033000
0.0000.00
2,00:00:00
2.00000000
Child
K251:000
3,0010000
12
10.511.722
14.240.00
3.00
7,00003000
3.0.0.00
NIA
Neumopothic
K6210000
6,001000
9,3310.94
10.00.000
10,000.000
4,30110.5
9,0000000
1.350-42
NIA
Eanhggake
0L00-8000
400H0000
3,02+0.00
3,00+0.00
100+0.00
0.40:4O4R
BL00-8L00
EL00-BUU0
0
Cunter
20010000
3,0010000
3,0010.00
3,00100000
2,00100000
0 50.0
2,00E0000
0.0010000
2,00200:00
0.00:00.00
Survey
200100000
4.0010000
5.0210.00
5,00100000
3 3,0010.00
3.001:.335
2,0050000
1831000
2,00210:00
0.00.0..00
Asia
1S2000
4.0310000
5,0210.00
4.40
3,0010.00
1,40:10488
8,00
0.310-477
N/A
NIA
Asi-M
1001:000
4.0210000
5,031000
4 0.9210.99
3,0010.0)
2,00:10000
0,00 1.00-0000
8.00-0.00
S 00-00000
Child
5.00-3004
5,000000
12.2+1466
11,40043
14.40043
2.82++084
5,002.664
LA0+0U00
N/A
NIA
Nammsahic
10,0000.00
6,00H000
10.00.000
10.0+0.00
3,00:00000
10,00+0.00
LA0+0U00
N/A
NIA
Table A3: Comparison with causal discovery methods, showing mean and std dev of Dtop Neuropathic subgraph (1k samples), PC Algorithm returns eyclic graphs in MEC). Human experiments not conducted for Neuropathic, Child (due to feasibility issues) and Asia; hence rows marked as N/A.
runs, (For the
E.2 Dtop SHD: BETTER MEASURE EFFECT ESTIMATION ERROR
As discussed in Sec 3.2 of the main paper, we show herein that Dtop has a strong correlation with effect estimation error and hence a valid metric for effect inference.
Asia
Child
0.5
0.3
00.2
Facczia otart
0.1
0.1
0.0
0.0
Dop
Figure A2: Drop vs. CACE. EACE increases as Dtop increases, aligning with theoretical observations.
Cancer
Dao
SHD
SHD
eaoB
Dzop
CACK
0.00
0.00
2
0.00
0.25
0.00
2
0.50
Asia
Deop 0
SHD 3
SHD
Dzep
CaCR
0.00
0.14
0.00
2
0.22
0.57
Survey
Deop 0
SHD = 2
SHD
Dtop
0.00
0.00
0.00
0.25
0.03
0.50
Table A4: cACE DDD given Droy Daoy given SHD)
21

---

Published as a conference paper at ICLR 2025
E.3 LLMs USED IN POST PROCESSING FOR GRAPH DISCOVERY
We conducted some experiments where we utilised discovery algorithms like PC creating skeletons of the graph and employed LLMs for orienting the undirected edges. The idea was utilise LLMs ability to correctly estimate the causal direction while leveraging PC algorithm'ss 's ability to give skeleton which could be oriented in post processing setup. We that LLM ended up giving improved results as compared to PC alone.
E.4 TRIPLET VS PAIRWISE QUERY STRATEGIES
In continuation to the discussion Sec 5 of the main paper, we include Tables A5 for more details. The pairwise strategy also shows flaws when LLMs used as noisy experts. In many cases, pairwise querying yields cycles due to which Drop cannot be computed. In particular, for the Child dataset with 20 nodes, pairwise querying of LLMs yields extremely high number of cycles (see Table A5). L.M output tends to overconnect, resulting in high SHI. Overall, among the prompting strategies, the chain of thought prompt performs the best: it has the lowest number of cycles for Child and Neuropathic datasets. This indicates that in-context examples and thain-of-thought reasoning help to increase the accuracy of causal order output, but other contextual cues do matter.
Dataset
Dcop
SHD
IN/TN
Cycles
Base Prompt
Earthquake
0
7
0/5
0
Cancer
0
0/5
0
Survey
0/6
Asia
21
0/8
Asia-M
15
0/7
Child
177
0/20
>>33
Neuropathic
212
0/22
>>5k
All Directed Edges
Earthquake
1
0/5
0
Cancer
0/5
Survey
0/6
Asia
21
0/8
Asia-M
0
13
0/7
0
Child
139
0/20
300
Neuropathic
194
0/22
>> 1k
One Hop Iteration
Earthquake
0
0/5
0
Cancer
0
0/5
Survey
(1/
Asia
Asia-M
0
14
0/7
0
Child
167
0/20
400
Neuropathic
204
0/22
> >4
Table A5: Comparison various querying strategics for only LLM-based setups, providing different contextual cues in each setup about the graph. IN: Isolated Nodes, TN:Total Nodes.
22

---

Published as a conference paper at ICLR 2025
Dataset
Drop
SHD
IN/TN
Cycles
Chain of Thought
Earthquake
0
4
0/5
0
Survey
I
9
2/6
Asia
0/8
Asia-M
13
0/7
Child
138
0/20
>>500
Neuropathic
64
0/22
5
Triplet Query
Farthquake
0
4
0/5
Cancer
I
6
0/5
Survey
0
9
0/6
Asia
I
0/8
Asia-M
I
11
0/7
Child
138
0/20
391
Child (+ Cycle Remover)
1
28
10/20
0
Neuropathic
151
0/22
Neuropathic(+ Cycle remover)
24
13/20
Table A6: Triplet query output using variable names with their descriptions (Cancer not included since CoT prompt has examples from this graph). IN: Isolated Nodes, TN:Total Nodes. Since calculating total number of cycles in a DAG is computationally challenging (NP Hard), we find lower bound cycles present in each graph based on total k lenght cycles in each setting, where k=5. If i scaled up, the number of such unique cycles in the LLM output will also scale significantly. Lower bound helps us make comparison with number of cycles in outputs like in Triplet strategy, where numbers are comparatively smaller and can be calculated easily.
Dataset
SHD
Dtop
Cycles
IN/TN
Base Prompt
Asia
18
Child
148
>> 10k
Farthquake
4
0
0
Survey
7
0
Neuropathic
178
10k
0
Covid
33
15
Alzheimers
30
0
Table A7: Final result of using performing base pairwise querying strategy with GPT-4. These results show how using a superior model in pairwise querying does not lead to complete removal of cycles, further highlighting the impact of triplet strategy.
23

---

Published as a conference paper at ICLR 2025
Dataset
Metric
Pairwise (Base)
Pairwise (CoT)
Triplet
Using Phi-3
Drop
0
SHD
17
11
13
Asia
Cycles
0
IN/TN
1/8
1/8
Drop
7
SHD
28
28
25
Alzheimers
Cycles
11
0
IN/TN
0/11
0/11
0/11
Drop
17
SHD
142
80
69
Child
Cycles
=10k
59
0
IN/TN
0/20
0/20
0/20
Using Llama3
Drop
2
SHD
22
23
17
Asia
Cycles
71
20
0
IN/TN
0/8
0/8
0/8
Drop
5
SHD
41
29
24
Alzheimers
Cycles
1144
0
IN/TN
1/11
0/11
1/11
Diop
12
SHD
167
151
129
Child
Cycles
>10k
71
0
IN/TN
0/20
0/20
0/20
Table A8: (Top) Results using Phi-3 (Bottom) Performance of triplet method using Llama3 (8b) models vs CoT pairwise vs base pairwise query strategy on multiple benchmark datasets across diff' metrics: Dtop- SHD, (Num of) Cycles, IN (Isolated Nodes), TN (Total Nodes). When num of cycles>0O, cannot be computed, hence Dzop is given by '-'. Triplet consistently outperforms the pairwise (base as well as CoT) strategy across metrics & datasets, especially by significant amounts larger graphs like Child.
Graphs
Dtop
SHD
Cycles
Isolated Nodes
LLM Calls
Number of Nodes
Complesity
Quadruplet
Asia
0
0
70
(
Covid
19
0
330
O(n3)
Alzheimers
0
O(n3)
Triplet
Asia
1
0
286
O(n))
Covid
0
30
0
0)
165
11
O(n
Alzheimers
4
28
0
165
11
O(n")
Table A9: Analyzing the performance differences between using triplets and quadruplets, we found no significant difference in the quality of the final graph output. However, the number of LLM API calls more than doubles when shifting from triplets to quadruplets, leading to a substantial increase in cost.
24

---

Published as a conference paper at ICLR 2025
Dataset
Metric
Triplet GPT-4
Triplet GPI:3.5-Turbs
Asia
Diop
0
SHD
10
14
Cycles
0
0
IN/TN
0/8
0/8
Alzheimers
Dian
4
4
SHD
23
28
Cycles
0
0
IN/TN
0/11
0/11
Child
Diag
1
1
SHD
24
28
Cycles
0
0
IN/TN
6/20
10/20
Table A10: Results of running GPT-4 for orienting triplet subgraphs, and then re-using GPT-4 for resolving clashes during merging phase. These results cover graph discovery on Asia, Alzheimers and Child graphs. Upgrading to superior model (GPT-4) leads to better results for all three graphs on triplet strategy.
Graph
Sample size
Before LLM prior
After LLM Prior
250
18
16
500
16
15
Child
1000
14
13
5000
13.5
12
10000
9.66
6
250
3.83
3
500
3.6
3
Earthquake
1000
3.6
3
5000
1.16
0.66
10000
0
0
250
0
500
3.83
3.83
Cancer
1000
2.6
2.6
5000
2.3
2.3
10000
2
2
250
7.5
5
Asia
5000
10000
2
250
4.5
4
500
4
Asia-M
1000
5.5
5
5000
4
4
10000
4
4
250
27
26
500
31
29
Neuropathic
1000
41
40
5000
55
53
Table A11: Comparison of SHD Values Before and After Incorporating LLM Priors Using the PC Algorithm Across Various Graphs
Finally, the triplet prompt provides the most accurate causal order. For small-scale graphs, it produces cycles and consistently produces minimal Dtop (ranging from 0 while also producing isolated nodes. Even for medium- size graphs like Child and Neuropathic, the LLM output includes significantly fewer cycles than the pairwise strategy, which were removed leading to a significant and accurate causal order used further as prior. That said, we do see that isolated nodes in the output increase after cycles are removed for medium graphs (all graphs connected,
outputting isolated node is error). Considering L.MSs virtual experts, this indicates that there are some nodes which LLM expert cannot determine the causal order. This is better tradeoff than providing the wrong causal order, which can confuse downstream algorithms. Overall, we conclude that the triplet query strategy provides the most robust causal order predictions. Additional results showing the error introduced by the I.L with respect to ground truth order are shown in two different settings in Tables A15 and A16.
25

---

Published as a conference paper at ICLR 2025
Dataset
Dtop
SHD
IN
Cycles
Alzheimers
14
0
0 1
Covid
36
0
Table A12: Results of hybrid approach where the PC algorithm integrates an LLM-derived prior (GPT-4) obtained via BFS for Alzheimer's and COVID graphs. The prior directly provides edge orientations, which guide the imitial graph structure, while PC subsequently orients remaining edges. Unlike triplet that used only causal order, this approach incorporates the full graph as prior. The PC algorithm is further supported by large observational dataset of 10,000 samples. The results show that PC + BFS (GPT-4) is also outperformed by Triplet method (GPT-3.5). Specifically, PC+BFS yields cycle and higher SHD Covid dataset. On the Alzheimers dataset, PC+BFS is comparable: yields higher Dtop but lower SHD.
Dataset
Samples
LLM
Ground Truth
PC (Average MFC)
250
1.00+0.00
0.00=10.00
2.0011.00
Asia
1000
3.00==0.00
2.00++0.00
3.00+0.00
10000
3.00=+0.00
3.00++0.00
3.00++0.00
250
5.00=#0000
5.00=0000
6.50+0..00
Child
1000
6.00=+0.00
6.00=+0.00
8.43++0.00
10000
9.00==0.00
9.00==0.00
9.75+t0.00
Table A15:
Comparing Drop of final graph using L.I order vs Ground truth order as prior PC algorithm for Child and Asia graph, averaged over runs
samples
Context
Base prompt
Past iteration orientations
Markov Blanket
(Avg. MEC)
Deop SHD
8.0
9.61
14.33
12.66
14.0
17.0
10000 samples
Droy
6.33
9.66
6.0
7.67
SHDD
9.0
13.33
8.33
12.0
Table A13: PC + LLM results where LLM is used to orient the undirected edges of the skeleton PC returns over different data sample sizes. We show how LLMs can be used in post processing setup for edge orientation besides having the capability of acting as strong prior for different discovery algorithms.
Dataset
Samples
(ATE(S1)
(ATE(S2)
(ATE(Sa)
A12
A13
250
0.70:10.40
0.70
0.69-1.33
0.00.10.00
0.00-10.00
500
0.64
0.64=0.39
0.64=0.38
0.00=0.00 0.00+0.00
Asia
1000
0.59+10.2
0.59=0.32
0.59=0.32
0.00=0.00 0.00#0000
5000
0.59+0.30
0.59=0.30
0.59=0.22
0.00+0.00
0.00=0.00
10000
0.49:10.00
0.49-10.00
0.49-10.00
0.00-10.00
0.00-10.00
Table A16: Results Asia dataset. Here we test the difference in the estimated causal effect of lung dyspnoeu when the causal effect is estimated using the backdoor S /smoke/ the causal effect estimated when variables that precede treatment variable in two possible topological orders as backdoor sets: S2 = [asia, smoke), S2= [asia, tub, smoke). A12, Ars refers to the absolute difference between the pairs EAræ (Si), (Ar)(S2) and (Are(Si). (Are(S,) respectively. From the last two columns, we observe that using the variables that come before the treatment node in topological order as backdoor set does not result the deviation causal effects from the ground truth effects.
26

---

Published as a conference paper at ICLR 2025
BFS
BFS + Statistics
Dataset
GPT-3.5
GPT-4
GPT-3.5
GPT-4
Dtop
SHD
IN
Cyc
Drop
SHD
IN
Cyc
Drop
SHD
IN
Cyc
Drop
SHD
IN
Cyc
Asia
2
7
0
0
1
0
23
0
33
0
3
0
Alzh.
5
17
2
0
34
27
1
17
14
Child
40
6
30
52
2
21
2
27
Covid
28
0
4 5
20
0
30
0
15
32
10
Table A A17: Comparison of BFS and Statiiitss approaches using GPr--.5-turbo and GPT-4, Datasets used: Asia, Alzheimers, Child, Covid. Metries: topological distance ( (Drop), structural hamming distance (SHD), Isolated Nodes (IN), and cycle count (Cyc).
F QUERY STRATEGIES: MORE DETAILS AND EXAMPLES
As stated in Sec. E.4, we follow earlier efforts in studying pairwise query strategies in experiments. Beyond the basic query strategy, we also study its augmentation with additional contextual information. In summary, we study four types of pairwise queries, which we describe helow.
Basic prompt. This is the simplest technique. We directly ask the expert to find the causal direction between given pair of variables (Kiciman et al., 2023).
Chain-of-Thoughh (+ In-context Learning). Based encouraging results of providing in-context examples in prompts for various I.IM tasks (Brown et al., 2020), we include 3 examples the ordering task that we expect the expert to perform on. Effectively, we provide example node pairs with their correct causal ordering before asking the question about the given nodes. Each example answer also contains an explanation of the answer, generated using high-cost expert (GPT-4, in our experiments). Adding the explanation provides the expert with additional reasoning information when deciding the causal order (Wei et al., 2022). To avoid overfitting, we select node pairs from graphs that are not evaluated in our study, as additional input. Node pairs with and without direct edges were equally chosen for this purpose. Examples of an expert's (LLM's in this case) answers (and their explanations) using this query strategy are shown in tables below,
Iterative Context. Here, we provide previously oriented pairs as context in the prompt. Since the expert has access to its previous decisions, we expect that it may avoid creating eycles through its predictions.
One hop iterative Context. Providing previously oriented pairs may become prohibitive for large graphs. Here we provide the information of connections with neighbouring nodes of the node pair being inspected as additional context in the query.
Cost Estimation Analysis: Pairwise vs. Triplet for LLMs
Triplet method ensures scalability hy optimizing most calls to cheaper and smaller model (like GPT-3.5-Turho) while improving performance. The triplet pipeline boosts accuracy through multiple context switches (varying the third node) for better pairwise orientation. Strategic use of GPT-4 for conflict resolution enhances effectiveness and controls costs. For a 100-node graph. pairwise orientation using GPT-4 costs an estimated $574, while our triplet strategy. leveraging both GPT-4 and GPT-3.5 T-rbbo, reduces costs to $55. Although our triplet method involves more calls, it optimally uses GPT-4 for error correction, significantly improving performance while keeping costs low.
Tradeoff Between Increased Nodes: Gains vs. Complexity- -Driven Errors As we increase the number of
nodes in the prompt, there is tradeoff:: Adding more nodes provides more context and thus is beneficial, but more nodes in the L.M's 's prompt also lead to higher error and higher computational cost. Therefore, we tackled this question empirically by comparing pairwise, triplet, and quadruplet- based prompts. As Table A9 shows, using a quadruplet prompt slightly increases accuracy but leads to significant increase in the number of LLM calls. In contrast, the increase in accuracy (especially cycle avoidance) is substantial when moving from pairwise to the triplet method. Given these considerations, we decided to go with the Triplet prompt, as it allows for adding more context with minimal increase in prompt complexity and total number of L.M calls. Note that future iterations of language models might be able to handle longer context hetter with more improvements, therefore the will vary with model size, architecture and data the model is trained on. Since we don't 't have the information about this, it will be difficult to model accurately. However, with the LLMs that we have tried (GPT-4, GPT-3.5, Phi-3 and LLama3), we do not see an increased error when using the triplet prompt compared the pairwise prompt.
G CAUSAL GRAPHS USED IN EXPERIMENTS
Figures A3-A7 show the causal graphs and details we considered from BNI repository (Scutari & Denis, 2014).
27

---

Published as a conference paper at ICLR 2025
Dataset
Graph
Data for Variables
BN Learn Datasets (Asia, Cancer, Earthquake, Survey, Child) Neuropathic Pain
Real-world graphs from scientific studies
Synthetic data generation based on bnlearn library
Real-world graph constructed with consen- sus from medical experts Tu ct al. (2019). In- cludes domain-specifec variables as Right LI Radiculopathy, Topical Dysfunction, DLS
Synthetic data generation based on Tu et al. (2019)
L5-S1. etc. (see Fig. A8)
Alzheimers Dataset
Real-world graph constructed with consen- sus from medical experts Abdulaal et al. (2023). Constructed in 2023, after the train- ing cutoff date of GPT-3.5 and GPT-4 mod-
No data is available
cls used.
Covid-19 Dataset
Real-world graph constructed by experts understand effect of Covid- 19 respiratory (2023). Constructed in 2022, after the training cutoff' date GPT- 3.5 and GPT-4 models used.
No data is available.
system Mascaro
Table A18: Details about the datasets used for evaluation.
Dataset
Number of Nodes
Number of Edges
Description
(used context)
Asia
8
8
Model the possible respiratory problems someone can have who has recently visited Asia and experiencing shortness breath
Cancer
4
Model the relation between various variables responsible for causing Cancer and its possible outcomes
Earthquake
5
5
Model factors influencing the probability of a burglary
Survey
Model hypothetical survey whose aim investigate the usage patterns of different means of transport
Child
20
25
Model congenital heart disease babies
Neuropathic Pain
Diagnosis (subgraph)
22
25
For neuropathic pain diagnosis
Table A19: Overview of datasets used
28

---

Published as a conference paper at ICLR 2025
Figure A3: Earthquake Bayesian network. Abbreviations/Descriiiinss Burglary: burglar entering, Earthquake: earthquake hitting, Alarm: home alarm going off in house, JohnCalls: first neighbor call inform the alarm sound, Marycalls: second neighbor to call inform the alarm sound.
XIWI
Dysprooa
Figure A4: Cancer Bayesian network. Abbreviations/Desrripions Pollution: exposure pollutants, Smoker:
smoking habit, Cancer: Cancer, Dyspnoca: Dyspnoea, Xray: getting positive xray result.
29

---

Published as a conference paper at ICLR 2025
A
O
Figure A5: Survey Bayesian network. Abbreviations: A=Age/Age of people using transport, S=Sex/male female, E=Education/upp to high school university degree, O=Occupatton/epploye self-employed, R=Residence/the size of the city the individual lives recorded as either small or big, T=Travel/the of transport favoured by the individual.
asia
smoka
tub
lung
eilher
Xray
dysp
Figure A6: Asia Bayesian network. Abbrviations/Deccritions asia=visit to Asia/visiting Asian countries
with high exposure pollutants, smoke=smoking habit, tub=tuberculosis, lung=lung cancer, either=either tuberculosis lang cancer, bronc=bronchitis, dysp=dyspnoea, xray=getting positve xray result.
30

---

Published as a conference paper at ICLR 2025
BirhAsphyxil
Disease
Sick
Flow
CardiacMixing
LungParench
LungFlow
LVH
Age
Grunting
HypDistrib
HypoxialnO2
Co2
ChestXray
LVHrepori
GrunfingRoport
LowerBodyO2
RUG02
CC2Raport
XrayReport
Figure A7: Child Bayesian network. Abbreviations: BirthAsphyxia: Lack of oxygen the blood during the infant's birth, HypDistrib: Low oxygen
equally distributed around the body, HypoxialnO2: Hypoxia
when breathing oxygen, CO2: Level of carbon dioxide in body, ChestXray: Having a chest x-ray, Grunting:
Grunting in infants, LVHreport: Report of having left ventricular hypertrophy, LowerBodyy2: Level of oxygen in the lower body. RUQO2: Level of oxygen in the right upper quadricep muscle, CO2Report: A document
reporting high levels of CO2 levels in blood, XrayReport: Report having chest x-ray, Disease: Presence an illness. GeuntingReport: Report of infant grunting, Age: Age of infant at disease presentation, LVH: Thickening of the left ventricle, DuctFlow: Blood flow across the ductus arteriosus, CardiacMixing: Mixing of oxygenated and deoxygenate blood, LangParench: The state of the blood vessels in the lungs, LungFlow: Low blood flow in the lungs, Sick: Presence of an illness
31

---

Published as a conference paper at ICLR 2025
Figure A8: For Neuropathic dataset, consider sub-graph induced by of the nodes, containing the following nodes corresponding edges taken from https: http:::///bseeaabll... the-ccmpl etee causal -graph----- srraph-o--aee of www.nnututtcccccco wwwthhiiippaiiiiaeeeeeeee agnosis: 'right C7', 'right elhow trouble'. 'left shoulder trouble', 'left bend of arm problem'. 'right shoulder trouble', 'right hand problem'. "left medival elbow problems', 'right finger trouble', 'left neck problems', 'left wrist problems' "left shoulder problem'. 'right neck', 'right wrist problem', 'right shoulder problem 'discoligment injury C6 C7', 'left hand problem', 'left C7', 'right arm band', 'left lower arm disorders', 'neck pain', 'left finger trouble', 'left arm'. did not use descriptions for the nodes of Ncuropathic graph.
com/ @turuibo/
32

---

Published as a conference paper at ICLR 2025
age
sex
ventricular volume
APOE4
brain volun
moca
av45
education
tau
slice number
brain MRI
Figure A9: The Alzheimer's dataset is Bayesian Network developed by Abdulaal, Ahmed, et al. in collaboration with five domain experts, as detailed in their paper "Causal Modelling Agents: Causal Graph Discovery through Synergising Metadata-and Data-driven Reasoning" (ICLR 2024). The dataset includes the following variables: age, which represents the age of the patient; sex, indicating the biological sex of the patient; APOE4, which measures the expression level of the APOE4 gene; education, reflecting the patient's educational attainment in years; av45, measuring the beta amyloid protein level using Florbetapir F 18; tau, indicating phosphorylated-tar deposition; brain volume, representing the total brain matter volume the patient; Ventricular Volume, indicating the total ventricular volume of the patient; and moca, which is the Montreal Cognitive Assessment Score.
33

---

Published as a conference paper at ICLR 2025
Virus enters URT (1)
URT epithelial infection (2)
Infection of offactory epithelium (3)
Alveolar epithelial infection (5)
Anosmia and/or ageusia
Alveolar endothelial infection (6)
Systemic immune/infiam. response (8)
Viremia (7)
Pul. capillary leakage (9)
Productive cough (11)
Dry cough (10)
Infection related
Pulmonary related
Signs and symptoms
URT upper respiratory tract pul. pulmonary inflam. inflammatory
Figure A10: Respiratory causal DAG taken from Mascaro S, Wu Y, Woodberry O, et al. Modeling COVID-19 disease processes by remote elicitation of causal Bayesian networks from medical experts. BMC Med Res Methodol. Here, Virus enters upper respiratory tract (URT): SARS-CoVas viral particles inhaled and attach to upper respiratory tract mucosal surface. The size of the viral inoculum dependent on exposure related factors, not included in the current model, Upper respiratory tract (URT) epithelial infection: Viral infection of upper respiratory tract epithelial cells +/- signaling an immune response and leading local inflammation. Infection of olfactory epithelium: Viral infection of the olfactory epithelial cells + leading to impaired olfaction, Ageusia and/or anosmia: Loss of the ability taste and/or smell, Alveolar epithelial infection: Viral infection of the alveolar cells, +/- inducing an immune response which leads to local inflammation. Alveolar endothelial infection: Viral infection of the endothelial cells of the capillaries
terminal airways, +/- inducing immune response which leads to local inflammation Viremi: Presence of SARS- CoV-2 in blood allowing for systemic dissemination of the virus., Systemic immune/nflammatorr (inflam.) response: Activation of innate and/or adaptive immune system by the presence of virus at one or more body site/s. Manifest by the release of pro- +/- anti-inflammatory markers in blood by immune-related cells, Pulmonary capillary leakage: Leakage of plasma from the pulmonary capillaries into the terminal airways., leakage Dry cough: Cough without the production of mucus or phlegm., Productive cough: Cough with the production of mucus phlegm.
34

---

Published as a conference paper at ICLR 2025
Question: For a causal graph used model relationship of various factors and outcomes related cancer with the following nodes: I'Pollution', 'Cancer', 'Smoker', 'Xray', 'Dyspnoea', Which cause-and-effee relationship is more likely between nodes 'smoker" and "cancer??
A. changing the state of node 'smoker' causally affects change in another node 'cancer'.
B. changing the state of node 'cancer' causally affects a change in another node 'smoker'.
C. There is no causal relation between the nodes 'cancer' and 'smoker'.
Make sure to first provide a grounded reasoning for your answer and then provide answer the following format: (Ansser)A/BB/((AAAnnn A/B/C(/Answeer). It is very important that you output the final Causal graph within the tags like (Answer) A/B/C(/Answer) otherwise your answer will not be processed.
The causal effect "mmoker" directing to "cancer" is based on strong evidence from epidemi- ological studies linking smoking to an inereased risk of developing cancer. Smoking introduces harmful substances into the respiratory system, leading to cellular damage and mutation, which significantly raises the likelihood of cancer development in the lungs respiratory tract, subsequently impacting the occurrence of respiratory problems like shortness of breath. Therefore answer is (Answer) A A:/Answer)
Answer:
For a causal graph used to model relationship of various factors and outcomes related to cancer with the following nodes: ['Pollution', , 'Cancer', 'Smoker', "Xray". 'Dyspnoca'] J. Which cause-and and-effect relationship is more likely between nodes 'xray' and 'dyspnoea"?
Question:
A. changing state node 'xra'' causally affects change in another node 'dyspnoea'.
B. changing the state of node 'dyspnoca' causally affects a change in another node 'xray'.
C. There no causal relation between the nodes 'xray" 'dyspnoea'.
Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: (AnSSwrr)A/BCC//AAnsss is very important that you output the final Causal graph within the tags like (Answer) A//CC//AsCCC/CC otherwise your answer will not be processed.
Answer: Reasoning behind the lack of causal relation between X-ray and dyspnoea is that X-ray and dyspnoea are both effects of having cancer, but they do not directly cause affect each other. X-ray is diagnostic test that help detect cancer in the lungs or other organs, while dyspnoea is symptom of cancer that involves feeling short of breath. Therefore, X-ray and dyspnoea are not causally related, but they are both associated with cancer. Therefore answer is AnnswerCC/AAnseee
Question: For a causal graph used to model relationship of various factors and outcomes related to cancer with the following nodes: ['Pollution', 'Cancer', 'Smoker', 'Xray', 'Dyspnoca'), Which cause-and-effect relationship is more likely between nodes 'xra'' and "cancer'?
A. changing the state of node 'xray' causally affects change in another node 'cancer'.
B. changing the state of node 'cancer' causally affects a change another node 'xray".
C. There is no causal relation between the nodes 'xray" and 'cancer'.
Make sure first provide grounded reasoning for your answer and then provide the answer in the following format:
is very important that you output the final Causal graph within the tags like (Answer)A//B///// otherwise your answer will not be processed.
Table A20: Chain of Thought Prompt
35

---

Published as a conference paper at ICLR 2025
The causal effect of cancer on X-ray is that X-rays are often used to diagnose or detect cancer in different parts of the body, such as the bones, lungs, breasts, or kidneys X-rays use low doses of radiation to create pictures of the inside of the body and show the presence, size, and location of tumors, X-rays can also help monitor the response treatment having cancer may increase the likelihood of getting an X-ray as part of the diagnostic process or follow-up care. Therefore answer is (Answer) B (/Answer))
Answer:
check for signs of recurrence. Therefore,
Question: For causal graph used model relationship of various factors and outcomes related to cancer with the following nodes: ['Pollution', 'Cancer', 'Smoker', "Xray', 'Dyspnoea') J. Which cause-and-effect relationship is more likely between nodes ppollution" and 'cancer'?
A. changing the state of node 'pollution' causally affects change in another node 'cancer'.
B. changing the state of node 'cancer' causally affects a change in another node "pollution'.
C. There is no causal relation between the nodes 'pollution' and 'cancer'.
Make sure to first provide a grounded reasoning for your answer and then provide the answer the A/B/C
following format: graph within the tags like (Answer)A/B/C(/Ansss otherwise your answer will not be processed.
It very important that you output the final Causal
Answer: The causal effect of pollution cancer is that air pollution contains carcinogens (cancer- causing substances) that may be absorbed into body when inhaled and damage the DNA cells. Another possible reasoning is that particulate matter (tiny dust-like particles) in air pollution may cause physical damage the cells in the lungs, leading to inflammation and oxidative stress and eventually cell mutations. A third possible reasoning is that air pollution may create an inflamed environment in the lungs that encourages the prolifcration of cells with existing cancer-driving mutations. These are some of the hypotheses that rescarchers have proposed to explain how air pollution may cause cancer, but more studies are needed to confirm them. Therefore answer is Ansswrr/A//Anweee
Question: For causal graph used model relationship of various factors and outcomes related to cancer with the following nodes: ['Pollution', 'Cancer', 'Smoker', 'Xray', 'Dyspnoea"] J. Which cause- audse-an--oee effect -nf-fffeetf relationship is more likely between nodes "pollution" and 'smoker'?
A. changing the state of node 'pollution' causally affects change in another node 'smoker'.
B. changing the state of node 'smoker' causally affects change in another node 'pollution'.
C. There is no causal relation between the nodes 'pollution' and 'smoker'.
Make sure first provide grounded reasoning for your answer and then provide the answer in the following format: AAn/ A/B/C graph within the tags like A/B/CC
It very important that you output the final Causal otherwise your answer will not be processed.
Answer: Reason behind the lack of causal relation between pollution and smoker that pollution and smoking are both independent risk factors for respiratory problems, but they not directly cause or affect each other. Pollution and smoking both contribute to air pollution, which can harm the health of people and the environment. However, pollution is mainly caused by human activities such as burning fossil fuels, deforestation, industrial processes, while smoking is personal choice that involves inhaling tobacco smoke. Therefore, pollution and smoker not causally related, but they are both associated with respiratory problems. Therefore answer is
Table A21: Chain of Thought Prompt (continued.)
36

---

Published as a conference paper at ICLR 2025
For causal graph used for modeling factors causing Coronary Heart Discases with the following nodes: [ ('Family Discase", 'Gene', 'Smoking', 'Blood Pressure', "Coronary Heart Discase", 'Headache') J. Which cause-and-effect relationship is more likely between nodes 'Family Discase' and "Gene"?
Question:
A. changing the state of node FFailyy Disease' causally affects a change in another node 'Gene'.
B. changing the state of node "Gene" causally affects change in another node 'Family Disease'.
C. There is no causal relation between the nodes FFamily Disease' and 'Gene'.
Make sure to first provide grounded reasoning for your answer and then provide the answer in the following format: (Answer) A/BCC//Ar///C(/,sswwerr.. It is very important that you output the final Causal graph within the tags like (Answer) A/B/C(/Answerr otherwise your answer will not be processed.
Answer: Reason behind the causal effect of family disease on gene is that family disease is a term that refers to discases health conditions that in the family, meaning that they are influenced genetic factors. Gene is term that refers to the basic unit of heredity that carries information or function. Family disease can affect gene by altering the type or frequency of genes that are inherited by the offspring from their parents. For example, some family diseases are caused by deterministic genes, which are genes that guarantee the development of disease if they present in person's 's genome. Other family discases influenced by risk genes, which genes that increase the likelihood of developing a disease but do not guarantee it. Therefore, family disease can causally direct to gene by changing the genetic makeup of a person or a population. Therefore answer is (Answer) A A/Answerr).
specific trait
Question:
For causal graph used for modeling factors causing Coronary Heart Discases with the following nodes: ['Family Discase", 'Gene', 'Smoking'. 'Blood Pressure', "Coronary Heart Disease", "Headache') 1, Which sause-and-effeee relationship is more likely between nodes 'Coronary Heart Disease' and 'Gene"?
A. changing the state of node 'Coronary Heart Disease' causally affects change in another node 'Gene".
B. changing the state of node 'Gene' causally affects a change in another node CCronary Heart Disease".
C. There is no causal relation between the nodes 'Coronary Heart Disease' and 'Gene'.
Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: (Ansserr//B/CC(/Annweer It is very important that you output the final Causal graph within the tags like (Answer)./BB/C(/Annns otherwise your answer will not be processed.
Answer: Possible reasoning behind the causal effect of gene on coronary heart discase is that gene is term that refers to the basic unit of heredity that carries information for specific trait or function. Gene can affect coronary heart disease by influencing the structure and function of the blood vessels, the metabolism and transport of lipids (fats) in the blood, the inflammation and clotting processes, the response to environmental factors such as smoking or diet. For example, some genes code for proteins that regulate the cell cycle and growth of the cells that line the arteries, which can affect their susceptibility to damage plaque formation. Other genes code for proteins that control the synthesis and clearance of cholesterol other lipids, which can affect their levels and deposition in the arteries. Therefore, gene can causally direct to coronary heart disease by modifying the biological pathways that contribute to the development or progression the disease. Therefore answer is (Answer) B B/aneerrr)/AAsaa (/Anwer)
Table A22: Chain of Thought Prompt (continued.)
37

---

Published as a conference paper at ICLR 2025
For causal graph used modeling factors causing Coronary Heart Diseases with the following nodes: [ ('Family Disease', 'Gene', 'Smoking'. 'Blood Pressure', 'Coronary Heart Disease', "Headache'J, J. Which cause-and and-effect relationship is more likely between nodes "Blood Pressure' and "Smoking"?
Question:
A. changing the state of node 'Blood Pressure' causally affects a change another node 'Smoking'.
B. changing the state of node 'Smoking' causally affects change in another node 'Blood Pressure'.
C. There no causal relation between the nodes 'Blood Pressure" and 'Smoking'.
Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: (Answer) ABBCC////B/C(//nsswrrr is very important that you output the final Causal graph within the tags like (Answer) A/B/C(/Answerr) otherwise your answer will not be processed.
Answer: Possible reasoning behind the causal effect of smoking on blood pressure is that smoking is habit that involves inhaling tobacco smoke, which contains nicotine and other harmful chemicals. Smoking affect blood pressure by activating the sympathetic nervous system (SNS), which is the part of the nervous system that controls the body's response stress danger. When the SNS activated. it releases hormones such as adrenaline and noradrenaline, which cause the heart to beat faster and harder, and the blood vessels to constrict. This results in a temporary increase in blood pressure, which can last for 15 to 20 minutes after each cigarette. Therefore, smoking causally direct to blood pressure by stimulating the SNS and increasing the cardiac output and vascular resistance. Therefore answer is B
Question: For causal graph used for modeling factors causing Coronary Heart Diseases with the following nodes: [ ('Family Discase', 'Gene', 'Smoking'. 'Blood Pressure', 'Coronary Heart Discase", 'Headache"), Which causo-and-effect relationship more likely between nodes 'Headache' and 'Smok- ing'?
A. changing the state of node 'Headache' causally affects change in another node 'Smoking'.
B. changing state of node 'Smoking' causally affects a change in another node 'Headache'.
C. There is no causal relation between the nodes 'Headache" and 'Smoking".
Make sure first provide grounded reasoning for your answer and then provide the answer in the following format: within the tags like (Answer) A/B/C(/Answer) otherwise your answer will not be processed.
is very important that you output the final Causal graph
One possible reasoning behind the lack of causal relation between headache and smoking is that headache and smoking are both associated with various health conditions, but they not directly cause or affect each o2her12. Headache is a term that refers to pain or discomfort in the head, scalp, or neck, which can have many possible causes, such as stress, dehydration, infection, injury, or medication. Smoking is habit that involves inhaling tobacco smoke, which contains nicotine and other harmful chemicals, which increase the risk of diseases such cancer, heart disease, stroke, and lung disease. Therefore, headache and smoking are not causally related, hut they are both linked to different health problems. Therefore the answer is
Answer:
Table A23: Chain of Thought Prompt (continued.))
38

---

Published as a conference paper at ICLR 2025
For causal graph used modeling factors causing Coronary Heart Diseases with the following nodes: ('Family Disease', 'Gene', 'Smoking', 'Blood Pressure', "Coronary Heart Disease', "Hcadache"l I. Which cause-and-effoct relationship is more likely between nodes 'Hcadache' and "Smok- ing'?
Question:
A. changing the state node 'Headache' causally affects a change in another node 'Smoking'.
B. changing the state of node 'Smoking' causally affects change in another node "Headache'.
C. There no causal relation between the nodes 'Headache" and 'Smoking".
Make sure to first provide a grounded reasoning for your answer and then provide the answer in the following format: (Answer) A/BCC((A///B( It is very important that you output the final Causal graph within the tags like (AuSWAr)//BBCC(/AAsver otherwise your answer will not be processed.
Answer: One possible reasoning behind the lack of causal relation between headache and smoking is that hcadache and smoking are both associated with various health conditions, but they do not directly cause affect each other. Hcadache is term that refers pain discomfort in the head, scalp, or neck, which can have many possible causes, such stress, dehydration, infection, injury, or medication. Smoking a habit that involves inhaling tobacco smoke, which contains nicotine and other harmful chemicals, which can increase the risk of diseases such as cancer, heart disease, stroke, and lung disease. Therefore, headache and smoking are not causally related, but they are both linked to different health problems. Therefore the answer
C(/Anwee)
For causal graph used for modeling factors causing Coronary Heart Diseases with the following nodes: [ ['Family Disease", 'Gene', 'Smoking'. 'Blood Pressure', 'Coronary Heart Disease', 'Headache'] 1, Which cause-and-effect relationship is more likely between nodes CCornnary Heart Disease' and "Smoking"?
Question:
A. changing the state of node 'Smoking" causally affects a change in another node 'Coronary Heart Disease'.
B. changing the state of node CCoronary Heart Discase" causally affects a change in another node 'Smoking".
C. There is no causal relation between the nodes 'Coronary Heart Disease' and 'Smoking'.
Make sure to first provide a grounded reasoning for your answer and then provide answer the following format: within the tags like (Answer) A/B/C(/Answrrr) otherwise your answer will not be processed.
is very important that you output the final Causal graph
Answer: Possible reasoning behind the causal effect of smoking on coronary heart discase is smoking damages the heart and blood vessels by raising triglycerides, lowering HDL, increasing blood clotting, and impairing blood flow to the heart. This can lead to plaque buildup, heart attacks, and death. Therefore answer is (Answer) (/Answer).
Question: For causal graph used context with the following nodes: nodes, Which cause-and-effect relationship is more likely between nodes X and Y?
A. changing the state of node X causally affects change in another node Y.
B. changing state of node causally affects change in another node X.
C. There is no causal relation between the nodes X and Y.
Make sure to first provide a grounded reasoning for your answer and then provide answer the following format: (Answer) A/BCC//A///((/ is very important that you output the final Causal graph within the tags like (Answer) A/B/C//Answer) otherwise your answer will not be processed.
Table A24: Chain of Thought Queries (continued.))
39

---

Published as a conference paper at ICLR 2025
Which sause-and-effed relationship is more likely?
A. changing the state of node which says X causally affects change in another node which says Y.
B. changing the state of node which says Y causally affects change in another node which says X.
C. There is causal relationship between node and
Make sure first output a factually grounded reasoning for your answer. X and Y nodes of a Causal Graph. The causal graph is sparse and acyclic in nature. So option C could be chosen if there is some uncertainity about causal relationship between X and Y.
First give your reasoning and after that please make sure to provide your final answer within the tags A/BCeer It very important that you output your final answer between the tags like (Answer) otherwise your response will not be processed.
Table A25: Base Queries
For the nodes X and Y which form edge in Causal Graph, you have to identify which cause-and-effect relationship is more likely between the nodes of the edge. This will be used to rearrange the nodes in the edge to create directed edge which accounts for causal relation from one node to another the edge.
A. changing the state of node causally affects a change another node
B. changing the state of node Y causally affects a change in another node X.
C. There is no causal relation between the nodes X and Y.
You can also take the edges from the skeleton which have been rearranged to create directed edge to account for causal relationship between the nodes: directed_ _ddges. Make sure to first output factually grounded reasoning for your answer. First give your reasoning and after that please make sure to provide your final answer within the tags (Answer) A/AnSwr/.... It very important that you output your final answer between the tags like (Answer) ABB/C//A/BB/C//Answer otherwise your response will not be processed.
Table A26: Iterative orientation Queries
40

---

Published as a conference paper at ICLR 2025
For the following undirected edge in a Causal Graph made of nodes X and Y, you have to identify which cause-and-effect relationship is more likely between the nodes of the edge. This will be used to rearrange the nodes in the edge to create directed edge which accounts for causal relation from one node to another the edge.
A. changing the state of node X causally affects a change in another node Y.
B. changing the state of node Y causally affects change in another node X.
C. There is causal relation between the nodes X and Y.
You can also take the other directed edges of nodes X: X X_edges and Y: Y_edges of the Causal graph as context to redirect the edge to account for causal effect.
Make sure to first output factually grounded reasoning for your answer. First give your reasoning and after that please make sure to provide your final answer within the tags (Answer) (Answer)A/B/C//Aeee (/AnSw/.....r).. It is very important that you output your final answer between the tags like (AnSswe)//BB/C//AAwwwr otherwise your response will not be processed.
Table A27: Iterative One Hop Queries
Identify causal relationships between the given variables and create directed acyelic graph to [context). Make sure to give reasoning for your answer and then output the directed graph in the form of a list of tuples. where each tuple is directed edge. The desired output should be in the following form: [( [('A', 'B'), ('B', 'C')] where first tuple represents directed edge from Node to Node BB, second tuple represents directed edge from Node 'B' to Node 'C'and on.
If a node should not form any causal relationship with other nodes, then you can add it as an isolated node of the graph by adding seperotely. For example, if "C' should be an isolated node graph with nodes 'A', 'B,, 'C', then the final DAG representation should be like [('A', 'B'). (CC").
Use the description about the node provided with the nodes in brackets to form a better decision about the causal direction orientation between the nodes.
is very important that you output the final Causal graph within the tags <An- swer> </Answwrrottheeevisee your answer will not be processed.
Example:
Imput: Nodes: ['A', 'B', 'C', DD');
Description of Nodes: [ídescription of Node A), (description of Node B), (description of Node
C), (description of Node D)J
Output: <Answer>!('s A', ('C),"C" (')''/////Asswe>>>>
Question:
Imput: Nodes: (Triplet Nodes Input]
Description of Nodes: (Description of Each Node from the Triplet)
Output:
Table A28: The triplet query template, which includes concise context of the graph, the input triplet nodes and their respective heart disease in babies". the three nodes may be ( ('HypoxialnO2', 'Grunting'. 'GruntingRepor" )) and their node descriptions are ("hypoxia when breathing oxygen", "grunting in infants", ", "report of infant grunting" ] respectively.
As an example, for the Child graph, the context "to model congenital
41

---

Published as a conference paper at ICLR 2025
Input: ("HypDistrib', 'LowerBodyO2")
Answer: Low oxygen equally distributed around the body affect the level of oxygen in lower body by reducing the amount oxygen available for circulation. Therefore, the answer is
Input: ('Disease', 'LungFlow")
Answer: Infant methemoglohinemia condition where blood unable to carry enough oxygen, which can Icad to low blood flow in the lungs. Therefore, the answer is (Answer) A A/Answer).
Input: ("CardiacMixing', "HypDistrib')
Answer: Mixing of oxygenated and deoxygenated blood is a common cause of congenital heart disease in babies. This mixing can occur due to structural defects in the heart, such as hole between the heart chambers abnormal connections between the blood vessels. This mixing can lead to low oxygen areas equally distributed around the body. Therefore, the answer (Answer) (/Answer).
Table A29: Example LLM (GPT-3.5-turbbe outputs for estimating the causal direction between given pair of nodes (with their descriptions) using the CoT Prompt (refer Table A20 in Appendix for the prompt). Given pair (X,Y) Option A refers X causing option B refers Y causing X, and option C refers causal relationship between variables. The pairs shown are from Child graph from BNLeamn repository (Scutari Denis, 2014), that is used for modeling Congenital Heart Disease in infants.
Input: ('Right C7', DDscolligmtt injury C6-C7")
Answer: Discoligment injury C6-C7 can cause compression of the nerve roots that exit the spinal cord at the C7 level, which can lead to symptoms such as pain, numbness, and weakness in the right C7 dermatome. Therefore, the answer is (Answer)B//Answer)..
Input: ('Right C7', 'Left C7')
Answer: Right C7 and left C7 are both parts of the cervical spine and not known directly influence each other. Therefore, the answer (.nswerCC/AAnswes
Input: ('Right elbow trouble', 'Left shoulder trouble")
Answer: There is no direct causal relationship between right elbow trouble and left shoulder trouble. They may both be symptoms of larger underlying condition, but they do not directly cause or affect each other. Therefore the answer is (Ansssr)C(/Annserr,
Table A30: Example LLM (GPT-3.5-uul (ttt.-)-turb) reasoning outputs for estimating causal directionality between different pairs of nodes using CoT queries (refer Table A20 for the query) for Neuropathic subgraph (used for pain diagnosis).
42


Reviews:
[{"review_id": "9yqwr13niE", "reviewer": "Reviewer_WUxn", "summary": "This paper considers optimal ways of querying imperfect experts (e.g., LLMs) when the aim is to discover a causal DAG over a set of variables. The key proposed idea is to query imperfect experts about the causal ordering over variables instead of graphical relationships like edges. The paper shows that certain ways of orienting edges based on queries to experts can lead to errors in the overall DAG, while the causal ordering from a perfect expert will never be incorrect. The paper proposes ways of integrating causal ordering into known causal discovery algorithms, and then focuses on extracting orderings from imperfect experts. To minimize errors by imperfect experts, the paper proposes querying for the order over triplets of variables while enforcing acyclicity, proving that this makes fewer errors than querying about pairs of variables (under some assumptions). The work then studies this querying strategy empirically using a variety of hand-curated known causal DAGs. They study both human and LLM imperfect experts, and find evidence that the triplet querying strategy improves causal discovery and effect estimation performance over both the pairwise querying strategy and using no experts at all.", "questions": "+ Why do you think the premise of LLMs as $\\epsilon$-imperfect experts makes sense more generally in causal discovery, given that $\\epsilon$ *should* large if we're solving a truly interesting causal discovery problem?\n\n+ Can you the details about the empirical studies that I mention above?\n\n\nEdit: The authors satisfactorily addressed my questions and concerns in their response; I'm happy to thus raise my score and support their work.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The technical as well as written clarity of the paper is excellent. Technically, the notation is clear, all preliminaries and results are well-defined and well-explained. The paper is also written in a way that is easy to read and follow.\n\n+ The technical result about the optimality of the triplet querying strategy over the pairwise querying strategy is novel, to the best of my knowledge, and could have an impact on causal discovery algorithms more broadly.\n\n+ The empirical studies are comprehensive, and the initiative to contrast both humans and LLM imperfect experts is impressive since studies with humans are time-consuming and costly.", "weaknesses": "+ In the broader context of causal discovery, I question the idea that LLMs are $\\epsilon$-imperfect experts. Existing results including those in this work consistently consider causal discovery problems that involve standard medical knowledge, or general knowledge about the world that's fairly well-established by now. However, causal discovery as a field ought to be about discovering new knowledge, in domains where we have very only faint hypotheses about variables relate, e.g., consider abstract variables that represent aspects of human behavior like \"trust\" or \"aggression.\" In these settings, we should expect $\\epsilon$ to be so large as to render LLM experts unreliable. I'm curious to see the authors better justify and contextualize the role of LLMs in genuinely challenging causal discovery problems.\n\n+ The premise in this paper is that querying experts with questions of the form \"does A cause B\" and directing the edge A->B if the expert says yes leads to incorrect graphs. However, the query \"does A cause B\" isn't by itself inherently flawed -- it's already a question about causal ordering -- the policy to orient edges based on ancestral causal relations is what's flawed. Nevertheless, the introduction and parts of section 3 (e.g., definition 3.1) seem to critique the query itself, when really, the issue is that responses to this query should not be used to directly orient edges (e.g., the perfect expert in 3.1 is defined based on a flawed policy). Therefore, the argumentation strategy in the paper feels like setting up a strawman argument. Note, though, that this is just a critique about the storytelling; the technical results are valid and make sense regardless. One could pivot the story slightly to argue that the flaw isn't with the \"does A cause B\" style of query, but in ensuring that this is interpreted only as a causal ordering.\n\n+ There is a bit of extra context that I would have liked in the empirical studies: 1) can you contextualize the datasets and graphs that are studied further and discuss whether they are semi-synthetic, purely realistic, etc.? 2) Despite the results in Figure 3 on synthetic data, it would be useful in the broader empirical studies to compare the causal ordering queries to the flawed policy of orienting edges based no responses to \"does A cause B\"-style questions. (I apologize if this comparison is indeed there and I missed it -- if that ended up being the case, please point me to the correct figure or table in the empirical studies.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T6zPH5pdLL", "reviewer": "Reviewer_5DRM", "summary": "The authors aimed to find a better interface to use domain knowledge for causal discovery, including LLM, but not restricted. For LLM, this is done via a pairwise prompting strategy. The authors found that expert knowledge often falls short in distinguishing direct and indirect causal relations between pairwise variables, correspondingly many cycles in causal graphs, but their causal order is relatively precise. Given this finding, the authors suggest using expert knowledge of causal order instead of causal relation. While integrating the strategy of using causal order and LLM prompting, the authors propose to introduce an auxiliary variable to improve the conventional pairwise prompting strategy by avoiding cycles within the triplet. Experimental results support that the proposed prompting methods increase the robustness and performance.", "questions": "- While domain expert knowledge is independent of the number of observations, it significantly affects the causal discovery algorithm. Therefore, it raises curiosity about how much performance improvement the proposed method can achieve in situations of data scarcity. Could you conduct an experiment with a very small dataset as a showcase for a data scarcity regime? It would be very interesting.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-written. Motivation, method, and experiments are logically straightforward.\n- The attempts to utilize LLM for causal discovery have identified certain problems, such as the difficulty in distinguishing between direct and indirect causal relationships, which consequently lead to cycles in the causal graph. The authors convincingly argue that these issues are inherent to the pairwise prompting approach.\n- The authors propose using causal order to leverage domain expert knowledge in causal discovery without encountering the issues associated with pairwise prompting. Additionally, it presents a novel prompting strategy to support this approach.\n- When the proposed method was utilized, performance improvement was consistently observed across experiments. Additionally, the experimental design accommodates node size scaling from small to large datasets, allowing for a comprehensive understanding of how the method is affected by node size.", "weaknesses": "Overall, I found this work very interesting, though a few questions remain unanswered. Addressing these issues could potentially lead to a higher score.\n\n- It is hard to determine whether triplet prompting is effective for high-performance LLMs (such as GPT-4, Claude3-opus). It appears that the prompting strategy and LLM can be applied orthogonally, so it is weird the corresponding results are omitted. Could you add results for Triplet prompting + high-performance LLMs in Table 2, to clarify this matter?\n- In the experimental design related to the downstream causal discovery algorithm, it is difficult to determine how the proposed method's advantages change with varying observations. Could you add corresponding experiments with varying sample sizes to clarify this matter?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5A8BPul2iW", "reviewer": "Reviewer_HLpf", "summary": "LLMs are often used as an expert for finding causal graphs, but they can’t distinguish between direct and indirect edges. Authors proposed to use causal order as a more stable version of it, along with a triplet prompt approach. \n- Their proposed approach facilitates the recovery of causal order despite imperfect experts. \n- Prompt strategy that introduces auxiliary variables that introduces an auxiliary variable for every variable pair and instructs the LLM to avoid cycles within this triplet. \n- Triplet prompt leads to fewer cycles than pairwise\t\n- Causal order is a simpler structure that can still encode helpful information for down-stream tasks;", "questions": "Q1: Can you clarify the experimental setup in Table 4? The title shows ‘ours - < causal discovery > + LMM’. But in line 279 it is described that “causal order is used to reduce search space of causal discovery methods”. In that sense, should it be LLM + < causal discovery > instead? As the LLM with triplet is used to reduce the search space inside the causal discovery method. \n\nQ2: Can you describe in more details the causal effect component? How is the causal order used in that context? Is it used to drop variables (aka, no informative features or down-stream features)? How are the counterfactuals estimated? \n\nQ3: Another ablation study that would be helpful is to see the robustness to the number of variables. Most causal discovery methods struggle to scale for larger graphs / number of features. How LLMs + triplet could be adopted to reduce search space and improve this current causal discovery bottleneck?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Well-written motivation and discussion of limitations; \n\nGood discussion on the utility of the causal order, and related works. \n\nSound experiments with empirical evidence of the effectiveness of the proposed method.", "weaknesses": "Despite a sound experimental analysis, there a few details missing, i.e., how many repetitions were done in Tables 1 and 2, \n\nOne potential area not explored/mentioned is what happens with a large number of features. As the proposed method decreases the feature space, would the proposed method also facilitate the adoption of causal discovery areas that contains larger graphs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dFZ2Q0CAuV", "reviewer": "Reviewer_W4GZ", "summary": "This paper proposes a new technique for causal discovery algorithms by leveraging triplets. The authors argue that traditional causal discovery algorithms based on pairwise variable relationships cannot handle situations involving mediating variables and can lead to circular structures in causal graphs. Thus, they emphasize the importance of utilizing causal order, meaning modeling based on the topological order among variables. Causal order is effective for downstream tasks, and the triplet approach can efficiently identify causal order. The authors discuss the error of estimated causal order from both empirical and theoretical perspectives, especially in the presence of imperfect experts.", "questions": "1. The current method produces a causal order rather than a full causal graph. While the authors suggest that, in the absence of confounders, this causal order can offer a superset of backdoor adjustment sets, the presence of “imperfect experts” in real-world applications may lead to additional error accumulation in the causal order. This raises uncertainty about whether the derived causal order is sufficiently reliable for downstream tasks, such as causal effect estimation, and how significantly these biases could impact the results. Experimental validation would be valuable to assess the practicality and risks of this trade-off.\n2. In the triplet method, the authors employ a four-step process to determine causal order. Unlike the pairwise approach, each iteration uses a third auxiliary variable \\(C\\) to determine the causal edge direction between \\(A\\) and \\(B\\), followed by aggregation across all variables. My understanding is that the pairwise approach is simply a special case of this method where the number of auxiliary variables equals zero. The authors could further clarify the distinctions and differences between the pairwise approach under this four-step method and the current approach.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The approach introduces the use of causal order as the output for causal discovery algorithms employed by LLMs, with D_{top} topological divergence as a metric. This novel approach brings a fresh perspective and insight into the fields of LLMs and causal discovery.", "weaknesses": "1. Typo error: Lines 333-334 seem to have missed a closing parenthesis in O(V^3).\n2. Complexity considerations: According to the paper, the triplet-based method has a complexity of O(V^3), whereas the traditional pairwise method has a complexity of O(V^2). For larger-scale graphs, the time complexity of these methods poses significant limitations for causal discovery.\n3. The approach of using LLMs to simply determine causal relationships between variables is challenging to implement effectively in real-world applications. It requires the large model itself to possess a high level of expert knowledge, which is often difficult to achieve, as real causal inference typically demands deep domain-specific expertise and nuanced understanding beyond general-purpose language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "az9kCxW53K", "reviewer": "Reviewer_JgEV", "summary": "This paper explores the use of large language models (LLMs) for causal inference tasks, particularly causal discovery. The authors present two key arguments: (1) For both LLMs (termed as imperfect experts) or perfect experts, it is more effective to identify the topological order of causal variables rather than directly attempting to discover the full causal graph. They demonstrate that this approach yields more robust results and remains valuable for downstream tasks such as causal discovery or treatment effect estimation. (2) Since LLMs are imperfect experts, relying on existing pairwise prompts to determine causal order may still result in cycles. To address this, the authors propose a triplet-based prompt design. They empirically validate their claims on several datasets.\n\n**Claim**: I have research experience in causality. However, I have limited experience in LLMs or LLMs for causality. Hence, I might miss something especially when it comes to evaluation of novelty or comparison with existing works.", "questions": "1. Experiment - Missing reference to Table 2 in the main paper.\n2. Line 114 - Could the authors provide some justification on why the redundancy leads to a more reliable order?\n3. Line 52 - Would using triplet prompt lead to a more efficient method in this sense?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Regarding the first key contribution, I find the proposal of identifying causal orders rather than full causal graphs to be both useful and insightful.\n2. Overall, the paper is easy to follow. While there are some typos, the writing remains generally clear.", "weaknesses": "I will present my main concerns here. Minor concerns or questions can be found in the Question section.\n\n**Significance of contribution**\n\nI have several concerns regarding the methodology section:\n1. 3.1 - While I agree with the intuition that identifying causal order is easier and leads to fewer errors, I’m not sure the result in this subsection is particularly interesting by itself. It seems almost obvious, as knowing the correct causal order is a prerequisite for identifying the causal graph. What matters more is whether the reduced chance of errors still preserves the usefulness of the method.\n2. 3.2 - This section should have provided evidence addressing the question above. However, I find it difficult to assess the significance of the method presented. Proposition 3.2 offers a sufficient condition for backdoor adjustment, but applying this condition broadly to include everything that satisfies it seems impractical.\n3. 4.1 - Should these be considered as contributions of the paper?\n4. 4.2 - The authors try to justify why triplet prompt is better than pairwise prompt here. One thing unclear to me is that, as more variables are provided in the context (as the authors also mentioned), there is a higher chance of LLMs making mistakes. In practice, does that mean an $\\epsilon$-expert LLM would become an $\\epsilon’$-expert where $\\epsilon’>\\epsilon$? If so, would this become a trade-off and how should we interpret this tradeoff?\n5. 4.2 - Another question regarding this theorem: It seems that the major problem the authors try to resolve is to avoid cycles. However, Proposition 4.1 focuses on the error of edge prediction given the assumption of acyclicity. Hence, I am not sure if this theorem provides insight on how to resolve the key problem of this paper.\n\n**Experiment**\n\nMy main concern is the lack of comparison with existing LLM-based methods. For example, while it’s valuable to verify that providing causal order helps with downstream causal tasks, do we know if integrating causal order into causal discovery outperforms directly using LLMs to identify the causal graph or providing other types of inputs to a causal discovery algorithm? Could the authors justify their choice not to compare against existing baselines, such as those mentioned in Section 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)