Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
Authors: Haoang Chi
He Li
Wenjing Yang
Feng Liu
Long Lan
Xiaoguang Ren
Tongliang Liu
Bo Han
Venue: NeurIPS 2024 poster (2024)
Keywords: trustworthy machine learning
LLM
causal reasoning
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Causal reasoning capability is critical in advancing large language models (LLMs) towards artificial general intelligence (AGI). While versatile LLMs appear to have demonstrated capabilities in understanding contextual causality and providing responses that obey the laws of causality, it remains unclear whether they perform genuine causal reasoning akin to humans. However, current evidence indicates the contrary. Specifically, LLMs are only capable of performing shallow (level-1) causal reasoning, primarily attributed to the causal knowledge embedded in their parameters, but they lack the capacity for genuine human-like (level-2) causal reasoning. To support this hypothesis, methodologically, we delve into the autoregression mechanism of transformer-based LLMs, revealing that it is not inherently causal. Empirically, we introduce a new causal Q&A benchmark named CausalProbe 2024, whose corpus is fresh and nearly unseen for the studied LLMs. Empirical results show a significant performance drop on CausalProbe 2024 compared to earlier benchmarks, indicating that LLMs primarily engage in level-1 causal reasoning.To bridge the gap towards level-2 causal reasoning, we draw inspiration from the fact that human reasoning is usually facilitated by general knowledge and intended goals. Inspired by this, we propose G$^2$-Reasoner, a LLM causal reasoning method that incorporates general knowledge and goal-oriented prompts into LLMs' causal reasoning processes. Experiments demonstrate that G$^2$-Reasoner significantly enhances LLMs' causal reasoning capability, particularly in fresh and fictitious contexts. This work sheds light on a new path for LLMs to advance towards genuine causal reasoning, going beyond level-1 and making strides towards level-2.

Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
Haoang 2hi He Li Wenjing Yang"! Feng Liu², Long Lan', Xiaoguang Ren', Tongliang Liu', Bo Han
Intelligent Game and Decision Lab, National University of Defense Technology, University Melbourne, University of Sydney, Hong Kong Baptist University (haoangchi (18, ffeegliii mmmammiil....mmmm com, ug__nut@126... com [lihe_111 117, wenj wenjing. yang, long.lan) (annlannnnnntt edu. en, tongliang liu@sydney...... edu. au, bhanml @oomm..omp...m hkbu, edu. hk
Abstract
Causal reasoning capability is critical in advancing large language models (LLMss) toward strong artificial intelligence. While versatile LLMs appear to have demon- strated capabilities in understanding contextual causality and providing responses that obey the laws of causality, it remains unclear whether they perform genuine causal reasoning akin humans. However, current evidence indicates the con- trary. Specifically, LLMs are only capable of performing shallow (level-1) causal reasoning, primarily attributed to the causal knowledge embedded in their parame- ters, but they lack the capacity for genuine human like (level-2) causal reasoning. To support this hypothesis, methodologically. we delve into the autoregression mechanism of transformer-basee LLMs, revealing that it is not inherently causal. Empirically, we introduce a new causal Q& benchmark called CausalProbe-2001 whose corpora are fresh and nearly unseen for the studied LLMs. The LLMs exhibit significant performance drop on CausalProbe-220 benchmarks, indicating the fact that they primarily engage in level-1 causal rea- soning. To bridge the gap towards level-2 causal reasoning, we draw inspiration from the fact that human reasoning is usually facilitated by general knowledge and intended goals. We propose Ge-Reasoner, a method that incorporates general knowledge and goal-oriented prompts into LLMs causal reasoning processes. Experiments demonstrate that G2 G--Reasoner significantly enhances LLMs' causal reasoning capability, particularly in fresh and counterfactual contexts. This work sheds light on new path for LLMs to advance towards genuine causal reasoning. going beyond level-1 and making strides towards level-2.
compared to earlier
Introduction
The emergent of large language models (LLMs), such as GPT 4 [45], Gemini 1.5 [16], and Claude [2], have significantly changed the paradigm of how people work and do research in recent years, demonstrating competitive abilities (LLMs) instruction following [1, 7, 45, 60], in-context learning [8, 12, 67], reasoning [63, 70], coding [10, 18, 51] and etc. LLMs have demonstrated remarkable abilities processing and generating human-like text, leading to a belief that they may possess intelligence akin to human cognition. Reasoning is an essential component of human intelligence and a prominent characteristic that distinguishes humans from other species [47]. Many recent works
"Equal contribution
1 Correspnnding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
(c)
"Id" XX.
CausalProbe 2024
common case recently
"context" "Peterborough mayor Nick Sandford has admitted to using help write speeches, specifically utllzing ChurGPT stratogy and
renote village limited had new road constructed nearby. What the cause of the inprovenent in children's 's academic performance in this village following the construction of the new road?
implementing guidelines for future use..
"question: : "What the reult of Peterborough mayor Nick Sandford using ChatGPT help write his speachee""
A: The Isproved academic performance is likely due to increased access to educational resources, such as libraries or tutoring centers, facilitated by easier road.
public engagement speediwwiting
"choice 'Improved
"hokee 3% implimmntation of an Al strategy by the diy council."
Choice 4:
DDvelopment of ethical guidelincs for Al use in the
transportation
council."
(b)
imagined case
(d)
the result developing railway stations as social hibs and providing a range of public services
alongside ide tickets?
A: Developing railway stations as social hubs and providing a range of public services alongside tickets fosters community engagesent, enhances public transportation accessibilsty, and promotes econonic development.
X
The motivation of this work. (a) LLMS work well on common causal reasoning tasks, whose topics usually are widely discussed. (b) LLMs struggle to tackle rare tasks, whose corpora are possibly brand new for them. (c) Here is an example of the CausalProbe 2024 benchmark that is introduced to examine the true level of causal reasoning in LLMs, including an easy one- choice version (CausalProbe--B EEoobee-), a hard one-choice version (CausalProbe-HF H)) and an uncertain multiple-choice version (CausalProbe-Mi They have analogous formats but different construction strategies. (d) Compared to previous causal Q&A benchmarks, the studied LLMs exhibit a significant performance drop on CausalProbe 2024, (*) represents our benchmarks.
Figure 1:
have dived into evaluating and improving LLMs' general reasoning capabilities, such as logical reasoning [1 20] and mathematical reasoning [22, 57].
In this work, we focus on causal reasoning, an advanced reasoning form [31]. In the context of LLMs, causal reasoning is to discern the cause-and-effect relationships that govern the physical world from a text [43, 54]. Current LLMs appear to have demonstrated some degree of causal reasoning capabilities [73]. Sometimes, when asked about the cause or effect of given text, LLMs can accurately provide responses satisfying the laws of causality that originate from the physical world. For example, in Figure I(a), the LLM can work out reasonable causes (i.e., "increased access to educational resources") for academic performance improvement. Faced with such encouraging performances, we have to pose question:
Does this reflect LLMs' genuine causal reasoning capability or only a "mirage"?
The answer leans more towards the latter. We find that LLMs are adept at qualitatively solving causal reasoning tasks related to common knowledge (e.g.. Figure I (a)), but they struggle to tackle more advanced task types (e.g., Figure (b)), such as discovering new causal knowledge and estimating specific causal quantities. For example, in Figure 1(b), when asked about the effect of an imagined, unusual case ("developing railway stations as social hubs..."), the LLM's answer ("enhance public transportation accessibility"") is clearly irrelevant with such action. Recent works [64, 72, 73] also came to a similar conclusion. Given the performance differences on tasks of varying difficulty, we propose a hypothesis: the apparent (level-1) causal reasoning capabilities of LLMs can be primarily attributed to associated knowledge from their training corpora, rather than engaging in genuine, human-like (level-2) reasoning. We will justify this hypothesis from two aspects.
From a methodological perspective, the widely-used transformer-based LLMs essentially perform next-token prediction [35, 61], which is realized in an autoregressive manner. Autoregressive model [71] originates from time-series analysis with an important assumption: the current value is determined by past values, not related to future values. For texts, however, the fact that the current token depends on the past tokens does not necessarily mean that there is a causal relationship between them. In addition, philosopher David Hume raised a viewpoint that scquential causality is not equivalent to logical causality [21]. Therefore, this mechanism makes LLMS good at reusing the causal knowledge in their training corpora but often makes them struggle to comprehend and generate texts that capture genuine causal knowledge. We also use structural causal models (SCMs) to formally account for this intuitive conclusion (see Section 4.1).
From an empirical perspective, to validate the hypothesis that I.LMs only possess level-1 causal reasoning capabilities, we introduce new causal question & answer (Q&A) benchmark (Figure

---

I(c)), named CausalProbe 2024, whose corpora was made public later than the release of the studied LLMs. 3 Given that the training data is rarely disclosed, we can, at a minimum, guarantee that the corpora of CausalProbe 2024 is not included verbatim in the training data of the studied L.L.s. Compared to earlier causal Q&A benchmarks, such as COPA [48], e-CARE [14] and CausalNet [4], all the studied LLMs (e.g., LLaMA 2 7B, LLaMA 3 8B, GPT 3.5 turbo, Claude 3 Opus) exhibit a significant performance drop on CrusalProbe 2024 (Figure (d)). As the earlier benchmarks are potentially part of the training data, these longitudinal comparisons largely support our hypothesis.
Existing studies about LLMs' causal reasoning mainly focused on only assessments [15, 26, 49] and designing prompt-based based approaches [4, 25]. These studies have taken significant step forward in advancing LLMs' causal reasoning. However, they overlooked two basic principles of human reasoning: general knowledge and intention. For example, when we reason about a mathematical problem, we always take the basic axioms as reference, with the ultimate goal as guidance. Inspired by this fact, we propose causal reasoning framework for LLMs called G²-Reasoner, which incorporates general knowledge and goal-oriented prompts during reasoning. Specifically, we use the retrieval-augmmented generation (RAG) to incorporate external knowledge bases. Then we stimulate the LLMs to consistently discern correct causal relationships in contexts to reach the final responses. Experiments show that Ge-Reasoner significantly enhances LLMs causal reasoning capabilities towards level-2, especially on fresh even fictitious tasks (e.g.. CausalProbe 2024), which is consistent for both open-source and closed-sourc. LLMs.
2 Related Work
In this section, we review the related works, including LLMs' reasoning, LLMs' causal reasoning, and LLMss causal reasoning benchmarks. In Appendix D, we discuss the related works in detail.
2.1 Reasoning in Large Language Models
Reasoning ability is crucial to LLMS'' performance on tasks such as theorem proving, problem- solving, and robotics [41, 58]. [58] propose comprehensive review of the foundation models' reasoning, they discuss reasoning tasks including commonsense reasoning, mathematical reasoning. logical reasoning, and causal reasoning. Commonsense reasoning refers to the reasoning process that utilizes commonsense and daily life experiences [9]. Several commonsense question-answerrin datasets have been proposed to test LLMs commonsense reasoning ability [19, 29, 74, 75J. Causal reasoning is the process of identifying and understanding the cause-and-effect relationships between variables or events, which involves identifying potential causes and effects within system context [58]. One distinctive feature of causal reasoning is that involves counterfactual reasoning, which means reasoning within hypothetical scenario. Our work focuses on the LLMs' causal reasoning, particularly counterfactual reasoning.
2.2 Causal Reasoning in Large Language Models
Causal Reasoning tasks include causal discovery, cause attribution, and causal effect estimation [30, 58, 65]. Causal discovery aims recover the latent causal structure of variables. Cause attribution refers to uncovering potential causes behind process, while causal effect estimation aims investigate the effect of cause variables [26, 76]. While LLMs already show the ability uncover causal relationships from context, their abilities have some limitations [37, 4, 39, 77]. Counterfactual reasoning is an essential task of causal reasoning. The difference between counterfactual reasoning and other causal tasks is counterfactual reasoning involves reasoning in hypothetical scenarios [28, 58]. Several studies conclude that LLMs have limitations when encountering hypothetical scenarios in counterfactual reasoning [33, 66].
"To double ensure the freshness of the corpora in CausalProbe 2024, we use a membership inference attack approach (i.e., determining whether an arbitrary sample is part of a given LLM's training data), Min- K % Prob [53]. to evaluate it and earlier benchmarks. The results shown in Section 6.1.
3

---

Jack learned the new programming language at home because he was not able to go to school, which was closed because of the rain.
rain
cannot go to school
learn...o ..at home
school closure
school closure
learn.. at home
cannot go to school
rain
(a) ground truth causal relationship
(b) sequential causal relationship
Figure 2: An diagram of illustrating how autoregression fails to capture the correct causal knowledge.
2.3 Causal Reasoning Benchmarks in Large Language Models
There have been extensive studies on causal reasoning benchmarks. Existing causal reasoning benchmarks are mainly causal question-answering datasets. [6] employs language rules to extract causal questions from ten large dneestion-answeriig datasets form the CausalQA. CRAB [50] is dataset that aims to assess LLMs' abilities to understand causal relationships among real-world events.
FCR [69] is labeled dataset that includes 24K question -anseernnn pairs. Cladder [57] is dataset that involves symbolic questions and corresponding ground truth answers, [57] employs causal graphs and structural causal models to generate the dataset. CausalProbe 2024 is different from the above benchmarks, its contents are based on the latest and authoritative information, which is unlikely be encompassed by the pre-training corpora of L.Mss.
3 Problem Formalization
In this section, we introduce and clarify the necessary definitions used in this work. First, we provide formal definition for causal reasoning in the context of LLMs. Then, introduce two levels of causal reasoning capability to reveal the limitations of LLMs in this aspect. Last, we use causal language depict the causal reasoning of LLMs.
In this work, the scope of causal reasoning of LLMs we study is reasoning about causal knowledge in textual form, distinguish from the numerical form in statistical causal inference [23].
Definition (causal reasoning in LLMs). In the context of large language models, the causal reasoning consists of two aspects:
comprehend the given contexts and discern the causal relationship within them;
responds to the causalityrrlatted queries, obeying the contexts and objective laws of causality.
To reach the major conclusions of this work, we first categorize I.I.M'' causal reasoning capability into two levels, motivated by the results of cognition science [56] and causality science [17]-
Definition 2 (level-1 causal reasoning). Level-1 causal reasoning involves retrieving causal knowl- edge embedded in model parameters and contextual information. This form of reasoning typically fast and well-suited for handling simple cause-and and-effect relationships.
Definition (level-2 causal reasoning). Level-2 causal reasoning leverages sophisticated reasoning mechanisms and internal parametric knowledge and contexts to deduce causal knowledge, including new/unseen causal knowledge. This form of reasoning is typically slow and capable of deriving new causal knowledge.
The above two definitions are inspired by 'Thinking, Fast and Slow' [27]. Level-2 causal reasoning is not necessarily always better than level-1. Ideally, the interplay and adaptive switching and of these two levels of causal reasoning are crucial for LLMs to work both rapidly and reliably. In this work, we aim to explore the causal reasoning capabilities current LLMs terms of these two levels and dig into the underlying reasons.
Remark 1. In this work, we only consider simple rype of causal reasoning tasks that contain single cause-effect pair. The cases of multiple cause-effect pairs and mediators are excluded. In addition, we primarily consider qualitative causal reasoning tasks (e.g., causal discovery [4]), rather than quantitative ones (e.g., trealment effect estimation (25]). In summary, the causal reasoning lasks that we focus on can be categorized into two type: 1) what is the reason ...; 2) what the result Due to the complexity of natural language, there many different sentence patterns express it.

---

goal-oriented prompt
You are in intelligent caucal reaccner. To arrive at the correct ansaer, teep caretally aratyzing the seatlaele infermation coutal reforence usefal.
logically selated general krowledge
General trouledge: (retrieved information]
Contest (it availasie): (background ot the question]
Question:
Pick one choace. Arawant [cholce 1D]
Cholces: aou xox xay
RAG
world knowledge
effect
natural language
(7
general knowledge
embedding
model
causel question
"questiinn:: "ahst to the resute the surge in interest in artificial
Intelliipecee (AT) in terns of medta coverage"",
frowlodge
X
"cholce_1" "Increased
"chotce 27: "answrr":
Large language model
retripser
vector
dotabase
embeddinng model
Figure 3: The diagram G2 Reasoner. This framework consists two modules. One module is retrieval-augmented generation (RAG) system to retrieve general knowledge that is related to the causal question. Another is goal-oriented prompt steer LLMs to race toward the ultimate goal of causal reasoning.
Figure 4:
The causal graph that depicts the data generation mechanism of causal reasoning in LLMs.
4 LLMs cannot Perform Genuine Causal Reasoning
In this section, we aim to explore the real causal reasoning capability of current LLMs, in terms of our pre-defined two capability levels: level-1 (Definition 2) and level-2 (Definition 3). We study this problem from both a methodological perspective and an empirical perspective.
4.1 Autoregressive LLMs are not Necessarily Causal
Although autoregressive LLMs have achieved great successes, recently, they began to be doubted by this community [34, 42]. We aim to study why the autoregressive mechanism prevents LLMs from acquiring level-2 causal reasoning capabilities. Essentially, the decoder-only transformer-bast based LLMs that are widely used today to perform next-token prediction, are trained with an autoregressive loss [35]. statistics, autoregression model is based on a fundamental assumption: in a sequence, the current value is determined by past values, not related to future values. However, causal knowledge expressed through natural languages does not necessarily satisfy this assumption. This is because sequential causality is not equivalent to logical causality [21]. For example, due to the variability of natural language, sentence patterns may lead to false sequential causal relationships, and Figure shows toy instance. This sentence consists of four concepts, i.e., "rain", "school closure", "cannot go to school", and "learn the new programming language at home". However, we can easily find that the sequential causal relationship in this sentence is totally incorrect. Thus, autoregressive LLMs suffer from capturing logical causal knowledge in complex texts, restricting their generalization abilities on unseen tasks. Note that due to the vast training corpus of LLMs, they may have encountered different textual expressions with the same semantic meaning, thus still being able to infer correct causal relationships in some complex tasks.
Assume that c = (c1, (21, .., (x) is tokenized context and (Wi, u2,...., w/) are already generated tokens, LLMs can obtain distribution of the next token Wy+1: P(w++1le, W1, W2..... .., Wt), which is usually obtained by a softmax function. Then, LLMS predict the next token from this distribution in sampling method, such as greedy sampling and beam search. Autoregressive training makes LLMs memorize common causal knowledge expressions. Based on the above discussion, there two major issues: 1) If the context is not sequentially causal and unfamiliar for LLMs, they tend to misunderstand the causal knowledge in c; 2) If P(w /c, W1, W2,..., w) is large* but the text represented by (..., W We, w(+1) inconsistent with the laws of causality
the context c, I.MSs tend to respond an incorrect causal reasoning result. In contrast, if the context e and generated (..., -.-..WW111
W 1, We; w;+1) express correct causal knowledge and are familiar for LLMss, L.LMs will perfectly address this task. Thus, the autoregression mechanism makes LLMs' causal reasoning primarily rely on correct causal knowledge in large number of training corpora, i.c., the level-1 causal reasoning. In the following, we will validate this hypothesis from an empirical perspective.
4 Probbbbyy because the token sequence (..., Wt-1, 1, We, w(+1) appears frequently in the training corpora.
5

---

Table 1: The data cut-off time comparison of the studied LLMs and CausalProbe 2024 benchmark.
LLaMA 2 7B chat LLaMA 3 8B instruct
GPT 3.5 turbo
Claude opus
CausalProbe 2024
Sep 2022
Mar 2023
Sep 2021
Aug 2023
Jan 2024
4.2 Empirical Study on Causal Reasoning Capabilities of LLMs
In this section, we study the capability boundary of LLMs' causal reasoning for causal reasoning. Recall the hypothesis that LLMs are only capable of performing level-1 reasoning, primarily attributed to the causal knowledge embedded in their parameters during training, but struggle to perform genuine causal reasoning in complex uncommon cases. Therefore, in order to physically avoid LLMS seeing the same corpora as the training data, we introduce new causal question & answer (Q&A) benchmark, named CausalProbe 2024 (Figure I(c)). The corpora used to construct CausalProbe 2024 are published later than January 1, 2024, which is later than the training data cut-off times of all the studied LLMs (Table 1). Thus, the studied LLMs should not have seen corpora that is the same or very similar to CausalProbe 2024. The construction details of CausalProbe 2024 are presented in Section 6.1. CrusalProbe 2024 consists of three datasets: CausalProbe 2024 Easy (abbr. CausalProbe--i CausalProbe 2024 Hard (abbr,, CausalProbe HH), and CausalProbe 2024 Multiple-choice (abbr, CausalProbe-M),, These three datasets progressively probe whether LLMs can solve novel causal reasoning tasks, whether they can distinguish misleading false causal relationships, and whether they can reason about multiple causal relationships within scenario.
To validate whether LLMs heavily rely on the causal knowledge embedded in their parameters, we employ three earlier causal Q&A benchmarks as a comparison, i.e., COPA [48], e-CARE [14], and CausalNet [4]. The corpora of COPA is released earlier than 2011. 1. An example of their comparison is shown in Figure 9. The corpora of e-CARE (i.e., GenericsKB [5]) is released earlier than 2020. CausalNet also a fresh causal Q&A benchmark and its corpora are generated by ChatGPT, serving as intermediate comparison. The corpora of COPA and c-CARE are likely to exactly be the training data of the studied models. For CausalNet, we have reason to speculate that its corpora may be similar to the training data of ChatGPT,, although a large temperature hyperparameter can bring some creativity. Their detailed introduction in presented in Appendix F. In terms of the format, CausalProbe 2024 is consistent with them, including ID, question, choices, and answer. In addition, Prose 2024 additionally provides contexts as the background knowledge of questions. In Section 6. we discuss the reasonability of providing contexts rather than only providing questions." To doubly guarantee the freshness of CausalProbe 2024, we employ a LLM's membership inference attack method, Min-K % Prob [53]. to detect how much the corpora of a benchmark potentially comes from LLM's training data. The results are shown in Table 3, showing that the corpora of CausalProbe 2024 are fresher for the studied LLMs than other benchmarks.
From Figure (dd), we can observe significant performance drops on CausalProbe-E and CausalProbe- H for all four studied LLMs. Especially for CausalProbe-H, Claude 3 opus, current SOTA LLM, only achieves the average exact match less than 70%. The popular and competitive open-source LLM, LLaMA 2 7B chat, just get it half right. As the corpora of CausalProbe 2024 comes from news, it is close to everyday life and hardly consists of professional concepts and unfamiliar words. Thus, the main cause of performance degradation is the freshness of corpora, indicating the fact that LLMs only are capable of doing level-1 causal reasoning, instead of genuine level-2 causal reasoning. The empirical result is in perfect agreement with the analysis derived from the autoregressive mechanism.
Ge-Ressoner: A General-Knowledge-Asiiee and Goal-Driven Reasoner
Until now, we have discovered of the limitation of LLMs' causal reasoning capabilities from both methodological and empirical perspectives. To move towards level-2 causal reasoning, we seek inspiration from human beings. Human reasoning processes, including causal reasoning, are driven by specific reasoning tasks and supported by extensive foundational knowledge acquired throughout life, following established reasoning patterns such as deductive and inductive reasoning [44). When
'In COPA, E-ECARE, and CausalNet, their data also contain "context" or "premise". However, they are not real background knowledge, but part of the questions themselves. In CausalProbe 2024, the "question" is equivalent to "context" plus "question" in the other three benchmarks.

---

solving plane geometry proof problems, students apply three basic axioms as criteria while working toward the target proposition. Unlike humans, LLMs perform causal reasoning through next-token predictions based solely on patterns learned during training, without necessary knowledge to guide their reasoning. Drawing inspiration from causal inference, we formalize textual causal reasoning tasks using causal graph, as shown below.
Causal model for causal reasoning in LLMs. Based on Definition I, we use a causal graph (Figure 4) to formally depict the textual causal reasoning task of LLMs. Based on this formalization, we can identify the key factors for LLMS to perform causal reasoning well. Given a cause semantic variable X and effect semantic variable Y, naturally, we have X + Y. The fact that X causing Y is driven by the laws of the physical world or imagined/virtual worlds C. Then, the semantic variables X and Y are transformed into a variable T that represents the natural language, through mapping h, i.e., h(X, e) = T. where E is random exogenous variable. In this formalization, can be viewed as a language system to transform two causal concepts into a paragraph of text. The variable c represents various factors in generating readable text from causal concepts X and Y. such as language type, context, and mode of expression (e.g., active or passive voice). While linguistic variability (e) enriches the diversity and flexibility of natural language, it poses challenges for LLMs' causal reasoning. Consider the causal relationship between "smoking" (X) and "lung cancer" (Y). This relationship can be expressed in various ways, such as: 1) "A history of smoking is a common risk factor for lung cancer," and 2) "Knowing that smoking greatly increases the risk of lung cancer, why take the risk?" Although these sentences convey the same underlying causal relationship, they differ significantly in their linguistic structure and expression (i.e., different e). The natural language expression T encapsulate atee both the fundamental causal relationship between X and Y, as well as the contextual nuances and linguistic style represented by c.
We can easily find that the causal graph (Figure 4) is exactly a causal model with a confounding variable and conditioned collider [46]. T is conditioned because the observed natural language descriptions in textual causal reasoning tasks are inherently deterministic. Following a fundamental principle of causal inference, when the collider T is conditioned (T =To), it creates association between X and Y. Thus, natural language descriptions of causal reasoning tasks provide valuable information for determining causal relationships." 7 For cause-to-effec tasks, given Py (the distribution of possible effects Y generated by LLMs), our objective is as follows:
arg max P[Y|X- = Xo. T = Too C] Y-Py
where Xo is the cause described in causal reasoning question. However, to account for the confounding variable C, we can apply the total probability formula, i.e.,
arg max PP
BC-PoPP = Xo, T= To: C] = arg max Y-Py
P[Y|X = Xo. T = =To):
(2)
where Pe is the general knowledge base. While strictly ensuring the validity of Eq. (2) would require a complete general knowledge base, which is impractical, Eq. (2) nonetheless provides an insightful approach to addressing causal reasoning problems in LLMs and is helpful in practice (Section 6.3).
Goal-driven prompt. As discussed in Section 4.1, 1, the autoregressive nature of L.LMs hinders their understanding of correct causal relationships. As generated sequences lengthen, L.MMs tend to lose coherence and deviate from their initial targets [40]. To maintain focused generation, we design a goal-driven prompt that guides LLMs in identifying correct causal relationships during the decoding process, as shown in Figure 3.
Motivated by human reasoning mechanism and causal graph theory, we propose a novel causal reasoning framework (Figure 3), named Ge-Reasone, which involves general knowledge as basis and intended goals as guide. Specifically, G² Ge-Resone leverages small (~~16 Mb) general knowledge Q&A dataset as the knowledge base, enabling the model to draw upon related knowledge
'In this work, we focus on textual causal reasoning tasks, instead the numerical ones like classical causal inferenceediscoveeyy
NNoe that collider 1' is useful in our problem, although common causal inference tasks treat it bias. *General knowledge dataset: https//hugginggiac.co wtt.:giinnnicee.oottttttttsesaee coviattaees/Musuuulllllnen Saaam//eneral-Knan Knowledge. We only use its answers as the retrieval document, because the answers are declarative sentences that are more suitable to construct knowledge base. In addition, the answers have contained the entity information in the questions.
7

---

Table Results of the studied LLMs on causal Q&A benchmarks. The metric exact match (EM). "Vanilla" denotes doing inference directly. "C-E", "C- H" represnt CausalProbe- E and CausalProbe- H. The standard deviations are presented in Appendix G.
Table 3: Results training data detection using Min- K % Prob [53]. We conduct this evaluation on LLaMA 2 7B and LLaMA 3 8B. The metric is the average neg- ative log-likelihood indicates better freshness. "C-E"" "C-H" represnt
dataset. A smaller value
LLaMA 2LLaMA 3.5
CausalProbe-E and CausalProbe-H.
Vanilla 0.752
0.937
0.948
0.991
LLaMA 2 LLaMA 3
CoT
0.812
0.944
0.951
0.991
CОРА
RAG
0.757
0.990
Min-10% Prob
13.27
16.64
Gº Reussnne
0.813
0.948
0.953
0.990
COPA
Min-20% Prob
10.57
12.21
Min-30% Prob
8.97
10.32
Vanilla
0.684
0.778
0.814
0.861
e-CARE
CoT
0.697
0.770
0.802
0.864
Min-10% Prob
13.08
14.48
RAG
G ReRsssoerrr
0.701
0.779
0.821
0.849
e-CARE
Min-20% Prob
11.20
12.98
Min-30% Prob
9.92
10.89
Vanilla
0.673
0.857
0.897
0.933
CausalNet
CoT
0.666
0.767
0.874
0.910
Min-10% Prob
10.84
11.3
RAG
0.650
0.898
0.922
Neusal Min-20% Prob
8.84
9.45
G Reasoner
0.681
0.855
0.898
0.929
Min-30% Prob
7.51
8.00
Vanilla
0.616
0.715
0.732
0.758
CiT
0.636
0.720
0.737
0.753
Min-10% Prob
9.34
9.03
C-B (*)
RAG
0.621
0.741
0.756
C-E
Min-20% Prob
7.27
7.29
G Reasoner
0.642
0.718
0.746
0.758
Min-30% Prob
5.90
5.69
Vanilla
0.565
0.652
0.671
0.692
Min- 10% Prob
9.93
9.70
CrT 0.573
0.644
0.667
0.701
C-H(C)
RAG 0.575
0.678
0.685
C-H
Min-20% Prob
7.86
7.77
G² Reasoner
0,582
0.658
0.693
0.696
Min-30% Prob
6.65
6.49
reference when performing causal reasoning tasks. Specifically, when querying LLMs causal reasoning question, we first retrieve related general knowledge from a vector database constructed with the general knowledge dataset, through the pipeline of retrieval-augme augmmentedd generation (RAG). Using the retrieved general knowledge as a basis, we employ the proposed goal-oriented prompt to steer LLMs to perform causal reasoning in targeted manner, rather than aimlessly generating answers. This is the first step taken to advance LLMs towards level-2 causal reasoning.
Experiments
In this section, we will discuss two points: 1) the construction of the CausalProbe 2024 benchmark and its superiority; 2) the main implementation details 3) the result analysis of G²-Reasoner and baseline methods. The CausalProbe 2024 benchmark and the source codes are presented in this URL: https://gg htw.:ttuu....aaaaaaaeeooo wowm/aaaagg9/////// PCtesaaaaaoeeeee22222
6.1 Construction of CausalProbe 2024 and its superiority
We will introduce the construction procedure of the CausalProhe 2024 benchmark and provide a brief analysis. In general, the procedure contains two stages: 1) collecting the latest web articles to form a corpora; 2) employing an LLM to generate the question-answer pairs from the corpora.
Corpora collection. To ensure the quality of the collected corpora, we choose two well-known media: British Broadcasting Corporation (BBC) and the Guardian." 9 BBC [Link] is a British public service broadcaster founded in 1922, one of the oldest and largest broadcasting companies in the world. The Guardian [Link] is national newspaper in the UK, established in 1821, with its online edition being particularly influential. We downloaded the latest articles from BBC and the Guardian. Specifically, the downloaded articles are from January 1, 2024, to April 29, 2024, later than the releases of the studied LLMs (Figure 1). The articles cover categories such as technology, environment, business, health, world news, culture, and climate, statistics of the downloaded articles are shown Table 10.
NOTICE: According to the licenses from the BBC and The Guardian, CausalProbe 2024 can only be used for non-profit purposes. All rights to the corpora used in CausalProbe 2024, including copyrights, are owned by the BBC and The Guardian.

---

Bechmark construction. We generated two sub-datasets from the same corpora using two different strategies. Since it's expensive to create them manually, we employ GPT 3.5 turbo as an assistant to automatically generate them. The overall pipeline diagram is shown in Figure 8.
CausalProbe-E. We construct this dataset following the format of the CausalQA [6]. In detail, we first query the GPT 3.5 turbo to analyze example data in CausalQA, then offer GPT 3.5 turbo articles and ask it to generate Q&A pairs similar to the example data. Finally, the highest-quality Q&A pairs are selected as the reference, which is adopted in the prompt template of generating Q&A data. The related prompt templates are shown in Figure 14a, 14d.
CausalProbe-H. We construct CausalProbe-HF H with another strategy. Specifically, first, we provide GPT 3.5 turbo an example article and ask to summarize this article; second, we ask it to extract several cause-effect pairs based on the summary; third, we ask it to create some incorrect cause-effect pairs regarding this summary: finally, multiple multi-choice questions are generated according to these correct and incorrect cause-effect pairs. The highest-quality multi-choice Q&A data is selected as the reference, which is adopted in the prompt template of generating Q&A data. The related prompt templates are shown in Figure 14. The made-up fake cause-effect pairs can be used to examine the LLMs' genuine causal reasoning capability when encountering counterfactual disturbance term.
CausalProbe-M. We construct uncertain multiple-choice version, with the number of correct answers for each question ranging from to 4. LLMs are required to distinguish the correctness or incorrectness of each causal proposition, avoiding LLMS relying on random guess to get correct answers. We provided GPT-4o mini 10 the prompt templates of CausalProbe-HI and additional prompt to realize varying number of answers, shown in Figure 14. To ensure each question really has at least one correct answer, we manually checked the correctness of the provided answers.
Due to the broad range of topics in the corpus, it may contain ethically questionable content such as conflicts. Therefore, used LLMs to filter the unethical questions from the preliminary generated data. Moreover, to comply with the context length limit of GPT 3.5 turbo, we removed articles exceeding 15,000 characters. After filtering, our dataset contains 3,461 unique Q&A data. We also provide statistical analysis and more details of CausalProbe 2024 in Appendix I.
Superiority of CausalProbe 2024. 1) Contextual information. As we have simply mentioned, earlier three causal Q&A benchmarks lack the necessary background knowledge for each question. Although these benchmarks greatly promote the development of LLMs' causal reasoning, their formats are a little bit unreasonable. Even for humans, when we perform reading comprehension tests, we also need full context as a reference. Although we find that many of these questions can be answered by us, this heavily relies on our knowledge reserve, which is similar to LLMs. The knowledge reserve contributes their level-1 1 causal reasoning capability. Thus, we involve the contexts into CausalProbe 2024 as briefly as possible, which is more realistic, and they significantly improves the performance (Figure 5 and 6). 2) Hierarchical capability assessment. Three datasets of CausalProbe 2024 hierarchically assess causal reasoning ability levels. CausalProbe-FF assesses the genuine causal reasoning ability on novel problems. Furthermore, CausalProbe-H assesses the ability to identify misleading deceptive causal propositions in new tasks. Last, CausalProbe-M assesses the ability to identify multiple valid causal statements in novel tasks, greatly ensuring that LLMa cannot obtain right answers through random guessing. By evaluating LLMS on these three datasets in sequence, we can determine their actual level of causal reasoning ability.
Guarantee the freshness of CausalProbe 2024, To doubly guarantee the freshness of CausalProbe 2024 than earlier benchmarks, we LLM's training data detection method, Min-K% Prob [53], to evaluate them as mentioned before. The detailed introduction for this method is in Appendix E. As the APIs of GPT and Claude no longer provide the logarithmic likelihood input texts, only make this evaluation on open-source LLMs. The results in Table 3 show that CrusalProbe 2024 is fresher. Here K denotes the tokens in the bottom K percent of log-likelihood.
We employ several measures to control the data quality of CausalProbe 2024. The detailed information is shown in Appendix H.
During the contrustion of CausalProbe-Xe M. OpenAl released the GPT-4o series models, which were more powerful and cheaper than GPT-3.5 turbo, we adopted GPT-4o mini
9

---

Figure 5:
CausalProoe-E w/ and Figure 6: CausalProbe-H w/ and w/o contexts
Figure 7:
Result comparison be-
w/o contexts
tween exact match and partial match
6.2
Implementation Details
All the experiments are conducted on the Ubuntu 20.04 system and NVIDIA RTX A6000 GPUs. For closed-source LLMs, i.e., GPT 3.5 turbo and Claude 3 opus, we call the API provided by their officials. We provide more implementation details in Appendix C.
6.3 Result Analysis
To evaluate Ge-Reasoner, we compare it with three common LLM reasoning methods, i.e., vanilla, ofain-of-thought (CoT) [63], and retrieval- -aggmeened generation (RAG) [32]. The "vanilla" refers to directly perform zero-shot inference. We evaluate four LLMs on CausalProbe 2024 and other three causal Q&A benchmarks, whose detailed introduction is presented in Appendix F. The full results are shown in Table 2. First, the LLMs' performance decreases monotonically as the benchmark corpora become more fresh. A counterintuitive phenomenon is that CoT usually perform a little worse than vanilla on three earlier benchmarks. This may indicate that CoT cannot improve reasoning on simple or common tasks for LLMs. For G²-Reasoner, it mostly can outperform the vanilla method. However, limited by the scale of the general knowledge base and the power of vector databases, G Ge-Reasoner cannot significantly outperform the baselines, which is the direction of our future efforts. In fact, RAG is just the ablated GD ReeReson,,, and it usually cannot reach the vanilla method, indicating the effectiveness of goal-oriented prompt.
We evaluate the four LLMs on CausalProbe-M, where they showed a more significant performance decline compared to their results on CausalProbe-E and -H. Under exact match, which required all correct answers
precisely identified, all models struggled. However, when using partial match, where missing some correct options was acceptable but selecting incorrect options was not, GPT and Claude performed relatively well, achieving accuracy rates of approximately 75% and 85% respectively. While these results demonstrate that current LLMs cannot fully comprehend each causal proposition, revealing limitations in their causal reasoning abilities, there is an encouraging finding: the models rarely make false positive errors when identifying causal relationships.
Note that Reasonee'''s performance relies on general knowledge bases. The reported results were obtained using a very small knowledge base (around 16 MB), yet G²-Reasoner generally achieved non-marginal improvements. If use significantly larger one, such as Wikipedia API, performance can be boosted a lot. However, due to resource constraints, we couldn't 't repeat all experiments with it.
Conclusion and Future Outlook
This work investigates the causal reasoning capabilities of LLMS and argues that current LLMs are limited to level-1 causal reasoning. To verify this hypothesis, we introduce a new causal Q& benchmark, CausalProbe 2024, revealing that LLMs struggle with causal reasoning unseen contexts and primarily rely on the causal knowledge in training data. To fill this gap, we proposes G² Ge-Reason,r, method that incorporates general knowledge and goal-oriented prompts into causal reasoning of LLMs. Experiments demonstrate that G2 Ge-Reasoner can enhance the causal reasoning capability, particularly in fresh and counterfactual contexts. This work provides valuable insights into the current state of LLMs' causal reasoning and offers a promising attempt to move towards level-2 causal reasoning, bringing LLMs closer to reaching genuine causal reasoning capabilities. While this work takes important step forward, it still does not enable LLMs to achieve level-2 causal reasoning. Further research in this field could potentially lead to significant advancements towards stronger artificial intelligence.
10

---

8 Acknowledgements
This work was partially supported by the National Natural Science Foundation of China (Grant No. 62372459, No. 62376282, No. 91948303). We express our sincere gratitude to Dr. Jie Tan for her valuable discussions and support from the National Natural Science Foundation of China (Grant No. 62402499).
References
[1] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag. Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. Palm 2 technical report. arXiv, 2305.10403 2023.
[2] Anthropic. The claude model family: Opus, sonnet, haiku. https: //www. anthropic. hom/nnws/cae claude- 3- famt Ly, 2024.
[3] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In ICLR, 2024.
[4] Swagata Ashwani, Kshiteesh Hegde, Nishith Reddy Mannuru, Mayank Jindal, Dushyant Singh Sengar, Krishna Chaitanya Rao Kathala, Dishant Banga, Vinija Jain, and Aman Chadha. Cause and effect: Can large language models truly understand causality? arXiv preprint arXiv:2402.18133 18139, 2024.
[5] Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. Genericskb: A knowledge base of generic statements. arXiv preprint arXiv:2005.00660 2020.
[6] Alexander Bondarenko, Magdalena Wolska, Stefan Heindorf, Lukas Blübaum, Axel- Cyrille Ngonga Ngomo, Benno Stein, Pavel Braslavski, Matthias Hagen, and Martin Potthast. Causalqa: A benchmark for causal question answering. In Proceedings of the 29th International Conference Computational Linguistics, pages 2208-3308, 2022.
[7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[8] Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya K. Singh, Pierre H. Richemond, James L. McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In NeurlPS, 2022.
[9] Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators. Advances in Neural Information Processing Systems, 36, 2024.
[10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex
11

---

Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McCirew Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377, 2021.
[11] Ishita Dasgupta, Andrew K. Lampinen, Stephanic C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L., McClelland, and Felix Hill. Language models show human-like content effects on reasoning tasks. arXiv preprint arXiv:2207.0700 2, 2023.
[12] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. survey on in-context learning, arXiv preprint XXX::2301.0023:: 2023.
[13] Mathijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08228/ 2024.
[14] Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. e-care: new dataset for exploring explainable causal reasoning. In ACL, 2022.
[15] Jinglong Gao, Xiao Ding, Bing Qin, and Ting Liu. Is ChatGPT a good causal reasoner? comprehensive evaluation. In Findings of EMNLP, 2023.
[16] Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
[17] Alison Gopnik and Laura Schulz. Causal Learning: Psychology, Philosophy, and Computation. Oxford University Press, 04 2007.
[18] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv:2401. 2...22441.449,6 2024.
[19] Willis Guo. Armin Toroghi, and Scott Sanner. Cr-It-kgqa: knowledge graph question answering dataset requiring commonsense reasoning and long-tail knowledge. arXiv preprint arXiv:2403.01395 2024.
[20] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In EMNLP, 2023.
[21] David Hume. A treatise of human nature. Clarendon Press, 1896.
[22] Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. ACL, 2023.
[23] Guido W. Imbens and Donald B. Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, 2015.
[24] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint 20Xx::2122091111 2021.
[25] Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fer- nando Gonzalez, Max Kleiman- Kleinan-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. CLadder: Assessing causal reasoning in language models. In NeuriPS, 2023.
[26] Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona T. Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation? In ICLR, 2024.
[27] D. Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011.
12

---

[28] Daniel Kahneman and Dale T Miller. Norm theory: Comparing reality to its alternatives. Psychological review, 93(2):136, 1986.
[29] Mayank Kejriwal, Henrique Santos, Ke Shen, Alice M Mulvehill, and Deborah L McGuinness. A noise audit of human-labelen benchmarks for machine commonsense reasoning. Scientific Reports, 14(1),8609, 2024.
[30] Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. Transactions Machine Learning Research, 2024.
[31] Deanna Kuhn. The development of causal reasoning. Wiley Interiscciiiy Reviews: Cognitive Science, 3(3):327-335, 2012.
[32] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS, 2020.
[33] Jiaxuan Li, Lang Yu, and Allyson Ettinger. Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios. arXiv preprint arXiv:2305.16577 2023.
[34] Chu-Cheng Lin, Aaron Jaech, Xin Li, Matthew R. Gormley, and Jason Eisner. Limitations of autoregressive models and their alternatives. In NAACL-IILT, 2021.
[35] Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In ICLR, 2018.
[36] Ruinan Liu, Guisheng Yin, and Zechao Liu. Learning walk with logical embedding for knowledge reasoning. Information Sciences, 667:120471, 2024.
[37] Xiaoyu Liu, Paiheng Xu, Junda Wu, Jiaxin Yuan, Yifan Yang, Yuhang Zhou, Fuxiao Liu, Tianrui Guan, Haoliang Wang, Tong Yu, al. Large language models and causal inference in collaboration: A comprehensive survey. arXiv preprini arXiv:2403. 2024.
[38] Stephanie Long, Alexandre Piché, Valentina Zantedeschi, Tibor Schuster, and Alexandre Drouin. Causal discovery with language models as imperfect experts. arXiv preprint arXiv:2307.02390 2023.
[39] Stephanie Long, Tibor Schuster, Alexandre Piché, Université de Montreal, ServiceNow Re- search, et al. Can large language models build causal graphs? arXiv preprint arXiv:2303.05279, 2023.
[40] Taiming Lu, Muhan Gao, Kuai Yu, Adam Byerly, and Daniel Khashabi. Insights into Ilm long-context failures: When transformers know but don't tell. In Findings of EMNLP, 2024.
[41] Christopher D Manning. Human language understanding & reasoning. Daedalus, 151 138, 2022.
[42] R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained solve. arXiv preprint arXiv:2309.11 13608, 2023.
[43] Philipp Mondorf and Barbara Plank. Beyond accuracy: Evaluating the reasoning behavior of large language models survey. arXiv preprint AxXi:22404.08666 2024.
[44] Allen Newell, Herbert Alexander Simon, et al. Human problem solving, volume 104, Prentice- hall Englewood Cliffs, NJ, 1972.
[45] OpenAl. GPT-4 technical report. arXiv, 2303.08774, 2023.
[46] Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009.
[47] David Premack. Human and animal cognition: Continuity and discontinuity, Proceedings of the National Academy of Sciences, 138111):1881-11867 2007.
13

---

[48] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In AAAI, 2011.
[49] Angelika Romanou, Syrielle Montariol, Debjit Paul, Leo Laugier, Karl Aberer, and Antoine Bosselut. CRAB: Assessing the strength of causal relationships between real-world events, In EMNLP, 2023.
[50] Angelika Romanou, Syrielle Montariol, Debjit Paul, Leo Laugier, Karl Aberer, and Antoine Bosselut, Crab: Assessing the strength of causal relationships between real-world events. arXiv preprint arXiv:231 1..42814.444 2023.
[51] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. / /2950, 2024.
[52] Christopher Michael Rytting and David Wingate. Leveraging the inductive bias of large language models for abstract textual reasoning. In NeurIPS, 2021.
[53] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In ICLR, 2024.
[54] Steven Sloman. Causal Models: How People Think about the World and Its Alternatives. Oxford University Press, 08 2005.
[55] Peiyang Song, Kaiyu Yang, and Anima Anandkumar. Towards large language models as copilots for theorem proving lean. arXiv preprint Ttt::2404.12534.. 2024.
[56] Keith E. Stanovich and Richard F. West. Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23(5):645- 665, 2000.
[57] Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with language models.
In ACL, 2023.
[58] Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. A survey of reasoning with foundation models. arXiv preprint arXhv:2312.11562 2023.
[59] Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. Can llms learn from previous mistakes? investigating llms' errors to boost for reasoning. arXiv preprint arXiv:2403.20044 2024.
[60] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2307 307.09288. 2023.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
14

---

[62] Yuxuan Wan, Wenxuan Wang, Yiliu Yang. Youliang Yuan, Jen-tse Huang. Pinjia He, Wenxiang Jiao, and Michael R Lyu. A & b= & a: Triggering logical reasoning failures in large language models. arXiv preprint arXiv:4401.00557, 2024.
[63] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thouggh prompting elicits reasoning in large language models. In NeurIPS, 2022.
[64] Moritz Willig, Matej Zečević, Devendra Singh Dhami, and Kristian Kersting. Can foundation models talk causality? In UAI 2022 Workshop on Causal Representation Learning, 2022.
[65] Anpeng Wu, Kun Kuang, Mingin Zhu, Yingrong Wang, Yujia Zheng, Kairong Han, Baohong Li, Guangyi Chen, Fei Wu, and Kun Zhang. Causality for large language models. arXiv preprint arXiv:2410.13319 2024.
[66] Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks, arXiv preprint arX1v:2307.02477 2023.
[67] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In ICLR, 2022.
[68] Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrieval augmented language models. Advances in Neural Information Processing Systems, 36, 2024.
[69] Linyi Yang, Zhen Wang, Yuxiang Wu, Jie Yang, and Yue Zhang. Towards fine-grained causal reasoning and qa. arXiv preprint arXiv:2204.0744: 2022.
[70] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurlPS, 2023.
[71] G Udny Yule. Why do we sometimes get consense-correlationn between time-series? study in sampling and the nature of time-series. Journal of the royal statistical society, 89(1):1663, 1926.
[72] Matej Zečević, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting, Causal parrots: Large language models may talk causality but are not causal. Transactions on Machine Learning Research, 2023.
[73] Cheng Zhang, Stefan Bauer, Paul Bennett, Jiangfeng Gao, Wenbo Gong, Agrin Hilmkil, Joel Jennings, Chao Ma, Tom Minka, Nick Pawlowski, and James Vaughan. Understanding causality with large language models: Feasibility and opportunities. arXiv preprint arXiv:2304.08 2304.05524, 2023.
[74] Yunxiang Zhang and Xiaojun Wan. Situatedgen: Incorporating geographical and temporal contexts into generative commonsense reasoning. Advances in Neural Information Processing Systems, 36, 2024.
[75] Danna Zheng, Mirella Lapata, and Jeff Z Pan. Archer: A human- text-to-sql dataset with arithmetic, commonsense and hypothetical reasoning. arXiv preprint arXiv:2402.12554 2024.
[76] LYU Zhiheng, Zhijing Jin, Rada Mihalcea, Mrinmaya Sachan, and Bernhard Schölkopf. Can large language models distinguish cause from effect? In UAI 2022 Workshop on Causal Representation Learning, 2022.
[77] Yu Zhou, Xingyu Wu, Beicheng Huang, Jibin Wu, Liang Feng, and Kay Chen Tan. Causalbench: A comprehensive benchmark for causal learning capability of large language models. arXiv preprint arX 2024.
15

---

Supplement to "Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?"
Content
Limitations
16
B Broader Impacts
17
Full Implementation Details
17
D Related Work
17
D.I
Reasoning in Large Language Models
17
D.2 Causal Reasoning in Large Language Models
18
D.3 Causal Reasoning Benchmarks Large Language Models
18
E
LLMs' Training Data Detection: Membership Inference Attack
18
F
A Detailed Introduction of Used Benchmarks
19
F.1 CausalNet
19
F2 COPA
19
F3
c-CARE
19
G Additional Results
19
G.1 Standard Deviations of Main Results
19
H Quality Control
19
Additional Introduction and Analysis of CausalProbe 2024
L1 Statistic of CausalProbe 2024
21
I.2 Addition Analysis
I.3 Potential bias
22
A
Limitations
In this section, we discuss the limitations of our work, which include two parts. The first limitation is that the proposed causal reasoning method for LLMs only step forward, but cannot realize the genuine causal reasoning. On the other hand, enabling LLMs to perform causal reasoning like humans challenging and requires further research. The second limitation is that we can not fully confirm the CausalProbe 2024 is unseen for LLMs. Specifically, the contents of CausalProbe 2024 are still part of human knowledge, and LLMs may have seen comparable information. For example, LLMs may not know one latest event, but their pre-training data may include similar event. Thus, fully excluding the pre-training data from datasets challenging, and we already ensure the CausalProbe 2024 is unseen for LLMs to large extent.
16

---

B Broader Impacts
In this section, we discuss both the potential positive societal impacts and negative societal impacts of our work. The positive societal impacts include that our work facilitates understanding LLMs' current causal reasoning abilities. Besides, our work a step forward for improving LLMs' causal reasoning. The potential negative societal include that our work cannot promote LLMs' causal reasoning abilities to the human level, thus, LLMs' causal reasoning results may still have errors.
C Full Implementation Details
All the experiments are conducted on the Ubuntu 20.04 system and NVIDIA RTX A6000 GPUs. For closed-source LLMs, i.e., GPT 3.5 turbo and Claude 3 opus, we call the API provided by their officials.
Hyper-paree saper-parameters. For LLM inference, we set the temperature parameter 1.0 for closed-source LLMs all the time, and set it as O for open-source LLMs all the time. For CoT reasoning, we set the maximal new tokens as 128, and we set it as 50 for all other cases.
Benchmark. All the used benchmarks are the json or jsonl format. If their original versions are such formats, we convert them jsson jsonl format. For CausalNet, there exist few missing values, and we have manually remove the invalid data. The filtered version of CausalNet will be uploaded in anonymous link.
ReRResoner. Ge-Reasoner uses the RAG system to involve external general knowledge, We use the Faiss package [13] to construct the vector database. We use the Meta's Contriever [24] as the information retriever to perform vector database retrieval. For the construction of RAG system, we refer to the realization of self-RAG [3]. For the external general knowledge base, we use the an- swers of general knowledge dataset (https: / htth/hhggggacceeeeeeeee co/dataset (aatasets/uskkume /a///MMuuiPl Patuuummmlllumm/mmmmm General- wwownnowwwlllloee edge) to form a document. We use Contriever and Faiss to embed and retrieve this general knowledge base.
Training data detection. We follow the official repository of Min-K K Prob to perform detection, without any modification. However, the Text-Da Dixt-DaVinii003 seems be deprecated and current GPT 3.5 and GPT 4 are no longer support returning the input's log likelihood, we have to only perform this detection on open-source LLMs. We will continue explore how perform on OpenAl's alternative.
D Related Work
In this section, we review the related works in detail, including LLMs' reasoning, LLMs' causal reasoning, and LLMs' causal reasoning benchmarks.
D.1 Reasoning in Large Language Models
Reasoning ability is crucial to LLMs' performance on tasks such as theorem proving, problem-solving. and robotics [41, 58). Increasing interest in strong artificial intelligence has triggered debates whether L.LMs master reasoning abilities. [58] propose a comprehensive review of the foundation models' reasoning. they discuss reasoning tasks including commonsense reasoning, mathematical reasoning, logical reasoning, and causal reasoning. Commonsense reasoning refers to the reasoning process that utilizes commonsense and daily life experiences [9]. Rather than specialized knowledge, commonsense reasoning relies more on universally accepted knowledge. Commonsense question answering an important method commonsense reasoning test. Several commonsense question- answering datasets have been proposed to test LLMs' commonsense reasoning ability [19, 29, 74, 75]. Different from commonsense reasoning, mathematical reasoning requires the ability to master symbolic forms and formal definitions. Automated theorem proving is a commonly used task to test L.MMs mathematical reasoning ability [58]. [55] propose a framework for L.MMs' theorem proving, which helps humans to prove theorems. LeanDojo [68] is mathematical reasoning benchmark
17

---

consisting of theorems and proofs. Logical reasoning refers the process of using formal logic principles and rules to make deductions, draw conclusions, and solve problems [36, 58]. It involves applying logical rules and syllogistic reasoning to evaluate the validity of arguments and propositions. [62] propose an automatic evaluation framework for L.LMs' logical reasoning ability, and find L.MSs have difficulty in performing logical reasoning well. [59] shows that LLMs can learn from mistakes in logical reasoning. Causal reasoning is the process of identifying and understanding the cause- and-effect relationships between variables or events, which involves identifying potential causes and effects within a system or context [58]. One distinctive feature of causal reasoning is that it involves counterfactual reasoning, which means reasoning within hypothetical scenario. [33] inspect LLMs' ability to distinguish between hypothetical and real-world scenarios. Our work focuses on the LLMs' causal reasoning, particularly counterfactual reasoning.
D.2 Causal Reasoning in Large Language Models
Causal Reasoning tasks include causal discovery, cause attribution, and causal effect estimation [30, 58]. Causal discovery aims to recover the latent causal structure of variables, which could be a structural causal model (SCM). Cause attribution refers to uncovering potential causes behind process, while causal effect estimation aims to investigate the effect of cause variables [26, 76]. While LLMs already show the ability to uncover causal relationships from context, their abilities have some limitations [37]. [4] suggests that LLMs are struggling to perform causal discovery tasks in hypothetical scenarios. [39] find GPT-3 has limitations in uncovering the causal structure among medical context. [77] illustrate the boundaries of LLMs' performance on causal discovery. Moreover, [38] indicates that LLMs'' expertise may contain errors, which could undermine causal reasoning. Therefore, it's essential to incorporate external expert knowledge into LLMs to enhance LLMs' causal reasoning abilities. Counterfactual reasoning is an essential task of causal reasoning. LLMs with high counterfactual reasoning abilities have the strength make predictions under various circumstances. The difference between counterfactual reasoning and other causal tasks is counterfactual reasoning involves reasoning in hypothetical scenarios [28, 58]. For example. "If cats were vegetarians. what results would happen?" is a counterfactual reasoning question [33]. Several studies conclude that LLMs have limitations when encountering hypothetical scenarios counterfactual reasoning [33, 66].
D.3 Causal Reasoning Benchmarks in Large Language Models
There have been extensive studies on causal reasoning benchmarks. Existing causal reasoning benchmarks are mainly causal question-answering datasets. CausalQA [6] is a dataset containing 1.1 million causal questions and answers. [6] employs language rules to extract causal questions from ten large question-answeriine to assess LLMs' abilities to understand causal relationships among real-world events. [50] extract real-world stories during the past ten years and create CRAB based on these real-world stories. FCR [69] is human-labeled dataset that includes 24K
datasets form the CausalQA. CRAB [50] is dataset that aims
pairs. [69] collect data from Yahoo and employ crowd workers to generate the causal questions. However, the data they collected are between December 2020 and July 2021, which may be included in the pre- pre-training corpora of the latest LLMS (e.g. LLaMA 3). Cladder [57] is a dataset that involves symbolic questions and corresponding ground truth answers, [57] employs causal graphs and structural causal models to generate the dataset. CausalProbe 2024 is different from the above benchmarks, as its contents are based on the latest and authoritative information, which is unlikely to be encompassed by the pre-training corpora of L.MMs.
E LLMs' Training Data Detection: Membership Inference Attack
In our work, we employ pre- pre-training data detection method [53] to confirm further that LLMs'
pre-training data do not include our datasets to a large extent. MIN-K % PROB [53] is a pre-training data detection method. This method is based on simple assumption: "When encountering an unseen example, the large language model (LLM) is likely to find a few words with low probabilities, while words in familiar example are less likely to be assigned such low probabilities." " Let x I ...... Ly be a token sequence, and for one token 24 E x, the log-like probability given its preceding tokens can be calculated as:
Dogp(zs/zn 22..... II-1)
18

---

Then, each token's like probability can be computed. Let MINN-K%( represent the set composed of the %K tokens with the lowest log probabilities among all tokens in sequence æ. The average log-like probability of this set is:
Min-K % Prob(x)
1 N
Σ
log 22. Ti-1))
E.EMIN-K%((r
where N the number of elements in (IN))K((2)) In end, conditions on a threshold e, Min-Ke K% Prob(x)) > 6, then this sequence of text x is determined not in pre-training data, otherwise x is determined in pre-training data. One advantage of this method is that it does not need any information about the pre-training data nor does it need to train reference model in advance.
F
A Detailed Introduction of Used Benchmarks
F1
CausalNet
The CausalNet [4] is an LLM-generated dataset designed to support studies in causal reasoning and counterfactual analysis. With 1000 meticulously selected scenarios, CausalNet offers wide range of causal and counterfactual inquiries, helping researchers delve into the complexities of cause-and-effee dynamics across different contexts. Each question-answering pair in the CausalNet contains background context, causal-effect question, counterfactual question, and the choices and answers of the question. However, compared with CausalProbe 2024, the background context of the CausalNet is limited, which may provide sufficient basic information for LLMs to answer the questions better.
F.2 COPA
The Choice Of Plausible Alternatives (COPA) evaluation [48] is dataset including 1000 causal questions. The COPA aims test LLMs' commonsense causal reasoning abilities. [48] employs an authoring methodology to secure broad topics, correct language, and high consensus among human evaluators. To be specific, they utilize several regulations to secure language clarity. Besides, they choose two different sources of questions that are confirmed broad to ensure the breadth. However, the articles they selected are mainly from August and September 2008, which may encompassed by LLMs' pre-training corpus.
F33 e-CARE
The CAusal REasoning dataset (e-CARE) [14] is howwding-soureingg based dataset, which contains 21K multi-choice causal reasoning questions. This dataset not only tests LLMs' ability to choose the correct causal statements but also examines the LLMs' ability to explain their choice. The CARE includes 21 21324 question-
question-answering answering pairs and 13048 corresponding explanations. [14] collect statements of world knowledge and ask crowd workers to generate the causal facts according to the instructions. Then, the causal questions are generated by crowd workers based on these causal facts. However, LLMS' pre-training corpus may contain the collected statements about world knowledge.
G Additional Results
G.1 Standard Deviations of Main Results
Due to the space limitation, we present the standard deviations of the results in Table here. For two closed-source LLMss, i.e., GPT 3.5 turbo and Claude 3 opus, we find that setting the temperature value to 1.0 can achieve slightly better results than We repeat these experiments three times, and the average results together with their standard deviations are shown in Table 6. The standard deviations are generally small.
H Quality Control
We have employed several measures to control the quality of CausalProbe 2024.
19

---

Raw Matetia
CausalProbe-H
CausalProbe M
Cousel Denchmarks
The Guardian BBC B
Uler
Step Article Summary
Step Question Generation
Artide)) please summactar,
Prompt Template of CausiPrasaa-11,
GPFT
Example
[Sammary]]
Renomilas, the camber of connect answera is NOT flaod..
Atticle
User
Stepa: Cornect Cacaal Pairs Extrection
Litt the cause and oflect pains in this [Summnrry]
CausalProbe-
pair) (Qoestion
Cansel effect puir]: (Causal J. Effect
EPT
Step 1: Exomple Illsstration
Correct Answer
Correct Annerr
Uner
[Eesmple) ask for coross [Chsio__]
Stepg: Incorrect Cansal Pairs Making Up
Falee Answers
User
you undkrstandP......
GPT
Can you also unke the fales cuner and sfllct pairs using this [Summury P...
scems (Camso)
Unecerect Caussl-ellext pai)] (Causal L Eflxc_
GPT
Step 2: Answer Verification
Step 2:
Question Generation
Uset
[Artile), loced
sinilar questions
User
Step4: Question Generation
User
[QA pair), mamuilly verified the sumber cornast attewers is unt suro.
and
Can you make multiple choice test using (Causal:
effect patz) and [incoerect Causal effect pair]?
[ammuryy....
[QA paic]:
(QA Quseetonn Answer
[Quaskaa J; Answer_
Summary + QA
Figure 8: The pipeline of constructing Pruse 2024. CausalProbe 2024 consists of three datasets: CausalProbe-E, CausalProbe-H, and CausalProbe-M. They follow different strategies. In particular, CausalProbe-H introduces made-up fake cause-effect pairs, and can be used to examine the LLMs' genuine causal reasoning capability when encountering counterfactual disturbance term. CausalProbe-M features questions with varying numbers of correct options, preventing LLMs from providing right responses through random guessing.
Preparation: Our corpus is sourced from two famous media with high quality. We further performed an initial cleaning of the corpus using regular expressions. Subsequently, we used the Google DLP API detect sensitive information (such as pornography, violence, advertisements, etc.) in the corpus and removed any violating content.
We use GPT 3.5 Turbo, one of the best LLMs at the time, to construct the benchmark from the prepared corpus. To improve its quality, we experimented with different prompt templates and adopted the best.
Production:
We write Python scripts exclude incomplete/garbled items. Then re-organize them to Json format for ease of reading. Finally we went through all items manually to find out problematic ones and excluded them.
Verification:
We have made additional efforts to enhance the quality following a crowdsoureing pipeline. We recruited 17 volunteers, all of whom hold a master's degree or higher and are currently engaged in frontline research. Additionally, all volunteers are fluent in English. In the preliminary tests, we randomly sampled 20 questions from CausalProbe-H and asked each volunteer to answer them. We then recorded each volunteer's 's answer and their perceived difficulty level about this task (on scale of 1-10, with 10 being the most difficult). The selection criteria required the perceived difficulty level to be no more than 7 and the accuracy rate to be no less than 80%. Ultimately, 13 out of the 17 volunteers met these criteria and were selected as qualified, and the test results are shown in Table 4.
Given the limited time and human resource, we performed quality control on a subset of CausalProhe 2024. Specifically, we randomly sampled 260 questions from CausalProbe-H and assigned each question to 3 volunteers randomly, using the Algorithm 1. Each volunteer received a total of 60 questions. After receiving their feedback, we treated those questions correctly answered by no less than two volunteers as high-quality data. Finally, 232 out of 260 questions were filtered out (temporarily called CausalProbe- HQ). achieving the qualification rate of 89.2%. Next, we use CausalProbe-HHi to evaluate four LLMs used in our paper again, whose results are shown in Table 5. The results show that all four LLMs still perform poorly on this subset, suggesting that their failure is primarily due to limited causal reasoning abilities rather than the enrors in the benchmark. We will continuously perform quality control on all data through the aforementioned crowdsoureing pipeline and update the quality-controll data our GitHub repository.
20

---

Algorithm Question assignment algorithm
Require: List of all question IDs Q, List of volunteers P
1: Shufile Q randomly
2: Create Sip] for each volunteer E P to store their assigned questions
3: for all question q E Q do
4: Petigible + {p E P |S|p]| < {00
5: while Paligiblel < do
6: Shuffle P randomly
7: Peligrble {p E
[Spp] < 60)
8:
end while
9:
Assign question g to the first volunteers Petigible
10: iori=11 to do
S[Pettgibie[]] < S(Peligible [i] {4}
12: end for
13: end for
Table 4: Statistical results of volunteer selection.
ID
8
10
13
16
Diff level
5
6
5
8
6
5
4
6
7
4
5
5
4
Acc (%)
80
75
100
80
90
70
100
85
90
70
95
85
95
100
85
Qualified
x
x X
x
4
/
Table 5: Results on subset of high-quality data.
Dataset
LLaMA 2
LLaMA 3
GPT
Claude
CausalProbe-H
0.565
0.652
0.671
0.692
ProseI 0.547
0.660
0.698
0.733
I Additional Introduction and Analysis of CausalProbe 2024
I.1 Statistic of CausalProbe 2024
CausalProbe 2024 consists of 6,922 Q&A data, including two sub-datasets, each containing 3,461 Q&A data. Each sub-dataset covers topics such technology, culture, business, climate, and world news. The statistic of CausalProbe 2024 in terms of the topics is shown in Figure 11. Moreover, all questions can be categorized into "asking for cause' and 'asking for effect,' thus examining the LLMs' ability to reason about causes and effects in questions, respectively.
For CausalProbe-M we visualize the distribution of the number of correct options in its Q&A data, shown in Figure 12. We find that two and three correct options occupy the majority and the distribution is simialr to be Gaussian distribution. We also visualize the distribution of query types (i.e., asking for cause or effect) Figure 13.
I.2 Addition Analysis
In this subsection. we discuss how the CausalProbe dataset tests the causal reasoning ability of LLMs. As discussed in Section 6. we collect the latest and authoritative web articles and utilize them to generate the dataset. To be specific, the downloaded articles are from January 1, 2024, April 29, 2024, later than the releases of many LLMs. Consequently. these latest articles are unlikely directly incorporated into the pre-training corpus of some open- source LLMs (e.g. LLaMA 2 and LLaMA). Afterward, the questions generated by these articles are probably not included in pre-training corpus of many LLMs. Therefore, LLMS are less likely to provide correct answers using their pre-training knowledge directly, which forces them to utilize their causal reasoning ability. Moreover, the questions cover a wide range of categories and encompass many real-world events that are highly discussable, which tests the causal reasoning ability of LLMs from multiple perspectives.
21

---

Table 6: Results with standard deviations of the studied LLMs on four causal Q&A benchmarks. The metric is exact match (EM). "Vanilla" denotes doing inference directly. "C-E", "C-H" represnt CausalProbe-F and CausalProbe-HH For GPT 3.5 turbo and Claude opus, the temperature value set to 1.0.
LLaMA 2
LLaMA
GPT 3.5 turbo
Claude opus
Vanilla
0.752
0.937
0,943
0.992=0.002 0.991=0.000
CoT
0.812
0.944
0.951+00.001
COРA
RAG
0.757
0.912
0.939+-0.005
0.988=0.003
G Reaassene
0.813
0.948
0.953++00002
0.993=0.001
Vanilla
0,684
0.778
0,811
0.857
CoT
0.697
0.770
0.807+0.006
0.863#0.002
e-CARE
RAG
0.687
0.760
0.806+1.0004
0.842=0.005
G2 -e-assoee
0.701
0.779
0.000
0.854=0.003
Vanilla
0.673
0.857
0.895+00.003
0.930+0.002
CoT
0.666
0.767
0.872+0.003
0.907
CausalNet
RAG
0,650
0,860
0.894
0.910=0.004
G² ReRessoeer
0.681
0.855
0.000
0.929
Vanilla
0.616
0.715
0.730=10.002
0.755
CoT
0.636
0.720
0.740
0.751
C-E (*)
RAG
0.621
0.704
0.743
0.756
G2 -eResonee
0.642
0.718
0.747+00.003
0.762
Vanilla
0.565
0.652
0.670110.001
0.688
CoT
0.573
0.644
0.662++0.004
0.700
C-H (*)
RAG
0.575
0.655
0.678+0.001
0.687+0.001
G² ---Resooner
0.582
0.658
0.690#0.000
0.701
I.3
Potential bias
As known for us, any machine model has its own inductive bias [52]. Similarly, using GPT 3.5 turbo to build CausalProbe 2024 introduces potential biases. They mainly consist of the following types of bias:
Model bias: GPT-3.5 turbo may inherit biases and limitations from its own training data.
Generation bias: GPT-3.5 turbo may frequently produce certain types of questions or answers.
Language and cultural bias: Both CausalProbe 2024's English news corpus and GPT-3.5 Turbo's predominantly English training data may introduce Western or Anglophone biases.
To mitigate these potential biases, we propose the following measures:
Diversification of data sources: We can incorporate diverse corpus sources and more methods, such as manually created Q&A pairs or data generated by different LLMs.
Manual check: We can conduct thorough human review and curation for CausalProbe 2024's biases.
22

---

COPA
CausalNet
"id": "I",
"index": "cause-effect=e",
"asks-for"; "cause",
unespected tobarco anong teenagers. At the same time, there was an Increase in online educational content about health, and popular celebrity spoke against seoking,",
"context"; new poliry country
"context": "My body shadow
the
"choice_ 8":
"ask-for": "cause",
rising.
"label":
"choice 1": "The grass was cut.
id8": policy change wininal
"anseer"; B
"choice_id1"; celebrity's
charged teenagers' perceptions significantly
CausalProbe 2024
102":
educational content
reason
"Id":
e-CARE
"context"; "Sourralists facing criticisn for contributing
surrounding capabilities and
intelligence (AI) accurstely reporting Hisitations. surge interest experts calling sore anced reporting ing hight both positive negative aspects AI.
"Index":
"train-"",
has
coverage,
"premise":
light
today.",
"ask-for": "effect",
"question": "What is the result of the surge in interest in artificial intelligence (AI)
"typethesisi"; "The reats of rany
In terss of sedia coverage?",
plants are not moistened by rain.".
"choice_ 1": "Incrwased serutiny on AT Journalists.",
"choice_2" "More balanced reporting the positive and negative
"typethesis2"; "Tourists have ripples."
responsibilitt technologies by fournalists.',
report accurately.",
understanding
"label";
"answee":
Figure 9: Examples of four Causal Q&A benchmarks used in this work. These four benchmarks come from different corpora. and exhibit the similar format and difficulty. particular, CausalProbe 2024 additionally provides the context for each question, which describes the question's 's background knowledge. Note that other three datasets also have "context" or "premise", but actually, this a part of the question itself, not the background knowledge.
Figure 10: Statistics of the down- loaded articles in terms of their top-
ics.
Categories of Question Topics
Categories
BBC
The Guardian
12%
6%
25%
Science
220
251
Culture
381
Climate
121
31%
12%
Technology
340
14%
Health
204
Environment
329
Science and Technolocy Cuiture Businass World News a Health Others
Business
101
352
World News
69
934
Figure
The statistic of the number of each topic's data in CausalProbe 2024.
Others
75
Total
967
2702
Effect
81%
Number of correct arswors
Figure 13: The statis- tic of query types in each Q&A data of CausalProbe-M.
Figure 12: The statistic of the number of correct options in each Q&A data of C.
23

---

Amiile Summary
User
Airlne adverts banned over "roenwasshin clains
sdlens isr oe fraee uuttanee and Frieed have lime han rasine cn mumenn about ailand meurzann
Causal Pains Extrection
Lis the cause and effect pars this summary.
andiuttanss cuiuaa Rihll cad and sanie Seniroore anceeaan""
Sum,, hare are the cause and efeet paint
Cause: Mitoecing endhonmenta idaiss by Air France. Lutttanox and Eilhod
Piccse summarkk this artides
Offect Banning of adhens by the Advert ding Stancants auttority (8S4)
baned ebloutin beat
wxsut Aiachhnng arb autherr poo0).
Couse: free ous crticin faood Lufthanca for greenwoshing
The UK governeet maraans 10% sustinaade
Elece famoad of ad by calthare and Eihsad.
The prompt of summarizing articles.
(b) The prompt of extracting cause-effect pairs.
Questions Generation
Make Up False Causal Pairs
Usar
multiple choice best using
User
you palls using summaryy?
Of course, here are the fube cause and effect pains.
Sare, rudtple
GPT
Standaris iss Author (ASA).
false Effect Mis exding cnvrrmmental daies by France, Lifthense, and Elhaar.
Patee Caucet forn of adtwetts the Adtenttiing
Standdanto Aushortty (484))
Lutte stua Cihee.
M iieecing enwictemnna ciima by dir frunne.
adownts
enaloameent
Fules Cauaet famoral of ids tuftheree and tiched. fale thect Previcus crticam fsoed by Lutthenne
Rope for call toresion theo catton focppeer
Armerr:
Luthaasa tthitd
(c) The prompt of generating incorrect cause-effect pairs.
(d) The prompt of generating causal Q&A data.
Figure 14: The prompt templates used in constructing CausalProbe 2024.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately and roundly reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss two limitations: 1) our proposed causal reasoning method for LLMs is only a step forward, but cannot realize the genuine causal reasoning; 2) we cannot strictly guarantee that our proposed CausalProbe 2024 benchmark is absolutely unseen for LLMs.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
25

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We clearly list the implementation details and provide the main codes and datasets through anonymous github link.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide codes and data reproduce the main results in this paper through an anonymous github link.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips. cc/ Sublicc/guies/Codeee
ceapbbmmssioooooocyyy. for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. wttpcc/pclpccpcceeeeeeeooo gut////id/////e// htt////eeeeeesiiiieeeeeee ssionnolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the test details necessary to understand the results. However, our paper does not include model training.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: This paper does not involve model training, and only invovle LLMs inference. We set the temperature to very small value to ensure the preciseness of causal reasoning, and thus the inference results are stable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Yes. We provide the compute workers, memory, time of execution in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https : htt///eerrpp../e wttppp/bic//tiiiccccieieeeeellle
Answer: [Yes]
Justification:
We strictly obey the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We specially discuss the broader impacts in section. The positive societal impacts include that our paper help this community correctly understand current LLMs' causal reasoning capability. The negative societal impacts include that we still cannot improve L.LMs causal reasoning capability to reach the human level, thus the reasoning results may still be incorrect.
28

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: To mitigate risks of unsafe content, the dataset was filtered to only include articles from reputable news sources, specifically the BBC and The Guardian. These news organizations have their own strict editorial standards and content moderation practices to avoid publishing explicit, harmful or illegal content.
However, as an added safeguard, the scraped text was automatically scanned for profane, explicit, offensive language, and any articles containing such content were removed from the dataset. The dataset was also manually spot-checked to verify the automated filtering effective at identifying and removing unsafe text.
While these measures significantly reduce the risk of unsafe content, it is still possible that some inappropriate text evaded detection. The dataset is intended only for research purposes by responsible parties. We emphasize that the dataset may still contain offensive language or content, and should be used with caution. By publishing this dataset, we do not endorse or take responsibility for any inappropriate content that may be present.
We believe making this dataset available the research community, with these caveats and safeguards, provides benefits that outweigh the risks. However, we are open to feedback on additional steps we could take to further reduce the potential for misuse of this data. Responsible use and deployment of models trained on this data is crucial.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
29

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We list the copyright and the license and terms of use in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide a document in the anonymize link.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing and crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
30

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "5l3heoY7xb", "reviewer": "Reviewer_ZNie", "summary": "This paper studies the autoregressive mechanism of the transformer-based LLMs and their ability to reason causally and introduces a new Q&A-based benchmark named CausalProbe-2024 with around 3k Q&A pairs (believed to be unseen by the existing LLMs). The paper further introduces a reasoning strategy that considers general knowledge and the goal of performing a task called G2reasoner. The proposed G2-reasoner is based on a RAG based framework that encodes the information related to general knowledge in the form of a vector database, and the goal orientation is simulated by modifying the prompt that steers an LLM to be an intelligent causal reasoner and provides the correct causal relationship between the events and reason about them. \n\nThe paper reports the empirical evaluation results in over 3 widely used benchmarks (COPA, e-CARE, and CausalNet) along with the two versions of the proposed Causal Probe dataset.", "questions": "* The assumption behind the autoregressive models made in line 159, “in a sequence, the current value is determined by past values, not related to future values,” may not be entirely correct. Specifically, highlighting “the current value is not related to future values” may be incorrect. \n\n* Line 148 raises a very good point regarding the complexity of natural language, and there can be many different ways or sentence patterns to express the same information. However, the evaluation metrics used in the paper is Exact match (Table 2, Figure-5, Figure-6).  It would be good to consider a better evaluation metric while performing the evaluations. \n\n* The claims made in the para [line 52 to line 61] and figure 2 are not completely justified in paras [line 156 to line 180], concluding on the statement [lines 178 to 180] “Thus, the autoregression mechanism makes LLMs’ causal reasoning primarily rely on correct causal knowledge in a large number of training corpora, i.e., the level-1 causal reasoning”. It would be great if the authors could provide a more detailed justification and talk about level 2 as well. Currently, it is not very clear if the statements say that autoregressive training is sufficient for achieving level-1 causal reasoning. \n\n* Equation 2 removes the dependency over the confounding variable using the total probability formula, where the $P_C$ is the general world knowledge base that may not always be available. Moreover, even if the general knowledge base is available, applying the total probability formula would need it to be complete, covering all the possibilities, which may not be feasible in general for natural language descriptions. I may have understood it incompletely, but it would be good if you could share some more thoughts and assumptions behind equation 2 to make it more transparent to the reader. \n\n* The use of GPT 3.5 turbo for constructing the benchmark may add a bias in the dataset construction. It would be good to highlight such biases and talk about them in detail. \n\n\nMinor Suggestions:\n* The use of fancy terms like AGI would be better avoided since the definitions of these terms are unclear and still under construction.", "limitations": "The limitations of the work are highlighted in the appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper raises an interesting question regarding the causal reasoning in LLMs, and the research goal established by the paper with the underlying question regarding causality is timely. \n\n* The paper’s primary strength comes from creating a newly created benchmark for facilitating the study of causal reasoning in LLMs when compared to humans. Formulating a formal causal reasoning benchmark in a natural language is an extremely challenging task, and this work will take a small step toward making the causal reasoning evaluation effective. \n\n* The paper highlights the training data detection using MinK% Prob in Table 3 for the existing causal benchmarks/datasets, which will be helpful for future research. With evaluation and comparison with the widely used causal reasoning benchmarks, the paper highlights some findings for the open-weight and popular proprietary models.", "weaknesses": "* The definitions of level 1 and level 2 are not concretely stated. Though lines 48 to 51 vaguely describe it, and the expanded version of the definitions is present in section 3, line 126 to line 135, it would be better to make them more concrete. In the current version, the definition is not very clear when it talks about complex or unknown tasks. A clear distinction between the levels would help the reader/research community to explore it further in future works. \n\n* The results of the studied LLMs on four causal Q&A benchmarks, as shown in Table 2, highlight very little improvement of the G2-reasoner compared to the other approaches. Moreover, the motivation/inspiration taken for the G2 reasoner is generic and not causal specific (specifically the context and the RAG part) and may help improve the performance over tasks where causal reasoning is not required. \n\n* Figure 5 and Figure 6 shows the performance improvements when provided with the context, which is generally true for all the LLM evaluations, and the observation may not be causal reasoning specific. It may be noted that the G2-reasoner approach may also be applicable to other non-causal tasks and may have a similar result in other non-causal benchmarks. The proposed strategy will be more reliable when there is a direct effect over the causal-reasoning benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for  your constructive comments and they are valuable for improving our paper. In the following, I will address your concerns one-by-one.\n\n> Q1. Weakness (1)\n\nA1. Thank you for point out this ambiguity. Here we provide more clear definitions for two causal reasoning levels.\n- **Level-1** causal reasoning refers to retrieving existing causal knowledge embedded in model parameters and the given contexts. It is usually fast and suitable for handling simple causal relationships. Current LLMs are at this level.\n- **Level-2** causal reasoning mimics human cognition, allowing LLMs to use powerful reasoning mechanisms and existing knowledge to infer causal relationships, even with unfamiliar tasks. This process is slower but capable of handling unknown causal knowledge.\n\nWe think that level-2 reasoning is not always better, as it is less efficient and cost-effective. Ideally, LLMs would adaptively choose the appropriate reasoning mode based on task difficulty. However, our paper shows that LLMs lack level-2 causal reasoning capabilities, preventing this adaptability.\n\nWe have added a link in lines 48-51 to the formal definitions of level-1 and -2 causal reasoning, i.e., Def. 2 and Def. 3.\n\n> Q2. Weakness (2)\n\nA2. We get your concern and clarify it for you. Our paper focuses on **textual causal reasoning tasks**, instead of the numerical ones like classical causal inference/discovery. There is a significant gap in addressing textual tasks with numerical causal methods. Instead, **G$^2$-Reasoner is specifically designed for textual causal reasoning of LLMs** with clear motivations:\n- In Sec. 4, we showed that sequential causality differs from logical causality in natural language, a similar view also proposed by philosopher David Hume [1]. As the autoregressive nature of LLMs, they inherently learn sequential causality. To address this, we proposed a **goal-driven prompt** to steer LLMs in identifying correct cause-effect pairs and reaching the intended goal during decoding.\n- In Sec. 5, we model the data generation process of a textual causal reasoning task with a SCM (Figure 4). This SCM shows that complete **general knowledge essentially drives  causal relationships**. Without it, LLMs may contradict objective causal laws, rendering final reasoning conclusions meaningless.\n\nThus, G$^2$-Reasoner's performance relies on an general knowledge base. The reported results were obtained using a **very small knowledge base (around 16 MB), yet G$^2$-Reasoner generally achieved non-marginal improvements**. If we use a complete one, such as Wikipedia API, performance can be boosted a lot. However, due to resource constraints, we couldn't repeat all experiments with Wikipedia API. Instead, we are creating a more complete offline general knowledge base and will open source it, which is helpful for LLM causal reasoning and other fields.\n\nIn addition to LLM's causal reasoning, we also believe that these two motivations are helpful for general LLM reasoning tasks.\n\n[1] David Hume. A treatise of human nature. Clarendon Press, 1896.\n\n> Q3. Weakness (3)\n\nA3. The 'context' in CausalProbe 2024 **is not a technical contribution for boosting performance but rather a contribution in terms of more reasonable evaluation benchmarks**. Previous causal Q&A benchmarks did not provide contexts. However, we have figured out that LLM's causal reasoning may be meaningless without specific contexts. Figure 5 & 6 exactly highlight this contribution. The causal insights behind G$^2$-Reasoner is discussed in Q2. We also think that G$^2$-Reasoner and necessary contexts will be applicable for non-causal tasks of LLMs.\n\n> Q4. Question (1)\n\nA4. Sorry, we don't fully understand your concerns. For widely-used decoder-only LLMs, the masked self-attention naturally determines that the current token is only influenced by the previous ones and the prompt. We are happy to further discuss this interesting problem with you.\n\n> Q5. Question (2)\n\nA5. Thank you for your insightful comment. We understand your desire for better metrics to measure text complexity. However, our benchmarks are in the form of multiple-choice questions, and the **final outputs are only option IDs**, making it difficult to introduce other metrics.\n\n>Q6. Question (3)\n\nA6. To reach this main conclusion, we constructed CausalProbe 2024, which **has a format and difficulty similar to previous benchmarks** but has a **completely new corpus (see Tables 1 and 3)**, which was released later than the LLMs' data cut-off time. Instead, the previous benchmarks may have been training data. It allows CausalProbe 2024 to test an LLM's **true** causal reasoning ability, and the effect of the LLM's memorized knowledge are partially exluded. Figure 1(d) shows that four LLMs perform significantly worse on CausalProbe 2024 than the previous benchmarks, indicating that autoregressive LLMs only achieve level-1 causal reasoning but not level-2.\n\n> Q7. Question (4)\n\nA7. Your understanding for the Eq. (2) is correct. Ideally, a complete general knowledge base $P_C$ is needed to ensure the total probability formula holds. However, obtaining an absolutely complete $P_C$ is challenging, limiting G$^2$-Reasoner's performance as discussed in Q2.\n\n> Q8. Question (5)\n\nA8. Thank you for your insightful feedback on the potential biases introduced by using GPT-3.5 Turbo to construct our benchmark. We discuss it here:\n- **Model bias**: GPT-3.5 Turbo may inherit biases and limitations from its training data.\n- **Generation bias**: The model may frequently produce certain types of questions or answers.\n- **Language and cultural bias**: Using English news corpora may introduce a Western or Anglophone bias.\n\nWe will include this discussion in Sec. 5 of our paper.\n\n> Q9. Minor Suggestions\n\nA9. Thank you for your valuable suggestion. 'AGI' is an unclear concept and using it in a research paper is inappropriate. We used 'AGI' three times and have replaced them with more conservative terms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b24QUqSGrQ", "reviewer": "Reviewer_yorJ", "summary": "The paper investigates the causal reasoning capabilities of LLMs and argues that current LLMs are limited to shallow (level-1) causal reasoning. To support this claim, the authors introduce a new benchmark, CausalProbe-2024, which reveals that LLMs struggle with causal reasoning in fresh and unseen contexts. To address this, the authors propose G2-Reasoner, method that incorporates general knowledge and goal-oriented prompts to enhance causal reasoning capabilities. Experiments show that G2-Reasoner improves performance, particularly in novel and counterfactual scenarios.", "questions": "Just a minute comment, for better readability, I would appreciate it if the authors could an identifier(eg bold) in the Tables 2,3 just to make it easier to read.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper goes beyond simple retrieval from LLM memory causal reasoning tasks. \n\n - They attempt to establish the sensitivity of the model for pretraining\n\n- The authors also propose a new benchmark CausalProbe-2024 to evaluate level-2 reasoning.\n\n- The authors identify fundamental limitation in the current architecture of LLMs as authoregressive next token prediction mechanism (which is intuitive though) for causal reasoning.", "weaknesses": "- The long-term viability of this approach is uncertain. Continuous advancements in model architecture and training techniques might be required to truly enable level-2 reasoning, and the proposed method might only be a temporary solution. However I understand that this paper still makes a good progress.\n\n- Unless I missed it, I did not find much detail on RAG since it is so knowledge-specific. It isnt always aplicable. Also the results dont show a big improvement for G2 reasoner. It then begs the question how dependent was the model the knowledge of RAG. \n\n- paper primarily addresses simple, single cause-effect pairs which is a bit limiting\n\n- the variances arent given", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for constructive comments and they are valuable for improving our paper. In the following, I will address your concerns one-by-one.\n\n> Q1. The long-term viability of this approach is uncertain. Continuous advancements in model architecture and training techniques might be required to truly enable level-2 reasoning, and the proposed method might only be a temporary solution. However I understand that this paper still makes a good progress.\n\nA1. We partially agree with this comment. In the future, more powerful model architectures, training methods, or new forms of AI may eventually achieve genuine causal reasoning even surpass humans. However, this may **take a long time**. Therefore, our current efforts in causal reasoning for LLMs are **meaningful at the current development stage**.\n\n> Q2. Unless I missed it, I did not find much detail on RAG since it is so knowledge-specific. It isnt always aplicable. Also the results dont show a big improvement for G2 reasoner. It then begs the question how dependent was the model the knowledge of RAG.\n\nA2. Thank you for this insightful comment. The implementation details of RAG was presented in Appendix C.\n\nIn our paper, RAG incorporates general knowledge into LLM causal reasoning. As you noted, complete and high-quality knowledge bases are crucial for RAG performance. Our reported results for G$^2$-Reasoner are based on a **very small general knowledge base (about 16 MB), yet it achieved non-marginal performance improvement**. If we use a complete one, such as Wikipedia API, the performance can be boosted a lot. However, due to resource constraints, we couldn’t repeat all experiments with Wikipedia API. Instead, we are creating a more complete offline general knowledge base and will open source it, which is helpful for LLM causal reasoning and other fields.\n\n> Q3. Paper primarily addresses simple, single cause-effect pairs which is a bit limiting.\n\nA3. Thank you for your insightful comment. Our paper is **the first to comprehensively explore the boundary of LLMs' causal reasoning abilities**. We created a benchmark with over 6,000 multiple-choice questions. However, our benchmark **is not simply established on single cause-effect pairs**. To pick the correct option, especially for CausalProbe-H, LLMs are also required to identify incorrect but confusing cause-effect pairs. Our benchmark construction method was deliberately designed to achieve this goal, and the details were presented in Sec. 5 \"Bechmark construction\" and Figure 7. Following your suggestion, we will construct more diverse and complex causal reasoning tasks for LLMs in the future.\n\n> Q4. The variances arent given.\n\nA4. Of course, thank you for your suggestion! Due to budget constraints at the time, we ultimately did not repeat the experiment multiple times to obtain standard deviations. Currently, we have repeated the experiments of closed-source models (GPT 3.5 Turbo and Claude 3 opus) **anthor two times** and the results together with standard deviations are shown in Table 4 in PDF.\n\n> Q5. Just a minute comment, for better readability, I would appreciate it if the authors could an identifier(eg bold) in the Tables 2,3 just to make it easier to read.\n\nA5. Thank you for this reminder. We have highlighted the best results under each benchmark and each LLM in **bold** in Table 2 and Table 3. In addition, we have completely polished our paper again to make it easier to read.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v65ZUSqbtM", "reviewer": "Reviewer_NPEc", "summary": "The authors proposed a new causal reasoning framework, to improve the causal reasoning capacity of LLMs, with inspiration drawn from causal graph theory and human reasoning process. The work utilized \"general world knowledge\" as a component to take a step closer to making LLMs perform a more human-like causal reasoning process. The authors also developed a new benchmark and carried out experiments evaluating how different LLMs perform on their benchmark.", "questions": "see weaknesses.", "limitations": "The author didn't provide a separate limitations section. One limitation the authors mentioned in the paper is that they only consider causal reasoning tasks with single cause-effect pair. Another limitation is that their approach doesn't enable LLMs to achieve level-2 causal reasoning, but rather provides insights into it.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The authors draw inspiration from the human causal reasoning process and provide an analysis of why LLMs cannot perform \"genuine\" causal reasoning, from not only a methodological perspective but also an empirical perspective.", "weaknesses": "1. The key hypothesis of this work is that LLM is capable of \"level-1\" reasoning, but lacks the capacity of \"level-2\" reasoning. Given that this work is centered around this hypothesis, a clearer definition and illustrative example of level-1 and level-2 should be provided.\n2. I suggest the authors provide an example of a \"random exogenous variable\" in line 238. I can understand what this variable is from the causal graph framework, but it would be helpful to better clarify it in your setting.\n3. Can you provide a more detailed explanation of how equation (1) plays a role in your G2 reasoner, and how can you compute this term?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for constructive comments and they are valuable for improving our paper. In the following, I will address your concerns one-by-one.\n\n> Q1. The key hypothesis of this work is that LLM is capable of \"level-1\" reasoning, but lacks the capacity of \"level-2\" reasoning. Given that this work is centered around this hypothesis, a clearer definition and illustrative example of level-1 and level-2 should be provided.\n\nA1. Thank you for point out this ambiguity. Here we provide more clear definitions for two causal reasoning levels.\n- **Level-1** causal reasoning refers to retrieving existing causal knowledge embedded in model parameters and the given contexts. It is usually fast and suitable for handling simple causal relationships. Current LLMs are at this level.\n- **Level-2** causal reasoning mimics human cognition, allowing LLMs to use powerful reasoning mechanisms and existing knowledge to infer causal relationships, even with unfamiliar tasks. This process is slower but capable of handling unknown causal knowledge.\n\nNote that level-2 reasoning is not always better, as it is less efficient and cost-effective. Ideally, LLMs would adaptively choose the appropriate reasoning mode based on task difficulty. However, our paper shows that LLMs lack level-2 causal reasoning capabilities, preventing this adaptability.\n\n> Q2. I suggest the authors provide an example of a \"random exogenous variable\" in line 238. I can understand what this variable is from the causal graph framework, but it would be helpful to better clarify it in your setting.\n\nA2. Of course. The formula $h(X,Y,\\epsilon)=T$ represents a natural language generation process that contains cause-effect information. Here, $X$ and $Y$ represent the concepts of cause and effect, respectively, and $T$ is the textual expression of this causal relationship through the mapping $h$. The variable $\\epsilon$ represents **various factors** in generating readable text from the causal concepts $X$ and $Y$, **such as language type, context, and mode of expression (e.g., active or passive voice)**. While $\\epsilon$ contributes to the **diversity and flexibility of natural language**, it also complicates LLM's causal reasoning from a linguistic perspective.\n\nFor example, consider the concepts \"smoking\" ($X$) and \"lung cancer\" ($Y$). With different $\\epsilon$, we can get different natural language expressions: 1) \"A history of smoking is a common risk factor for lung cancer.\" 2) \"Knowing that smoking greatly increases the risk of lung cancer, why take the risk?\" Both sentences imply the same causal relationship but differ linguistically.\n\n> Q3. Can you provide a more detailed explanation of how equation (1) plays a role in your G2 reasoner, and how can you compute this term?\n\nA3. Certainly. We are happy to explain the role and motivation bebind Eq. (1). This equation represents the task of **inferring the most probable effect from a cause given a context**, in the language of statistics. To achieve this goal, we need the cause ($X$), the natural language expression of the causal proposition ($T$), and general knowledge ($C$). In our setup, $X$ and $T$ are known, so we need access to a complete general knowledge base ($C$). With this, LLMs can ideally reason out the correct effect, and mathematically, the total probability formula in Eq. (2) holds. \n\nHowever, we emphasize that what Eq. (1) and Eq. (2) provides is the **technical motivation** from a causal inference perspective, and we **do not need to calculate them explicitly**. We use such technical motivation to design the G$^2$-Reasoner, which integrates the general knowledge base into LLMs' causal reasoning processes through RAG.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TkQXcj8XdE", "reviewer": "Reviewer_ZgYU", "summary": "The paper is concerned with the question of whether LLM can perform causal reasoning on a human-like level by incorporating contextual information in their decision and answer process. The authors argue that many everyday causal inferences are not purely logical but take into account general knowledge and intention. Therefore, a distinction between 'shallow' (level-1) and 'human-like' (level-2) causal reasoning is proposed. Two versions of a novel  \"CausalProbe-2024\" causal Q&A data set for benchmarking level-2 reasoning capabilities of LLM are presented. The presented benchmark is sourced from recent news sources. The authors, therefore, claim that the presented data is unlikely to be contained in the training set of the tested LLM (LLaMa 2/3, GPT-3.5 Turbo, and Claude 3).\n\nTo tackle level-2 causal reasoning, the authors propose a theory on how cause-effect pairs get instantiated into natural language sentences and further argue that a general world context affects the instantiated cause and effect nodes. To reason in this kind of setting, the authors propose a \"$G^2$-Reasoner\" to incorporate general knowledge and inherent goals via additional context information retrieved from a general knowledge base.\n\nEvaluating multiple LLMs on the CausalProbe data set shows a strong deterioration in performance compared to earlier causal reasoning datasets. The proposed $G^2$-reasoner helps improve reasoning capabilities over a naive baseline approach and performs on par with Chain-of-Thought or retrieval-augmented generation approaches.", "questions": "My questions mainly concern the weaknesses mentioned above. In detail, I would like the authors to comment on the following:\n\nI) Regarding (2): could the authors clarify the measures that were imposed to ensure the quality of the collected data samples? How do the authors view the possibility of multiple valid answers with regard to the proposed data set format and evaluation results?\n\n\nII) Regarding (1) and (2), I would like to ask the authors to clarify the expected effects of the context on the causal relation. In detail, could the authors comment (e.g., in the context of epistemic reasoning / modal logic) how 'reasonable/possible' and 'correct/necessary' answers would be distinguished in their setup and which implications would follow for their data set design?\n\n\nIII) Regarding (3): is the partition of the samples performed by human judgment or via some automated metric? Which criteria are used to judge sample difficulty?\n\n\nIV) Regarding (4): Can the authors quantify the possible randomness-induced variance for the closed-source models?", "limitations": "In their paper, the authors correctly address the causal reasoning limitations of LLMs in general and their $G^2$-reasoner in detail.\n\nThe authors seem to have not performed or do not mention a human-backed review of the data set contents. The quality in terms of answer texts and evaluation results could be improved by establishing such a process.\n\nThe data set seems to be free of ethically questionable content; However, the applied filtering process was not disclosed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, the author's proposal of assessing the causal reasoning capabilities of LLM in context is a valid contribution and improves in realism and difficulty over existing benchmarks. The work is well embedded into existing related work on the causal reasoning capabilities of LLMs. Relations to existing causal Q&A data sets are drawn, and previous examinations of LLM causal reasoning capabilities are discussed.\n\nThe novel data set is constructed by extracting information from recent news articles of two major news sources, covering events that lie after the information cut-off of all tested LLMs (after Jan 1st, 2024). To support the thesis of querying the LLMs on unseen information, LLama models are tested via the Min-K% Prob technique on whether the models might have memorized the collected information. Compared to older datasets, CausalProbe seems to be composed of more novel information than COPA, e-CARE, or CausalNet data sets.\n\nThe authors contribute a possible explanation of how causal information is instantiated in natural language via a high-level SCM. This involves a general context, drawing information from world knowledge, and incorporating the naturally occurring diversity of expressing such relations.\n\nThe experimental setup and individual steps of the $G^2$-reasoner are described clearly. Prompt templates are provided. The approach is compared to a 'vanilla' approach, Chain of Thought (CoT), and a retrieval-augmented generation (RAG), giving the LLMs access to a general knowledge base.", "weaknesses": "While I agree with the authors that benchmarking LLMs for causal relations in context is a more difficult task than pure logical causal reasoning, the paper remains vague on the particular effect that the additional context might impose in terms of the cause-effect pairs as discussed in Sec. 5 / Fig. 4.\n\n(1) Generally, the described distinction between level-1 and -2 causal reasoning (def. 2 and 3) seems to be about how LLMs draw information from either memorization (information embedded in the model weights) or dynamically from the input text. LLMs are known to have difficulties when reasoning over non-memorized facts. In this regard, the general shortcomings of LLMs are non-surprising and have already been presented in previous works (e.g., Kiciman et al., 2024; Jin et al., 2024; Zecevic et al., 2023). In that regard, the proposed $G^2$-reasoner seems to feature no explicit mechanism to help improve on the causal reasoning of LLMs other than providing general background knowledge. No alternative prompt variations or combinations with, e.g. CoT or fine-tuning, are presented to improve results.\n\n\n(2) My main concern is the quality of the collected data set itself. While the authors do clearly describe the automated process of extracting information from credible news sources, no human involvement in quality checking is mentioned. Given the more implicit nature of causal relations considered in this paper, many answers can not be directly inferred but need to be extrapolated from the texts (as intended by the authors). However, checking on the first ten entries of CausalProbe-H seems to reveal a generally poor quality of the samples. In detail, I get the impression that multiple answer options per sample can be valid. Given that only a single answer is indicated as 'correct' by the data set, I question the significance of the reported results. In detail:\n\n* id 3: correct marked answer 2 is implicit, but so could be 3 and 4.\n* id 4: 1, 2 and 4 are possible answers with partial contribution to the effect.\n* id 5 - 1 and 2 are both correct.\n* id 7 - 2 and 4 describe partial factors that could both be valid.\n\nThe main problem seems to stem from the unclear definition of what counts or does not count as a causal factor for a specific scenario. As a result, alternative answers --other than the one indicated by the dataset-- might contribute to the causal effect relation and, therefore, could be viewed as reasonable answer options. The authors might have either adopted an answer format capable of accepting multiple correct answers or might impose a more extensive quality checking to prevent the appearance of alternative valid answer options.\n\n\n(3) The partition of samples into causalProbe-S and -H is unclear to me. The authors mention that \"the highest-quality Q&A pairs are selected as the reference\" (l.289) but do not describe under which criteria this selection is done.\n\n\n(4) The authors mention in Appendix (C) \"Full Implementation details\" to have set the temperature to \"1.0 for closed-source LLMs [...], and set it as 0 for open-source LLMs [...].\". Non-zero temperature evaluations might introduce additional variance to the results. However, the authors do not quantify the possible variance in in their evaluation results.\n\n\nMinor comments:\n\n* Typo in the caption of table 3: \"represnt\".\n\n* While I agree in general with the discussion in Sec. 4.1 about causal event sequence and sequence of appearance, it dismisses possible generalization capabilities of LLMs. Claims 1) and 2) are provided without reference or experimental evidence. The authors might want to tune down these claims.\n\n* I would advise not captioning Fig. 2 (b) as a \"sequential causal relationship\" but rather as a \"sequence of appearance\" since --as the authors mention correctly-- word sequences do not need to imply any causal relations inherently.\n\n* Typo \"we proposes G2-Reasoner\" (l.340)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for constructive comments and they are valuable for improving our paper. In the following, I will address your concerns one-by-one.\n\n> Q1. The paper remains vague on the particular effect that the additional context might impose in terms of the cause-effect pairs as discussed in Sec. 5 / Fig. 4.\n\nA1. In my understanding, what you refer to as 'the additional context' means general knowledge. We model a textual causal reasoning task through a SCM in Fig. 4. General knowledge, i.e., variable $C$, **drives the causal relationship between two events**. Without it, the derived causal relationship may contradict the objective causal laws. Thus, general knowledge serves as a **guide** for LLMs' causal reasoning.\n\n> Q2. Weakness (1)\n\nA2. First, level-2 causal reasoning refers to the **genuine complex reasoning** akin to human cognition, rather than merely retrieving information from sources. \n\nOur paper's main contribution is **exploring the boundary of LLM causal reasoning abilities and identifying the causes of their limitation by CausalProbe 2024**. Previous works did not systematically studied this. We also proposed the G$^2$-Reasoner for LLM's **textual causal reasoning**. It involves general knowledge and a goal-oriented prompt: the former serves as a guide, and the latter steers LLMs to identify correct cause-effect pairs and reach the intended goal, alleviating off-target generation.\n\nCoT and finetuning are both effective for LLMs. While finetuning works well for specific tasks, it often leads to catastrophic forgetting, conflicting with our general-purpose goal. CoT is versatile and as a baseline in our paper. Incorporating CoT into G$^2$-Reasoner only boosts performance slightly, so we didn't combine it with G$^2$-Reasoner to avoid additional inference costs.\n\n> Q3. Weakness (2) & Question (I)\n\nA2. Upon review, we found that some questions might have more than one correct option if we only consider the \"problem\" itself. However, within the given \"context\", the unique answer provided by us is usually the most relevant option.\n\nFirst, we discuss the works we **have done** to ensure the quality. To further improve our benchmark, we have taken **two additional steps** based on your comment. **Please refer to the *Author Rebuttal* for more details.**\n\n> Q4. Weakness (3) & Question (III)\n\nA4. We clarify that CausalProbe-E and -H **are not created by partioning existing data** based on a difficulty metric. Instead, they are **constructed from scratch** using different strategies but the same corpus. We presented the methods for constructing them in **Section 6.1 and Figure 7**. I am glad to introduce their differences here.\n\nCausalProbe-E follows the format of CausalQA [1]. We provided GPT with original corpus and samples from CausalQA and asked GPT to generate Q&A data by imitating them.\n\nThe construction of CausalProbe-H is more complex. We provided GPT with an article and asked it to generate several cause-effect pairs. These pairs were classified as true (accurate information) or false (distortions). GPT then generated Q&A pairs from these cause-effect pairs. Three co-authors rated the Q&A pairs for quality, selecting the best one. This top-quality Q&A pair, along with its source article, was used as an in-context example. We then used this in-context example with a new article to prompt GPT to generate new data. Unlike CausalProbe-E, CausalProbe-H tests LLMs' ability to reason and identify erroneous causal information.\n\n[1] Bondarenko et al. Causalqa: A benchmark for causal question answering. COLING 2022.\n\n> Q5. Weakness (4) & Question (IV)\n\nA5. During the experimental phase, we discovered that setting the temperature of the closed-source models to 1.0 slightly improved performance compared to 0, albeit with variance. Due to budget constraints, we did not initially repeat the experiment to obtain the standard deviation. Now, we have repeated the experiments **two more times**, and the  standard deviations are shown in Table 4 in PDF.\n\n> Q6. Minor comments\n\nA6. We have polished our paper again to correct the remaining typos and grammar issues.\n\nFor Sec. 4.1 and the caption of Fig. 2 (b), we have modified them following your advice: 1) we have weakened our claims 1) and 2): \"Based on the above discussion, there are two **possible** issues: 1) ...\"; 2) we have changed \"sequential causal relationship\" to \"sequence of appearance\".\n\n> Q7. Question (II)\n\nA7. The context serves as a **reference** for specific causal reasoning tasks. Without constraints, causal or general reasoning can be aimless, especially for LLMs. Given only a question, LLMs might find multiple \"reasonable\" options. However, with the context provided, LLMs can more easily identify the \"correct\" option, which is the optimal answer. In our benchmark design, we provided context for each question, **distinct from previous causal Q&A benchmarks**. Figure 5 & 6 compare the performances w/ and w/o contexts, showing that context helps LLMs' causal reasoning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Here, we mainly present the work we **have done** and the **new efforts** added for data quality control.

> Current quality control for CausalProbe 2024.

We discuss our efforts for ensuring the benchmark's quality here, which have not been discussed in detail in our paper. We have merged them into our paper. Our efforts are threefold:
- **Preparation**: Our corpus is sourced from two famous media with high quality. We further performed an initial cleaning of the corpus using regular expressions. Subsequently, we used the Google DLP API to detect sensitive information (such as pornography, violence, advertisements, etc.) in the corpus and removed any violating content.
- **Production**: We use GPT 3.5 Turbo, one of the best LLMs at the time, to construct the benchmark from the prepared corpus.  To improve its quality, we experimented with different prompt templates and adopted the best.
- **Verification**: First we used Python scripts to exclude incomplete/garbled items. Then we re-organize them to .json format for ease of reading. Finally we went through all items to find out problematic ones and excluded them.

> Newly added quality control.

We have performed **additional quality control** following reviewer ZgYU's comment. This quality control followed a crowdsourcing pipeline. We recruited 17 volunteers, all of whom hold a master’s degree or higher and are currently engaged in frontline research. Additionally, all volunteers are fluent in English. In the preliminary tests, we randomly sampled 20 questions from CausalProbe-H and asked each volunteer to answer them. We then recorded each volunteer's answer and their perceived difficulty level about this task (on a scale of 1-10, with 10 being the most difficult). The selection criteria required the perceived difficulty level to be **no more than 7** and the accuracy rate to be **no less than 80\%**. Ultimately, **13 out of the 17 volunteers** met these criteria and were selected as qualified, and the test results are shown in Table 1 in PDF.

Given the limited time available during the author response phase, we performed quality control on a subset of CausalProbe 2024. Specifically, we **randomly sampled 260 questions** from CausalProbe-H and assigned each question to 3 volunteers randomly, using the Algorithm 1 in PDF. Each volunteer received a total of 60 questions. After receiving their feedback, we treated those questions correctly answered by no less than two volunteers as high-quality data.  Finally, 232 out of 260 questions were filtered out (temporarily called CausalProbe-HQ), achieving **the qualification rate of 89.2\%**. The randomly sampled data IDs and the high-quality ones among them is shown in Table 5 in PDF. Next, we use CausalProbe-HQ to evaluate four LLMs used in our paper again, whose results are shown in Table 2 in PDF. The results show that all four LLMs still perform poorly on this subset, suggesting that their failure is primarily due to **limited causal reasoning abilities rather than the errors in the benchmark**.

We will continue to perform quality control on the remaining data and eventually open-source the fully quality-controlled version.

> Newly added indefinite multi-choice version of CausalProbe 2024.

In addition, we also constructed an **indefinite multi-choice version of CausalProbe 2024**, named as **CausalProbe-M**, following reviewer ZgYU's comment. Its corpus is the same as CausalProbe 2024 and we designed a prompt template to generate it using GPT-4o mini. **We have uploaded CausalProbe-M to the anonymous link attached in our paper**. CausalProbe-M consists of 3441 Q&A data. The number of correct options is **indefinite**, ranging from 1 to 4, and the statistic is shown in Figure 1 in PDF. The ratio of query types (cause or effect) is shown in Figure 2 in PDF. 

 We also sampled a subset of CausalProbe-M to perform quality control like the above. The **evaluation results** on CausalProbe-M are shown in Table 3 in PDF. All four LLMs experienced a **more significant performance drop** than CausalProbe-E and -H, under the exact matching (i.e., all the correct answers are exactly picked). However, for the partial matching (i.e., missed options are allowed but incorrect options are not allowed), GPT and Claude performed relatively well, achieving near 75% and 85% accuracy rates, respectively. This suggests that **LLMs cannot fully figure out the causal information in each option**, implying their limited causal reasoning abilities from a new perspective. However, it is gratifying that **LLMs make fewer false positive errors**.

Meta-review: This work explores the question of whether LLM can perform causal reasoning on a human-like level by incorporating contextual information. The work introduces a new benchmark, CausalProbe-2024, which reveals that LLMs struggle with causal reasoning in fresh and unseen contexts.

This work generated a lot of discussions both during the rebuttal phase as well as behind the scenes and all the reviewers agreed on the utility of the work. The reviewers agree that "the new perspectives on how cause-effect relations manifest in real-world data pose an important, but inherently tricky topic to capture". They also agreed on certain areas of improvement such as making the provided definitions for level 1 and level 2 more concrete and not using terms such as AGI which have an unclear defintion (which the authors mention that it has been removed). 

I read the paper and overall I think this is a valuable contribution to the growing area in the intersection of causality and LLM. I recommend acceptance.
Decision: Accept (poster)