Title: Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
Venue: NeurIPS 2024
Accepted: yes
Keywords: trustworthy machine learning

LLM

causal reasoning
Abstract: Causal reasoning capability is critical in advancing large language models (LLMs) towards artificial general intelligence (AGI). While versatile LLMs appear to have demonstrated capabilities in understanding contextual causality and providing responses that obey the laws of causality, it remains unclear whether they perform genuine causal reasoning akin to humans. However, current evidence indicates the contrary. Specifically, LLMs are only capable of performing shallow (level-1) causal reasoning, primarily attributed to the causal knowledge embedded in their parameters, but they lack the capacity for genuine human-like (level-2) causal reasoning. To support this hypothesis, methodologically, we delve into the autoregression mechanism of transformer-based LLMs, revealing that it is not inherently causal. Empirically, we introduce a new causal Q&A benchmark named CausalProbe 2024, whose corpus is fresh and nearly unseen for the studied LLMs. Empirical results show a significant performance drop on CausalProbe 2024 compared to earlier benchmarks, indicating that LLMs primarily engage in level-1 causal reasoning.To bridge the gap towards level-2 causal reasoning, we draw inspiration from the fact that human reasoning is usually facilitated by general knowledge and intended goals. Inspired by this, we propose G$^2$-Reasoner, a LLM causal reasoning method that incorporates general knowledge and goal-oriented prompts into LLMs' causal reasoning processes. Experiments demonstrate that G$^2$-Reasoner significantly enhances LLMs' causal reasoning capability, particularly in fresh and fictitious contexts. This work sheds light on a new path for LLMs to advance towards genuine causal reasoning, going beyond level-1 and making strides towards level-2.

Unveiling Causal Reasoning in Large Language

Abstract

Causal reasoning capability is critical in advancing large language models (LLMs) toward strong artificial intelligence. While versatile LLMs appear to have demonstrated capabilities in understanding contextual causality and providing responses that obey the laws of causality, it remains unclear whether they perform genuine causal reasoning akin to humans. However, current evidence indicates the contrary. Specifically, LLMs are only capable of performing shallow ( level1 ) causal reasoning, primarily attributed to the causal knowledge embedded in their parameters, but they lack the capacity for genuine human-like ( level2 ) causal reasoning. To support this hypothesis, methodologically, we delve into the autoregression mechanism of transformer-based LLMs, revealing that it is not inherently causal. Empirically, we introduce a new causal Q&A benchmark called CausalProbe-2024, whose corpora are fresh and nearly unseen for the studied LLMs. The LLMs exhibit a significant performance drop on CausalProbe-2024 compared to earlier benchmarks, indicating the fact that they primarily engage in level1 causal reasoning. To bridge the gap towards level2 causal reasoning, we draw inspiration from the fact that human reasoning is usually facilitated by general knowledge and intended goals. We propose G 2 -Reasoner, a method that incorporates general knowledge and goal-oriented prompts into LLMs' causal reasoning processes. Experiments demonstrate that G 2 -Reasoner significantly enhances LLMs' causal reasoning capability, particularly in fresh and counterfactual contexts. This work sheds light on a new path for LLMs to advance towards genuine causal reasoning, going beyond level1 and making strides towards level2 .

1 Introduction

The emergent of large language models (LLMs), such as GPT 4 [45], Gemini 1.5 [16], and Claude 3 [2], have significantly changed the paradigm of how people work and do research in recent years, demonstrating competitive abilities (LLMs) in instruction following [1, 7, 45, 60], in-context learning [8, 12, 67], reasoning [63, 70], coding [10, 18, 51] and etc. LLMs have demonstrated remarkable abilities in processing and generating human-like text, leading to a belief that they may possess the intelligence akin to human cognition. Reasoning is an essential component of human intelligence and a prominent characteristic that distinguishes humans from other species [47]. Many recent works

Q:

A remote village with limited internet access recently

had a new road constructed nearby. What is the cause of

the improvement in children's academic performance in this

village following the construction of the new road?

A: The improved academic performance is likely due to

increased access to educational resources, such as

libraries or tutoring centers, facilitated by easier

transportation via the new road.

Q: What is the result of developing railway stations as

social hubs and providing a range of public services

alongside tickets?

A: Developing railway stations as social hubs and providing

a range of public services alongside tickets fosters

community engagement, enhances public transportation

accessibility, and promotes economic development.

CausalProbe 2024

"id": xxx,

"context": "Peterborough mayor Nick Sandford has admitted to

using AI to help write his speeches, specifically utilizing ChatGPT. The

city council is considering implementing an AI strategy and ethical

guidelines for future use.",

"question": "What is the result of Peterborough mayor Nick

Sandford using ChatGPT to help write his speeches?",

"choice_1": "Increased public engagement with civic events.",

"choice_2": "Improved efficiency in speechwriting for the mayor.",

"choice_3": "Implementation of an AI strategy by the city council.",

"choice_4": "Development of ethical guidelines for AI use in the

council.",

"answer": 2

common case

imagined case

(a)

(c)

(d)

(b)

have dived into evaluating and improving LLMs' general reasoning capabilities, such as logical reasoning [11, 20] and mathematical reasoning [22, 57].

In this work, we focus on causal reasoning, an advanced reasoning form [31]. In the context of LLMs, causal reasoning is to discern the cause-and-effect relationships that govern the physical world from a text [43, 54]. Current LLMs appear to have demonstrated some degree of causal reasoning capabilities [73]. Sometimes, when asked about the cause or effect of a given text, LLMs can accurately provide responses satisfying the laws of causality that originate from the physical world. For example, in Figure 1(a), the LLM can work out reasonable causes (i.e., 'increased access to educational resources') for academic performance improvement. Faced with such encouraging performances, we have to pose a question:

Does this reflect LLMs' genuine causal reasoning capability or only a 'mirage'?

The answer leans more towards the latter. We find that LLMs are adept at qualitatively solving causal reasoning tasks related to common knowledge (e.g., Figure 1 (a)), but they struggle to tackle more advanced task types (e.g., Figure 1 (b)), such as discovering new causal knowledge and estimating specific causal quantities. For example, in Figure 1(b), when asked about the effect of an imagined, unusual case ('developing railway stations as social hubs...'), the LLM's answer ('enhance public transportation accessibility') is clearly irrelevant with such an action. Recent works [64, 72, 73] also came to a similar conclusion. Given the performance differences on tasks of varying difficulty, we propose a hypothesis: the apparent ( level -1) causal reasoning capabilities of LLMs can be primarily attributed to associated knowledge from their training corpora, rather than engaging in genuine, human-like ( level -2) reasoning. We will justify this hypothesis from two aspects.

From a methodological perspective, the widely-used transformer-based LLMs essentially perform next-token prediction [35, 61], which is realized in an autoregressive manner. Autoregressive model [71] originates from time-series analysis with an important assumption: the current value is determined by past values, not related to future values. For texts, however, the fact that the current token depends on the past tokens does not necessarily mean that there is a causal relationship between them. In addition, philosopher David Hume raised a viewpoint that sequential causality is not equivalent to logical causality [21]. Therefore, this mechanism makes LLMs good at reusing the causal knowledge in their training corpora but often makes them struggle to comprehend and generate texts that capture genuine causal knowledge. We also use structural causal models (SCMs) to formally account for this intuitive conclusion (see Section 4.1).

From an empirical perspective, to validate the hypothesis that LLMs only possess level -1 causal reasoning capabilities, we introduce a new causal question & answer (Q&A) benchmark (Figure

1(c)), named CausalProbe 2024, whose corpora was made public later than the release of the studied LLMs. 3 Given that the training data is rarely disclosed, we can, at a minimum, guarantee that the corpora of CausalProbe 2024 is not included verbatim in the training data of the studied LLMs. Compared to earlier causal Q&A benchmarks, such as COPA [48], e-CARE [14] and CausalNet [4], all the studied LLMs (e.g., LLaMA 2 7B, LLaMA 3 8B, GPT 3.5 turbo, Claude 3 Opus) exhibit a significant performance drop on CausalProbe 2024 (Figure 1(d)). As the earlier benchmarks are potentially part of the training data, these longitudinal comparisons largely support our hypothesis.

Existing studies about LLMs' causal reasoning mainly focused on only assessments [15, 26, 49] and designing prompt-based approaches [4, 25]. These studies have taken a significant step forward in advancing LLMs' causal reasoning. However, they overlooked two basic principles of human reasoning: general knowledge and intention. For example, when we reason about a mathematical problem, we always take the basic axioms as a reference, with the ultimate goal as guidance. Inspired by this fact, we propose a causal reasoning framework for LLMs called G 2 -Reasoner, which incorporates general knowledge and goal-oriented prompts during reasoning. Specifically, we use the retrieval-augmented generation (RAG) to incorporate external knowledge bases. Then we stimulate the LLMs to consistently discern correct causal relationships in contexts to reach the final responses. Experiments show that G 2 -Reasoner significantly enhances LLMs' causal reasoning capabilities towards level -2, especially on fresh even fictitious tasks (e.g., CausalProbe 2024), which is consistent for both open-source and closed-source LLMs.

2 Related Work

In this section, we review the related works, including LLMs' reasoning, LLMs' causal reasoning, and LLMs' causal reasoning benchmarks. In Appendix D, we discuss the related works in detail.

2.1 Reasoning in Large Language Models

Reasoning ability is crucial to LLMs' performance on tasks such as theorem proving, problemsolving, and robotics [41, 58]. [58] propose a comprehensive review of the foundation models' reasoning, they discuss reasoning tasks including commonsense reasoning, mathematical reasoning, logical reasoning, and causal reasoning. Commonsense reasoning refers to the reasoning process that utilizes commonsense and daily life experiences [9]. Several commonsense question-answering datasets have been proposed to test LLMs' commonsense reasoning ability [19, 29, 74, 75]. Causal reasoning is the process of identifying and understanding the cause-and-effect relationships between variables or events, which involves identifying potential causes and effects within a system or context [58]. One distinctive feature of causal reasoning is that it involves counterfactual reasoning, which means reasoning within a hypothetical scenario. Our work focuses on the LLMs' causal reasoning, particularly counterfactual reasoning.

2.2 Causal Reasoning in Large Language Models

Causal Reasoning tasks include causal discovery, cause attribution, and causal effect estimation [30, 58, 65]. Causal discovery aims to recover the latent causal structure of variables. Cause attribution refers to uncovering potential causes behind a process, while causal effect estimation aims to investigate the effect of cause variables [26, 76]. While LLMs already show the ability to uncover causal relationships from context, their abilities have some limitations [37, 4, 39, 77]. Counterfactual reasoning is an essential task of causal reasoning. The difference between counterfactual reasoning and other causal tasks is counterfactual reasoning involves reasoning in hypothetical scenarios [28, 58]. Several studies conclude that LLMs have limitations when encountering hypothetical scenarios in counterfactual reasoning [33, 66].

Jack learned the new programming language at home because he was not able to go to school, which was closed because of the rain.

rain

school closure

cannot go to school

learn...at home

rain

school closure

cannot go to school

learn...at home

(a) ground truth causal relationship

(b) sequential causal relationship

2.3 Causal Reasoning Benchmarks in Large Language Models

There have been extensive studies on causal reasoning benchmarks. Existing causal reasoning benchmarks are mainly causal question-answering datasets. [6] employs language rules to extract causal questions from ten large question-answering datasets to form the CausalQA. CRAB [50] is a dataset that aims to assess LLMs' abilities to understand causal relationships among real-world events. FCR [69] is a human-labeled dataset that includes 24K question-answering pairs. Cladder [57] is a dataset that involves symbolic questions and corresponding ground truth answers, [57] employs causal graphs and structural causal models to generate the dataset. CausalProbe 2024 is different from the above benchmarks, as its contents are based on the latest and authoritative information, which is unlikely to be encompassed by the pre-training corpora of LLMs.

3 Problem Formalization

In this section, we introduce and clarify the necessary definitions used in this work. First, we provide a formal definition for causal reasoning in the context of LLMs. Then, we introduce two levels of causal reasoning capability to reveal the limitations of LLMs in this aspect. Last, we use causal language to depict the causal reasoning of LLMs.

In this work, the scope of causal reasoning of LLMs we study is reasoning about causal knowledge in textual form, distinguish from the numerical form in statistical causal inference [23].

Definition 1 (causal reasoning in LLMs) . In the context of large language models, the causal reasoning consists of two aspects:

comprehend the given contexts and discern the causal relationship within them;

responds to the causality-related queries, obeying the contexts and objective laws of causality.

To reach the major conclusions of this work, we first categorize LLMs'causal reasoning capability into two levels, motivated by the results of cognition science [56] and causality science [17].

Definition 2 ( level -1 causal reasoning) . Level1 causal reasoning involves retrieving causal knowledge embedded in model parameters and contextual information. This form of reasoning is typically fast and well-suited for handling simple cause-and-effect relationships.

Definition 3 ( level -2 causal reasoning) . Level2 causal reasoning leverages sophisticated reasoning mechanisms and internal parametric knowledge and contexts to deduce causal knowledge, including new/unseen causal knowledge. This form of reasoning is typically slow and capable of deriving new causal knowledge.

The above two definitions are inspired by 'Thinking, Fast and Slow' [27]. Level -2 causal reasoning is not necessarily always better than level -1. Ideally, the interplay and adaptive switching and of these two levels of causal reasoning are crucial for LLMs to work both rapidly and reliably. In this work, we aim to explore the causal reasoning capabilities of current LLMs in terms of these two levels and dig into the underlying reasons.

Remark 1. In this work, we only consider a simple type of causal reasoning tasks that contain a single cause-effect pair. The cases of multiple cause-effect pairs and mediators are excluded. In addition, we primarily consider qualitative causal reasoning tasks (e.g., causal discovery [4]), rather than quantitative ones (e.g., treatment effect estimation [25]). In summary, the causal reasoning tasks that we focus on can be categorized into two type: 1) what is the reason · · · ; 2) what is the result · · · . Due to the complexity of natural language, there are many different sentence patterns to express it.

"id": 1,

"question": "What is the result of

the surge in interest in artificial

intelligence (AI) in terms of media

coverage?",

"choice_1": "Increased

scrutiny...",

"choice_2": "...",

"answer": 2

causal question

Large language model

goal-oriented prompt

You are an intelligent causal reasoner. To arrive at the correct

answer, keep carefully analyzing the available information and

logically inferring the most probable causal relationship during

reasoning. Related general knowledge can be a reference if useful.

General knowledge

: [retrieved information]

Context (if available)

: [background of the question]

Question:

…

Choices

: 1. xxx 2. xxx 3. xxx

Pick one choice. Answer: [choice ID]

general

knowledge base

vector

database

retriever

embedding

model

Knowledge

ID

embedding model

C

X

Y

T

C

world

knowledge

X

cause

Y

effect

T

natural

language

4 LLMs cannot Perform Genuine Causal Reasoning

In this section, we aim to explore the real causal reasoning capability of current LLMs, in terms of our pre-defined two capability levels: level -1 (Definition 2) and level -2 (Definition 3). We study this problem from both a methodological perspective and an empirical perspective.

4.1 Autoregressive LLMs are not Necessarily Causal

Although autoregressive LLMs have achieved great successes, recently, they began to be doubted by this community [34, 42]. We aim to study why the autoregressive mechanism prevents LLMs from acquiring level -2 causal reasoning capabilities. Essentially, the decoder-only transformer-based LLMs that are widely used today to perform next-token prediction, are trained with an autoregressive loss [35]. In statistics, autoregression model is based on a fundamental assumption: in a sequence, the current value is determined by past values, not related to future values. However, causal knowledge expressed through natural languages does not necessarily satisfy this assumption. This is because sequential causality is not equivalent to logical causality [21]. For example, due to the variability of natural language, sentence patterns may lead to false sequential causal relationships, and Figure 2 shows a toy instance. This sentence consists of four concepts, i.e., ' rain ', ' school closure ', ' cannot go to school ', and ' learn the new programming language at home '. However, we can easily find that the sequential causal relationship in this sentence is totally incorrect. Thus, autoregressive LLMs suffer from capturing logical causal knowledge in complex texts, restricting their generalization abilities on unseen tasks. Note that due to the vast training corpus of LLMs, they may have encountered different textual expressions with the same semantic meaning, thus still being able to infer correct causal relationships in some complex tasks.

Assume that c = ( c 1 , c 2 , . . . , c k ) is a tokenized context and ( w 1 , w 2 , . . . , w t ) are already generated tokens, LLMs can obtain a distribution of the next token w t +1 : P ( w t +1 | c , w 1 , w 2 , . . . , w t ) , which is usually obtained by a softmax function. Then, LLMs predict the next token from this distribution in a sampling method, such as greedy sampling and beam search. Autoregressive training makes LLMs memorize common causal knowledge expressions. Based on the above discussion, there are two major issues: 1) If the context c is not sequentially causal and unfamiliar for LLMs, they tend to misunderstand the causal knowledge in c ; 2) If P ( w ∗ t +1 | c , w 1 , w 2 , . . . , w t ) is large 4 but the text represented by ( . . . , w t -1 , w t , w ∗ t +1 ) is inconsistent with the laws of causality or the context c , LLMs tend to respond an incorrect causal reasoning result. In contrast, if the context c and generated ( . . . , w t -1 , w t , w ∗ t +1 ) express correct causal knowledge and are familiar for LLMs, LLMs will perfectly address this task. Thus, the autoregression mechanism makes LLMs' causal reasoning primarily rely on correct causal knowledge in a large number of training corpora, i.e., the level -1 causal reasoning. In the following, we will validate this hypothesis from an empirical perspective.

4.2 Empirical Study on Causal Reasoning Capabilities of LLMs

In this section, we study the capability boundary of LLMs' causal reasoning for causal reasoning. Recall the hypothesis that LLMs are only capable of performing level -1 reasoning, primarily attributed to the causal knowledge embedded in their parameters during training, but struggle to perform genuine causal reasoning in complex or uncommon cases. Therefore, in order to physically avoid LLMs seeing the same corpora as the training data, we introduce a new causal question & answer (Q&A) benchmark, named CausalProbe 2024 (Figure 1(c)). The corpora used to construct CausalProbe 2024 are published later than January 1, 2024, which is later than the training data cut-off times of all the studied LLMs (Table 1). Thus, the studied LLMs should not have seen corpora that is the same or very similar to CausalProbe 2024. The construction details of CausalProbe 2024 are presented in Section 6.1. CausalProbe 2024 consists of three datasets: CausalProbe 2024 Easy ( abbr. , CausalProbe-E), CausalProbe 2024 Hard ( abbr. , CausalProbe-H), and CausalProbe 2024 Multiple-choice ( abbr. , CausalProbe-M). These three datasets progressively probe whether LLMs can solve novel causal reasoning tasks, whether they can distinguish misleading false causal relationships, and whether they can reason about multiple causal relationships within a scenario.

To validate whether LLMs heavily rely on the causal knowledge embedded in their parameters, we employ three earlier causal Q&A benchmarks as a comparison, i.e., COPA [48], e-CARE [14], and CausalNet [4]. The corpora of COPA is released earlier than 2011. An example of their comparison is shown in Figure 9. The corpora of e-CARE (i.e., GenericsKB [5]) is released earlier than 2020. CausalNet is also a fresh causal Q&A benchmark and its corpora are generated by ChatGPT, serving as an intermediate comparison. The corpora of COPA and e-CARE are likely to exactly be the training data of the studied models. For CausalNet, we have reason to speculate that its corpora may be similar to the training data of ChatGPT, although a large temperature hyperparameter can bring some creativity. Their detailed introduction in presented in Appendix F. In terms of the format, CausalProbe 2024 is consistent with them, including ID , question , choices , and answer . In addition, CausalProbe 2024 additionally provides contexts as the background knowledge of questions. In Section 6.1, we discuss the reasonability of providing contexts rather than only providing questions. 5 To doubly guarantee the freshness of CausalProbe 2024, we employ a LLM's membership inference attack method, MinK % Prob [53], to detect how much the corpora of a benchmark potentially comes from a LLM's training data. The results are shown in Table 3, showing that the corpora of CausalProbe 2024 are fresher for the studied LLMs than other benchmarks.

From Figure 1(d), we can observe significant performance drops on CausalProbe-E and CausalProbeH for all four studied LLMs. Especially for CausalProbe-H, Claude 3 opus, a current SOTA LLM, only achieves the average exact match less than 70%. The popular and competitive open-source LLM, LLaMA 2 7B chat, just get it half right. As the corpora of CausalProbe 2024 comes from news, it is close to everyday life and hardly consists of professional concepts and unfamiliar words. Thus, the main cause of performance degradation is the freshness of corpora, indicating the fact that LLMs only are capable of doing level -1 causal reasoning, instead of genuine level -2 causal reasoning. The empirical result is in perfect agreement with the analysis derived from the autoregressive mechanism.

5 G 2 -Reasoner: A General-Knowledge-Assisted and Goal-Driven Reasoner

Until now, we have discovered of the limitation of LLMs' causal reasoning capabilities from both methodological and empirical perspectives. To move towards level -2 causal reasoning, we seek inspiration from human beings. Human reasoning processes, including causal reasoning, are driven by specific reasoning tasks and supported by extensive foundational knowledge acquired throughout life, following established reasoning patterns such as deductive and inductive reasoning [44]. When

solving plane geometry proof problems, students apply three basic axioms as criteria while working toward the target proposition. Unlike humans, LLMs perform causal reasoning through next-token predictions based solely on patterns learned during training, without necessary knowledge to guide their reasoning. Drawing inspiration from causal inference, we formalize textual causal reasoning tasks 6 using a causal graph, as shown below.

Causal model for causal reasoning in LLMs. Based on Definition 1, we use a causal graph (Figure 4) to formally depict the textual causal reasoning task of LLMs. Based on this formalization, we can identify the key factors for LLMs to perform causal reasoning well. Given a cause semantic variable X and an effect semantic variable Y , naturally, we have X → Y . The fact that X causing Y is driven by the laws of the physical world or imagined/virtual worlds C . Then, the semantic variables X and Y are transformed into a variable T that represents the natural language, through a mapping h , i.e., h ( X,Y,ϵ ) = T , where ϵ is a random exogenous variable. In this formalization, h can be viewed as a language system to transform two causal concepts into a paragraph of text. The variable ϵ represents various factors in generating readable text from causal concepts X and Y , such as language type, context, and mode of expression (e.g., active or passive voice). While linguistic variability ( ϵ ) enriches the diversity and flexibility of natural language, it poses challenges for LLMs' causal reasoning. Consider the causal relationship between "smoking" ( X ) and "lung cancer" ( Y ). This relationship can be expressed in various ways, such as: 1) "A history of smoking is a common risk factor for lung cancer," and 2) "Knowing that smoking greatly increases the risk of lung cancer, why take the risk?" Although these sentences convey the same underlying causal relationship, they differ significantly in their linguistic structure and expression (i.e., different ϵ ). The natural language expression T encapsulates both the fundamental causal relationship between X and Y , as well as the contextual nuances and linguistic style represented by ϵ .

We can easily find that the causal graph (Figure 4) is exactly a causal model with a confounding variable and a conditioned collider [46]. T is conditioned because the observed natural language descriptions in textual causal reasoning tasks are inherently deterministic. Following a fundamental principle of causal inference, when the collider T is conditioned ( T = T 0 ), it creates an association between X and Y . Thus, natural language descriptions of causal reasoning tasks provide valuable information for determining causal relationships. 7 For cause-to-effect tasks, given P Y (the distribution of possible effects Y generated by LLMs), our objective is as follows:

where X 0 is the cause described in a causal reasoning question. However, to account for the confounding variable C , we can apply the total probability formula, i.e.,

where P C is the general knowledge base. While strictly ensuring the validity of Eq. (2) would require a complete general knowledge base, which is impractical, Eq. (2) nonetheless provides an insightful approach to addressing causal reasoning problems in LLMs and is helpful in practice (Section 6.3).

Goal-driven prompt. As discussed in Section 4.1, the autoregressive nature of LLMs hinders their understanding of correct causal relationships. As generated sequences lengthen, LLMs tend to lose coherence and deviate from their initial targets [40]. To maintain focused generation, we design a goal-driven prompt that guides LLMs in identifying correct causal relationships during the decoding process, as shown in Figure 3.

Motivated by human reasoning mechanism and causal graph theory, we propose a novel causal reasoning framework (Figure 3), named G 2 -Reasoner, which involves general knowledge as a basis and intended goals as a guide. Specifically, G 2 -Reasoner leverages a small ( ∼ 16 Mb) general knowledge Q&A dataset 8 as the knowledge base, enabling the model to draw upon related knowledge

as a reference when performing causal reasoning tasks. Specifically, when querying LLMs a causal reasoning question, we first retrieve related general knowledge from a vector database constructed with the general knowledge dataset, through the pipeline of retrieval-augmented generation (RAG). Using the retrieved general knowledge as a basis, we employ the proposed goal-oriented prompt to steer LLMs to perform causal reasoning in a targeted manner, rather than aimlessly generating answers. This is the first step taken to advance LLMs towards level -2 causal reasoning.

6 Experiments

In this section, we will discuss two points: 1) the construction of the CausalProbe 2024 benchmark and its superiority; 2) the main implementation details ; 3) the result analysis of G 2 -Reasoner and baseline methods. The CausalProbe 2024 benchmark and the source codes are presented in this URL: https://github.com/Haoang97/CausalProbe-2024 .

6.1 Construction of CausalProbe 2024 and its superiority

We will introduce the construction procedure of the CausalProbe 2024 benchmark and provide a brief analysis. In general, the procedure contains two stages: 1) collecting the latest web articles to form a corpora; 2) employing an LLM to generate the question-answer pairs from the corpora.

Corpora collection. To ensure the quality of the collected corpora, we choose two well-known media: British Broadcasting Corporation (BBC) and the Guardian. 9 BBC [Link] is a British public service broadcaster founded in 1922, one of the oldest and largest broadcasting companies in the world. The Guardian [Link] is a national newspaper in the UK, established in 1821, with its online edition being particularly influential. We downloaded the latest articles from BBC and the Guardian. Specifically, the downloaded articles are from January 1, 2024, to April 29, 2024 , later than the releases of the studied LLMs (Figure 1). The articles cover categories such as technology, environment, business, health, world news, culture, and climate, statistics of the downloaded articles are shown in Table 10.

Bechmark construction. We generated two sub-datasets from the same corpora using two different strategies. Since it's expensive to create them manually, we employ GPT 3.5 turbo as an assistant to automatically generate them. The overall pipeline diagram is shown in Figure 8.

CausalProbe-E . We construct this dataset following the format of the CausalQA [6]. In detail, we first query the GPT 3.5 turbo to analyze an example data in CausalQA, then we offer GPT 3.5 turbo articles and ask it to generate Q&A pairs similar to the example data. Finally, the highest-quality Q&A pairs are selected as the reference, which is adopted in the prompt template of generating Q&A data. The related prompt templates are shown in Figure 14a, 14d.

CausalProbe-H . We construct CausalProbe-H with another strategy. Specifically, first, we provide GPT 3.5 turbo an example article and ask it to summarize this article; second, we ask it to extract several cause-effect pairs based on the summary; third, we ask it to create some incorrect cause-effect pairs regarding this summary; finally, multiple multi-choice questions are generated according to these correct and incorrect cause-effect pairs. The highest-quality multi-choice Q&A data is selected as the reference, which is adopted in the prompt template of generating Q&A data. The related prompt templates are shown in Figure 14. The made-up fake cause-effect pairs can be used to examine the LLMs' genuine causal reasoning capability when encountering counterfactual disturbance term.

CausalProbe-M . We construct an uncertain multiple-choice version, with the number of correct answers for each question ranging from 1 to 4. LLMs are required to distinguish the correctness or incorrectness of each causal proposition, avoiding LLMs relying on random guess to get correct answers. We provided GPT-4o mini 10 the prompt templates of CausalProbe-H and additional prompt to realize varying number of answers, shown in Figure 14. To ensure each question really has at least one correct answer, we manually checked the correctness of the provided answers.

Due to the broad range of topics in the corpus, it may contain ethically questionable content such as conflicts. Therefore, we used LLMs to filter the unethical questions from the preliminary generated data. Moreover, to comply with the context length limit of GPT 3.5 turbo, we removed articles exceeding 15,000 characters. After filtering, our dataset contains 3,461 unique Q&A data. We also provide statistical analysis and more details of CausalProbe 2024 in Appendix I.

Superiority of CausalProbe 2024. 1) Contextual information. As we have simply mentioned, earlier three causal Q&A benchmarks lack the necessary background knowledge for each question. Although these benchmarks greatly promote the development of LLMs' causal reasoning, their formats are a little bit unreasonable. Even for humans, when we perform reading comprehension tests, we also need full context as a reference. Although we find that many of these questions can be answered by us, this heavily relies on our knowledge reserve, which is similar to LLMs. The knowledge reserve contributes their level -1 causal reasoning capability. Thus, we involve the contexts into CausalProbe 2024 as briefly as possible, which is more realistic, and they significantly improves the performance (Figure 5 and 6). 2) Hierarchical capability assessment. Three datasets of CausalProbe 2024 hierarchically assess causal reasoning ability levels. CausalProbe-E assesses the genuine causal reasoning ability on novel problems. Furthermore, CausalProbe-H assesses the ability to identify misleading or deceptive causal propositions in new tasks. Last, CausalProbe-M assesses the ability to identify multiple valid causal statements in novel tasks, greatly ensuring that LLMa cannot obtain right answers through random guessing. By evaluating LLMs on these three datasets in sequence, we can determine their actual level of causal reasoning ability.

Guarantee the freshness of CausalProbe 2024. To doubly guarantee the freshness of CausalProbe 2024 than earlier benchmarks, we use an LLM's training data detection method, MinK %Prob [53], to evaluate them as mentioned before. The detailed introduction for this method is in Appendix E. As the APIs of GPT and Claude no longer provide the logarithmic likelihood input texts, we only make this evaluation on open-source LLMs. The results in Table 3 show that CausalProbe 2024 is fresher. Here K denotes the tokens in the bottom K percent of log-likelihood.

Weemploy several measures to control the data quality of CausalProbe 2024. The detailed information is shown in Appendix H.

LLaMA 2 7B

LLaMA 3 8B

GPT 3.5 turbo

Claude 3 opus

LLMs

0

10

20

30

40

50

60

70

80

Average exact match

w/o context

w/ context

LLaMA 2 7B

LLaMA 3 8B

GPT 3.5 turbo

Claude 3 opus

LLMs

0

10

20

30

40

50

60

70

80

Average exact match

w/o context

w/ context

LLaMA 2 7B

LLaMA 3 8B

GPT 3.5 turbo

Claude 3 opus

0

10

20

30

40

50

60

70

80

90

Metric value

Exact match

Partial match

6.2 Implementation Details

All the experiments are conducted on the Ubuntu 20.04 system and NVIDIA RTX A6000 GPUs. For closed-source LLMs, i.e., GPT 3.5 turbo and Claude 3 opus, we call the API provided by their officials. We provide more implementation details in Appendix C.

6.3 Result Analysis

To evaluate G 2 -Reasoner, we compare it with three common LLM reasoning methods, i.e., vanilla, chain-of-thought (CoT) [63], and retrieval-augmented generation (RAG) [32]. The 'vanilla' refers to directly perform zero-shot inference. We evaluate four LLMs on CausalProbe 2024 and other three causal Q&A benchmarks, whose detailed introduction is presented in Appendix F. The full results are shown in Table 2. First, the LLMs' performance decreases monotonically as the benchmark corpora become more fresh. A counterintuitive phenomenon is that CoT usually perform a little worse than vanilla on three earlier benchmarks. This may indicate that CoT cannot improve reasoning on simple or common tasks for LLMs. For G 2 -Reasoner, it mostly can outperform the vanilla method. However, limited by the scale of the general knowledge base and the power of vector databases, G 2 -Reasoner cannot significantly outperform the baselines, which is the direction of our future efforts. In fact, RAG is just the ablated G 2 -Reasoner, and it usually cannot reach the vanilla method, indicating the effectiveness of our goal-oriented prompt.

We evaluate the four LLMs on CausalProbe-M, where they showed a more significant performance decline compared to their results on CausalProbe-E and -H. Under exact match, which required all correct answers to be precisely identified, all models struggled. However, when using partial match, where missing some correct options was acceptable but selecting incorrect options was not, GPT and Claude performed relatively well, achieving accuracy rates of approximately 75% and 85% respectively. While these results demonstrate that current LLMs cannot fully comprehend each causal proposition, revealing limitations in their causal reasoning abilities, there is an encouraging finding: the models rarely make false positive errors when identifying causal relationships.

Note that G 2 -Reasoner's performance relies on general knowledge bases. The reported results were obtained using a very small knowledge base (around 16 MB), yet G 2 -Reasoner generally achieved non-marginal improvements. If we use a significantly larger one, such as Wikipedia API, performance can be boosted a lot. However, due to resource constraints, we couldn't repeat all experiments with it.

7 Conclusion and Future Outlook

This work investigates the causal reasoning capabilities of LLMs and argues that current LLMs are limited to level -1 causal reasoning. To verify this hypothesis, we introduce a new causal Q&A benchmark, CausalProbe 2024, revealing that LLMs struggle with causal reasoning in unseen contexts and primarily rely on the causal knowledge in training data . To fill this gap, we proposes G 2 -Reasoner, a method that incorporates general knowledge and goal-oriented prompts into causal reasoning of LLMs. Experiments demonstrate that G 2 -Reasoner can enhance the causal reasoning capability, particularly in fresh and counterfactual contexts. This work provides valuable insights into the current state of LLMs' causal reasoning and offers a promising attempt to move towards level -2 causal reasoning, bringing LLMs closer to reaching genuine causal reasoning capabilities. While this work takes an important step forward, it still does not enable LLMs to achieve level -2 causal reasoning. Further research in this field could potentially lead to significant advancements towards stronger artificial intelligence.

References

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. Palm 2 technical report. arXiv , 2305.10403, 2023.

Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://www.anthropic. com/news/claude-3-family , 2024.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In ICLR , 2024.

Swagata Ashwani, Kshiteesh Hegde, Nishith Reddy Mannuru, Mayank Jindal, Dushyant Singh Sengar, Krishna Chaitanya Rao Kathala, Dishant Banga, Vinija Jain, and Aman Chadha. Cause and effect: Can large language models truly understand causality? arXiv preprint arXiv:2402.18139 , 2024.

Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. Genericskb: A knowledge base of generic statements. arXiv preprint arXiv:2005.00660 , 2020.

Alexander Bondarenko, Magdalena Wolska, Stefan Heindorf, Lukas Blübaum, AxelCyrille Ngonga Ngomo, Benno Stein, Pavel Braslavski, Matthias Hagen, and Martin Potthast. Causalqa: A benchmark for causal question answering. In Proceedings of the 29th International Conference on Computational Linguistics , pages 3296-3308, 2022.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS , 2020.

Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya K. Singh, Pierre H. Richemond, James L. McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In NeurIPS , 2022.

Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators. Advances in Neural Information Processing Systems , 36, 2024.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex

Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. Language models show human-like content effects on reasoning tasks. arXiv preprint arXiv:2207.07051 , 2023.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234 , 2023.

Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, PierreEmmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281 , 2024.

Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. e-care: a new dataset for exploring explainable causal reasoning. In ACL , 2022.

Jinglong Gao, Xiao Ding, Bing Qin, and Ting Liu. Is ChatGPT a good causal reasoner? a comprehensive evaluation. In Findings of EMNLP , 2023.

Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 , 2024.

Alison Gopnik and Laura Schulz. Causal Learning: Psychology, Philosophy, and Computation . Oxford University Press, 04 2007.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence. arXiv preprint arXiv:2401.14196 , 2024.

Willis Guo, Armin Toroghi, and Scott Sanner. Cr-lt-kgqa: A knowledge graph question answering dataset requiring commonsense reasoning and long-tail knowledge. arXiv preprint arXiv:2403.01395 , 2024.

Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In EMNLP , 2023.

David Hume. A treatise of human nature . Clarendon Press, 1896.

Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. In ACL , 2023.

Guido W. Imbens and Donald B. Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction . Cambridge University Press, 2015.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118 , 2021.

Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. CLadder: Assessing causal reasoning in language models. In NeurIPS , 2023.

Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona T. Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation? In ICLR , 2024.

D. Kahneman. Thinking, Fast and Slow . Farrar, Straus and Giroux, 2011.

Daniel Kahneman and Dale T Miller. Norm theory: Comparing reality to its alternatives. Psychological review , 93(2):136, 1986.

Mayank Kejriwal, Henrique Santos, Ke Shen, Alice M Mulvehill, and Deborah L McGuinness. A noise audit of human-labeled benchmarks for machine commonsense reasoning. Scientific Reports , 14(1):8609, 2024.

Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality. Transactions on Machine Learning Research , 2024.

Deanna Kuhn. The development of causal reasoning. Wiley Interdisciplinary Reviews: Cognitive Science , 3(3):327-335, 2012.

Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS , 2020.

Jiaxuan Li, Lang Yu, and Allyson Ettinger. Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios. arXiv preprint arXiv:2305.16572 , 2023.

Chu-Cheng Lin, Aaron Jaech, Xin Li, Matthew R. Gormley, and Jason Eisner. Limitations of autoregressive models and their alternatives. In NAACL-HLT , 2021.

Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In ICLR , 2018.

Ruinan Liu, Guisheng Yin, and Zechao Liu. Learning to walk with logical embedding for knowledge reasoning. Information Sciences , 667:120471, 2024.

Xiaoyu Liu, Paiheng Xu, Junda Wu, Jiaxin Yuan, Yifan Yang, Yuhang Zhou, Fuxiao Liu, Tianrui Guan, Haoliang Wang, Tong Yu, et al. Large language models and causal inference in collaboration: A comprehensive survey. arXiv preprint arXiv:2403.09606 , 2024.

Stephanie Long, Alexandre Piché, Valentina Zantedeschi, Tibor Schuster, and Alexandre Drouin. Causal discovery with language models as imperfect experts. arXiv preprint arXiv:2307.02390 , 2023.

Stephanie Long, Tibor Schuster, Alexandre Piché, Université de Montreal, ServiceNow Research, et al. Can large language models build causal graphs? arXiv preprint arXiv:2303.05279 , 2023.

Taiming Lu, Muhan Gao, Kuai Yu, Adam Byerly, and Daniel Khashabi. Insights into llm long-context failures: When transformers know but don't tell. In Findings of EMNLP , 2024.

Christopher D Manning. Human language understanding & reasoning. Daedalus , 151(2):127138, 2022.

R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638 , 2023.

Philipp Mondorf and Barbara Plank. Beyond accuracy: Evaluating the reasoning behavior of large language models - a survey. arXiv preprint arXiv:2404.01869 , 2024.

Allen Newell, Herbert Alexander Simon, et al. Human problem solving , volume 104. Prenticehall Englewood Cliffs, NJ, 1972.

OpenAI. GPT-4 technical report. arXiv , 2303.08774, 2023.

Judea Pearl. Causality . Cambridge University Press, 2 edition, 2009.

David Premack. Human and animal cognition: Continuity and discontinuity. Proceedings of the National Academy of Sciences , 104(35):13861-13867, 2007.

Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI , 2011.

Angelika Romanou, Syrielle Montariol, Debjit Paul, Leo Laugier, Karl Aberer, and Antoine Bosselut. CRAB: Assessing the strength of causal relationships between real-world events. In EMNLP , 2023.

Angelika Romanou, Syrielle Montariol, Debjit Paul, Leo Laugier, Karl Aberer, and Antoine Bosselut. Crab: Assessing the strength of causal relationships between real-world events. arXiv preprint arXiv:2311.04284 , 2023.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 , 2024.

Christopher Michael Rytting and David Wingate. Leveraging the inductive bias of large language models for abstract textual reasoning. In NeurIPS , 2021.

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In ICLR , 2024.

Steven Sloman. Causal Models: How People Think about the World and Its Alternatives . Oxford University Press, 08 2005.

Peiyang Song, Kaiyu Yang, and Anima Anandkumar. Towards large language models as copilots for theorem proving in lean. arXiv preprint arXiv:2404.12534 , 2024.

Keith E. Stanovich and Richard F. West. Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences , 23(5):645-665, 2000.

Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with language models. In ACL , 2023.

Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. A survey of reasoning with foundation models. arXiv preprint arXiv:2312.11562 , 2023.

Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. Can llms learn from previous mistakes? investigating llms' errors to boost for reasoning. arXiv preprint arXiv:2403.20046 , 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv , 2307.09288, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Yuxuan Wan, Wenxuan Wang, Yiliu Yang, Youliang Yuan, Jen-tse Huang, Pinjia He, Wenxiang Jiao, and Michael R Lyu. A & b== b & a: Triggering logical reasoning failures in large language models. arXiv preprint arXiv:2401.00757 , 2024.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS , 2022.

Moritz Willig, Matej Zeˇ cevi´ c, Devendra Singh Dhami, and Kristian Kersting. Can foundation models talk causality? In UAI 2022 Workshop on Causal Representation Learning , 2022.

Anpeng Wu, Kun Kuang, Minqin Zhu, Yingrong Wang, Yujia Zheng, Kairong Han, Baohong Li, Guangyi Chen, Fei Wu, and Kun Zhang. Causality for large language models. arXiv preprint arXiv:2410.15319 , 2024.

Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477 , 2023.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In ICLR , 2022.

Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrieval-augmented language models. Advances in Neural Information Processing Systems , 36, 2024.

Linyi Yang, Zhen Wang, Yuxiang Wu, Jie Yang, and Yue Zhang. Towards fine-grained causal reasoning and qa. arXiv preprint arXiv:2204.07408 , 2022.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS , 2023.

G Udny Yule. Why do we sometimes get nonsense-correlations between time-series?-a study in sampling and the nature of time-series. Journal of the royal statistical society , 89(1):1-63, 1926.

Matej Zeˇ cevi´ c, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting. Causal parrots: Large language models may talk causality but are not causal. Transactions on Machine Learning Research , 2023.

Cheng Zhang, Stefan Bauer, Paul Bennett, Jiangfeng Gao, Wenbo Gong, Agrin Hilmkil, Joel Jennings, Chao Ma, Tom Minka, Nick Pawlowski, and James Vaughan. Understanding causality with large language models: Feasibility and opportunities. arXiv preprint arXiv:2304.05524 , 2023.

Yunxiang Zhang and Xiaojun Wan. Situatedgen: Incorporating geographical and temporal contexts into generative commonsense reasoning. Advances in Neural Information Processing Systems , 36, 2024.

Danna Zheng, Mirella Lapata, and Jeff Z Pan. Archer: A human-labeled text-to-sql dataset with arithmetic, commonsense and hypothetical reasoning. arXiv preprint arXiv:2402.12554 , 2024.

LYU Zhiheng, Zhijing Jin, Rada Mihalcea, Mrinmaya Sachan, and Bernhard Schölkopf. Can large language models distinguish cause from effect? In UAI 2022 Workshop on Causal Representation Learning , 2022.

Yu Zhou, Xingyu Wu, Beicheng Huang, Jibin Wu, Liang Feng, and Kay Chen Tan. Causalbench: A comprehensive benchmark for causal learning capability of large language models. arXiv preprint arXiv:2404.06349 , 2024.

Supplement to 'Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?'

Content

A Limitations

In this section, we discuss the limitations of our work, which include two parts. The first limitation is that the proposed causal reasoning method for LLMs is only a step forward, but cannot realize the genuine causal reasoning. On the other hand, enabling LLMs to perform causal reasoning like humans is challenging and requires further research. The second limitation is that we can not fully confirm the CausalProbe 2024 is unseen for LLMs. Specifically, the contents of CausalProbe 2024 are still part of human knowledge, and LLMs may have seen comparable information. For example, LLMs may not know one latest event, but their pre-training data may include a similar event. Thus, fully excluding the pre-training data from datasets is challenging, and we already ensure the CausalProbe 2024 is unseen for LLMs to a large extent.

B Broader Impacts

In this section, we discuss both the potential positive societal impacts and negative societal impacts of our work. The positive societal impacts include that our work facilitates understanding LLMs' current causal reasoning abilities. Besides, our work is a step forward for improving LLMs' causal reasoning. The potential negative societal include that our work cannot promote LLMs' causal reasoning abilities to the human level, thus, LLMs' causal reasoning results may still have errors.

C Full Implementation Details

All the experiments are conducted on the Ubuntu 20.04 system and NVIDIA RTX A6000 GPUs. For closed-source LLMs, i.e., GPT 3.5 turbo and Claude 3 opus, we call the API provided by their officials.

Hyper-parameters. For LLM inference, we set the temperature parameter as 1 . 0 for closed-source LLMs all the time, and set it as 0 for open-source LLMs all the time. For CoT reasoning, we set the maximal new tokens as 128 , and we set it as 50 for all other cases.

Benchmark. All the used benchmarks are the .json or .jsonl format. If their original versions are not such formats, we convert them to .json or .jsonl format. For CausalNet, there exist a few missing values, and we have manually remove the invalid data. The filtered version of CausalNet will be uploaded in our anonymous link.

G 2 -Reasoner. G 2 -Reasoner uses the RAG system to involve external general knowledge. We use the Faiss package [13] to construct the vector database. We use the Meta's Contriever [24] as the information retriever to perform vector database retrieval. For the construction of RAG system, we refer to the realization of self-RAG [3]. For the external general knowledge base, we use the answers of a general knowledge dataset ( https://huggingface.co/datasets/MuskumPillerum/ General-Knowledge ) to form a document. We use Contriever and Faiss to embed and retrieve this general knowledge base.

Training data detection. We follow the official repository of MinK Prob to perform detection, without any modification. However, as the Text-DaVinci-003 seems to be deprecated and current GPT 3.5 and GPT 4 are no longer support returning the input's log likelihood, we have to only perform this detection on open-source LLMs. We will continue to explore how to perform on OpenAI's alternative.

D Related Work

In this section, we review the related works in detail, including LLMs' reasoning, LLMs' causal reasoning, and LLMs' causal reasoning benchmarks.

D.1 Reasoning in Large Language Models

Reasoning ability is crucial to LLMs' performance on tasks such as theorem proving, problem-solving, and robotics [41, 58]. Increasing interest in strong artificial intelligence has triggered debates on whether LLMs master reasoning abilities. [58] propose a comprehensive review of the foundation models' reasoning, they discuss reasoning tasks including commonsense reasoning, mathematical reasoning, logical reasoning, and causal reasoning. Commonsense reasoning refers to the reasoning process that utilizes commonsense and daily life experiences [9]. Rather than specialized knowledge, commonsense reasoning relies more on universally accepted knowledge. Commonsense question answering is an important method of commonsense reasoning test. Several commonsense questionanswering datasets have been proposed to test LLMs' commonsense reasoning ability [19, 29, 74, 75]. Different from commonsense reasoning, mathematical reasoning requires the ability to master symbolic forms and formal definitions. Automated theorem proving is a commonly used task to test LLMs' mathematical reasoning ability [58]. [55] propose a framework for LLMs' theorem proving, which helps humans to prove theorems. LeanDojo [68] is a mathematical reasoning benchmark

consisting of theorems and proofs. Logical reasoning refers the process of using formal logic principles and rules to make deductions, draw conclusions, and solve problems [36, 58]. It involves applying logical rules and syllogistic reasoning to evaluate the validity of arguments and propositions. [62] propose an automatic evaluation framework for LLMs' logical reasoning ability, and find LLMs have difficulty in performing logical reasoning well. [59] shows that LLMs can learn from mistakes in logical reasoning. Causal reasoning is the process of identifying and understanding the causeand-effect relationships between variables or events, which involves identifying potential causes and effects within a system or context [58]. One distinctive feature of causal reasoning is that it involves counterfactual reasoning, which means reasoning within a hypothetical scenario. [33] inspect LLMs' ability to distinguish between hypothetical and real-world scenarios. Our work focuses on the LLMs' causal reasoning, particularly counterfactual reasoning.

D.2 Causal Reasoning in Large Language Models

Causal Reasoning tasks include causal discovery, cause attribution, and causal effect estimation [30, 58]. Causal discovery aims to recover the latent causal structure of variables, which could be a structural causal model (SCM). Cause attribution refers to uncovering potential causes behind a process, while causal effect estimation aims to investigate the effect of cause variables [26, 76]. While LLMs already show the ability to uncover causal relationships from context, their abilities have some limitations [37]. [4] suggests that LLMs are struggling to perform causal discovery tasks in hypothetical scenarios. [39] find GPT-3 has limitations in uncovering the causal structure among medical context. [77] illustrate the boundaries of LLMs' performance on causal discovery. Moreover, [38] indicates that LLMs' expertise may contain errors, which could undermine causal reasoning. Therefore, it's essential to incorporate external expert knowledge into LLMs to enhance LLMs' causal reasoning abilities. Counterfactual reasoning is an essential task of causal reasoning. LLMs with high counterfactual reasoning abilities have the strength to make predictions under various circumstances. The difference between counterfactual reasoning and other causal tasks is counterfactual reasoning involves reasoning in hypothetical scenarios [28, 58]. For example, 'If cats were vegetarians, what results would happen?' is a counterfactual reasoning question [33]. Several studies conclude that LLMs have limitations when encountering hypothetical scenarios in counterfactual reasoning [33, 66].

D.3 Causal Reasoning Benchmarks in Large Language Models

There have been extensive studies on causal reasoning benchmarks. Existing causal reasoning benchmarks are mainly causal question-answering datasets. CausalQA [6] is a dataset containing 1.1 million causal questions and answers. [6] employs language rules to extract causal questions from ten large question-answering datasets to form the CausalQA. CRAB [50] is a dataset that aims to assess LLMs' abilities to understand causal relationships among real-world events. [50] extract real-world stories during the past ten years and create CRAB based on these real-world stories. FCR [69] is a human-labeled dataset that includes 24K question-answering pairs. [69] collect data from Yahoo and employ crowd workers to generate the causal questions. However, the data they collected are between December 2020 and July 2021, which may be included in the pre-training corpora of the latest LLMs (e.g. LLaMA 3). Cladder [57] is a dataset that involves symbolic questions and corresponding ground truth answers, [57] employs causal graphs and structural causal models to generate the dataset. CausalProbe 2024 is different from the above benchmarks, as its contents are based on the latest and authoritative information, which is unlikely to be encompassed by the pre-training corpora of LLMs.

E LLMs' Training Data Detection: Membership Inference Attack

In our work, we employ a pre-training data detection method [53] to confirm further that LLMs' pre-training data do not include our datasets to a large extent. MIN-K % PROB [53] is a pre-training data detection method. This method is based on a simple assumption: 'When encountering an unseen example, the large language model (LLM) is likely to find a few words with low probabilities, while words in a familiar example are less likely to be assigned such low probabilities." Let x = x 1 , x 2 , . . . , x n be a token sequence, and for one token x i ∈ x , the log-like probability given its preceding tokens can be calculated as:

Then, each token's log-like probability can be computed. Let MIN-K% ( x ) represent the set composed of the %K tokens with the lowest log probabilities among all tokens in sequence x . The average log-like probability of this set is:

where N is the number of elements in MIN-K% ( x ) . In the end, conditions on a threshold ϵ , if MinK %Prob ( x ) > ϵ , then this sequence of text x is determined not in pre-training data, otherwise x is determined in pre-training data. One advantage of this method is that it does not need any information about the pre-training data nor does it need to train a reference model in advance.

F A Detailed Introduction of Used Benchmarks

F.1 CausalNet

The CausalNet [4] is an LLM-generated dataset designed to support studies in causal reasoning and counterfactual analysis. With 1000 meticulously selected scenarios, CausalNet offers a wide range of causal and counterfactual inquiries, helping researchers delve into the complexities of cause-and-effect dynamics across different contexts. Each question-answering pair in the CausalNet contains a background context, a causal-effect question, or a counterfactual question, and the choices and answers of the question. However, compared with CausalProbe 2024, the background context of the CausalNet is limited, which may not provide sufficient basic information for LLMs to answer the questions better.

F.2 COPA

The Choice Of Plausible Alternatives (COPA) evaluation [48] is a dataset including 1000 causal questions. The COPA aims to test LLMs' commonsense causal reasoning abilities. [48] employs an authoring methodology to secure broad topics, correct language, and high consensus among human evaluators. To be specific, they utilize several regulations to secure language clarity. Besides, they choose two different sources of questions that are confirmed broad to ensure the breadth. However, the articles they selected are mainly from August and September of 2008, which may be encompassed by LLMs' pre-training corpus.

F.3 e-CARE

The CAusal REasoning dataset (e-CARE) [14] is a crowding-sourcing-based dataset, which contains 21K multi-choice causal reasoning questions. This dataset not only tests LLMs' ability to choose the correct causal statements but also examines the LLMs' ability to explain their choice. The eCARE includes 21324 question-answering pairs and 13048 corresponding explanations. [14] collect statements of world knowledge and ask crowd workers to generate the causal facts according to the instructions. Then, the causal questions are generated by crowd workers based on these causal facts. However, LLMs' pre-training corpus may contain the collected statements about world knowledge.

G Additional Results

G.1 Standard Deviations of Main Results

Due to the space limitation, we present the standard deviations of the results in Table 2 here. For two closed-source LLMs, i.e., GPT 3.5 turbo and Claude 3 opus, we find that setting the temperature value to 1.0 can achieve slightly better results than 0. We repeat these experiments three times, and the average results together with their standard deviations are shown in Table 6. The standard deviations are generally small.

H Quality Control

We have employed several measures to control the quality of CausalProbe 2024.

User

[Example], ask for cause

, …

{Choice_i}

Can you understand?...

It seems [Cause

]…

Answer

[Article], I need summary,

…, create similar

questions

and Answer.

[Summary

]…

[QA pair]:

{Question_i ; Answer_i}

Question

Example

Summary + QA

Article

Causal

Benchmarks

Step 1: Example Illustration

Step 2: Question Generation

GPT

User

GPT

Step 1: Article Summary

Step2: Correct Causal Pairs Extrcation

Step3: Incorrect Causal Pairs Making Up

Step4: Question Generation

User

[Article], please summarize

, …

[Summary

]…

GPT

User

List the cause and effect pairs in this [Summary

], …

[Causal-effect pair]:

{Causal_i ; Effect_i}

GPT

User

Can you also make the false cause and effect pairs

using this [Summary]?...

{Causal_i ; Effect_i}

GPT

User

Can you make a multiple-choice test using [Causal-

[QA pair]:

{Qusetion_i ; Answer_i}

GPT

CausalProbe-E

Raw Matetia

CausalProbe-H

User

[Prompt Template of

CausalProbe-H

], …

Remember, the number of

correct answers is NOT

fixed...

[QA pair]:

{Question ;

Correct_Answer_1

Correct_Answer_2...

False_Answers

}

[QA pair], manually verified

the number of correct

answers is not zero.

Step 1: Question Generation

Step 2: Answer Verification

GPT

User

CausalProbe-M

Preparation : Our corpus is sourced from two famous media with high quality. We further performed an initial cleaning of the corpus using regular expressions. Subsequently, we used the Google DLP API to detect sensitive information (such as pornography, violence, advertisements, etc.) in the corpus and removed any violating content.

Production : We use GPT 3.5 Turbo, one of the best LLMs at the time, to construct the benchmark from the prepared corpus. To improve its quality, we experimented with different prompt templates and adopted the best.

Verification : We write Python scripts to exclude incomplete/garbled items. Then we re-organize them to Json format for ease of reading. Finally we went through all items manually to find out problematic ones and excluded them.

We have made additional efforts to enhance the quality following a crowdsourcing pipeline. We recruited 17 volunteers, all of whom hold a master's degree or higher and are currently engaged in frontline research. Additionally, all volunteers are fluent in English. In the preliminary tests, we randomly sampled 20 questions from CausalProbe-H and asked each volunteer to answer them. We then recorded each volunteer's answer and their perceived difficulty level about this task (on a scale of 1-10, with 10 being the most difficult). The selection criteria required the perceived difficulty level to be no more than 7 and the accuracy rate to be no less than 80%. Ultimately, 13 out of the 17 volunteers met these criteria and were selected as qualified, and the test results are shown in Table 4.

Given the limited time and human resource, we performed quality control on a subset of CausalProbe 2024. Specifically, we randomly sampled 260 questions from CausalProbe-H and assigned each question to 3 volunteers randomly, using the Algorithm 1. Each volunteer received a total of 60 questions. After receiving their feedback, we treated those questions correctly answered by no less than two volunteers as high-quality data. Finally, 232 out of 260 questions were filtered out (temporarily called CausalProbe-HQ), achieving the qualification rate of 89.2%. Next, we use CausalProbe-HQ to evaluate four LLMs used in our paper again, whose results are shown in Table 5. The results show that all four LLMs still perform poorly on this subset, suggesting that their failure is primarily due to limited causal reasoning abilities rather than the errors in the benchmark. We will continuously perform quality control on all data through the aforementioned crowdsourcing pipeline and update the quality-controlled data on our GitHub repository.

Algorithm 1 Question assignment algorithm

I Additional Introduction and Analysis of CausalProbe 2024

I.1 Statistic of CausalProbe 2024

CausalProbe 2024 consists of 6,922 Q&A data, including two sub-datasets, each containing 3,461 Q&A data. Each sub-dataset covers topics such as technology, culture, business, climate, and world news. The statistic of CausalProbe 2024 in terms of the topics is shown in Figure 11. Moreover, all questions can be categorized into 'asking for cause' and 'asking for effect,' thus examining the LLMs' ability to reason about causes and effects in questions, respectively.

For CausalProbe-M, we visualize the distribution of the number of correct options in its Q&A data, shown in Figure 12. We find that two and three correct options occupy the majority and the distribution is simialr to be Gaussian distribution. We also visualize the distribution of query types (i.e., asking for cause or effect) in Figure 13.

I.2 Addition Analysis

In this subsection, we discuss how the CausalProbe dataset tests the causal reasoning ability of LLMs. As discussed in Section 6.1, we collect the latest and authoritative web articles and utilize them to generate the dataset. To be specific, the downloaded articles are from January 1, 2024, to April 29, 2024, later than the releases of many LLMs. Consequently, these latest articles are unlikely to be directly incorporated into the pre-training corpus of some open-source LLMs (e.g. LLaMA 2 and LLaMA). Afterward, the questions generated by these articles are probably not included in the pre-training corpus of many LLMs. Therefore, LLMs are less likely to provide correct answers using their pre-training knowledge directly, which forces them to utilize their causal reasoning ability. Moreover, the questions cover a wide range of categories and encompass many real-world events that are highly discussable, which tests the causal reasoning ability of LLMs from multiple perspectives.

I.3 Potential bias

As known for us, any machine model has its own inductive bias [52]. Similarly, using GPT 3.5 turbo to build CausalProbe 2024 introduces potential biases. They mainly consist of the following types of bias:

Model bias : GPT-3.5 turbo may inherit biases and limitations from its own training data.

Generation bias : GPT-3.5 turbo may frequently produce certain types of questions or answers.

Language and cultural bias : Both CausalProbe 2024's English news corpus and GPT-3.5 Turbo's predominantly English training data may introduce Western or Anglophone biases.

To mitigate these potential biases, we propose the following measures:

Diversification of data sources : We can incorporate diverse corpus sources and more methods, such as manually created Q&A pairs or data generated by different LLMs.

Manual check : We can conduct a thorough human review and curation for CausalProbe 2024's biases.

COPA

"id": "1",

"asks-for": "cause",

"context": "My body cast a shadow

over the grass.",

"choice_0": "The sun was

rising.",

"choice_1": "The grass was cut.",

"answer": 0

e-CARE

"index": "train-0",

"premise": "There is a light rain

today.",

"ask-for": "effect",

"hypothesis1": "The roots of many

plants are not moistened by

rain.",

"hypothesis2": "Tourists have seen

many ripples.",

"label": 1

CausalNet

"index": "cause-effect-0",

"context": "A new policy in a country led to the unexpected decline in tobacco use among

teenagers. At the same time, there was an increase in online educational content about

health, and a popular celebrity spoke against smoking.",

"ask-for": "cause",

"label": 1,

"choice_id0": "The policy change had a minimal impact compared to other factors.",

"choice_id1": "The celebrity's influence changed teenagers' perceptions more significantly

than expected.",

"choice_id2": "Online educational content was the sole reason for this decline."

CausalProbe 2024

"id": 1,

"context": "Journalists are facing criticism for contributing to the hype surrounding

artificial intelligence (AI) and not accurately reporting on its capabilities and

limitations. The surge in interest in AI has led to increased media coverage, with some

experts calling for more balanced reporting that highlights both the positive and negative

aspects of AI.",

"question": "What is the result of the surge in interest in artificial intelligence (AI)

in terms of media coverage?",

"choice_1": "Increased scrutiny on AI by journalists.",

"choice_2": "More balanced reporting on the positive and negative aspects of AI.",

"choice_3": "Greater responsibility on the media to report on AI accurately.",

"choice_4": "Improved understanding of AI technologies by journalists.",

"answer": 3

Statisticsofthecorrectanswers

1600

1400

Number of questions

1200

1000

800

600

400 -

200

D

2

3

Numberofcorrectanswers

Categories of the Question Topics

12%

6%

25%

12%

31%

14%

■Science and TechnologyCultureBusiness World NewsHealth Others

Effect

19%

Cause

81%

(a) The prompt of summarizing articles.

(b) The prompt of extracting cause-effect pairs.

(c) The prompt of generating incorrect cause-effect

pairs.

(d) The prompt of generating causal Q&A data.

ArticleSummary

User

Airlineadvertsbannedover'greenwashing'claims

Adverts for Air France, Lufthansa and Etihad have been banned

for misleading consumers about the airlines' environmental impact.

Air France's ad said it wascommitted to protecting the

sustainably". Etihad’s ad used the words "environmental advocacy".

Please summarize this article.

Adverts from Air France, Lufthansa, and Etihad have been

GPT

banned for misleading consumers about their environmental

impact by the Advertising Standards Authority (ASA).

The UK government aims to mandate 10% sustainable

aviation fuel by 2030.

Causal PairsExtrcation

User

List the cause and effect pairs in this summary.

Sure,here are the cause and effect pairs:

GPT

1. Cause: Misleading environmental claims by Air France,

Lufthansa, and Etihad.

Effect:Banning of adverts by the Advertising Standards

Authority (ASA).

2.Cause:Previous criticism faced by Lufthansa for

greenwashing.

Effect: Removal of ads by Lufthansa and Etihad.

QuestionsGeneration

User

Canyou make amultiple-choicetest,using the cause-

effectpairs and false cause-effectpairs?

The answer to thismultiple-choice test shouldbe the

correctcause-effectpair.

Sure,here's a multiple-choice test:

GPT

What led to the banning of adverts by the Advertising

StandardsAuthority (ASA)?

A) Misleading environmental claims by Air France,

Lufthansa, and Etihad.

B) Previous criticism faced by Lufthansa for greenwashing.

C) Increased scrutiny on corporate environmental claims.

D) Requirement for airlines to reduce their carbon footprint.

Answers:

A) Misleading environmental claims by Air France,

Lufthansa, and Etihad.

Make Up False CausalPairs

User

Canyoualsomakethefalsecauseandeffect

pairs using this summary?

Of course,here are the false cause and effect pairs:

GPT

1.False Cause:Banning of adverts by the Advertising

Standards Authority (ASA).

False Effect: Misleading environmental claims by Air

France, Lufthansa, and Etihad.

2.False Cause: Removal of ads by Lufthansa and Etihad.

False Effect:Previous criticism faced by Lufthansa for

greenwashing.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately and roundly reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We discuss two limitations: 1) our proposed causal reasoning method for LLMs is only a step forward, but cannot realize the genuine causal reasoning; 2) we cannot strictly guarantee that our proposed CausalProbe 2024 benchmark is absolutely unseen for LLMs.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We clearly list the implementation details and provide the main codes and datasets through a anonymous github link.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide codes and data to reproduce the main results in this paper through an anonymous github link.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the test details necessary to understand the results. However, our paper does not include model training.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: This paper does not involve model training, and only invovle LLMs inference. We set the temperature to a very small value to ensure the preciseness of causal reasoning, and thus the inference results are stable.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes. We provide the compute workers, memory, time of execution in Appendix. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We strictly obey the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We specially discuss the broader impacts in a section. The positive societal impacts include that our paper help this community to correctly understand current LLMs' causal reasoning capability. The negative societal impacts include that we still cannot improve LLMs' causal reasoning capability to reach the human level, thus the reasoning results may still be incorrect.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: To mitigate risks of unsafe content, the dataset was filtered to only include articles from reputable news sources, specifically the BBC and The Guardian. These news organizations have their own strict editorial standards and content moderation practices to avoid publishing explicit, harmful or illegal content.

However, as an added safeguard, the scraped text was automatically scanned for profane, explicit, or offensive language, and any articles containing such content were removed from the dataset. The dataset was also manually spot-checked to verify the automated filtering was effective at identifying and removing unsafe text.

While these measures significantly reduce the risk of unsafe content, it is still possible that some inappropriate text evaded detection. The dataset is intended only for research purposes by responsible parties. We emphasize that the dataset may still contain offensive language or content, and should be used with caution. By publishing this dataset, we do not endorse or take responsibility for any inappropriate content that may be present.

We believe making this dataset available to the research community, with these caveats and safeguards, provides benefits that outweigh the risks. However, we are open to feedback on additional steps we could take to further reduce the potential for misuse of this data. Responsible use and deployment of models trained on this data is crucial.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We list the copyright and the license and terms of use in our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide a document in the anonymize link.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing and crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is concerned with the question of whether LLM can perform causal reasoning on a human-like level by incorporating contextual information in their decision and answer process. The authors argue that many everyday causal inferences are not purely logical but take into account general knowledge and intention. Therefore, a distinction between 'shallow' (level-1) and 'human-like' (level-2) causal reasoning is proposed. Two versions of a novel  \"CausalProbe-2024\" causal Q&A data set for benchmarking level-2 reasoning capabilities of LLM are presented. The presented benchmark is sourced from recent news sources. The authors, therefore, claim that the presented data is unlikely to be contained in the training set of the tested LLM (LLaMa 2/3, GPT-3.5 Turbo, and Claude 3).\n\nTo tackle level-2 causal reasoning, the authors propose a theory on how cause-effect pairs get instantiated into natural language sentences and further argue that a general world context affects the instantiated cause and effect nodes. To reason in this kind of setting, the authors propose a \"$G^2$-Reasoner\" to incorporate general knowledge and inherent goals via additional context information retrieved from a general knowledge base.\n\nEvaluating multiple LLMs on the CausalProbe data set shows a strong deterioration in performance compared to earlier causal reasoning datasets. The proposed $G^2$-reasoner helps improve reasoning capabilities over a naive baseline approach and performs on par with Chain-of-Thought or retrieval-augmented generation approaches.\n\nSTRENGTHS:\nOverall, the author's proposal of assessing the causal reasoning capabilities of LLM in context is a valid contribution and improves in realism and difficulty over existing benchmarks. The work is well embedded into existing related work on the causal reasoning capabilities of LLMs. Relations to existing causal Q&A data sets are drawn, and previous examinations of LLM causal reasoning capabilities are discussed.\n\nThe novel data set is constructed by extracting information from recent news articles of two major news sources, covering events that lie after the information cut-off of all tested LLMs (after Jan 1st, 2024). To support the thesis of querying the LLMs on unseen information, LLama models are tested via the Min-K% Prob technique on whether the models might have memorized the collected information. Compared to older datasets, CausalProbe seems to be composed of more novel information than COPA, e-CARE, or CausalNet data sets.\n\nThe authors contribute a possible explanation of how causal information is instantiated in natural language via a high-level SCM. This involves a general context, drawing information from world knowledge, and incorporating the naturally occurring diversity of expressing such relations.\n\nThe experimental setup and individual steps of the $G^2$-reasoner are described clearly. Prompt templates are provided. The approach is compared to a 'vanilla' approach, Chain of Thought (CoT), and a retrieval-augmented generation (RAG), giving the LLMs access to a general knowledge base.\n\nWEAKNESSES:\nWhile I agree with the authors that benchmarking LLMs for causal relations in context is a more difficult task than pure logical causal reasoning, the paper remains vague on the particular effect that the additional context might impose in terms of the cause-effect pairs as discussed in Sec. 5 / Fig. 4.\n\n(1) Generally, the described distinction between level-1 and -2 causal reasoning (def. 2 and 3) seems to be about how LLMs draw information from either memorization (information embedded in the model weights) or dynamically from the input text. LLMs are known to have difficulties when reasoning over non-memorized facts. In this regard, the general shortcomings of LLMs are non-surprising and have already been presented in previous works (e.g., Kiciman et al., 2024; Jin et al., 2024; Zecevic et al., 2023). In that regard, the proposed $G^2$-reasoner seems to feature no explicit mechanism to help improve on the causal reasoning of LLMs other than providing general background knowledge. No alternative prompt variations or combinations with, e.g. CoT or fine-tuning, are presented to improve results.\n\n\n(2) My main concern is the quality of the collected data set itself. While the authors do clearly describe the automated process of extracting information from credible news sources, no human involvement in quality checking is mentioned. Given the more implicit nature of causal relations considered in this paper, many answers can not be directly inferred but need to be extrapolated from the texts (as intended by the authors). However, checking on the first ten entries of CausalProbe-H seems to reveal a generally poor quality of the samples. In detail, I get the impression that multiple answer options per sample can be valid. Given that only a single answer is indicated as 'correct' by the data set, I question the significance of the reported results. In detail:\n\n* id 3: correct marked answer 2 is implicit, but so could be 3 and 4.\n* id 4: 1, 2 and 4 are possible answers with partial contribution to the effect.\n* id 5 - 1 and 2 are both correct.\n* id 7 - 2 and 4 describe partial factors that could both be valid.\n\nThe main problem seems to stem from the unclear definition of what counts or does not count as a causal factor for a specific scenario. As a result, alternative answers --other than the one indicated by the dataset-- might contribute to the causal effect relation and, therefore, could be viewed as reasonable answer options. The authors might have either adopted an answer format capable of accepting multiple correct answers or might impose a more extensive quality checking to prevent the appearance of alternative valid answer options.\n\n\n(3) The partition of samples into causalProbe-S and -H is unclear to me. The authors mention that \"the highest-quality Q&A pairs are selected as the reference\" (l.289) but do not describe under which criteria this selection is done.\n\n\n(4) The authors mention in Appendix (C) \"Full Implementation details\" to have set the temperature to \"1.0 for closed-source LLMs [...], and set it as 0 for open-source LLMs [...].\". Non-zero temperature evaluations might introduce additional variance to the results. However, the authors do not quantify the possible variance in in their evaluation results.\n\n\nMinor comments:\n\n* Typo in the caption of table 3: \"represnt\".\n\n* While I agree in general with the discussion in Sec. 4.1 about causal event sequence and sequence of appearance, it dismisses possible generalization capabilities of LLMs. Claims 1) and 2) are provided without reference or experimental evidence. The authors might want to tune down these claims.\n\n* I would advise not captioning Fig. 2 (b) as a \"sequential causal relationship\" but rather as a \"sequence of appearance\" since --as the authors mention correctly-- word sequences do not need to imply any causal relations inherently.\n\n* Typo \"we proposes G2-Reasoner\" (l.340)\n\nQUESTIONS:\nMy questions mainly concern the weaknesses mentioned above. In detail, I would like the authors to comment on the following:\n\nI) Regarding (2): could the authors clarify the measures that were imposed to ensure the quality of the collected data samples? How do the authors view the possibility of multiple valid answers with regard to the proposed data set format and evaluation results?\n\n\nII) Regarding (1) and (2), I would like to ask the authors to clarify the expected effects of the context on the causal relation. In detail, could the authors comment (e.g., in the context of epistemic reasoning / modal logic) how 'reasonable/possible' and 'correct/necessary' answers would be distinguished in their setup and which implications would follow for their data set design?\n\n\nIII) Regarding (3): is the partition of the samples performed by human judgment or via some automated metric? Which criteria are used to judge sample difficulty?\n\n\nIV) Regarding (4): Can the authors quantify the possible randomness-induced variance for the closed-source models?\n\nLIMITATIONS:\nIn their paper, the authors correctly address the causal reasoning limitations of LLMs in general and their $G^2$-reasoner in detail.\n\nThe authors seem to have not performed or do not mention a human-backed review of the data set contents. The quality in terms of answer texts and evaluation results could be improved by establishing such a process.\n\nThe data set seems to be free of ethically questionable content; However, the applied filtering process was not disclosed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The authors proposed a new causal reasoning framework, to improve the causal reasoning capacity of LLMs, with inspiration drawn from causal graph theory and human reasoning process. The work utilized "general world knowledge" as a component to take a step closer to making LLMs perform a more human-like causal reasoning process. The authors also developed a new benchmark and carried out experiments evaluating how different LLMs perform on their benchmark.

STRENGTHS:
The paper is well-written and easy to follow. The authors draw inspiration from the human causal reasoning process and provide an analysis of why LLMs cannot perform "genuine" causal reasoning, from not only a methodological perspective but also an empirical perspective.

WEAKNESSES:
1. The key hypothesis of this work is that LLM is capable of "level-1" reasoning, but lacks the capacity of "level-2" reasoning. Given that this work is centered around this hypothesis, a clearer definition and illustrative example of level-1 and level-2 should be provided.
2. I suggest the authors provide an example of a "random exogenous variable" in line 238. I can understand what this variable is from the causal graph framework, but it would be helpful to better clarify it in your setting.
3. Can you provide a more detailed explanation of how equation (1) plays a role in your G2 reasoner, and how can you compute this term?

QUESTIONS:
see weaknesses.

LIMITATIONS:
The author didn't provide a separate limitations section. One limitation the authors mentioned in the paper is that they only consider causal reasoning tasks with single cause-effect pair. Another limitation is that their approach doesn't enable LLMs to achieve level-2 causal reasoning, but rather provides insights into it.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the causal reasoning capabilities of LLMs and argues that current LLMs are limited to shallow (level-1) causal reasoning. To support this claim, the authors introduce a new benchmark, CausalProbe-2024, which reveals that LLMs struggle with causal reasoning in fresh and unseen contexts. To address this, the authors propose G2-Reasoner, method that incorporates general knowledge and goal-oriented prompts to enhance causal reasoning capabilities. Experiments show that G2-Reasoner improves performance, particularly in novel and counterfactual scenarios.\n\nSTRENGTHS:\n- The paper goes beyond simple retrieval from LLM memory causal reasoning tasks. \n\n - They attempt to establish the sensitivity of the model for pretraining\n\n- The authors also propose a new benchmark CausalProbe-2024 to evaluate level-2 reasoning.\n\n- The authors identify fundamental limitation in the current architecture of LLMs as authoregressive next token prediction mechanism (which is intuitive though) for causal reasoning.\n\nWEAKNESSES:\n- The long-term viability of this approach is uncertain. Continuous advancements in model architecture and training techniques might be required to truly enable level-2 reasoning, and the proposed method might only be a temporary solution. However I understand that this paper still makes a good progress.\n\n- Unless I missed it, I did not find much detail on RAG since it is so knowledge-specific. It isnt always aplicable. Also the results dont show a big improvement for G2 reasoner. It then begs the question how dependent was the model the knowledge of RAG. \n\n- paper primarily addresses simple, single cause-effect pairs which is a bit limiting\n\n- the variances arent given\n\nQUESTIONS:\nJust a minute comment, for better readability, I would appreciate it if the authors could an identifier(eg bold) in the Tables 2,3 just to make it easier to read.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the autoregressive mechanism of the transformer-based LLMs and their ability to reason causally and introduces a new Q&A-based benchmark named CausalProbe-2024 with around 3k Q&A pairs (believed to be unseen by the existing LLMs). The paper further introduces a reasoning strategy that considers general knowledge and the goal of performing a task called G2reasoner. The proposed G2-reasoner is based on a RAG based framework that encodes the information related to general knowledge in the form of a vector database, and the goal orientation is simulated by modifying the prompt that steers an LLM to be an intelligent causal reasoner and provides the correct causal relationship between the events and reason about them. \n\nThe paper reports the empirical evaluation results in over 3 widely used benchmarks (COPA, e-CARE, and CausalNet) along with the two versions of the proposed Causal Probe dataset.\n\nSTRENGTHS:\n* The paper raises an interesting question regarding the causal reasoning in LLMs, and the research goal established by the paper with the underlying question regarding causality is timely. \n\n* The paper’s primary strength comes from creating a newly created benchmark for facilitating the study of causal reasoning in LLMs when compared to humans. Formulating a formal causal reasoning benchmark in a natural language is an extremely challenging task, and this work will take a small step toward making the causal reasoning evaluation effective. \n\n* The paper highlights the training data detection using MinK% Prob in Table 3 for the existing causal benchmarks/datasets, which will be helpful for future research. With evaluation and comparison with the widely used causal reasoning benchmarks, the paper highlights some findings for the open-weight and popular proprietary models.\n\nWEAKNESSES:\n* The definitions of level 1 and level 2 are not concretely stated. Though lines 48 to 51 vaguely describe it, and the expanded version of the definitions is present in section 3, line 126 to line 135, it would be better to make them more concrete. In the current version, the definition is not very clear when it talks about complex or unknown tasks. A clear distinction between the levels would help the reader/research community to explore it further in future works. \n\n* The results of the studied LLMs on four causal Q&A benchmarks, as shown in Table 2, highlight very little improvement of the G2-reasoner compared to the other approaches. Moreover, the motivation/inspiration taken for the G2 reasoner is generic and not causal specific (specifically the context and the RAG part) and may help improve the performance over tasks where causal reasoning is not required. \n\n* Figure 5 and Figure 6 shows the performance improvements when provided with the context, which is generally true for all the LLM evaluations, and the observation may not be causal reasoning specific. It may be noted that the G2-reasoner approach may also be applicable to other non-causal tasks and may have a similar result in other non-causal benchmarks. The proposed strategy will be more reliable when there is a direct effect over the causal-reasoning benchmark.\n\nQUESTIONS:\n* The assumption behind the autoregressive models made in line 159, “in a sequence, the current value is determined by past values, not related to future values,” may not be entirely correct. Specifically, highlighting “the current value is not related to future values” may be incorrect. \n\n* Line 148 raises a very good point regarding the complexity of natural language, and there can be many different ways or sentence patterns to express the same information. However, the evaluation metrics used in the paper is Exact match (Table 2, Figure-5, Figure-6).  It would be good to consider a better evaluation metric while performing the evaluations. \n\n* The claims made in the para [line 52 to line 61] and figure 2 are not completely justified in paras [line 156 to line 180], concluding on the statement [lines 178 to 180] “Thus, the autoregression mechanism makes LLMs’ causal reasoning primarily rely on correct causal knowledge in a large number of training corpora, i.e., the level-1 causal reasoning”. It would be great if the authors could provide a more detailed justification and talk about level 2 as well. Currently, it is not very clear if the statements say that autoregressive training is sufficient for achieving level-1 causal reasoning. \n\n* Equation 2 removes the dependency over the confounding variable using the total probability formula, where the $P_C$ is the general world knowledge base that may not always be available. Moreover, even if the general knowledge base is available, applying the total probability formula would need it to be complete, covering all the possibilities, which may not be feasible in general for natural language descriptions. I may have understood it incompletely, but it would be good if you could share some more thoughts and assumptions behind equation 2 to make it more transparent to the reader. \n\n* The use of GPT 3.5 turbo for constructing the benchmark may add a bias in the dataset construction. It would be good to highlight such biases and talk about them in detail. \n\n\nMinor Suggestions:\n* The use of fancy terms like AGI would be better avoided since the definitions of these terms are unclear and still under construction.\n\nLIMITATIONS:\nThe limitations of the work are highlighted in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}