Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation
Authors: Jinglong Gao
Xiao Ding
Bing Qin
Ting Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: Causal Reasoning
Large Language Models
Performance Evaluation
TL;DR: Assessing ChatGPT's causal reasoning ability, we find that ChatGPT is good at causal explanation, but has hallucinations in causal reasoning tasks.

Abstract: Causal reasoning ability is crucial for numerous NLP applications. Despite the impressive emerging ability of ChatGPT in various NLP tasks, it is unclear how well ChatGPT performs in causal reasoning.
In this paper, we conduct the first comprehensive evaluation of the ChatGPT's causal reasoning capabilities.
Experiments show that ChatGPT is not a good causal reasoner, but a good causal interpreter. Besides, ChatGPT has a serious hallucination on causal reasoning, possibly due to the reporting biases between causal and non-causal relationships in natural language, as well as ChatGPT's upgrading processes, such as RLHF. The In-Context Learning (ICL) and Chain-of-Thought (COT) techniques can further exacerbate such causal hallucination. Additionally, the causal reasoning ability of ChatGPT is sensitive to the words used to express the causal concept in prompts, and close-ended prompts perform better than open-ended prompts. For events in sentences, ChatGPT excels at capturing explicit causality rather than implicit causality, and performs better in sentences with lower event density and smaller lexical distance between events.

Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation.
Jinglong Gao Xiao Ding" Bing Qin Ting Liu
Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China {jlga xding, qinb, tliuj@ir. mit.edu...
Abstract
Event Causality identification (ECI)
Input: Minutes after woman was susperided and
Causal reasoning ability is crucial for numer- ous NLP applications. Despite the impressive emerging ability of ChatGPT various NLP tasks, is unclear how well CharGPT performs in causal reasoning. In this paper, we conduct the first comprehensive evaluation GPT's causal reasoning capabilities. Experi- ments show that ChatGPT not good causal reasoner, but good causal explainer. Besides, ChatGPT has serious hallucination causal reasoning, possibly due to the reporting biases between causal and non-causal relationships in natural language, well as ChatGPT'ss upgrad- ing processes, such as RLHH. The In-Context Learning (ICL) and Slain-of-Thouseee techniques can further exacerbate such causal hallucination. Additionally, the causal reason- ing ability of ChatGPT is sensitive to the words used to express the causal concept in prompts, and close-ended prompts perform better than open-ended prompts. For events in sentences, ChatGPT excels at capturing explicit causality rather than implicit causality, and performs bet- ter in sentences with lower event density and smaller lexical distance between events. The code is available on https://g/ thub. com/ ProgaanLLchattoTAAassaaaeeesss
escorted from her job at the Kraft Foods plant Philadelphia, returned opened fire, killing two women and critically injuring third co-worker before being taken into custody.
gun
Question: is there causal relationship between "suspended" and "injuring"
Chat-
Answer:
Causal Discovery (CD)
Multipi Choice
Input Event The fell unconscious.
Question:
cause input event
from the following options.
Option The astailant struck the in the head.
Option 2: The assailant took the mart' wallet.
(CoT)
Answer: Option
Binary Clessiffstton
Event A: The man fell uncanscious.
Event The assailant struck the the head.
Question: is there causal relationship between Event A and Event B
Causal Explanation Generation (CEG)
Cause: The assailant struck the man the head.
Effect: The man fell unconscious.
Question: why
Answer: Hit to head caused brain disruption. leading unconsciousness.
1 Introduction
Figure The forms of three causal reasoning tasks and the prompts we use. The content that requires ChatGPT to reply marked in red. The multiple-choice CD task also involves samples that ask for selecting the result of the input event. For such samples, modify the "cause" in the question to "result".
Causal reasoning ability crucial for numerous NLP applications. The recent causal reasoming sys- tems are mainly based on fine-tuned pre-trained lan- guage models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). However, their causal reasoning abilities rely supervised training using large amounts of annotated data.
In this paper, we conduct a comprehensive evaluation to demonstrate ChatGPT's causal rea- soning capabilities, involving four ttatc-of-the-art (SOTA) versions of ChatGPT: text-davinci-002, text-davinci-003, gpt-3.5-turbo and gpt-4.
Most recently, ChatGPT has achieved remark- able performance in various NLP tasks without the need for supervised training. However, there is currently no work that comprehensively evaluates ChatGPT's s ability in causal reasoning.
Firstly, we utilize the Event Causality Identifi cation (ECI) task as a comprehensive causal rea- soning benchmark. As shown in Figure 1, the ECI
"Corresponding Author

---

task aims determine whether there is causal relationship between two events sentence. This requires the ChatGPT to not only use extensive commonsense knowledge but also understand the complex context composed of multiple entities and events. Finally, ChatGPT must combine all infor- mation to identify causal relationships.
As the number of events in the sentence increases, and the lexical distance between events becomes greater, ChatGPT"s causal rea- soning performance decreases. Besides, Chat- GPT better at identifying explicit causality rather than implicit causality.
Open-ended generation prompts cannot im- prove ChatGPT's causal reasoning ability.
Secondly, employ Causal Discovery (CD) task for evaluation, which requires ChatGPT to possess broader and more specialized knowledge, yet does not necessitate consideration of complex contexts. As shown in Figure 1, two CD task for- mats are used: 1) mulriple-choice aims to select the cause or effect of the input event from two op- tions; 2) binary classification, aims to determine whether there is causal relationship between the input events. For binary classification, con- vert each multiple-choice example into two binary- classification examples by pairing the input event with each the two options. Our experiment in- dicates that binary classification is a more reliable evaluation method for ChatGPT.
2 Related Work
2.1
Causal Reasoning in NLP
Causal reasoning ability is important in NLP. Pre- vious work has made significant efforts to improve the causal reasoning ability of machines, such as incorporating external knowledge (Liu et al., 2020; Du al., 2021; ; Liu et al., 2023; Wang et al., 2023), conducting causal-specific pre-training (Li et al., 2021: Zhou et al., 2022), or applying data aug- mentation techniques (Li et al., 2020; Zuo et al., 2021a,b). However, these methods are highly de- pendent on annotated training data in specific do- mains and task formats. They perform poorly in scenarios where annotated data is scarce. In this pa- per, evaluate ChatGPT, which does not require training data.
Furthermore, as shown in Figure 1, we conduct Causal Explanation Generation (CEG) task test whether CharGPT can generate explanations for causal relations between events. This is typ- ically used to test whether machines truly under- stand the principles behind causality, which is cru- cial for building a reliable causal reasoning system.
2.2 Evaluation of ChatGPT's Capabilities
Recently, a large amount of work has conducted evaluations of ChatGPT's various capabilities, However, ChatGPT's causal reasoning capabilities have not been fully evaluated. Qin al. (2023) and Chan et (2023) only employed the multiple- choice CD format on the COPA dataset to evaluate ChatGPT, which consists of only 1,000 examples primarily focused on simple everyday causality. Besides, our experiments show that this multiple- choice format leads to an overestimation of Chat- GPT's performance. Furthermore, Kıeıman et al. (2023) claimed that ChatGPT achieved a 97% ac- curacy on the causal discovery task. However, they only required ChatGPT to determine the causal di- rection of 583 causal event pairs, without requiring to predict whether causality exists. This does not constitute a complete causal discovery task.
Key takeaways The key findings and insights are summarized as follows:
ChatGPT is not a good causal reasoner, but good causal explainer.
ChatGPT has a serious causal hallucination issue, where tends to assume causal relation- ships between events, regardless of whether those relationships actually exist.
The main reason of ChatGPT's causal hal- lucinations may be the reporting biases be- tween causal and non-causal relationships in natural language, as well as ChatGPT's up- grading processes, such as RLHF (Ouyang et al., 2022b). Besides, the ICL and CoT (Wei et al., 2022) prompts can further exacerbate the causal hallucination of ChalGPT.
In summary, previous evaluations only involved small-scale datasets and simple task formats, which overestimated ChatGPT's causal reasoning abili- ties. However, we conduct a comprehensive and objective evaluation of the ChatGPT's causal rea- soning abilitics, involving four different task forms and five widely-used causal reasoning datasets.
The causal reasoning ability of ChatGPT is sensitive to the words used to express the causal concept in the prompt.

---

3 Evaluation Settings
We conduct our experiments using OpenAl's official API', covering four progressive SOTA ver- sions of ChatGPT: text-davinci-002, text-davinci- 003, gpt-3.5-turbo and gpt-4. Specifically, text- davinci-002 was further trained using RLHF to obtain text-davinci-003, which was subsequently further trained using conversational data to obtain gpt-3.5-turbo. Although OpenAl has not disclosed gpt-4, gpt-4 has shown superior reasoning capabil- ities in various NLP tasks. For gpt-4, we sample 1000 instances from each dataset for evaluation. We set the temperature parameter to to minimize randomness.
3.1 Datasets and Evaluation Metries
3.1.1 Event Causality Identification
We conduct experiments on three widely used ECT datasets: 1) EventStoryLine v0.9 (ESC) (Caselli and Vossen, 2017), which contains 22 topics, 258 documents, 5,334 events and 1,770 causal event pairs; 2)
(CTB) (Mirza et al., 2014), which contains 184 documents, 6,813 events and 318 causal event pairs 3) MAVEN-ERE (Wang al., 2022), which contains 90 topics, 4,480 documents, 103,193 events and 57, 37,992 causal event pairs. Following previous works (Gao et al., 2019; Choubey and Huang, 2017), only the top 20 top- ics of ESC are used for evaluation. Besides, since MAVEN-ERE did not release the test set, we eval- uate ChatGPT on its development set. We adopt Accuracy, Precision (P), Recall (R), and Fl-score (F1) as the evaluation metrics.
3.3 Baselines
In this paper, all baseline methods for the three causal reasoning tasks are based on PLMs fine- tuned on the full training dataset.
For the ECI and the CD lask, we compare Chat- GPT with vanilla classification models based on BERT-Base (Devlin et al., 2019) and RoBERTa- Base (Liu et al., 2019). Their framework and train- ing process are consistent with previous work (Liu et al., 2020; Li et al., 2021).
3.1.2 Causal Discovery
We conduct experiments on two widely used CD datasets: 1) COPA (Roemmele et al., 2011), which is classic dataset for causal reasoning and consists of 1,000 multiple-choice questions that primarily focus on everyday life scenarios. 2) e-CARE (Du et al., 2022), contains 21,324 multiple-choice ques- tions covering a wide range of domains. We adopt Accuracy as the evaluation metric.
Besides, we compare ChatGPT with two SOTA ECI method: KEPT (Liu et al., 2023), based on BERT-Base, incorporated background and rela- tional information for causal reasoning; and DPJL (Shen ct al., 2022) based on RoBERTa-Baas intro- duced information about causal cue words and the relation between events into ECI model.
For the CEG task, we first compare ChatGPT with GRU-based Seq2Seq model (Chung et al., 2014) and GPT2 (Radford al., 2019). Their framework and training process are consistent with previous work (Du et al., 2022). Besides, we em- ploy recent LLM baselines by finetuning LLaMA 7B (Touvron et al., 2023) and FLAN-T5 11B (Chung et al., 2022) on the c-CARE dataset.
3.1.3 Causal Explanation Generation
We conduct experiments on e-CARE, which con- tains human annotated causal generations for 21,324 causal event pairs. Following the evaluation settings of e-CARE, we first adopt average-BLEU (n=4) (Papineni et al., 2002) and ROUGE-L (Lin, 2004) as the automatic evaluation metrics. Sec- ondly, sample 100 explanations generated by each version of ChatGPT on the e-CARE for hu- man evaluation. Specifically, we label whether the generated explanation can explain the correspond- ing causal fact to calculate the accuracy.
4 Experimental Results
4.1 Event Causality Identification
Table show the results on the three ECI datasets: ESC, CTB and MAVEN-ERE We find that: Firstly, ChatGPT, even the gpt-4, has been com- prehensively outperformed by baseline methods based on fine-tuned small PLMs. This indicates that ChatGPT is not a good causal reasoner in complex causal reasoning task like ECI.
3.2 Experiment Setting
For ChatGPT, we follow an instruction prrmupt scheme for ECI, CD and CEG tasks. Figure 1 shows the prompts employed for these three causal reasoning tasks. We evaluate ChatGPT's per- formance under zero-shot settings. Additional prompts and settings are discussed in $5.
/ttpt//r///a/.....m/// www.oo.enccccoooo com/ (accessed between 3/1/2023 and 4/28/2023)

---

Methods
ESC
CTB
MAVEN-ERE
P
R
F1
P
F1
P
R
F1
BERT-Base (Devlin et al., 2019)
38.1
56.8
45.6
41.4
45.8
43.5
52.5
75.6
61.9
RoBERTa-Base (Liu et al., 2019)
42.1
64.0
50.8
39.9
60.9
48.2
52.8
75.1
62.0
KEPT (Liu et al., 2023)
50.0
68.8
57.9
48,2
60.0
53.5
DPJL (Shen et al., 2022)
65.3
70.8
67.9
63.6
66.7
64.6
text-davinci-002 text-davinci-003
23.2
80.0
36.0
5.0
75.2
9.3
19.6
92.9
32.4
33.2
74.4
45.9
8.5
64.4
15.0
25.0
75.1
37.5
gpt-3.5-turbo gpt-4
27.6
80.2
41.0
6,9
82.6
12.8
19.9
85,8
32.3
27.2
94.7
42.2
6.1
97.4
11.5
22.5
92.4
36.2
Pos
Neg
Full
Neg
Full
Pos
Neg
Full
BERT-Base (Devlin et al., 2019)
59,5
83.6
79.7
46,4
87.8
86,2
75.7
86.9
85.1
Bass (Liu et al., 2019)
63.8
82.8
79.8
62.3
86.4
85.5
76.6
87.1
85.4
text-davinci-002
80.0
43.1
49.6
75.2
41.9
43.2
92.9
21.2
33.5
text-davinci-003
74.4
67.7
68.9
64.4
71.9
71.6
75.1
53.6
57.2
gpt-3.5-turbo
80.2
54,4
59.0
82.6
55.0
56.0
85.8
28.5
38.3
gpt-4
94.7
41.4
51.4
97.4
39.1
41.4
92.4
33.9
44.0
Table 1: Experimental results (%) the ECI task. P. R and F1 indicate Precision, Recall and F1 Fl-score, respectively. Pos, Neg and Full indicate accuracy on the causal pairs, non-causal pairs and all test datas, respectively.
Multiple Choice
Binary Classification
Methods
c-CARE
CОРA
c-CARE
CОРА
Full
Full
Pos
Neg
Full
Pos
Neg
Full
BERT-Base (Devlin et al., 2019)
75.4
75.4
65.7
95.2
65.0
59.5
88.9
54.5
RoBERTa-Base (Liu et al., 2019)
70.7
80.5
64.6
74.6
64.5
66.0
69.0
63.5
texf-davinci-002
78.4
94.4
18.5
95.2
56.8
55.6
92.4
74.0
1ext-davinci-003 gpt-3,5-turbo gpt-4
76.7
93.2
41.0
86,4
63.7
80.4
82.3
81.4
79.1
96.3
75.5
66.9
71.2
96.3
43.2
69.7
84.5
98.1
84.8
57.5
71.2
97.9
38.5
68.2
Table 2: Experimental results (%) the CD task. Pos, Neg and Full indicate accuracy the causal pairs, non-causal pairs and all test datas, respectively.
Secondly, the recall of ChatGPT is high, but the precision is low, indicating that a large number of non-causal event pairs are falsely identified as causal pairs. This is also why ChatGPT performs particularly poorly on the CTB dataset, which con- tains more non non-causal event pairs. The main rea- son for this may be that natural language contains large number of descriptions of causal relation- ships, mainly indicated by causal cue words such "lead to" and "therefore". However, natural lan- guage generally does not express which events are not causally related. Furthermore, since ChatGPT's ability comes from training on massive amounts of natural language text, such reporting bias between causal and non-causal event pairs in texts makes ChatGPT good at identifying causal event pairs but not in recognizing non-causal event pairs.
there much more negative examples than posi- tive examples in the ECI dataset, and the fine-tuned models have learned such data distribution.
4.2 Causal Discovery
Table 2 show the results on the two CD datasets: COPA and e-CARE. We find that:
Firstly, although ChatGPT performs well in multiple-choice, its performance is poor in binary classification. The main reason is that in multiple- choice, ChatGPT only needs to consider the option that shows the more obvious causal or non-causal relationship with the input event, while the other more difficult option can be ignored. However, pre- vious work (Qin et al., 2023; Chan al., 2023) only used multiple-choice to evaluate ChatGPT's causal reasoning ability, leading to misrepresentation that ChatGPT is good at causal reasoning.
Besides, can be observed that the fine-tuned small PLMs do better at identifying non-causal event pairs rather than causal ones. This is because
Secondly, compared to the ECI task, ChatGPT achieves higher accuracy on non-causal pairs in the

---

&CCARE ROUGE-I
Methods
AVG-BLEU
Human Evaluation
GRU-Seq2Seq (Chung et al., 2014)
18.7
21.3
0.0
GPT2 (Radford et al., 2019)
32.0
31.5
20.0
LLaMA 7B (Touvron et al., 2023)
40.3
37.1
63.0
FLAN-TS 11B (Chung et al., 2022)
38.5
42.8
66.0
text-davinci-003 gpt-3,5-turbo
10.55
37.49
83.0
7.32
40.31
82.0
gpt-4
6.47
39.77
85.0
Human Generation (Du et al., 2022)
35.51
33.46
Table 3: Experimental results (%) on the CEG task. "Human Generation" is the human annotated explanations provided by Du et al. (2022).
CD task. This is mainly because the non-causal pairs in the e-CARE and COPA datasets are gener- ated manually given input event, and they have simple structure and weak correlation with the in- put events, making them easier identify. This is also the reason why the fine-tuned small PLMs do better at identifying non-causal event pairs rather than causal ones.
Secondly, compared to "Human Generation", ChatGPT achieves better ROUGE-I score, which is text generation metric similar to "recall" in text classification. This is because ChatGPT tends generate explanations that are more complete and detailed. This was confirmed by the evalu- ators during our human evaluation process. This is also the reason why ChatGPT received a lower AVG-BLEU score, which is a text generation met- ric similar to "precision" in text classification.
Besides, compared to the ECI task, ChatGPT achieves slightly lower accuracy causal pairs in the e-CARE dataset. This is because e-CARE requires ChatGPT to grasp a wider range of knowledge, which involves not only commonsense knowledge in more scenarios, but also professional knowledge in certain fields, such as biology.
Thirdly, in manual evaluation, we find that the ex- planations generated by LLaMA and FLAN-T5 are highly correlated with the input events. However, the explanations might be mere repetitions of the input events or provide relevant but uninformative descriptions that cannot be used for explanation. This also a reason for the poor performance both LLaMA and FLAN-T5 in human evaluations.
More importantly, we notice that the upgrading process of ChatGPT (text-davinci-003-- 3gpt--..- turbo gpt-4) leads the ChatGPT become in- creasingly inclined to classify events having causal relationship, regardless of whether it is ac- tually correct or not. This may be due to the align- ment tax (Ouyang et al., 2022a) of RLHF. This indicates that while OpenAl (2023) mentioned that ChatGPT's upgrading process reduces the hallucination issue in various other tasks, it also makes the ChatGPT better at fabricating causal relationships. A preliminary analysis of the im- pact of RLHF on causal reasoning is provided in the Appendix
Besides, compared to ChatGPT, the explanations provided by LLaMA and FLAN-TS are noticeably shorter, as the gold explanations provided by e- CARE are very concise. However, ChatGPT excels in providing more comprehensive and detailed ex- planations in the zero-shot setting, This shows the advantage of ChatGPT's causal explanations com- pared to traditional fine-tuning methods.
Moreover, is worth noting that fine-tuned LLaMA, FLAN-T5 and ChatGPT achieve simi- lar ROUGE-I scores, but the two finetuned LLMs perform worse in our human evaluation. This is because that the fine-tuned LLaMA and FLAN-T5 may generate less informative explanations, e.g., mere repetitions of the input events. However, ChatGPT may offer valuable explanations for the input causal event pairs, but from different perspec- tives or in distinct syntactic forms compared to the gold explanations in e-CARE.
4.3 Causal Explanation Generation
Table 3 show the experimental results on the CEG task. It can be observed that:
Firstly, according to the human evaluation re- sults, the accuracy of causal explanations gener- ated by ChatGPT close to those generated by humans. This indicates that ChatGPT is a good causal explainer.

---

ESC
Firstly, when both and y are less than equal to 4, ICL mainly improves ChatGPT's accuracy on causal pairs, but decreases the accuracy on non- causal pairs. This may be because although ICL can stimulate the ChatGPT's abilities, ChatGPT is better at identifying causal event pairs. Therefore, ICL further exacerbates the imbalance of ChatGPT in identifying causal and non-causal pairs. In addition, "4 pos + 48 neg" achieves higher Full accuracy. However, this because it improves Neg accuracy at the expense of Pos accuracy, as the ESC dataset contains a larger proportion of non- causal pairs, making Neg accuracy have a greater impact on Full accuracy. A substantial improve- ment in the overall performance should not be a case of robbing Peter to pay Paul, rather than im- prove both the Pos and the Neg accuracy.
Methods
ICL Settings
Pos
Neg
Full
none
80.0
43.1
49.6
I pos neg
85.1
43.8
51.2
text- davinci- 002
2 pos neg
88.4
37.3
46.4
1 pos neg
88.7
38.3
47.2
2 pos neg
89.7
36.0
45.5
4 pos neg
78.8
51.0
55.9
none
80.2
54.4
59.0
1 pos neg
94.4
26.6
38.7
gpt- 3.5- turbo
2 pos neg
91.3
35.7
45.6
1 pos neg
92.6
31.0
41.9
2 pos neg
92.0
36.0
45.9
4 pos neg
82.2
50.3
56.0
none
74.4
67.7
68.9
1 pos 2 neg
92.4
35.3
45.2
2 pas 2 neg
96.3
26.1
38.5
1 pos + neg
95.9
27.1
39.3
text-
2 pas neg
98.4
18.2
32.4
davinci-
4 pos neg
92.8
38.2
47.9
003
4 pos + 48 neg
71.7
77.0
76.2
40 pos 20 neg
86.1
59.1
63.4
5.2 Chain-of-Thoouhh Prompting
top similar
85.3
50.1
55.6
top 4 similar
95.6
28.0
38.8
As shown in the Table 6, we analyze ChatGPT un- der different Chain-of-Thougee settings: 1) "-w/ CoT zero-shot": we employ the zero-shot CoT (Kojima et al., 2022) by adding "Let's think step by step" after the prompt; 2) "-w/ CoT x pos + neg": we manually annotate the reasoning chain for æ pos- itive and y negative training examples. They are selected demonstrations for in-context learning, and all test cases share the same demonstrations. We further analyze the error types of ChatGPT in the Appendix The examples of our used demon- strations and the reasoning chains generated by ChatGPT are presented in the Appendix E. Firstly, "-w/ CoT zero-shot" cannot effectively improve the performance of ChatGPT in the ECI task. This is because the quality of the reason- ing chain generated by zero-shot CoT is not high enough effectively guide the model.
Table 4: Performance of ChatGIT the ECI task with ICL. "none" indicates ChatGPT without ICL.
c-CARE
Methods
ICL Settings
Pos
Neg
Full
41.0
86.4
63.7
I pos neg
90.3
41.3
65.8
text- davinci-
2 pos 2 neg
74.6
64.1
69.3
003
pos neg
83.4
55.5
69.5
2 pos neg
86.6
49.6
68.1
pos neg
72.7
66.5
69.6
Table Performance of ChatGPT on the binary- classification CD task with ICL. "none" indicates Chat- GPT without ICI.
5
Analysis
Secondly, "-w/ CoT pos + y neg" improves ChatGPT's accuracy on causal pairs, but decreases its accuracy on non-causal pairs. Observing the reasoning chains generated by ChatGPT, we found that the ChatGPT generates lower quality chains for non-causal pairs than for causal pairs. This difference may worsen the imbalance of ChatGPT in identifying causal and non-causal event pairs.
5.1
In-Context Learning
As shown in the Table 4 and Table 5, we analyze ChatGPT under different In-Context Learning set- tings: 1) "x pos + y neg": we randomly select x positive and y negative training examples as demon- strations for in-context learning, and all test cases share the same demonstrations; 2) "top k similar": for each test case, we retrieve the top l most sim- ilar training examples its demonstrations. To compute the similarity, we first encode them with BaRT-large, then compute the cosine similarity of their embeddings. We further analyze the impact of the order and the label distribution of ICL demon- strations Appendix and C.
5.3 Ways of Expressing Causality in Prompts
As shown in Figure 2, we analyze the performance of ChatGPT on the ECI task using prompts that express the causal concept different ways: 1) "counterfactual" prompt based on the counter- factual causality view of Pearl (2009); 2) "one-

---

ESC
e-CARE
Methods
ICL Settings
Pos
Neg
Full
Pos
Neg
Full
text-
davinci-
none
74.4
67.7
68.9
41.0
86.4
63.7
003
zero-shot
32.8
42,4
37.0
47.4
80.7
64.0
4 pos neg
97.1
41.1
50.1
84.6
51.1
67.8
ww/ CoT
4 pos 4 neg
86.2
48.0
54.1
86.0
50.2
68.1
4 pos 8 neg
76.8
62.4
64.7
82.9
52.9
67.9
Table 6: Performance of ChatGPT on the ECI and the binary-classificttion CD task with the Chain-of-Thought prompts. "none" indicates ChatGPT without ICL.
Input: Minutes counterfactual
ESC
Methods
custody.
Pos
Neg
Full
Question: if "suspended" does not happen, will
"injuring" happen? ?
text-davinci-003
74.4
67.7
68.9
Answer: No
-w/ counterfactual
98.5
00.8
16.3
-w/ one-step
44.4
91.7
84.4
one-stop
-w/ trigger (lead to)
88.3
46.5
53.1
Input: Minutes after
custody.
-w/ trigger (give rise to)
90.8
53.7
59.5
-w/ trigger (bring about)
87.1
53.1
58.4
Question: is there a one-step causal relationship between "suspended" and "injuring"
-w/ trigger (result in)
83.3
59.5
63.3
-w/ trigger (in order to)
70.1
60.9
62.4
Answer: Yes
-w/ trigger (so that)
81.2
59.9
63.3
Input: Minutes triggar(<X>)
custody.
Table 7: Performance of ChatGPT in the ECI task using
Quastion: does "suspanded"
<X> "injuring"
prompts that express the causality in different ways.
Answer: Yes
Figure 2: Prompts that express causal concepts in vari- ous ways. The content that requires ChatGPT to reply is marked in red.
of causality triggered by each cue word are quite different. Therefore, causal cue words that have the same meaning to humans may represent different causal concepts to ChatGPT. This further indicates that it is challenging to accurately convey what causality means to ChatGPT through prompts.
step", we add constraint words "one-step" to alle- viate the issue of identifying non-causal event pairs causal; and "trigger<<X>)), we use different causal cue words <X> (e.g.. "lead to") to construct prompts. Results are shown in Table
5.4 Lexical Distance between Events
As shown in the Figure 3, we analyze the perfor- mance of ChatGPT on pairs events with different lexical distances in the ECI task. The "lexical dis- tance" refers to the number of words that separate two events within sentence.
Firstly, the "counterfactual" prompt makes al- most all non-causal pairs to be identificd as causal. This is mainly because ChatGPT's counterfactual reasoning results are not accurate.
Firstly, find that the event distances in- crease, ChatGPT is more inclined predict event pairs as non-causal. This may be because in natural language, the larger the distance between events, the less likely there is a causal relationship, and ChatGPT has learned this pattern.
Secondly, the "one-step" improves CharGPT's accuracy on non-causal pairs, but lowers its accu- racy causal pairs. This is because while con- straint words such as "one-step" can make the model more likely to predict event pairs non- causal, it does truly enhance ChatGPT's causal reasoning abilities.
Secondly, as the event distances increase, the Fl scores of ChatGPT decrease. This indicates that ChatGPT is good at identifying long-distance causal relationships. An outlier is the Fl score of gpt-4 the interval (25,30). This due to the fact that out 1000 test instances for gpt-4, there are only 35 examples within the interval [25,30),
Moreover, the performance of "trigger(<X>)" with different causal cue words is significantly dif- ferent. This may be due to the fact that during pre- training, ChatGPT mainly learns causal knowledge triggered by causal cue words, but the distributions

---

lext-davinci-002
gpt-8.5-turbo
text-davincc-002
gpt-3.5-turbo
lext-davincil-00
gpl-4
text lext-davinci-003
gPI-4
0.30
0.90
0.80
0.70
0.80
0.50
0.40
0.70
0.70
0.60
0.60
0.50
0.40
0.30
0.30
0.10
0.70
0.60
0.60
0.50
0.50
0.40
0.30
(9.5)
(5,10)
(10,15)
[15.20)
[20,25)
Target [20,0M]
(25,30)
Number
Figure 3: Performance of ChatGPT on pairs of events with different lexical distances in the ESC dataset.
Figure Performance of ChatGPT on sentences with different numbers of events in the ESC dataset.
text-davinci-002
gpt-3.5-turbo
leading to more random performance. However, all other results demonstrate that ChatGPTs' perfor- mance decreases the event distance increases.
text noxt-davincii-003
gpt-4
1.00
5.5 Density Events
Accuracy
0.885
As shown in the Figure 4, we analyze the perfor- mance of ChatGPT in sentences with different num- bers of events the ECI task. We find that: Firstly, as the event density increases, most ver- sions of ChatGPT more inclined to predict event pairs non-causal. This is mainly because as the event density increases, the context of events be- comes more complex, making more difficult to capture the correlations between events.
0.861
0.818
0.60
0,798
0.758
9,741
Explict Gausal Pairs
Implici Causel Pairs
Type of Causal Pairs
Figure Performance of ChatGPT on pairs of events with different types of causality in the ESC dataset.
Secondly, as the event density increases, the Fl scores of ChatGPT decreases. This indicates that ChatGPT is not good at handling complex situa- tions involving multiple events.
tifying explicit causality only requires recogniz- ing causal cue words, whereas identifying implicit causality requires reasoning with contextual infor- mation and commonsense knowledge.
5.6 Types of Causal Relationship
5.7 Prompts in the Form of Open-Ende Generation
As shown in the Figure we analyze the accuracy ChatGPT on pairs of events with different types causal relationships the ECI task: 1) Explicit Causality, which refers to causal relationships ex- plicitly triggered by causal cue words (c.g., "lead to"); 2) Implicit Causality, which refers to causal relationships expressed without causal cue words. can be observed that, compared to implicit causality, ChatGPT performs better on capturing explicit causality. This is mainly because iden-
Recently, Arora et al. (2023) revealed that open- ended prompts ("Who went to the park?") tend to yield better results for ChatGPT than prompts that restrict ChatGPT's outputs ("John went the park. True False?"). As shown in Table we analyze ChatGPT with open-ended prompts: 1) "open-ended A.1/2/3",, requires ChatGPT to gener- ate all the causal event pairs in the input sentence.

---

Open-ended
Open-ended
Input:
Minutes after ... taken into custody.
Input: Minutes after taken custody.
List the effect pairs input sentence.
Question:
Question: there token-level causal relationship in the sentence? If so, please extract it into this form: seffect If there are multiple causal relationships, add AND between causal pairs, and display No there causal relationship. Answer:<the response ChatGPT>
Answer:
1.<the response of ChatGPT>
Open-ended A.2
Input: Minutes after taken into custody.
Question: IF there is causal relationship between two events in the input sentence, extract the causal pair word level. there multiple causal pairs, add AND between them, otherwise None. For example: (accuse of) cause (death) AND (kill) cause (death)
Open-ended
Input: Minutes after taken custody. Question: the events in input sentence that are causally related to the event "suspended".
Answer:
the response of ChatGPT>
1. ste response of ChatGPT>
Figure 6: Prompts in the open-ended form. The content that requires ChatGPT to reply is marked in red.
ESC
Conclusion
Methods
Prompt
F1
In this paper, conduct comprehensive evalu- ation of ChatGPT's causal reasoning capabilities. Experiments show that: 1) ChatGPT is not good causal reasoner, but is good at causal explanation generation; 2) ChatGPT has a serious causal hallu- cination, possibly due to the causal reporting biases and ChatGPT's upgrading processes 3) The ICL and CoT techniques can further exacerbate such causal hallucination; 4) The ChatGPT is sensitive to the words used to express the causal concept in prompts, and open- ended causal reasoning prompts not suitable for ChatGPT; 5) For events sen- tences, ChatGPT excels at capturing explicit causal- ity, and performs better sentences with lower event density and smaller event distances.
close-ended
27.6
80.2
41.0
gpt- open-ended A.1
4.8
16.3
7.4
3.5- turbo
open-ended A.2
6.9
8.0
7.4
open-ended A.3
14.1
10.4
12.0
open-ended B
6.3
54.0
11.3
Table Performance of ChatGPT on ECI task with open-ended generation prompts. "close-ended" indi- cates the origin ECI prompt shown in Figure 1. It is worth noting that the "close-ended" prompt does not explicitly require ChatGPT to output Yes or No, but its question format guides ChatGPT almost always output Yes or No.
We designed three different prompts fully evalu- ate ChatGPT's performance. 2) "open-ended B". gives a target event the input sentence, and re- quires ChatGPT to generate events the input sentence that have causal relations with the target event. We employ a relaxed P. R, and Fl calculation for open-ended prompts. Specifically, predicted causal-effect pair is considered correct if at least token is shared between the predicted and the labeled cause, as well as between the predicted and the labeled effect. The formats of these prompts are shown Figure
Although there may be more delicate prompts that can further surpass reported results, believe that relying solely on prompts cannot fun- damentally solve the issues that ChatGPT faces in causal reasoning. We hope that this study can in- spire future works, such as addressing the causal hallucination issue of ChatGPT or further evaluat- ing ChatGPT in scenarios involving multi-factor and multi-modal causal reasoning.
Acknowledgements
It can be observed that the open-ended prompts decrease the performance of ChatGPT. This is be- cause open-ended prompts require ChatGPT to jointly perform the event extraction and the ECI task. However, previous studies (Gao et al., 2023; Wei et al., 2023) show that ChatGPT is not good at extracting events.
We would like to thank Jianbai Zhao for his ef- fort on prompt design and human evaluation, and the anonymous reviewers for their constructive comments, and gratefully acknowledge the sup- port the National Natural Science Foundation of China (U22B2059, 62176079). and the Natu-

---

ral Science Foundation of Heilongjiang Province (YQ2022F005).
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai. Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin. Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scaling instruction-finctuned language models.
Limitations
This work is comprehensive evaluation on the causal reasoning ability of ChatGPT, and it has several limitations. Firstly, ChatGPT's capabilities are constantly being updated, and current test rc- sults may change as ChatGPT evolves. Secondly, although OpenAl has provided rough introductions of different versions of ChatGPT, the implementa- tion details are unclear, making it difficult to deeply analyze why different versions of ChatGPT have different performance, and how each data and train- ing technique affects the performance of ChatGPT, Finally, there may still be prompts that can further outperform the results we reported, such as differ- ent questioning formats and more advanced ICL techniques, but we believe that relying solely on prompts cannot fundamentally solve the illusion problem that ChatGPT currently faces in causal reasoning.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence model- ing. arXiv preprint arXiv:1412.3555.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proc. of NAACI.
Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2021. ExCAR: Event graph knowledge enhanced explainable causal reasoning. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Vol- une 1: Long Pupers), pagess 2354-2363, Online. As- sociation for Computational Linguistics.
References
Simran Arora, Avanika Narayan, Mayee F Chen, Lau- rel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. 2023. Ask me anything: A sim- ple strategy for prompting language models. In The Eleventh International Conference on Learning Rep- resentations.
Li Du. Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022. e-care: a new dataset for exploring explainable causal reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 432-446.
Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023. Exploring the feasibility of chatgpt for event extraction. arXiv preprini wrXiv:2303.0383 033336
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph. Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Saufield-oodde Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a help- ful and harmless assistant with reinforcement learn- ing from human feedback.
Lei Gao, Prafulla Kumar Choubey, and Ruihong Huang. 2019. Modeling document-level causal structures for event causal relation identification. In Proc. of NAACL.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205.11912
Tommaso Caselli and Piek Vossen. 2017. The event
Emre Kıemaan Robert Ness, Amit Sharma, and Chen- hao Tan. 2023. Causal reasoning and large language models: Opening new frontier for causality.
StoryLine corpus: A new benchmark for causal and temporal relation extraction. In Proceedings of the Events and Stories in the News Workshop.
Zhongyang Li, Xiao Ding, Kuo Liao, Bing Qin, and Ting Liu. 2021. Causalbert: Injecting causal knowl- edge into pre- trained models with minimal supervi- sion. arXiv preprint arXiv:2107.c 0780999..
Chunkit Chan, Jiayang Cheng, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2023. Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations.
Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, and Benjamin Van Durme. 2020. Guided generation of cause and effect. In Proceedings of the Twenty- Ninth International Joins Conference on Artificial
Prafulla Kumar Choubey and Ruihong Huang. 2017. A sequential model for classifying temporal relations between intra-sentence events. In Proc. of EMNLP.

---

Intelligence, IJCAI-20, pages 3629-3636. Interna- tional Joint Conferences on Artificial Intelligence Organization. Main track.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon, 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Chin-Yew Lin. 2004. Rouge: package for automatic evaluation of summaries. In Text summarization
branches our, pages 74-81.
Shirong Shen, Heng Zhou, Tongtong Wu, and Guilin Qi. 2022. Event causality identification via deriva- tive prompt joint learning. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 2288-2299, Gyeongju, Republic of Korea, International Committee on Computational Linguistics.
Jian Liu, Yubo Chen, and Jun Zhao. 2020. Knowledge enhanced event causality identification with mention masking generalizations, In Proc. of IJCAI.
Jintao Liu, Zequn Zhang, Zhi Guo, Li Jin, Xiaoyu Li, Kaiwen Wei, and Xian Sun. 2023. Kept: Knowledge enhanced prompt tuming for event causality identifi- cation. Knowledge-Bast Based Systems, 259:1 159:11
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanow. 2019.
Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692..
Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang. Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. 2022. Maven- ere: unified large-scale dataset for event corefer- ence, temporal, causal, and subevent relation extrac- tion. In Proceedings of EMNLP.
Paramita Mirza, Rachele Sprugnoli, Sara Tonelli, and Manuela Speranza. 2014. Annotating causality in the TempEval-3 corpus. In Proc. of EACL.
OpenAI. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022a. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744. Curran Associates, Inc.
Zhaowei Wang. Quyet V. Do. Hongming Zhang, Jiayao Zhang, Weiqi Wang, Tianqing Fang. Yangqiu Song. Ginny Y. Wong, and Simon See. 2023. Cola: Con- textualized commonsense causal reasoning from the causal inference perspective.
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning large language models. In Advances in Neural Information Processing Systems.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022b. Training language models to follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang. and Wen- juan Han. 2023. Zero-shot information extraction via chatting with chatgpt.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computa- tional Linguisties, pages 311-318.
Yucheng Zhou, Tao Shen, Xiubo Geng, Guodong Long. and Daxin Jiang. 2022. ClarET: Pre-training a correlation-aware context-to-event transformer for event-centric generation and classification. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics ( (Volume 1: Long Papers), pages 2559-2575, Dublin, Ireland. Associa- tion for Computational Linguistics.
Judea Pearl. 2009. Causality. Cambridge university press.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver?
Xinyu Zuo. Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Weihua Peng, and Yuguang Chen. 2021a. a. Improving event causality identification via self- supervised representation learning on external causal statement. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 2162-2172. Online. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners. arXiy preprint 1rxiv:1412.3555

---

Xinyu Zuo, Pengfei Cao. Yubo Chen, Kang Liu, Jun Zhao, Weihua Peng, and Yuguang Chen. 2021b. LearnDA: Learnable knowledge-guidee data augmen- tation for event causality identification. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3558- 3558-3571, Online. Association for Computational Linguistics.

---

Methods
ESC
CTB
MAVEN-ERE
R
F1
P
R
F1
P
R
F1
FLAN-T5-L Large (Chung et al., 2022)
19.4
91.1
32.0
4.2
82.6
7.9
18.1
93.5
30.3
w/ RLHF
17.6
97.9
29.8
3,9
99.0
7.5
17.1
99.4
29.2
Pos
Neg
Full
Pos
Neg
Full
Pos
Neg
Full
FLAN-TS-Large (Chung al., 2022)
91.1
18.6
31.4
82.6
23.0
25.3
93.5
12.8
26.6
w/ RLHF
97.9
1.5
18.6
99.0
0.9
4.7
99.4
0.8
17.6
Table Experimental results (%) on the ECI task. P. R and F1 indicate Precision, Recall and Fl-score, respectively. Pos, Neg and Full indicate accuracy the causal pairs, non-causal pairs and all test datas, respectively.
A Effect of RLHF on Causal Reasoning
Demonstrations
Pos
Neg
Full
Proportion+
pos neg
21.8
26.7
It is necessary to explore why RLHF enhances the causal hallucination issue of ChatGPT. Due Ope- nAI's decision not to open-source ChatGPT, lack access to specific details about the data and experimental setup for ChatGPT's RLHE. As an al- ternative approach, we analyze the effect of RLHF with the Anthropic RLHF dataset (Bai et al., 2022), which is an open- RLHF dataset. As shown in Table 9, we test the zero-shot performance of FLAN-T5-Large (Chung al., 2022) with/without RLHF process on the Anthropic dataset.
pos neg
27.5
39.1
pos neg
88.6
36.3
49.5
25.2
pos neg
47.1
66.9
pos neg
95.3
35.7
83.7
80.6
Table 10: Experimental results (%) of "top 4 similar" in $5.1 with different labels of demonstrations on the ESC dataset.
each containing instances that only use 0, 1, 2, 3, or 4 causal demonstrations in their top similar demonstrations, respectively. Then, we present the performance of the "top similar" on these five subsets. "Proportion+" is the proportion of causal instances in the corresponding subset. "x pos neg" indicates the subset that only use x causal and y non-causal demonstrations.
can be find that the RLHF also exacerbates the causal hallucination issue LLAN-TS-Large. Through our analysis of the Anthropic dataset, this may be due to:
1. Among questions about "Why", only 10.29% include the word "not". This indicates that the majority of RLHF data guide the LLMs about causality rather than non-causality.
Firstly, we can observe that when using only causal non-causal demonstrations, the model achieves higher Pos accuracy. This might be be- cause that including only one classes prevents the model from contrasting the meanings of different labels, thus potentially confusing the model's 's un- derstanding of the task objectives.
2. In the gold responses, the frequency of the word "yes" is approximately twice that of the word "no". This could potentially increase the likelihood of the LLMs producing the positive label rather than the negative label.
Additionally, when using both causal and non- causal demonstrations, there is smaller change in Pos accuracy, while Neg accuracy increases the number of causal demonstrations rises. This might be because having more causal demonstra- tions helps the model understand the situations that truly involve causality, thus avoiding misclassify- ing non-causal instances as causal.
Both of these two characteristics in the RLHF data could potentially exacerbate ChatGPT's causal hal- lucination, leading it to assume causal relationships berween events, regardless of whether those rela- tionships actually exist.
B Effeet of the Label Distribution in ICL Demonstrations
C Effect of the Order of ICL Demonstrations
As shown in Table 10, we analyze the impact of the labels of ICL. demonstrations on performance of "top k similar" (described 15.1). For k=4, we first divide the ESC dataset into five subsets,
As shown in Table 12, we analyze the few-shot ChatGPT's performance under different orders of ICL demonstrations: 1) Causal first: from causal demonstrations to non-causal demonstrations; 2)

---

Error Type
text-davinci-003
gpt-3.5-turbo
gpt-4
Fake conditions
31
39
45
Incorrect basic event relationships
19
16
Wrong target event localization
15
13
Incorrect commonsense knowledge Other
12
23
17
Table 1I: The distribution (%) of causal reasoning error types ChatGPT.
Demonstrations
Pos
Neg
Full
Firstly, causal reasoning is comprehensive skill that requires commonsense knowledge, well the ability to understand basic event relationships, and to perform logical reasoning based on infor- mation. However, ChatGPT still not entirely reliable in these aspects, leading to the accumula- tion of errors. On the other hand, ChatGPT has encountered numerous causal event pairs in pre- training data, enabling it to associate many event pairs with potentially causal contexts. However, this context might not align with the input.
Zero-shot
74.4
67.7
68.9
Causal first
97.1
23.4
35.1
Non-causal first
92.8
29.0
39.2
Randem1
95.6
28.0
38.8
Random2
95.7
27.8
38.6
Random3
95.7
27.4
38.3
Table 12:
Experimental results (%) of "top similar" in $5.1 with different orders of demonstrations on the ESC dataset.
Non-causal first: from non-causal demonstrations causal demonstrations; Random: conduct three times experiments under random demonstra- tion orders. Besides, Zero-shot indicates the per- formance of ChatGPT under the zero-shot setting. Firstly, we find that Causal first is more inclined to classify event pairs as causal compared to Non- causal first. This might be because that the demon- strations located earlier have stronger impact the ChatGPT.
Besides, compared text-davinci-003. which not fine-tuned on dialog data, gpt-3.5-turbo and gpt-4 show a clearer tendency to fabricate addi- tional fake conditions to establish causality. This could be due to dialog data guiding them to pro- duce longer and more divergent responses, which deviate from the context provided in the original in- put. Additionally, gpt-4 introduces fewer incorrect commonsense knowledge, it has better grasp of knowledge.
Secondly, despite different orders, all of these few-shot settings make the ChatGPT more inclined to classify event pairs as causal compared to the zero-shot setting.
E Details of CoT Experiments
Figure 7 shows the examples of the demonstrations utilized in our few-shot CoT experiments. Figure 8 shows the examples of the reasoning chains gener- ated by ChatGPT.
D Error Analysis for the Causal Reasoning of ChatGPT
As shown in Table 11, we analyze the error types of ChatGPT's s causal reasoning by observing the rea- soning chains generated in the CoT setting. Specif- ically, we randomly select 100 instances of errors for each model on the ESC dataset, and then man- ually annotate them to analyze the types of errors among different models. Common error types in- clude: 1) Fabricating additional fake conditions to establish causality, even these conditions are not described or incorrect in the input sentence; 2) Misunderstanding basic event relationships such sub-events and temporal relationships between events; 3) Failing to accurately identify which two events the causal question is referring to; 4) Intro- ducing incorrect commonsense knowledge.

---

COT Demonstrations for the ECI Task
COT Demonstrations for the CR Task
An earthquake with magnitude of least 5, hit southern Iran on Sunday, killing at least 10 peaple and injuring many others.
INPUT:
Event A: This phenomenon is called speciation,
Event B: A species gradually adapted to its environment.
Question: is there causal relationship between Event and Event B
Question; the INPUT sentence, there causal relationship between "injuring" and 'earthquake"
Reasoning Process:
Reasoning Process:
Based on expertise in field biology, speciation often the result of environmental adaptation. 2. Therefore, there is causal relationship between Event A and Event Answer: Yes
1. From the INPUT, An earthquake hit southern Iran 2. From the INPUT, the earthquake is with a magnitude of at least 5.9
3. magnitude of least is dangerous enough for destroying buildings and causing things to fall.
4. Typically, high-magnitude earthquakes cause casualties.
injured. 6. Therefore, there is very high probability that the injuring is caused by the earthquake in the input sentence. Answer: Yes.
5. From the INPUT, there have been people
Power was restored to the afflicted villages on the Gulf island Qeshm after blackout caused by the quake. which struck on Sunday with force of about 6. on the Richter scale
INPUT:
Event A: The leader of the orphanage decided to find kind person to adopt
Event B: Her mother abandoned her.
Question: is there causal relationship between Event
causal relationship between 'struck" and "quake"?
Question: the INPUT sentence,
Reasoning Process:
Reasoning Process:
1. There is no correlation between Event A and Event B. there no correlation, then there must be causal relationship.
1. quake subject verb struck, and they together form the event that the earthquake struck the villages.
3. Therefore, there is no causal relationship between Event and Event B. Answer: No
2. Therefore, they are different component one event.
3. causal relationship is between events, rather than verb and its subject.
4. Therefore, there not causal relationship between "struck" and "quake".
Answer: No.
Figure 7: Examples of the CoT demonstrations.

---

Reasoning Chains Generated by ChatGPT for the ECI Task
Reasoning Chains Generated by ChatGPT for the CR Task
1. Correct Answer: Yes
1. Correct Answer: Yes
Tens thousands of people took to the streets in nationwide strike protest new taxes and government spending cuts demanded the International Monetary Fund and other European nations before heavily indebted Greece gets eurol10 billion $141 billion) ) loan package keep it from defaulting.
INPUT:
Event A: The swimmer saw shark fins in this water. Event chose leave here.
Question: there causal relationship between Event A and Event B
Reasoning Process:
1. Seeing shark fins the water is sign of danger and can cause fear in swimmer.
Question: In the INPUT sentence, is there causal relationship between 'protest' and "keep"?
2. Therefore, there causal relationship between Event A and Event
Reasoning Process:
From the INPUT, tens of thousands of people took to the streets nationwide strike to protest new taxes and government spendling cuts.
Answer: Yes
From the INPUT, the European nations demanded these austerity measures in order to 'keep" Greece from defaulting.
3. Typically, people protest or strike form of resistance against certain event.
4. From INPUT, the event the austerity measures demanded the European nations.
5. Therefore, there is causal relationship between "protest" and "keep". Answer: Yes.
2. Correct Answer: No
2. Correct Answer: No Event The teacher wants students think spontaneously.
INPUT: man who was fired last week from his job an accounting firm in suburban Detroit walked into his former office with shotgun on Monday and shot three people, killing one, before fleeing
Event B: Various elimination processes was adopted to achieve that.
leading police officers high speed chase, the authorities said. Question: In the INPUT sentence, is there a causal relationship between 'fleeing"
Question: there causal relationship between Event and Event B
'walked"?
Reasoning Process:
Reasoning Process:
1. The elimination processes used encourage students to think spontaneously.
1. From the INPUT, the man walked into his former office
with a shotgun.
2. Therefore, there causal relationship between Event A
2. Fram the INPUT, the fled lead police officers high-speed chase. 3. Typically, when people are facing with an intimidating situation, they will fled.
and Event
Answer: Yes
4. From the INPUT, the intimidating situation is walking
into his former office with shotgun.
5. Therefore, there a causal relationship between "fleeing" and "walked". Answer: Yes.
Figure 8: Examples of the reasoning chains generated ChatGPT.


Reviews:
[{"review_id": "9PjUM9j1kG", "reviewer": "Reviewer_MYMZ", "summary": "", "questions": "For the error cases, what are some common error types? Are there any patterns and are there some reasons? Does different model show similar distributions?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Error Analysis\n\nThanks for insightful suggestions. We will incorporate error analysis into the revised version.\n\nWe analyze the types of errors by observing the reasoning chains generated in the CoT setting. Common error types include:\n\n1. Fabricating additional fake conditions to establish causality, even if these conditions are not described or are incorrect in the input sentence.\n2. Misunderstanding basic event relationships such as sub-events and temporal relationships between events.\n3. Failing to accurately identify which two events the causal question is referring to.\n4. Introducing incorrect commonsense knowledge.\n\nWe believe that causal reasoning is a comprehensive skill that requires commonsense knowledge, as well as the ability to understand basic event relationships, and to perform logical reasoning based on information. However, ChatGPT is still not entirely reliable in these aspects, leading to the accumulation of errors. On the other hand, ChatGPT has encountered numerous causal event pairs in pre-training data, enabling it to associate any event pair with a potentially causal context. However, this context might not align with the input.\n\nWe randomly selected 100 instances of errors for each model and manually annotated them to analyze the types of errors among different models. Compared to text-davinci-003, which is not fine-tuned on dialog data, gpt-3.5-turbo and gpt-4 show a clearer tendency to fabricate additional fake conditions to establish causality. This could be due to dialog data guiding them to produce longer and more divergent responses, which deviate from the context provided in the original input. Additionally, gpt-4 introduces fewer incorrect commonsense knowledge, as it has a better grasp of knowledge.\n\n**Table. Distribution of Error Types across Various Models on 100 Error nstances.**\n\n| Error Type                          | text-davinci-003 | gpt-3.5-turbo | gpt-4 |\n| :---------------------------------- | :--------------: | :-----------: | :---: |\n| Fake conditions                     |        31        |      39       |  45   |\n| Incorrect basic event relationships |        19        |      17       |  16   |\n| Wrong target event localization     |        15        |      11       |  13   |\n| Incorrect commonsense knowledge     |        12        |      15       |   9   |\n| Other                               |        23        |      18       |  17   |\n\n\n\n## 2. Further Analyze on RLHF\n\ntext-davinci-002 was trained without RLHF. As described in lines 191-203, text-davinci-003, gpt-3.5-turbo, and gpt-4 are the results of further RLHF applied to text-davinci-002. By comparing text-davinci-002 (before RLHF) to other three models (after RLHF), we concluded in lines 280-293 that RLHF exacerbates the issue of causal hallucination in ChatGPT.\n\nOur work focused on ChatGPT, and due to OpenAI's decision not to open-source it, we lack access to specific details about the data and experimental setup for ChatGPT's RLHF. As a result, we did not provide further analysis about the RLHF of ChatGPT.\n\nAs an alternative approach, we will add RLHF analysis based on the Anthropic RLHF dataset [1], which is an open-source RLHF dataset. Some of our preliminary findings include:\n\n1. Among questions about \"Why\", only 10.29% include the word \"not\". This indicates that the majority of trianing data in the dataset guide the LLM about causality rather than non-causality.\n2. In the annotated gold responses, the frequency of the word \"yes\" is approximately twice that of the word \"no\". This could potentially increase the likelihood of the LLM producing the positive label rather than the negative label.\n\nBoth of these characteristics in the RLHF data could potentially exacerbate ChatGPT's causal hallucination, leading it to assume causal relationships between events, regardless of whether those relationships actually exist.\n\nA further analysis of why RLHF exacerbates the issue of causal hallucination requires a controlled variable analysis of the RLHF data, training iterations, base model, etc. This necessitates the training of LLM under various RLHF settings, which is time-consuming. Due to time constraints, we could not provide this analysis during the rebuttal period, but it will be included in the revised version. For instance, whether balancing the RLHF dataset can alleviate the causal hallucination issue in LLM.\n\n> [1] Bai Y, Jones A, Ndousse K, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback[J]. arXiv preprint arXiv:2204.05862, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper comprehensively evaluate the causal reasoning abilities of gpt models together with several insights summarized from experiments. \n2. The analysis is interesting and might lead to future improvements for causal reasoning.\n", "reasons_to_reject": "1. It would be interesting to show the error analysis from the results. For example, what are some common types for the hallucination?\n2. The authors claim that the hallucination might be from the RLHF. It might be great to have the some preliminary/toy analysis on that. For example, the same model before and after RLHF.", "questions_to_authors": "For the error cases, what are some common error types? Are there any patterns and are there some reasons? Does different model show similar distributions?", "paper_topic_and_main_contributions": "This paper perform a comprehensive study to evaluate the causal reasoning abilities (on event causality identification, causal discovery and causal explanation generation tasks) of GPT models including text-davinci-002, text-davinci-003, chatgpt and gpt4. They find that ChatGPT models can not do the causal reasoning so well and have serious hallucination issues. They also show that the hallucinations might be due to the RLHF. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ViAY8MWS69", "reviewer": "Reviewer_SCUQ", "summary": "", "questions": "a. As mentioned in the \"Key takeaways\" - ChatGPT is sensitive to the words used to express the causal concept in the prompt- could the authors elaborate more on the exact prompts used under the zero-shot setting for each of the tasks? I wonder if the questions asked in the prompt would affect the performance of the models.\n\nb. For model performance evaluation, could the authors elaborate more on how the responses of ChatGPT were mapped to the desired format that could be compared to the ground truth of different datasets? For example, previous studies have suggested that the direct inferencing results from ChatGPT models after adding CoT may contain extra argument and elaboration. It would be helpful if the authors could explain a bit more about their methods for processing the response.\n\nc. Table 1 and Table 2 provide the fine-grained results for ChatGPT on the pos, neg classes. How about the ones for the pre-trained baseline models? Would the fine-tuned models also do better at identifying causal event pairs but not in non-causal event pairs, or vice versa?\n\nd. For the \"top k similar\", could the authors elaborate more on whether the k examples are within one class (neg or positive) or from both classes? How might it influence the evaluation results?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. The forms of prompts.\n\n**As we mentioned in Sec 3.2 Experiment Setting (lines 186-188), we illustrated the exact prompts used under the zero-shot setting for each of the tasks in Figure 1.**\n\n**The prompt of the ECI task:**\n\nInput: \\<input sentence\\>\n\nQuestion: is there a causal relationship between “\\<event1\\>”and “\\<event2\\>” ?\n\nAnswer:\n\n**The prompt of the multiple choice CD task:**\n\n*(This task involves samples that ask for selecting the result of the input event. For such samples, we modify the word “cause” in the question to the word “result”.)*\n\nInput Event: \\<input event\\>\n\nQuestion: Please select the cause of the input event from the following options.\n\nOption 1: \\<option1\\>\n\nOption 2: \\<option2\\>\n\nAnswer:\n\n**The prompt of the Binary Classification CD task:**\n\nEvent A: \\<input event1\\>\n\nEvent B: \\<input event2\\>\n\nQuestion: is there a causal relationship between Event A and Event B ?\n\nAnswer:\n\n**The prompt of the CEG task:**\n\nCause: \\<input cause\\>\n\nEffect: \\<input effect\\>\n\nQuestion: why the cause can lead to the effect?\n\nAnswer:\n\n\n\n## 2. The impact of different questions in prompts.\n\nWe discuss if the questions asked in the prompt would affect the performance of ChatGPT in Sec 5.3 and Sec 5.7.\n\nIn Sec 5.3, we analyze the performance of ChatGPT with prompts that express the causal concept in different ways. We find that:\n\n1. The prompt based on the counterfactual causality view exacerbates the causal hallucination issue. \n2. Using the phrase \"one-step\" to restrict the captured causal relationships can improve performance on the Neg class, but significantly reduces performance on the Pos class.\n3. The performance differences of various causal cue words in prompts, such as \"lead to\" and \"result in\", are quite significant. This further indicates that it is challenging to accurately convey what causality means to ChatGPT through prompts.\n\nIn Sec 5.7, we analyze open-ended questions in the prompt, directly asking the ChatGPT to generate pairs of causal events in the text, rather than responding \"yes\" or \"no\" to specified target event pairs. We find that the open-ended prompts decrease the performance of ChatGPT. This is mainly because open-ended prompts require ChatGPT to jointly perform the event extraction and the causal reasoning task. However, previous studies show that ChatGPT is not good at extracting events.\n\n\n\n## 3. How the responses of ChatGPT were mapped to the desired format?\n\nOur goal is to reflect the actual performance of ChatGPT, avoiding inaccurate evaluation results stemming from wrong automated mapping rules.\n\n**For the CEG task:**\n\nno further text map is necessary; we directly use the responses of ChatGPT as its final prediction.\n\n**For the ECI task**: \n\nUnder the zero-shot setting, we found that out of 138,166 instances, only 96 responses do not begin with \"yes\" or \"no\". Therefore, responses starting with \"yes\" will be assigned the label 1, while other cases will be assigned the label 0.\n\nUnder the ICL and CoT settings, we construct demonstrations ending with the labels \"yes\" or \"no\". Therefore, responses from ChatGPT ending with \"yes\" will be mapped to label 1, while others will be mapped to label 0.\n\nFor the prompts discussed in Section 5.3, these prompts are directional. Therefore, for a pair of events, we reverse their order to obtain two responses from ChatGPT. If at least one of the two responses starts with \"yes\", it is assigned the label 1, while other cases will be categorized as label 0.\n\nFor the open-ended prompt discussed in Section 5.7, we added descriptions in the prompt about the output format, allowing for easy parsing into causal event pairs.\n\n**For the CD task:**\n\nMost responses begin with phrases \"option1\", \"option2\", \"yes\", or \"no\", which can be easily mapped to labels. For other response formats, we manually assigned labels for each different format of response, and the mapping results are recorded in the supplementary file \"CD_and_CEG_compute_score.py\".\n\n\n\n## 4. Would the fine-tuned models also do better at identifying causal event pairs but not in non-causal event pairs, or vice versa?\n\nThank you for constructive suggestions! As we primarily focus on ChatGPT, and there is no available previous performance reports for Pos and Neg acc with fine-tuned models, we did not provide them on Table 1 and Table 2. The following table presents these results, would be in the revised version.\n\n| For Table 1 (ECI) |         |   ESC   |          |         |   CTB   |          |         | MAVEN-ERE |          |\n| :---------------- | ------- | :-----: | :------: | :-----: | :-----: | :------: | :-----: | :-------: | :------: |\n| **Methods**       | **Pos** | **Neg** | **Full** | **Pos** | **Neg** | **Full** | **Pos** |  **Neg**  | **Full** |\n| BERT-Base         | 59.5    |  83.6   |   79.7   |  46.4   |  87.8   |   86.2   |  75.7   |   86.9    |   85.1   |\n| RoBERTa-Base      | 63.8    |  82.8   |   79.8   |  62.3   |  86.4   |   85.5   |  76.6   |   87.1    |   85.4   |\n\n**In the ECI task:** it can be observed that the fine-tuned models do better at identifying non-causal event pairs. This is because there are much more negative examples than positive examples in the ECI dataset, and the fine-tuned models have learned such data distribution.\n\n| For Table 2 (CD) |         | e-CARE  |          |         |  COPA   |          |\n| ---------------- | :-----: | :-----: | :------: | :-----: | :-----: | :------: |\n| **Methods**      | **Pos** | **Neg** | **Full** | **Pos** | **Neg** | **Full** |\n| BERT-Base        |  65.7   |  95.2   |   65.0   |  59.5   |  88.9   |   54.5   |\n| RoBERTa-Base     |  64.6   |  74.6   |   64.5   |  66.0   |  69.0   |   63.5   |\n\n**In the CD task:** we find that the fine-tuned models do better at identifying non-causal event pairs. This may be because, as mentioned in lines 268-272, the non-causal pairs in the e-CARE and COPA datasets are manually annotated given a input event, and they have a simple structure and weak correlation with the input events, making them easier to identify.\n\n## 5. The classes of demonstrations in \"top k similar\".\n\nI appreciate your valuable feedback. In the experiment, we did not constrain the class of the demonstrations to be Pos or Neg.\n\nBelow, we analyze the impact of the classes of demonstrations on performance of \"top k similar\". For k=4, we first divide the dataset into five subsets, each containing instances that only use 0, 1, 2, 3, or 4 Pos demonstrations in their top 4 similar demonstrations, respectively. Then, we present the performance of the \"top 4 similar\" on these five subsets. \"Proportion of Pos\" is the proportion (%) of Pos instances in the corresponding subset.\n\n| Used Demonstrations | Pos Acc | Neg Acc | Full Acc | Proportion of Pos |\n| ------------------- | :-----: | :-----: | :------: | :---------------: |\n| 0 pos 4 neg         |  98.4   |  21.8   |   26.7   |        6.4        |\n| 1 pos 3 neg         |  88.8   |  27.5   |   39.1   |       18.9        |\n| 2 pos 2 neg         |  88.6   |  36.3   |   49.5   |       25.2        |\n| 3 pos 1 neg         |  89.5   |  47.1   |   66.9   |       46.6        |\n| 4 pos 0 neg         |  95.3   |  35.7   |   83.7   |       80.6        |\n\nWe can observe that when using only Pos or Neg demonstrations, the model achieves a higher Pos acc. This might be because that including only one classes prevents the model from contrasting the meanings of different labels, thus potentially confusing the model's understanding of the task objectives.\n\nAdditionally, when using both Pos and Neg demonstrations, there is a smaller change in Pos acc, while Neg acc increases as the num of Pos demonstrations rises. This might be because having more Pos demonstrations helps the model understand the situations that truly involve causality, thus avoiding misclassifying non-causal instances as causal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Causal reasoning is an important task in the NLP community. Having a thorough evaluation of the available LLMs like chatGPT on this task would be informative for the community and also helps with the future improvement regarding the performance of the LLMs.\n\nCompared to existing evaluations that normally took a few hundred examples for evaluation, the current study covers more datasets and tasks specifically for causal reasoning tasks.\n\nBesides standard evaluation metrics (precision, f1, accuracy, etc.), the study also provides a detailed discussion and comparison of the model performance.", "reasons_to_reject": "There is not much algorithm breakthrough.", "questions_to_authors": "a. As mentioned in the \"Key takeaways\" - ChatGPT is sensitive to the words used to express the causal concept in the prompt- could the authors elaborate more on the exact prompts used under the zero-shot setting for each of the tasks? I wonder if the questions asked in the prompt would affect the performance of the models.\n\nb. For model performance evaluation, could the authors elaborate more on how the responses of ChatGPT were mapped to the desired format that could be compared to the ground truth of different datasets? For example, previous studies have suggested that the direct inferencing results from ChatGPT models after adding CoT may contain extra argument and elaboration. It would be helpful if the authors could explain a bit more about their methods for processing the response.\n\nc. Table 1 and Table 2 provide the fine-grained results for ChatGPT on the pos, neg classes. How about the ones for the pre-trained baseline models? Would the fine-tuned models also do better at identifying causal event pairs but not in non-causal event pairs, or vice versa?\n\nd. For the \"top k similar\", could the authors elaborate more on whether the k examples are within one class (neg or positive) or from both classes? How might it influence the evaluation results?", "paper_topic_and_main_contributions": "This study conducts a comprehensive evaluation of the ChatGPT family of models on their ability of four causal reasoning tasks from five datasets. The evaluation results suggest the current limitation of ChatGPT on this task. The findings would be insight for similar evaluations on LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 230 event -> even"}, "extra_text": {}}, {"review_id": "OS7Q2ivQ8I", "reviewer": "Reviewer_vWs8", "summary": "", "questions": "A. According to Table 4, “4 pos + 48 neg” does lead to better performance (76.2 to 68.9). Does this mean “overall accuracy did not improve” (line 331)?\n\nB. For Table 2, how do you interpret “AVG-BLEU” scores?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Hallucination in LLM has been discussed in [1] and [2].\n\n> [1] Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., ... & Fung, P. (2023). A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023.\n>\n> [2] Manakul, P., Liusie, A., & Gales, M. J. (2023). Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896.\n\nThanks for the review. Previous works [1] and [2] discussed the issue of illusions. However, our findings are not contained by [1] and [2]. Furthermore, we correct the misconception in [1], which believes that ChatGPT is good at causal reasoning.\n\n**For [1]:** [1] emphasizes the diversity of the evaluated tasks, while our work focuses on the comprehensive evaluation of causal reasoning tasks. For causal reasoning, [1] only tested 30 examples from the e-CARE [3] dataset and incorrectly claimed that ChatGPT achieved good performance, which our experiments demonstrate not to be the case. Besides, we identified a characteristic causal illusion issue in ChatGPT, where it tends to fabricate causal relationships between events. We further analyzed the possible reasons for such illusion and tested ChatGPT's performance in settings like in-context learning and chain-of-thought. Additionally, [1] lacks the experimental results of GPT-4.\n\n> [3] Du L, Ding X, Xiong K, et al. e-CARE: a New Dataset for Exploring Explainable Causal Reasoning[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022: 432-446.\n\n**For [2]:** There is no mention of any causal reasoning task in [2], and [2] focuses on offering a general solution to the illusion of ChatGPT rather than evaluating the causal reasoning capabilities. Besides, The hallucination issue of ChatGPT is only one aspect analyzed in our work. Therefore, compared to ours, [2] belongs to a different line of research.\n\n**Besides:** In addition to experiments and findings related to illusions, we also analyzed various factors influencing ChatGPT's causal reasoning ability, including the ways of expressing causality in prompts, the number of events in the sentence, the lexical distance between events, the type of causality (implicit or explicit), and whether an open-ended question format is utilized.\n\n\n\n## 2. Sensitivity to prompt is mentioned in [3] and [4].\n\nClearly, sensitivity to the prompt is a well-known characteristic of the ChatGPT, but it is necessary for a comprehensive evaluation to verify and report it in the context of causal reasoning tasks. Regarding prompt variations, we not only reported ChatGPT's sensitivity but also conducted a detailed analysis of the effects of different causal prompts, including the ways of expressing causality in prompts and whether an open-ended question format is utilized.\n\nSensitivity to the prompt is just one of our secondary contributions. Our work evaluated four causal reasoning task formats, five datasets, and four versions of ChatGPT, including GPT-4. Our other contributions can be summarized as follows:\n\n1. We empirically demonstrated ChatGPT's performance in causal reasoning.\n2. We summarized the characteristic of illusions in ChatGPT's causal reasoning.\n3. We analyzed potential reasons for such causal illusion issues.\n4. We investigated the impact of techniques like ICL and CoT on causal illusions of ChatGPT.\n5. We studied the influence of various prompt-agnostic factors, including event density, event spacing, and causal types.\n\n\n\n## 3. For the CEG task, the recent LLM baseline (e.g., LLaMA and FLAN-T5) should be added.\n\nThank you for constructive suggestions. Our work focused on ChatGPT, so there was no further discussion on the performance of other types of LLMs in causal reasoning. Instead, we used fine-tuned small PLMs as baselines to contrast the differences between zero-shot LLM and fine-tuned small PLMs. We will add the following CEG experiments with fine-tuned LLaMA 7B and FLAN-T5 11B in the revised version.\n\n| Method           | AVG-BLEU | ROUGE-l | Human Evaluation |\n| :--------------- | :------: | :-----: | :--------------: |\n| GRU-Seq2Seq      |   18.7   |  21.3   |        0         |\n| GPT2             |   32.0   |  31.5   |       20.0       |\n| **LLaMA 7B**     |   40.3   |  37.1   |       63.0       |\n| **FLAN-T5 11B**  |   38.5   |  42.8   |       66.0       |\n| text-davinci-003 |   10.1   |  37.5   |       83.0       |\n| gpt-3.5-turbo    |   7.3    |  40.3   |       82.0       |\n| gpt-4            |   6.5    |  39.8   |       85.0       |\n\nFor LLaMA 7B and FLAN-T5 11B, we utilize LoRa to perform parameter-efficient fine-tuning on the e-CARE dataset. We find that fine-tuned LLaMA and FLAN-T5 achieved significant better performance compared to GRU-Seq2Seq and GPT2. This is mainly due to the increased model scale enhancing the model's capability.\n\nIn manual evaluation, we find that the explanations generated by LLaMA and FLAN-T5 are highly correlated with the input events. However, the explanations might be mere repetitions of the input events or provide relevant but uninformative descriptions that cannot be used for explanation.\n\nBesides, compared to ChatGPT, the explanations provided by these two fine-tuned LLMs are noticeably shorter, as the gold explanations provided by e-CARE are very concise. However, ChatGPT excels in providing more comprehensive and detailed explanations in the zero-shot setting.\n\nFor AVG-BLEU, it is a text generation metric similar to \"precision\" in text classification tasks. Therefore, ChatGPT received a lower AVG-BLEU score because it generated detailed explanations that were not included in the annotations from e-CARE.\n\nFor ROUGE-l, it is a text generation metric similar to \"recall\" in text classification tasks. It's worth noting that fine-tuned LLMs and ChatGPT achieve similar ROUGE-l scores, indicating that they all have some gap from the gold explanations in e-CARE. This is because that the fine-tuned LLaMA and FLAN-T5 may generate less informative explanations compared to e-CARE, e.g., mere repetitions of the input events. However, ChatGPT may offer valuable explanations for the input causal event pairs, but from different perspectives or in distinct syntactic forms compared to the gold explanations in e-CARE.\n\n\n\n\n\n\n\n## 4. The impact of the order of few-shot demonstrations.\n\nIn the early experiments, we conducted multiple random tests on the ICL performance of the model. We shuffled the order of few-shot demonstrations, but the performance did not show significant changes. Therefore, there was no further discussion on the order. We will add the following discussion on order of few-shot demonstrations into the revised version.\n\nFor the few-shot setting \"top 4 similar\" in Sec 5.1, we analyze the model's performance under different demonstration orders:\n\n1. **Causal First**: from causal demonstrations to non-causal demonstrations.\n2. **Non-Causal FIrst**: from non-causal demonstrations to causal demonstrations.\n3. **Random**, conducting three times experiments under random demonstration orders.\n\n\"Zero-shot\" indicates the performance under the zero-shot setting.\n\n| Method           | Pos Acc | Neg Acc | Full Acc |\n| ---------------- | :-----: | :-----: | :------: |\n| Zero-shot | 74.4    | 67.7    | 68.9     |\n| Causal FIrst     | 97.1    | 23.4    | 35.1     |\n| Non-Causal FIrst | 92.8    | 29.0    | 39.2     |\n| random1          | 95.6    | 28.0    | 38.8     |\n| random2          | 95.7    | 27.8    | 38.6     |\n| random3          | 95.7    | 27.4    | 38.3     |\n\nWe find that Causal First is more inclined to classify event pairs as causal compared to Non-Causal First. This might be because that the demonstrations located earlier have a stronger impact on the model.\n\nIn addition, despite different orders, all of these few-shot settings make the model more inclined to classify event pairs as causal compared to \"zero-shot\".\n\n\n\n## 5. Lexical distances in section 5.4\n\nAs explained in lines 399-408, with the increase in the distance between events: (1) the proportion of non-causal events among all event pairs will rise; (2) the correlation between events decreases, causing the model to be more inclined to assign event pairs as non-causal. The combination of these two effects results in Full acc becoming an imbalanced metric, reflecting only the increase in accuracy of non-causal pairs but not reflecting the decrease in accuracy of causal pairs.\n\nTo further alleviate the confusion caused by the Full acc metric, we will add the F1 score as a more comprehensive metric in the revised version. The following table shows the changes in the F1 score of ChatGPTs across different distances between events. An outlier is the F1 score of GPT-4 at the interval [25,30). This is due to the fact that out of 1000 test instances for GPT-4, there are only 35 examples within the interval [25,30), leading to more random performance. However, all other results demonstrate that ChatGPTs' performance decreases as the event distance increases.\n\n| Model            | [1,5) | [5,10) | [10,15) | [15,20) | [20,25) | [25,30) | [30,+) |\n| :--------------- | :---: | :----: | :-----: | :-----: | :-----: | :-----: | :----: |\n| text-davinci-002 | 44.6  |  44.1  |  39.7   |  36.5   |  32.4   |  30.2   |  29.0  |\n| text-davinci-003 | 52.3  |  49.8  |  47.3   |  44.7   |  42.7   |  41.9   |  35.5  |\n| gpt-3.5-turbo    | 49.0  |  46.9  |  42.0   |  39.2   |  34.7   |  39.9   |  28.6  |\n| gpt-4            | 51.8  |  48.9  |  35.8   |  41.0   |  34.5   |  62.5   |  25.0  |\n\n\n\n## 6. Performance changes of \"4 pos + 48 neg\" in Table 4.\n\nThank you for careful review. This was an error in writing line 331, \"4 pos + 48 neg\" is one of our 29 few-shot experimental results, the overall accuracy of \"4 pos + 48 neg\" **does** improve. The error did not occur in Table 4, where the reported experimental results are reliable. Thank you for pointing out this error. We will fix it and clarify the confusion in the revised versions.\n\nIn the analysis of ICL and CoT (Table 4, 5, and 6), it can be observed that several methods, including \"4 pos + 48 neg\", outperform the zero-shot method in terms of Full acc. However, a common trend among these methods is that they enhance Pos acc to improve Neg acc, or vice versa. Besides, the same method performs inconsistently across different versions of ChatGPT.\n\nWe believe that the improvement of overall performance in causal reasoning ability should not show a compromise between identifying causal and non-causal pairs. Such behavior, including \"4 pos + 48 neg\", would merely be a case of robbing Peter to pay Paul, rather than a substantial improvement in the overall performance of causal reasoning.\n\n\n\n\n\n## 7. For Table 2, how do you interpret “AVG-BLEU” scores?\n\n**What is the AVG-BLEU that we used:**\n\nAs mentioned in line 172-183, we conducted experiments using the e-CARE dataset and employed the AVG-BLEU metric as described in the e-CARE paper for evaluation. Specifically, e-CARE first calculates the BLEU score for each test instance and then compute the average of all calculated BLEU scores to obtain the AVG-BLEU for the entire test dataset. We utilized the open-source code provided by e-CARE to compute the AVG-BLEU.\n\n**Explanation of the experimental results for AVG-BLEU in Table 2：**\n\n**AVG-BLEU** is an evaluation metric similar to precision. Roughly speaking, it calculates the ratio of the words appearing in both the generated output and the gold response text to the length of the generated output. As mentioned in line 302-307, ChatGPT tends to generate  more complete and detailed explanations. However, the explanations annotated in e-CARE are shorter and more concise, and the AVG-BLEU score will only consider the output part annotated in e-CARE as correct, treating all other output part as incorrect. Therefore, ChatGPT obtained relatively low AVG-BLEU scores. In addition, compared to text-davinci-003, both gpt-3.5-turbo and gpt-4 have a lower AVG-BLEU. This is because they have further trained by RLHF and tend to generate longer responses.\n\nOn the contrary, **ROUGE-l** is a metric similar to recall, focusing on whether the model's responses contain the annotated gold explanations. As a result, ChatGPT exhibits contrasting trends in ROUGE-l and AVG-BLEU.\n\n\n\n## 8. Suggestions for improving the presentation.\n\nThank you for valuable suggestion. We will include explanations for the terms \"causal interpreter\", \"hallucination\", and \"lexical distance\" in the revised versions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Given the popularity of ChatGPT, the topic has practical and timely significance.\n2. This paper presents comprehensive experiments and analysis to support main claims.", "reasons_to_reject": "1. This paper does not present sufficient novel findings. Hallucination in LLM has been discussed in [1] and [2]; sensitivity to prompt is mentioned in [3] and [4].\n2. For the CEG task, the baselines are too old to be effective. There are more recent baselines like FLAN-T5 and LLaMA.\n3. For analysis of in-context learning (sec 5.1), the order of few-shot demonstrations should be discussed, as it usually influences the performance.  \n4. This paper states that increasing lexical distances of events lead to worse causal reasoning performance (line 96-98, line 482-483). This is not supported by section 5.4 (line 405-408). In contrast, most models perform better as the distance increases (Figure 3).", "questions_to_authors": "A. According to Table 4, “4 pos + 48 neg” does lead to better performance (76.2 to 68.9). Does this mean “overall accuracy did not improve” (line 331)?\n\nB. For Table 2, how do you interpret “AVG-BLEU” scores?", "paper_topic_and_main_contributions": "This paper measures the causal reasoning abilities of GPT-3.5 and GPT-4. This paper states that evaluating ChatGPT in terms of causal reasoning is important, but not done by current work. To fill this gap, this paper measures 4 ChatGPT models on 5 benchmarks (classified into three types). Results show that these models are better at explaining causal relations instead of classifying them. Analysis shows that these models tend to hallucinate (supported with possible causes), and their performance is sensitive to prompts and the features (like numbers and distances) of causal events in a sentence.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., ... & Fung, P. (2023). A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023.\n\n[2] Manakul, P., Liusie, A., & Gales, M. J. (2023). Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896.\n\n[3] Zhao, Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021, July). Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning (pp. 12697-12706). PMLR.\n\n[4] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": "Typo: line 423, “that” to “than”; line 303, “that” to “than”.\n\nPresentation: it might be better if this paper interprets these terms before using them: “causal interpreter” (explained in sec 4.1.2), “hallucination” (explained in sec 4.1),  and “lexical distance” (sec 5.4)."}}]

Author rebuttal:

Meta-review: The reviewers generally agree that the paper provides a comprehensive evaluation of the causal reasoning abilities of GPT models, offering valuable insights for future improvements. However, they also point out that the paper lacks novelty, with some findings already discussed in previous studies. They suggest that the authors could have used more recent baselines for the Causal Explanation Generation task and discussed the order of few-shot demonstrations. The reviewers also note inconsistencies in the paper's claims and results, and suggest clarifications on certain terms and results. The authors' rebuttals address most of the concerns raised, providing detailed explanations and promising to include additional analyses in the revised version.
Decision: Accept-Findings