CRAB: Assessing the Strength of Causal Relationships Between Real-world Events
Authors: Angelika Romanou
Syrielle Montariol
Debjit Paul
Leo Laugier
Karl Aberer
Antoine Bosselut
Venue: EMNLP 2023 Main (2023)
Keywords: causal reasoning
benchmark
causal score
event causality
TL;DR: We introduce CRAB, a benchmark for evaluating reasoning capabilities of LLMs about causality between real-world events.

Abstract: Understanding narratives requires reasoning about the cause-and-effect relationships between events mentioned in the text. While existing foundation models yield impressive results in many NLP tasks requiring reasoning, it is unclear whether they understand the complexity of the underlying network of causal relationships of events in narratives. In this work, we present CRAB, a new Causal Reasoning Assessment Benchmark designed to evaluate causal understanding of events in real-world narratives. CRAB contains fine-grained, contextual causality annotations for ~2.7K pairs of real-world events that describe various newsworthy event timelines (e.g., the acquisition of Twitter by Elon Musk). Using CRAB, we measure the performance of several large language models, demonstrating that most systems achieve poor performance on the task. Motivated by classical causal principles, we also analyze the causal structures of groups of events in CRAB, and find that models perform worse on causal reasoning when events are derived from complex causal structures compared to simple linear causal chains. We make our dataset and code available to the research community.

CRAB: Assessing the Strength of Causal Relationships Between Real-World Events
Angelika Romanou, Syrielle Montariol," Debjit Paul," Léo Laugier, Karl Aberer, Antoine Bosselut EPFL
catttnamm.l.stnammee J@epfl.ch
Abstract
HIGH CAUSALITY
indipeddett
Understanding narratives requires reasoning about the cause-and-edd couse-and-effect relationships be- tween events mentioned the text. While existing foundation models yield impressive results in many NLP tasks requiring reasoning, it unclear whether they understand the com- plexity of the underlying network of causal re- lationships of events narratives. In this work, we present CRAB, new Causal Reasoning Assessment Benchmark designed to evaluate causal understanding of events in real-world narratives. CRAB contains fine-grained, con- textual causality annotations for ~ 2.7K pairs real-world events that describe various news- worthy event timelines (e.g., the acquisition of Twitter by Elon Musk). Using CRAB, we measure the performance of several large lan- guage models, demonstrating that most sys- tems achieve poor performance on the task. Motivated by classical causal principles, also analyze the causal structures of groups of events CRAB, and find that models perform worse causal reasoning when events are de- rived from complex causal structures compared simple linear causal chains. We make our dataset and code available the research com- munity.'
ney sibberiigtioo goinniiioo
Tvitter
E5
Twitter reinstates bamtad user accousts.
MEDIUM CAUSALITY
loss nany useze parmannntt:y
Elon Burks closes darl 44 billion
E6
bay Twitter,
ved ba
LOW CAUSALITY
indrect Bi
Nuck offered to buy E1
Pvitter de-1 the York Stock Excliange.
44 billion in April.
Story: /witter Acquisition
Timeline
E1
E2
E3
E5
E6
Figure I: Events from CRAB that lead to event E6, form- ing causal sub-structures with links of various causal strength.
However, building accurate causal mental mod- els of the situations depicted in narratives poses several complex challenges. First, human causality judgments are rarely binary. Instead, they fall on a spectrum depending on human perception of other mediating or confounding events (Pearl, 2009). For example, in Figure 1, E2 is a mediator event for the causal relationship of E1 and E3, likely affecting the human perception of the causal relationship be- tween E1 and E3. Second, causality judgments de- pend on the context depicting the events in question context that can affect perceptions of causality. For example, a high causal judgment might be as- signed between E4 and E6 in Figure 1. However, the introduction of new information about E5, an- other potential cause of E6, might downgrade the perceived intensity of a causal link between E4 and E6. Finally, because context is critically important to judging causal relationships about events, and most narratives offer an incomplete (and sometimes biased) reporting of particular scenarios, multiple sources may be required to paint an accurate pic- ture of the causal relationships between multiple
1 Introduction
Understanding narratives requires understanding the causc-and-effect relationships between inter- connected sub-events of those narratives. When reading text, humans immediately induce potential causal links between the events presented as part of larger scenario (Grunbaum, 1952; Pearl and Mackenzie, 2018). For example, in Figure 1, when reading an article about the acquisition of Twitter in 2022, a reader would implicitly assign causal links between events such E2: "Elon Musk closes 44 billion dollar deal buy Twitter" and "Twitter delists from the NYSE".
"Equal contribution Pttpss//ggtubb.cmm/psfinnnaaaa

---

interconnected events.
crucial for humans to comprehend the underlying factors driving specific events, helping them make sense of the world. predict outcomes, and take prac- tical actions toward those outcomes. Research in causal inference has attempted to formalize actual causality using causal models that map how hu- mans perceive and attribute cause and responsibil- ity to events and their outcomes. However, human perception of causality usually depends on back- ground context, implicit biases, epistemic state, and lack of information, making the task of ac- tual causality attribution challenging to formalize (Matute et al., 2015; Henne et al., 2021). Addi- tionally, in cases where the responsibility of an event can be attributed to more than one preceding event, observers tend to assign different attribu- tion to the contributing causes (Wolff' and Shepard, 2013). Therefore, when events are described with natural language. the causal judgments are not bi- nary but relative, enabling comparisons between causal events (Icard et al., 2017).
Addressing these challenges, we introduce CRAB, new Causal Reasoning Assessment Benchmark that contains fine-gres greiged contextual causality annotations of real-world events that hap- pened in the past ten years and received extensive media coverage. To collect the proposed bench- mark, we design crowdsoureing framework mo- tivated by standard causal principles from cogni- tive science (Cao et al., 2022) and actual causality (Halpem, 2016) that study how humans perceive and express causality and responsibility among events. Using this knowledge frame, we automat- ically extract the events of newsworthy stories by integrating large pre-trained LMs into the dataset creation loop. We then construct causal graphs combinations of inter-connected events forming different causal chains and frames, as presented in Figure 2 from the extracted events and assess the strength of the causal relationships between these events using human annotators.
Our resulting benchmark, CRAB, contains ~22.7 high-quality event pairs, their causal score, and the respective documents in which the events appeared. All the events are grouped into 1.8K causal frames and 352 causal chains. We use this benchmark assess the abilities of tatte-ofttheaar (SoTA) models understand and reason over the causal relationships of real-world events present in a set of contexts (i.e., online news articles). Our analysis reveals that LLMS can capture explicit causal statements through pre-training, but they face difficulty applying causal reasoning to new scenarios, limiting their generalization and accu- racy in offering predictions and explanations. We further stratify our results based on the structures of causal frames and chains, showing that they strug- gle with assessing the causality between events de- rived from complex causal structures compared to simple linear causal chains, especially when these events are extracted from different documents.
Humans tend to attribute different degrees of causality between contributory events, relying primarily on domain and commonsense knowledge (Kıeıman et al., 2023). Causality re- search refers to this set of candidate events that are relevant particular outcome event Causal Frame (Halpern, 2016). An example of this can be seen in Figure 1, where the causal frame of event E6 comprises events E2, and E5. We construct CRAB to collect causal frame subgraphs, where each event is associated with its potential causes, along with a causal score labeled by humans. In that way, subset of events can be directly com- pared using their scores, a proxy for their likelihood to be the cause of target event. Later in this paper, we explore the different types of Causal Frames ex- tracted from CRAB as presented in Figure 2 (left) and we stratify our analysis on LLM assessment based on these structures.
Causal Frame
2 Preliminaries on Causality
Causal Chain Another perspective for assessing the degree of causality between two events is to explore the chain of events that happened through time and led to an outcome event E (Pearl, 2009). We define the causal chain of outcome event the set of paths ending at E in the event's causal graph (see Section 3.1). In the example in Figure 1, the following causal chain can be considered: event E1 led to event E2, which led to event E3. In contrast with causal frames, the concept of the
this section, we define the main causality con- cepts that we use to create and analyze CRAB.
Actual Causality Actual causality refers to the causal relationship between specific events and their causes in the real world (Halpern, 2016) and seeks to understand the precise mechanisms by which one event leads to another, going beyond mere correlation. Understanding actual causality is

---

Causal Frames
Causal Chains
Strong Direct Causality (SD)
Strong Contributory Causality (SC)
Mediation
Direct Causality (D)
High
C2
High
C1
high
high
high
Medium-Low
C2
CI
Ci+1
no causality
low
Confounding
On
causality
High
Nigh
Contributory Causality (C)
Causality (N)
Mixed Causality (M)
C2
E
Low
C1
high
C1
high
nigh
Collider
Ci+1
C2
Nigh
Gi+1
Gn
medium
Cj+1
low
no causality
On
High
Figure 2: Different structures of causal frames (left; Halpern, 2016) and causal chains (right; Pearl, 2009) present CRAB. The patterns in structures are formulated based the different causal judgment scores among events. The colors of cause-nodes represent the causality strength they have towards the event-node
causal chain is heavily related to temporality since the events present in the chain are ordered not only based on causality but also by the time they took place. CRAB leverages both concepts of causal chains and temporality, providing testbed for lan- guage models to assess their ability to perform causal reasoning in different causal chain structures depicted in Figure (right).
Challenges Real-world causality identification
and assessment poses several challenges:
Subjectivity: Causal judgments may be affected by the implicit knowledge and bias of the reader, rather than the content the document. Readers may feel strong impression causation about an event, even when the real causal relation unclear based on the context (Wolff and Shepard, 2013).
3 Dataset Construction
Contextualization: Limited or subjective coverage of stories by the documents makes the automated creation of the story timelines and the respective causal graphs subject to incompleteness.
In this section, we give an overview of causal event linking and its associated challenges, and describe our approach for building our CRAB benchmark centered around these challenges. We create pus of documents from which we automatically extract events, build timeline, and collect causal- ity judgments between event pairs. The full data creation pipeline is described in Figure 3.
Temporality: Extracting events from single doeu- ment and ordering them based on their occurrence in time is a relatively straightforward task. How- ever, this task is more complex in multi-document setting as extracted events are grounded in different documents, which may be published on different dates. Consequently, their respective timelines may be interleaved.
3.1 Overview
We consider set of documents covering a news story. Each document reports several events associ- ated with that story. The tide-erdered list of these extracted events defines timeline associated with the story. From timeline and its set of documents, we can build causal event graph. Our goal is identify the causal relations between the events in the graph using the documents in which these events are mentioned.
While methods have been proposed for causal link identification and assessment, they mostly focus binary commonsense causality detection across short, single-document text (Zhang et al., 2023), which does not adequately address the unique chal- lenges of real-world causality judgment. In the fol- lowing sections, we present our approach for build- ing CRAB that addresses the above challenges.

---

Documents about a story
Story Graphs per Causal Score Threshold
Dataset Views
Cj > 20
Cj >
50
Cj > 80
Causal Frames
Doc 3
Doc 2
For E2
For
For
Ex
Doc 1
Ge
E4
Ce
E4
E1
E2
Event 3
E2
CN
E3
Event
Story Timeline
Causal Chains
Flon Nask reveals E1
Blon Mask closee dollara 44 billion deal to buy Twicter.
purchased more then 98 of Puitter's a shazee.
E3
E3
Elon Maak gata tho takeover teuporarily hold.
Twitter de-lists from New Stock Exchange.
E4
Figure 3: CRAB data pipeline overview: We collect documents covering newsworthy stories, create a timeline with the main events extracted from the documents for each story, and crowdsource human annotations on the causal judgments between the events (score 0 to 100). Based on these scores, we generate causal graph for each story that can be filtered on different causal score thresholds. CRAB can also be viewed from the perspective of causal frames and causal chains as presented in Scction 2. Same-color events originate from the same document.
3.2 Document Selection
and remove generations that represent claims or be- liefs by filtering the utterances with conversational verbs followed by quotes (e.g. "said" + "*"). Then, two experts manually evaluate the validity of all the remaining generated events. After filtering, our datasct contains 384 unique events.
We first select 20 stories about major events that happened around the world during the past 10 years, covering geopolitical, social, and environmental themes. We scrape the top 20 news articles per story from Google News AP1, filter the invalid ones and clip them to 250 words.² Our final corpus contains 173 documents, which we use for event extraction (see descriptive statistics and selection process details in Appendix A.1 and Table 17).
Timeline Construction & Event Disambiguation As causality is conditioned on temporality, we man- ually create a timeline of the extracted events for each the 20 storics, by considering documents associated with the story. While building these timelines, we disambiguate the events mentioned in different documents by merging differently phrased instances of the same event. Post-annotation, we refer to the merged instances in the causal graph while keeping all the textual versions of it in the dataset.
3.3 Event Extraction
We extract the main events mentioned in each docu- ment. In contrast to prior work (Shi and Lin, 2019), we generative approach to extract the main events given
single node
piece. Specifically, we prompt GPT-3 (text-davinci-00:: Ouyang et al., 2022) to extract the main events from given document (similar to Veysch et al., 2021, see Appendix Table 12 for exact prompts). Extensive experimental anal- ysis (see Table 10) confirms that this generative approach for event extraction leads to higher ex- traction precision and semantic granularity suited for our task.
3,4 Event Causality Linking
In the final stage of our pipeline, we collect causal- ity judgments about event pairs extracted from the documents related to a specific story (2730 pairs). We distinguish two settings: when both events in the pair are originally from the same doc- ument (in-document setting, 360 pairs) and when they originate from different documents (cross- document setting, 2370 pairs). Motivated by the way cognitive studies capture judgments about ac- tual causality (Gerstenberg et al., 2021), we define the causality between real-world events not as a single binary score but as a continuous value from
Extraction Verification Because generative methods come with the limitation of hallucinations and wrong outputs, we filter extracted events to keep only the valid generations. We first automat- ically filter events that contain less than 3 tokens,
Phttss//serpapi.com WWen clipping articles, we avoid dividing sentences.

---

O to 100, enabling finer analysis and predictions of causal judgment. We qualify 44 Amazon Me- chanical Turk workers, and for each pair of events, task 7 workers with providing a judgment for the causal link between the events (see Appendix A.3
Pairwise Event Causality Score
Type of pairs
Below 20
20-50
50-80
Above 80
In-doe Cross-doc
3.9%
25%
26.6%
44.4%
13.1%
37.6%
25.3%
24%
All pairs
11.9%
35.9%
25.5%
26.7%
for details).
Table I: Percentage of pairs present in the CRAB, per causality score class.
3.5 Causality Strength Validation
We divide the causality score into 4 classes (0-20,
20-50, 50-80, 80-100) to compute Krippendorff's a between our 44 annotators (see Appendix A.4).
the following section, are based on these 4 classes reported in Table
Krippendorff's a for all classes is 0.28, and 0.38 when considering only the furthest classes (0-20 and 80- 100). We average all annotators' causality scores for each event pair obtain unique scalar causal judgment and classify this event pair score into the 4 causal classes (high, medium, low, and no causality). Given the low agreement, we select pairs where the average score falls on the boundary of the 4 classes and the variance between annotators is high, to be validated by experts. This subset con- sists of 26.7% of the benchmark. This step is done by asking three expert annotators (NLP researchers who are familiar with the task of causal inference) further annotate event pairs' causal scores and classes. The experts, given the average causal score, were asked to choose which of the neighboring classes was a better class for the event pair, up- dating the score accordingly. The inter-rater agree- ment, using Krippendorph's alpha, between experts is 0.70. These expert- valliate causal scores along with the remaining low-variance samples used for CRAB.
4.2 Causal Structures
As discussed in Section 2 and based on expert an- notations of the pairwise causality between events, we view CRAB from the perspective of causal frame subgraphs. We stratify the dataset and get the causal frame of cach event. We categorize these subgraphs based on the causal links present in them. We observe different categories based on the strength of causal scores between the effect event and the potential causes (in-degree edges of the causal frame graph). Figure 2 (left) depicts the different types of causal frames present in the dataset, and Table 6 provides statistics relative to the number of chains and events for each one of the structure types.
Similarly to causal frames, we extract causal chains from CRAB based on the three causal structures; Mediation, Confounding, and Collision (Pearl, 2009), depicted in Figure (right). Descrip- tive statistics of the extracted causal chains that fit into these three cases are reported in Table 7.
4 Dataset Analysis
Experimental Setup
To evaluate how language models reason about causality, we define different experimental frame- works covering various causality assessment sce- narios, similar to Kıcıman et al. (2023). We investi- gate three tasks in ascending order of complexity in terms of causal structure: pairwise causality infer- ence, graded causality inference, and causal chain inference. This section describes the experimental setups associated with these tasks, and the models evaluated on them.
4.1 Descriptive statistics
CRAB consists of set of 173 documents regarding 20 different stories discussing newsworthy real- world events. It contains 384 extracted unique event instances and 2730 event pairwise causality scores (see Table 5 in Appendix for additional de- scriptive statistics). The experiments presented in
http://rooo//rrrpeedoorffrr "With httpps://pypt www.org/prrreeee propect/jrrrrrre ppeendr ff/, us- ing interval metric to calculate the pairwise distance.
"We note that the high number of annotators per sample makes the raw number of disagreements higher, lowering a. Moreover, contrary to classical annotation situations where a small number of annotators label each sample, annotation using Amazon Mechanical Turk involves many annotators participating in task, thus each sample being annotated by different workers, augmenting the variance and decreasing the agrocment rate.
5.1 Models
We use two decoder-only instructionn-ollloiing API-based models, GPT-3 (text-davinci-00s 175B size) and GPT-4, with the settings suggested by OpenAl (2023); a temperature of 0.3 and max-

---

imum length of 256 tokens. We additionally test CRAB using VJan-Alpaca-GPT4-22 (Chia al., 2023), an open-source 3B size encoder-decoder model fine-tuned on instruction-followiin datasets: Flan (Longpre et al., 2023) and GPT4-Alpaca."
event causally contributes to an effect. For example. following the example in Figure 1, multiple events are responsible for causing E6: "Twitter has lost many users permanently", with different causality scores. Thus, we go beyond pairwise causality and design two methods to prompt models to rank the events that contributed more to the effect. First, create a Multiple Choice Question (MCQ) An- swering task that asks the model to provide the most contributory to an effect cause among several events. We construct the dataset for the experiment by using the causal frames of each event and select- ing 4 possible causes. We then ask the model to select, based on these 4 choices, the cause with the highest causality score (see the prompt in Table 15 in Appendix).
5.2 Experiments
Pairwise Causality Inference To evaluate Pair- wise Causality Inference, we first prompt the model to generate a scalar causality score between two events given a context (the documents from which the events were extracted), mimicking the bench- mark human annotations. We compare it with the average annotators' score in CRAB. In initial ex- periments, this setup led to very poor performance all the models failed generate scalar value for quantifying causality. Therefore, we mapped the causality intervals to descriptions of different degrees of causality and augmented the prompt instructions with score ranges and their explana- tions. The 4 classes and definitions are as follows (i) High causality: a link between events that are definitely causally related to each other (causal score above 80), (ii) Medium causality: a link be- tween events that might be slightly causally related each other (causal score between 50 and 80), (iii) Low causality: a link between events that have a little causal connection (causal score between 20 and 50), and (iv) No causality: independent events (causal score lower than 20). The full prompt can be found in Table 16 in Appendix. We then evalu- ate the model's answer by mapping the generated score to the four classes and computing the Macro- Fl score (Pairwise Causality Score in Table 2). We also experiment with binary and multi-class classification tasks (Binary Pairwise Causality and Multi-class Pairwise Causality in Table 2, re- spectively), prompting the model to output causal- ity class instead of raw score. For multi-class, the generated output is letter matching one of 4 classes. For the binary classification, we ask the model whether an event caused another to happen based on the provided contexts, the answer being "yes" or "no". The prompts for these tasks can be found in Tables 13 and 14 in Appendix.
Second, we assess the responsibility of event pairs by stratifying the results of the pairwise causality experiment based on the type of events" causal frames. For each event, we extract its causal frame (i.c., all the potential causes of an event) and predict the class of causality between that event and all the potential causes in the causal frame. We compute both the average F1-score and the Exact Match (EM) between CRAB's causal class anno- tations (4 classes) and the causal class predictions across all event pairs in the causal frame. We report the scores in Table and stratify them based on the different causal frame structures strong direct (SD), direct (D), strong contributory (SC), contrib- utory (C), mixed (M) and no causality (N) (Figure 2 (left)).
Causal Chain Inference Pairwise causality pro- vides a strong indication of the causal relationship between two events. However, these events are usu- ally part of a larger chain of events with complex causal patterns. In this experiment, not only the relations between the causes and the effect in causal frames but also how the causes are related to one another. We stratify the results of the pairwise causality experiment based on the three causal chain structures we introduced in Sections 2 and 4; Mediation, Confounding, and Collider (Fig- ure 2 (right)). Similarly to the previous experiment, extract causal chains that fit the three patterns and compute the Fl-score and the Exact Match (EM) between CRAB's causal class annotations classes) and the causal class predictions for each edge in the causal chains. We report cach model's average scores in Table 4.
consider
Graded Causality Inference As described in Section 2, one of the main concepts in actual causal- ity is graded causation or responsibility (Halpemn 2016), which is the relative degree to which an
"https://instrucee carliionnttunnn--nn tuning-win witwiite th- gpt-4. 4. thub.
io/

---

Tasks
Models
pairs
In-doc
Cross-doc
Pre-Jan 2022
Post-Jan 2022
Pairwise Causality Score
Flan-Alpa Alpaca
21.6
14.9
22.4
22.0
21.3
GPT-3
25.8
24.4
25.4
26.6
25.2
GPT4
54.7
59.0
53.7
56.4
53.5
Multi-class Pairwise Causality
Flan-Alpaca
11.0
12.2
10.8
11.2
10.7
H/M/
GPT-3
35.0
27.4
34.9
35.0
34.5
L/N
GPT4
45.6
46.1
45.0
43.1
46.7
Binary Pairwise Causality
Flan-Alpaca
60.1
56.7
62.1
58.7
011
GPT-3
57.2
67.0
55.0
56.9
57.5
C
E
GPT-4
73.9
80.0
72.6
76.5
72.0
Graded Causality (MCQ)
Flan-Alpacu
39.9
53.2
29.3
44.1
35.4
GPT-3
59.7
70.9
50.7
64.5
54.5
GPT-4
53.8
67.3
43.1
63.3
44.5
Table 2: Macro Fi-scores SoTA LLMs all Pairwise Causality Inference tasks presented in Section 5.2 and the Graded Causality Inference MCQ task presented in Section 5.2. For the MCQ task, we stratify the results for in-doc and cross-doc based on whether the effect & correct cause are extracted from the same document.
All Frames
SD
D
C
M
MODELS
Fl
FM
FI
EM
F1
EM
F1
EM
F1
EM
F1
EM
FI
EM
Flan-Alpaca
10.2
0.8
5.0
0.1
19.8
11.1
3.7
0.0
8.0
0.1
13.0
0.7
6.4
3.0
GPT-3
28.8
3.3
29.1
4.1
30.5
16.6
31.9
3.3
31.0
29.0
0.0
19.3
9.1
GPT-4
38,9
6.1
45.2
16.7
39.8
11.1
40.6
6.7
38.3
3.1
36.8
1.3
39.4
24.2
Table 3: Macro F1 and EM scores SoTA LLMs for Graded Causality Inference (Section 5.2) stratified for different Causal Frame types. SD is for Strong Direct Causality; D is for Direct Causality; SC is for Strong Contributory Causality:
for Contributory Causality; N is No Causality: M is Mixed Causality. Green colors depict the F1 and EM highest scores of the Causal Frame types for each LLM and orange colors depict the lowest scores. Please refer to Figure for detailed visualization of the different Causal Frame structures.
Mediation
Confounding IMM
Collider
Causal Discovery Table 2 provides the pairwise causality inference scores for the three pairwise sub-tasks. All models perform poorly on CRAB, with GPT-4 showing a higher performance most of the tasks compared to GPT-3 and Flan-Alpaca. In the case of Binary Pairwise Causality inference, we measure GPT-3 and GPT-4's binary classifica- tion Macro-Fl scores for different thresholds (see Figure 5 in Appendix). Both models tend to predict causation between events with gold causal score above 50. Interestingly, if increase this gold score threshold and consider only causal events with a high causal connection, the performance all models drops. possible interpretation behind these results is that large language models are typ- ically quite good at identifying the distributional similarity between concepts. When it comes to making causal judgments, they may identify multi- ple related events causally associated with an out- come rather than a single consequential event. This means that they may misclassify events weakly re- lated to the event in question as causes. Ideally, when asked about a binarized causal relationship.
MODELS
Flan-Alpaca
49,4
25.4
38.1
10.0
29.3
8.2
GPT-3
40.2
10.6
38.1
9.7
28.0
GPT-4
20.9
Table 4: Macro Fl and EM scores SoTA LLMs for the Causal Chain Inference stratified in different Causal Chain structures as described Section 5.2. Green colors depict the F1 and EM highest scores of the Causal Chain types for each L.M and orange colors depict the lowest scores.
6 Experimental Results
the previous section, we introduced several lower-level tasks to investigate whether various LLMs can accurately assess pairwise causality judgments. Here, we describe the results from the perspective of Causal Discovery, Causal Respon- sibility Assessment, and Multi-document Causal Reasoning. We perform an in-depth analysis of LLM capabilities in different facets of causality, grounded to principles and dimensions of actual causality described Section

---

we would like models to provide only the main causes (strong relation between two events).
33% of the time, indicating that models themselves may be biased to prefer in-document causal links, even when humans identify a different causal link across multiple documents. This result suggests that models are able to capture causality when it is explicitly referred to in one context but struggle when it is implicitly inferred across documents.
Models also under-perform when assessing Multi-class Pairwise Causality inference, espe- cially when assessing medium and causality (Table 8). Further investigation of the results shows that these cases of misclassification, the model tends to predict high causality instead of medium causality, and medium causality instead of no causality, demonstrating that models tend to hallucinate stronger causal relationships than hu- mans perceive. Misclassified cases can be found in Figure 6. To assess the effect of fine-tuning on models' ability to learn the Causal Discovery task, we additionally fine-tuned one encoder-only and one decoder-only language model (DeBERTa-v3- large (He et al., 2021) and Llama2-7B (Touvron et al., 2023)). The experimental setup and analysis in Appendix B show that CRAB also challenges the current state-of-the-art fine-tuned methods.
Memorization vs. Generalization Due the lack of transparency of closed-source GPT mod- els, concerns arise regarding whether LLMs, pre- trained on extensive internet data, were subjected to the test set of benchmarks during their pre-training phase (Jacovi et al., 2023). This presents complex challenge in assessing the degree to which LLM performances change for events they might have seen during the pre-training phase. We study how the performance of GPT-3/4 varies when identify- ing causality for real-world events occurring pre- Jan 2022 and post-Jan 2022 (the official threshold date for their training data source). Even though we provide the context to these models, we still ob- serve substantial drop in performance for graded causality and pairwise causality score, suggesting that the performance of models can be affected by knowledge of the events memorized during their pretraining stage.
Assessing Responsibility Going beyond pair- wise causality and assessing whether LLMs can assign responsibility among potential causes of a specific event, we show that models fail to capture complex causal structures. Table 3 shows that mod- els perform better when assessing the causality of graph structures that contain causal scores that are well-separated from each other; high causal VS low or no causal ones (Strong Direct and Direct Causal Frames), compared to structures with scores of var- ious degrees (Mixed Causal Frames). Additionally, from Table 4, we notice common struggle among all models regarding the Collider case; causal diagram of two causes that contribute to effect but are not themselves related to each other.
Related Work
Causal Reasoning Benchmarks There has been extensive research on introducing challenging causal benchmarks in recent years. They focus on commonsense causal discovery and reasoning (Mooij et al., 2016; Kalainathan and Goudet, 2019; Bethard et al., 2008; Dalal et al., 2023), as well as assessing causal reasoning from the perspective of plausible alternatives and counterfactuals (Roem- mele et al., 2011; Frohberg and Binder, 2021; Sri- vastava al., 2022; O'Neill al., 2022). Similar to our work, there have been attempts to create benchmarks that incorporate the cross-document setup (Welbl et al., 2018; Tu et al., 2019) and causal structures (Jin et al., 2023a), but not on real-worid events. Finally, domain-specinn datasets that per- form causal reasoning in the medical (Nori et al., 2023) and the legal (Zhong et al., 2023; Choi al., 2023) domains have been introduced. CRAB dif- fers from prior work regarding the nature of its events, real-world events, and the type of causal reasoning that assesses LLMs, actual causality.
Multi-document Causal Reasoning In Table 2, we report results for causal score prediction for both event pairs that are found in the same docu- ment and event pairs found across documents. In all experimental settings, we see better performance for event pairs extracted from the same document (in-doc pairs). Based on the in-doc high causal score percentage reported in Table 1, models tend to perform well in the causal discovery of in-doc event pairs since documents usually express causal relationships in explicit way, likely because nar- rators seek to draw explicit causal links between events. Interestingly, in the MCQ setting, we find that GPT-4 wrongly assigns the highest responsi- bility to events that belong in the same document

---

Assessing Causal Reasoning Many existing studies investigate whether NLP models under- stand causality (Feder al., 2022; Jin et al., 2023b; Zhang et al., 2023), providing methods to quantify the causal abilities of language models to discover causal relationships in documents (Cao et al., 2022; Yu et al., 2019; Dalal et al., 2023) and extracting cause-effecct pairs as subtask of information ex- traction from text (Hidey and McKeown, 2016; Huang et al., 2021; Jin et al., 2023a; Nauta al., 2019). Another line of work studies the improve- ment of causal reasoning generation using instruc- tion prompting (Kıcıman et al., 2023), Chain-of- Thought (CoT; Wei et al., 2022), and prompt aug- mentation (Schick et al., 2023).
ered by the media, is nearly impossible to identify all possible mediating events in the causal graph. We are in a limited information situation, both from the viewpoint of the document (the document's s au- thor might be unaware of other mediating events) and of the model (we can not evaluate the amount of information stored in the weights of the model), The experiment regarding memorization and gen- eralization is first step toward investigating this limitation.
Acknowledgements
We thank Negar Foroutan, Reza Banaei, Deniz Bayazit, Beatriz Borges and Mete Ismayilzada for reading and providing comments on drafts of this paper. We also gratefully acknowledge the sup- port of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29), the EPFL Sci- ence Seed Fund, the EPFL Center for Imaging, Sony Group Corporation, and the Allen Institute for AI.
8 Conclusions
This work introduces CRAB, new causal reason- ing benchmark that contains fine-graie graiina contex- tual causality annotations for ~2.7K pairs of real- world events that describe various newsworthy sto- ries. Using CRAB, we explore how L.MS assess the causal relationships between events when the causal signal comes from different contexts. Addi- tionally, we assess LLMs performance in identify- ing and assessing complex causal structures. Our findings suggest that state-of-the-ee language mod- els perform poorly in pairwise causal inference and responsibility assignment when events are spread across documents. Furthermore, this weak per- formance is amplified when LLMs must identify causal relationships in complex causal structures rather than simple linear chains.
References
Steven Bethard, William Corvey, Sara Klingenstein, and James H Martin, 2008. Building a corpus of temporal-causal structure. In LRECC
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Angela Cao, Gregor Williamson, and Jinho Choi. 2022. A cognitive approach to annotating causal constructions a cross-genre corpus. In Proceedings of the 16th Lingusitic Annotation Workshop (LAW- XVI) within LREC2022, pages 151-159.
9 Limitations
As discussed in Section 3, actual causality poses great challenge when collecting human causal judg- ments about real-world events. We tried to mitigate these biases by introducing intermediate valida- tion steps throughout the data collection pipeline, keeping experts in the loop, and focusing on col- lecting objective causal assessments about stories grounded in the respective documents. However, several epistemic biases might remain since the news pieces that the causal judgments have been grounded might contain biases propagated by the reporter of the story. We try distribute this variance by using documents collected from vari- ous sources. An additional limitation of our study is related to the nature of the events of CRAB. Since we are collecting real-world stories that were cov-
Yew Ken Chia, Pengfei Hong, Lidong Bing. and Sou- janya Poria. 2023. Instructeval: Towards holistic evaluation of instruction-tune tuned large language mod- els. arXiv preprint 2xXv:2330.00757.
Jonathan Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz. 2023, Chatgpt goes to law school. Available at SSRN.
Dhairya Dalal, Paul Buitelaar, and Mihael Arcan. 2023. Calm-bench: A multi-task benchmark for evaluating causality-aware language models. In Findings the Association for Computational Linguistics: EACL 2023, pages 296-311.
Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins. and Benjamin Van Durme. 2020. Multi-sentence

---

argument linking. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics.
Zhijing Jin, Jiarui Liu, Zhiheng Lyu. Spencer Poff. Mrin- maya Sachan, Rada Mihalcea, Mona Diab, and Bern- hard Schölkopf. 2023b. Can large language models infer causation from correlation? arXiv preprint arXiv:2306.05832
Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty. Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. 2022. Causal inference in natural lan- guage processing: Estimation, prediction, interpreta- tion and beyond. Trunsactions of the Association for Computational Linguistics, 10:1 10:1138-11 1138-158.
Diviyan Kalainathan and Olivier Goudet. 2019. Causal discovery toolbox: Uncover causal relationships in python. arXiv preprint arXiv:1903.02278.
Emre Kıeıman, Robert Ness, Amit Sharma, and Chen- hao Tan. 2023, Causal reasoning and large language models: Opening new frontier for causality. arXiv preprint arXiv:2305.0003.
Jörg Frohberg and Frank Binder. 2021. Crass: A novel data set and benchmark to test counterfactual rea- soning of large language models. arXiv preprint arXiv:2112.11941.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Tay, Denny Zhou, Quoe V Lc, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 1..3801.16660 13688.
Tobias Gerstenberg, Noah D Goodman, David A Lagnado. and Joshua B Tenenbaum, 2021. A coun- terfactual simulation model of causal judgments for physical events. Psychological review, 128(5):936.
Helena Matute. Fernando Blanco. lon Yarritu, Marcos Díaz-Lago, Miguel A Vadillo, and Itxaso Barberia. 2015. Illusions of causality: how they bias our every- day thinking and how they could be reduced. Fron- tiers in psychology, 6:888.
Adolf Grunbaum. 1952. Causality and the science of human behavior. American Scientist, 4064):665-689,
Joseph Halpern. 2016. Actual Causality. MiT Press.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using clectra-style pre- training with gradient-disentangee ing. arXiv preprint arXiv: 1122211110000 11.0943.
Joris M Mooij. Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. 2016. Distin- guishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research, 17(1):1033 1204.
embedding shar-
Paul Henne, Kevin O' Neill, Paul Bello, Sangeet Khem- lani, and Felipe De Brigard. 2021. Norms affect prospective causal judgments. Cognitive Science, 12213..
Meike Nauta, Doina Bucur, and Christin Seifert. 2019. Causal discovery with attention based convolutional neural networks. Machine Learning and Knowledge Extraction, 1(1):112-340.
Christopher Hidey and Kathleen McKeown. 2016. Iden- tifying causal relations using parallel wikipedia arti- cles. In P Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pages
Harsha Nori, Nicholas King, Scott Mayer McKinney,
Dean Carignan, and Eric Horvitz. 2023. Capabili- ties of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375.
Yiyi Huang, Matthäus Kleindessner, Alexey Munishkin, Debvrat Varshney, Pei Guo, and Jianwu Wang. 2021. Benchmarking of data-driven causality discovery ap- proaches in the interactions of arctic sea ice and at- mosphere Frontiers in big Data, 4:642182.
OpenAl. 2023. Gpt-4 technical report, arXiv, pages 2303 3373408774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama. Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedhack. Advances in Neural Information Processing Systems, 11:27730-22744
Thomas F Ieard, Jonathan F Kominsky, and Joshua Knobe. 2017. Normality and actual causal strength. Cogmition, 161:80-93.
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. ArXiv, abs/2305.10160.
Kevin O' D'Neil,, Tadeg Quillien, and Paul Henne, 2022. A counterfactual model of causal judgment in double prevention. In Conference in computational cogni- rive neuroscience.
Judea Pearl. 2009, Causal inference in statistics: An overview. Statistics Surveys, 3:96-146.
Zhijing Jin, Yuen Chen, Felix Leeb, Ojasv Kamal, Zhi- heng Lyu, Kevin Blin, Fernando Gonzalez, Max Meeiaan-Weinee Luigi Gresele, Mrinmaya Sachan, al. 2023a. Cladder: Assessing causal reasoning in language models. preprint.
Judea Pearl and Dana Mackenzie. 2018. The book of why: new science of cause and effect. Basic books.

---

Nils Reimers and Iryna Gurevych. 2019, Sentence-bert, Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084.
Phillip Wolff and Jason Shepard. 2013. Causation. touch, and the perception force. In Psychology of learning and motivation, volume 58, pages 167-202. Elsevier.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Bei Yu, Yingya Li, and Jun Wang. 2019. Detecting causal language use in science findings. In Proceed- ings of the 2019 Conference on Empirieal Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 4664-4674.
Angelika Romanou, Panayiotis Smeros, Carlos Castillo, and Karl Aberer. 2020. Scilens news platform: sys- tem for real-time evaluation of news articles, arXiv preprint arXiv:2008. 200.:29908.1099.
Cheng Zhang. Stefan Bauer, Paul Bennett, Jiangfeng Gao, Wenbo Gong, Agrin Hilmkil, Joel Jennings, Chao Ma, Tom Minka, Nick Pawlowski, et al. 2023. Understanding causality with large language mod- els: Feasibility and opportunities. arXiv preprint arXiv:2304.0552.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves use tools. arXiv preprint arXiv:2302.04700
Mian Zhong, Shehzaad Dhuliawala, and Niklas Stochr. 2023. Extracting victim counts from text. arXiv preprimt arXiv:2302.1336 12367.
Peng Shi and Jimmy Lin. 2019. Simple hert models for relation extraction and semantic role labeling, arXiv preprint arXiv:1904.0522
Panayiotis Smeros, Carlos Castillo, and Karl Aberer. 2019. Scilens: Evaluating the quality of scientific news articles using social media and scientific litera- ture indicators. In The World Wide Web Conference. pages 1747-1758.
A
Dataset Construction Details
A.1 News Article Selection
Based on a selection of 20 distinctive stories, we crawl the web and select the top 20 news articles per story. When extracting articles related to a story that happened many years before, we noticed that the retrieved articles also covered recent events that were loosely related to the story's main events. Therefore, we a time window of 9 months when extracting the articles for each story keep only articles that have been published around the time that the respective story happened.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 20l.:2206604115.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almabairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023, Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint urXiv:2307.0922:
A.2 Event Extraction
Using a generative approach for event extraction has two main benefits, confirmed through exten- sive experimental analysis. First, when prompted correctly, generative models successfully output structured information the requested semantic abstraction, which leads to higher precision when extracting events. Second, the semantic granularity of the events we want to extract is between sen- tence and document level, mcaning that we aim for the main events covered in the article and not syntactic events as existing works use (Ebner et al., 2020).
Ming Tu, Guangtao Wang. Jing Huang, Yun Tang. Xi- aodong He, and Bowen Zhou. 2019. Multi-hop read- ing comprehension across multiple documents by reasoning over heterogeneous graphs. arXiv preprint arXiv:1905.07374.
Amir Pouran Ben Veyseh, Viet Dac Lai, Franck Dernon- court, and Thien Huu Nguyen. 2021. Unleash gpt-2 power for event detection. In Annual Meeting of the Association for Computational Linguistics.
Jason
Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning large language models. arXiv preprint arXiv:2201.1 1903.
Similarly to Smeros al. (2019) and Romanou et al. (2020), filter the top 20 news articles per story. We remove the ones with less than 100 words and those with paywalls or provide re-directions to the original referred news article. Finally, we clip the article to 250 words and round to the end of the
Johannes Welbi, Pontus Stenetorp. and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287- 302.

---

In-document data annotation
Cross-document
data annotation
timeline
timeline
Doc A
Event A1
Doc
Event A1
Doc B swent B1
enerAl
Event A2
Event
Event B1
Event B2
Event A3
Event A3
Event B3
Event A4
Event A4
in-document pairs
pross-dccument pairs
Event A1
Event A2
Event A1
Event A3
Event
Event B1
Event
Event
Event
Event B3
Event
Event A3
Event A1
Event A4
Event
Event B1
Event
Event
Event
A4
Event
Event
Event A2
Event
Event
Evont
Figure 4: Dataset construction pipeline. Once events are extracted from the documents (Section 3), we order them time and formulate story timelines. Conditioned on the temporal ordering, we create all combinations of events in the document and pass the in-document pairs to annotators. We perform similar process for the event pairs extracted from different documents (cross-doc), including an extra step of merging document timelines into one before taking the pair combinations.
sentence token. We end up using total number of 384 news documents as the main bed for event extraction. Figure 17 reports statistics about CRAB stories.
worker $0.80 for doing every 3 annotations for the in-document event pairs and $0.90 for doing 5 annotations for the cross-document pairs. Figures' 7 shows the screenshots of our Acceptance & Privacy Policy, and Figures 8, 9 and 10 depict the task instructions used for crowdsourcing along with the scripts for the in-doc and cross-doc tasks.
Documents
173
Events
384
Pairs
2730
In-document pairs
360
Cross-document pairs
2370
Stories
Causal frame
frames
pairs
In-doc
Cross-doc
Avg Timeline length
SD
24
85
25
60
Avg pairs per story
137
48
SC
254
228
C
789
103
686
Table 5: General descriptive statistics CRAB regard- ing event and event pairs stratified for both the in-doc and cross-doc cases along with statistics about the sto- ries and story timelines.
NC
33
100
18
82
MC
149
1386
156
1230
Table 6: Number of frames for different types causal frames (see Figure 2) along with the number of event pairs additionally stratified for the in- and cross-doc cases. is Strong Direct Causality;
for Direct Causality: SC for Strong Contributory Causality; C is for Contributory Causality: NC is for No Causality; MC is for Mixed Causality.
A.3 Crowdsourcing Causality Scores
For the in-document annotations, we show the an- notators the documents and ask them to assess event pairs. For the cross document ones, give 2 documents a time, along with event pairs. Fig- ure 4 depicts the pair creation process that served as input for Amazon Mechanical Turk experiments.
Causal chain
Chains
To select native English speakers, we focus on the group of workers whose locations are in the USA. We also ran 2-phase qualification where we evaluated the quality of annotators on our task and selected the ones with a higher than 80% score on our qualification task. Finally, , 44 out of 400 workers are selected as qualified. We pay each
Mediation Confounding Collider
774
924
115
Table 7: Number of chains different types causal chains (see Figure 2).

---

10
Causal Strength
Precision
Recall
F1
Support
Peported Utredhed
for Tate
GPT-3
High
0.57
0.61
0.59
722
Mid
0.35
0.38
0.36
685
Low
0.55
0.45
0.49
968
No 0.34
0.43
0.38
324
score
08
Table 8: Performance scores Pairwise Causality (4 classes) Inference task for each class for the GPT-4 model.
04
Causality classes
Size
[1, 2]
1310
0.04
Causal Score Threshold
2, 3]
1295
0.08
1429
0.12
Figure 5: F1 score per threshold for the Binary Pairwise Causality assessment described in Section 5.2. The scores reported in Table 2 correspond to causal score threshold of 50.
[1
1444
0.38
[1,2,3, 4]
2730
0.28
Table 9: Krippendorff's a for different groups of classes.
important to additionally evaluate our causal benchmark on different architectures and inference techniques, providing additional insight into the difficulty of the task. On that note, we fine-tuned DeBERTa-v3-large (He et al., 2021) and Llama2- 7B (Touvron al., 2023) models.
A.4 Inter-rater agreement
We have total of 44 workers scoring the causality between 2730 pairs of events. All pairs are an- notated by at least 7 annotators and, at most, 10 (around 21 11.3k annotations). We divide the causality score into equal classes and compute Krippen- dorff's a. We consider the ground truth the majority vote. Table shows Krippendorff's s for different groups of classes. As expected, the agrec- to discriminate between the lowest causality class and the highest one is the highest, while is harder for annotators to agree on discriminat- ing between nearby classes. Krippendorff's s a for all classes 0.27, and 0.33 for the further classes. We note that the high number of annotators per sample increases the raw number of disagreements. Moreover, contrary to classical annotation situa- tions where small number of annotators label each sample, the Amazon Turk settings involve many different annotators participating in a task. Thus each sample is annotated by different work- ers, augmenting the variance and decreasing the agreement rate.
We fine-ture both models on the 3 different pair- wise causality tasks presented in our paper. For each task, we create different train/test splits (75%/25% ratio) study the generalization ability of the models after fine-tuning; Date: we select 5 out of the 20 most recent stories for the test set and the rest for the train, Story: we randomly select 5 stories for the test set and the rest for the train, and Random: randomly split the event pairs dataset. regardless of the story or the date.
The results in Table 11 show that fine-tuned DeBERTa-large (encoder-only) models fail to per- form well on CRAB, showing that our benchmark challenges the current state-of-the-as fine-tuned methods. Compared to decoder-only models in few-shots setting, DeBERTa-large tends to un- derperform when splitting by story, except for the easier binary pairwise causality prediction task. Ad- ditionally, expected, the experiments with the random data split have higher scores, which vali- date the information leakage of context from the train to test set and verify that models rely on the context (articles) when assessing the causal re- lationship of the two events. A subsequent study on how fine-tuning improves pre-trained LLMs causal reasoning abilities would be interesting, and we hope that paper provides a strong benchmark for pursuing this research direction.
B Experimental Results with Tone-Tuned LMs
We initially evaluated proposed dataset decoder-only models because decoder-only models (especially API-based ones such as GPT-3 3.5 4) have become important pillars of AI products, mo- tivating researchers to benchmark their capabilities and identify their biases and limitations. However,

---

Few-shot event extraction on
Precision
Recall
Extractive summaries with SBERT (Reimers and Gurevych, 2019)
45,8%
31.9%
37.6%
Extractive summaries with GPT-3 (Brown et al., 2020)
63.8%
59. 1%
61.3%
Abstractive summaries with GPT-3 Brown et al. (2020)
74.2%
52.8%
61.7%
Original Document 5 par
75.0%
65.3%
69.8%
Table 10: Zero-shot performance score for Event Extraction using the GPT-3 model. Results are based on a manually constructed dataset of 88 articles. We different context settings by prompting the model with both providing the document in its original form as well as its summary. We found that prompting with the original text yields better quantitative and qualitative results.
Causality Tasks
Model
Split by Date
Split by Story
Random split
Pairwise Causality Score
DeBERTa-large
21.6
21.4
22,9
Ilama2 7B
24.3
26.3
32.8
Multi-class Pairwise Causality
DeBERTa-large
29,4
35.8
60.8
Llama2 7B
23.2
23.1
32.7
Binary Pairwise Causality
DeBERTa-large
62.5
74.2
76.6
Ilama2 7B
51.1
51.9
58.5
Table 11: Macro Fl Fl-scores on test set for fine-tuned models on all Pairwise Causality Inference tasks as presented in Section 5.2. We stratify the results for Date, Story and Random splits as described in paragraph B. The best performance for each causality task is bolded.
PROMPT: Event Extraction
You journalist who whats to extract the main events from articles. These events need to summarize the story of the article.
PROMPT: Multi-class Pairwise Causality Classes
You helpful assistant for causal relationship understanding.
Think about the cause-and-effect relationships related to context.
These events need be list format.
<DOCUMENT>
Context: <DOCUMENTS>
Event <EVENT 1>
Table 12: GPT-3 prompt for extracting events from documents.
Event <EVENT 2>
How much did event
event happen?
[A] High causality: Event 1 is definitely responsible for Event 2.
[8] Medium causality: Event might have been respansible for Event 2.
PROMPT: Binary Pairwise Causality
[C] Low causality: The context gives little indication that there is a connection between the two events, but background info might suggest low causal connection.
You are helpful assistant for causal relationship understanding. Think about the cause-and-effect relationships related to context.
[D] No causality: Events are somehow related but definitely NOT causally related.
Context: <DOCUMENTS>
Let's work this out in step-by-step way to be sure that we have the right answer. Then provide your final answer within the tags, <Answer>A/B/C/D<4/nswee
Event 1: <EVENT 1>
Event 2: <EVENT 2>
Did Event 1 cause Event 2 to happen?
Please answer in single word: no.
Table 14: Prompt for the Multi-class Pairwise Inference task.
Table 13: Prompt for the Binary Pairwise Inference task.

---

Predicting causality where is
Predicting causality where is causality
Twitter Acqusition
Riancial
5, Weiller's shares Lae open eazket, 355.20 musk oibsogioniiy has begun leying off kisable chunke of the conpany, toed Wetfter First
purehneed
comnddedd cohpañtes rere to lavortte ap euevioo method to cut as peopla rcacted bayans lase tos accommodata higher pricees [+co] markete still reellng from the enoecrment annouroeen seeent Leat.
company vas complenes aned last week celeter sold axplovess 1n an company would nossty acadtare by seall sbout their emp.iymmnt
by
[...]
E1
disc bas purclussed rasce thes 96 Twittec 'N
CRAI:: Mo caussity
E4
compenies to dedresse expenises an revenues
cousality
BP1-4 ligh causality
BPT-4 4 Vo causallly
E10
filed allegedly fallingg glve 60-day
E6
DE gevernaest announces to glash Laxca and increase borroving.
lavorts
Stay. Dollpsee of Sliken Valley Bank
Stary Arees wintesst
sedneslay; #illecn valley wank wao wall well-onpisationn isatdd indoced by very vergere capital commuiit that ave had served sus deiday and entsed see deporiti in the langnct T.F. hancins says custoeace of U.S. bants sborld have condidence that
reesidess joir volsosaro has instated thas the wreallian arena of blane He 201 zolsonaro and his gurerreent for turning biled defonectation, fos is an sddnnes at the coroed Ratione in leey hait the hasnes racntorast. sax anen aignsticans rine in the I...]
conserve:
Valley attempted ralse 92.25 biillon Fron Investors.
DRAR: No cousiity
Thousends iberate fires
DMAB High cassality
High causalty
lwvwging the Arasom
GPT- Na causality
Bidar speakts on the banking energency that their zoney will be sate.
Conservat-oonett, blane the goverunent for hastening deforestation.
E14
custoners
Hillee Neesneet The Anadaeyy of Motion Picture Arta and Sclences supac "Weet a out jettytyy.. Muinsreis's a former coapuny Eiler For bankevptay, scaual susgilto. Fres) since Rietto wont viral two weeks ago wake sarvey Reins-eda.
dhette Nousment
zers the bashteg that rocked tas varid. Since alattn wens vizai wabe
eilegations baz zeporta tapt eoce than 1:3 millios weean aun have used the hasasag countei that can bailitag scse watter aldbsgread ceally uaza into lasulny eovestons vares? dreate sooial Change
vicel ecmunt
haceonment
agstinl
abuse
Harewy Wel Weinstein has been nacked The board of company, with
DRAB: la causaity
CRAB: High causality
effect.
GPT causally
atillon Modenn and incn hare used Michoo
GPT causallty
ago.
#MeT0o viral two wooks ago.
Figure 6: Failed predictions of GPT-4 regarding CRAB on high and no causal classes.
Acceptance Privacy Policies (click expand/collapse)
Acceptance Policy
is no obligation to participate the not reject job unless observe evidence malicious behavior, as random clicks or vary short session times.
Privacy Policy
We may inckdentally collect some personal data for the purpose our research project, according to art. 36c and seq, the ETH Act. target is to process and publish only anonymized data, Raw data will be kept confidential and secure. Only anonymized or aggregated personal data may be shared with other research partners.
Having astablished this, however, should collect any personal data task.
using
Amazon Mechanical
Amazon.com.
81226, Seattle, 98108-1228
USA. Hence, the privacy policy of Amazon will apply for the processing of your personal information by Amazon.
If you wish to raise complaint on how we have handled your personal data, you want to know if we hold personal data about you, you can contact Data Protection Officer (5po@eppflch) who investigate the matter.
Figure 7: Acceptance and Privacy Policies script for the Amazon mTuuk experiment.

---

Thanks participating this HIT!
story events described in these news piece. The goal is to determine how much one event caused the other (causal relationshipl.
given segment
published
years. given pairs
You will define causal relationship much event causes another one) between the events choosing from (No causal relationship) to 100 (strong causal relationship) BASED ON THE NEWS PIECE.
The score can be interpreted as:
Score
definitely responsible
Score 51-80) Event A responsible for Event B, but not directly: there one more intermediate events between Event and happening.
Score 21-50 The context doesn't give any clue about connection between the events; there might be situation where there 8 small responsibilitt Event cause Event
Score 0-20 Events somehow related but definitely NOT causally related.
The EXAMPLE below showcases different causality scores various event pairs.
CAUSALITY SCORE
News Piece:
The online shop Book Depository due to close the end April.
Event Amazon eliminates number of positions.
(81-100)
Event 7: Book Depcsitory due close.
bookseller's parent company Amazon announced it had decided "eliminate" number across its Devices and Books businesses.
Event 7: Depository close.
rapidly over last several years.
"This year's 's review has been more difficult given uncertain economy
rapidly over the last several years" Amazon CEO wrote the end year. spate layofts reaction to hiring wave that got out of hand, when the world was COVID19 pandemic.
Event The world was struck by the COVID19 pandemic.
(-50)
Event 7: Book Depository due to close.
Event 2:
(21-50)
Event 3: Book Depository won two UK Startup
Amazon acquired Book Depository.
Book Depository was founded Crawford It was acquired by Amazon 2011 1. Then, 2012, Depository the Fast Growth Business Awards' Retai/I. Lalsue Business the Year Awards.
Event Depository founded in 2004 by Stuart Felton.
(0-20) -..e// Jo
Event &
its positions.
Along with the causality score, you will asked whether the second event would have happened the first event didn't occur. your causal score the previous question is relatively high, the second event probably couldn't have happened without the first one.
Figure 8: Amazon Turk instructions script. Annotators need perform the pairwise causality assessment task based on these instructions.

---

News
Layot's ars making readiiness most seamn be certared accund one sector: sech, The sech indurryy nomessd tos leyott biats 20z2, which the hiphoot winck the die- com bubble more than the decarie ago, accoming to "The Chai renger Peport Mor tech employees were
than
Companiea turn layotta surviaal moch cut coata when novanus and araits Ieaner
When
econonyy
Reneevee 2002 aase hightel inilaton rate
acconding the U.S. Buresu of
Laber Stattetics (1I.5) The aconamy softenedt people staned buying less to scconmochie these hightr prices The cod Biving jumpadd sigeiiicatll. and people and husimasnas has make putbacks, Tochhollog: compani sau price
services so compan had
Higher intweed cate affect how much a cormoon momomy sonts to horme di highee coste. These higher ratee directly impast acrture capiaa sis (VS and other funsiing of stanuce.
Comporess
inveat riskee cropp when sconomy to futurng
Investore want companie ee to decresse expensee revenues nlow cown. VGS are worted That profits be slown this your other the arge grow period.
of technoloay sugiiicantty many
pandemmi:
Wer the lapso of BVE, more venture capita ists and bonvss zre going be narvous acout
asing on tai star-up compan sos.
Ansoer the follwwig gassoiorr desuci informmaiion duscribed Ploco
what estart Event cause Exert happwr??
Event pandemicc
reduetry
Woutd Evart hure tapporsed,
Event wouldrrt have tsken place?
Event
Tacn industy liyoth ivaredaed by 64054 in 2600. The Fighisst since the dct-com buboia.
Figure 9: Amazon murk example script for annotating event pairs extracted from the same document. Based on the instructions, annotators need to read the narrative and assess the causal relationship of at most event pairs at a time. Due to space limitations, the figure depicts only one pair.
News Plese
News Ploce 2:
Layorts making
days, Enniland been
incrossed
foroad support UKmaaiiot posemment' weremnniiaaaaeen and increass somowing.
by ddir% 5122, which the lighest sance cist con--ca bubole move a You deeadine ago. ancovting to "The Challanger Repport Vore toch emproyees were uaid of 2022 tran 2020 and combiness.
last month that seci seolddlebiee bhshh trooes
central bank warmed Thaestay that thene "mmttett francca stabiity sharp gevertment wcross the econamy and
and get
paraion funde dump
assels raise cash.
When intiat the econamy by hard Juna 2002, consumars sats prize mcreasee of comzaned the hupical senual mite of noccinting
sump covennmntt bondl that promis to protect inweefor: drivdd gills
shrody
Laber deathetics (fi.5) The
soflrred
started
buying loss accommodkte these higher prices, The soot of jurpaed angn licarttyy and people and bus thesees had to make nuthadees iectrodtog companieee cne prlcee mechwasse noooosory.
"Dystunation this market, the prospoct of f-bliforring 'Ere salel dynamicc pose munanal rak LK finariiil stasalls said in statenment
lowe
oo porioc hot evalucto mare
borro cue higher costs. Those Higher ralos cirecty impect vanturo capitalltt art (VCh) and other funding startsss Companier not want invett vakler anoon when sconomy's future uncertaan Econcmio uncenaiddss cauae par les atusoo strategien..
Higher interest
how much
the
1900
millong
Eriedd gitts dua in 2001 the righes likd sinee October 2018.
aecording Rosoonns
Sturting Taossaay the Burk Erigland poluda indox linked energenzyy "These additiinal cperations progrem ammounced
Investoro vai't companiee len decraase excomaee revemuua slow dovnn, VCTa are worted that profita will be down this your wfter the large growth pertod.
furhhr backatop restore onderty man manket conaliiooa."
widediiithh benk seid the program would and as plenned on Friday desplt dalls io continue for another three weeks.
information desoribed in News Pleco
feppoan?
wartid bomno Higher comparies
100
Would Event Bi have happersec, Event woulden's hune talean place??
Dvent
UK govermmnn ann 1o00 to sizsh laves ircresse bernwing.
Figure 10: Amazon Turk example script for annotating event pairs extracted from different documents. Based on the instructions, annotators need to read both narratives and assess the causal relationship at most event pairs at time. Due space limitations, the figure depicts only one pair.

---

PROMPT: Graded Pairwise Causality MCQ
are a helpful assistant causal relationship understanding. Think about the cause-and-effect relationships related to context.
Context:
<DOCUMENTS>
Event: <EFFECT>
What is the likely cause this event?
[A] <CAUSE 1>
<CAUSE
<CAUSE
[D] <CAUSE
Let's work this in step-by-step way to be sure that have the right answer. Then provide your final answer wi within the tags, <AnswerrA/B///D<//ssseee
Table 15: Prompt for the Graded Pairwise Inference task.
PROMPT: Pairwise Causality Score
You helpful assistant for causal relationship understanding. Think about the cause-and-effect relationships related to context.
Context:
<DOCUMENTS>
Event 1: <EVENT 1>
Event 2: <EVENT 2>
What is the causality score between Event and Event 2 from 0 to 188?
Score above 68: Event is definitely responsible Event 2.
between 50-80: Event might have been responsible for Event 2.
Score lower than 50 Events are somehow related but definitely NOT causally related.
step-by-step way to that have the right answer. Then provide final answer within the tags, <enswer>scores</nssee..
this
Table 16: Prompt for the Pairwise Causality Score In- ference task.

---

Story
Number of pairs
Category
Twitter Acquisition
216
Business
Collapse of Silicon Valley Bank
177
Businoss,Economy
Gwyneth Paltrow ski crash
174
Lifestyle:Lega
Renewable energy Acquisition of Credit Suisse by UBS
66
Environmennt
147
Busnney
Second Pink Tide
108
Politics
Release of ChatGPT
130
Technology Business
Roe wade Overtumed
153
Society:Politics
Energy crisis
148
Economy
Heat waves
150
Environment
Tech Financial crisis
119
Bussomy.Business
Twitter Acquisition
117
Business
European floods
135
Environment
Ethereum Price
119
Business
Black Lives Matter riots
133
Society:Politics
Amazon rain forest wildfires
142
Environment:Politicc
Notre Dame Fire
163
Society
MeToo movement
216
Society:Politics
Equifax data breach
193
Bolitics:Business
Panama Papers
139
Politics
Table Stories present in CRAB along with statistics about the number of pairs event each contains.


Reviews:
[{"review_id": "Oodf4eGbuh", "reviewer": "Reviewer_d7EN", "summary": "", "questions": "1. What were the qualification task or acceptance criteria for expert annotators mentioned on the paper? What are their annotator agreement values? How are they different from Amazon mTurkers in terms of qualification?\n2. Why did not use encoder models? Causality predictions can be mapped as a QA model task?\n3. Why didn't you try fine-tuning the models? This can provide more insights about the difficulty of the task ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments, and for describing our benchmark creation process as thoroughly explained and analyzed. Below, we answer concerns and questions regarding the annotator agreement score, the expert annotation qualification and agreement, and the fine-tuning of encoder models to tackle the tasks.\n\n**Annotation agreement scores**: The reviewer points out the limitation of mTurk annotators agreement scores. To tackle this in our work, (1) not only did we use 7 to 10 annotators per event pair to make up for the inherent difficulty and subjectivity of the task, but also (2) we strengthened it with further expert annotation. Moreover, the low inter-rater agreement has to be taken with a pinch of salt, as the agreement score is usually computed for a unique set of annotators labeling the same samples, while with MTurk the samples are not necessarily labeled with the same set of annotators. \nFrom the beginning of this study, countering the subjectivity and complexity of the task to obtain robust causal scores from human annotators was a challenge at the heart of the research question. That is why we dedicated a considerable amount of time to carefully designing the task and the annotation guidelines, comparing several task designs (binary, multi-class, numerical causal score) and instructions before settling for the ones presented in the paper.\n\n**Expert qualification and agreement scores**: \nExpert annotators are NLP researchers who are familiar with the task of causal inference.  After the first round of annotations, we select event pairs for the second round according to two criteria:\n1) High variance in causal score between mTurk annotators.\n2) The average causal score falls close to a class threshold (from 0 to 100, the class thresholds are 20, 50, and 80; we provide the definitions of the classes to the annotators as guidance in the instruction).\n\nAccording to these criteria, 26.7% of the event pairs were re-annotated by experts. The inter-rater agreement, using Krippendorph’s alpha, is 0.70.\n\n**Fine-tuning encoder models**: \nThe reviewer mentions that trained methods, and more specifically encoder models, might have been more competitive than generative models for this task.  We provide 6 additional experimental settings and their results on fine-tuned encoder-only and decoder-only models. More specifically, we fine-tune the following SoTA models:\n- DeBERTa-v3-large (encoder-only)\n- Llama2-7B (decoder-only)\n\nWe fine-tune both models on the 3 different pairwise causality tasks presented in our paper. For each task, we create 3 different train/test splits (75%/25% ratio) to study the generalization ability of the models after fine-tuning: \n\n- Date: We select 5 out of the 20 most recent stories for the test set and the rest for the train.\n- Story: We randomly select 5 stories for the test set and the rest for the train.\n- Random: We randomly split the event pairs dataset, regardless of the story or the date. \n\nThe F1 scores of the 18 new experiments are included in the table below:\n\n| CAUSALITY TASKS                | MODEL         | Split by date | Split by Story | Random split |\n|--------------------------------|---------------|:-------------:|:--------------:|:------------:|\n| Pairwise Causality Score          | DeBERTa-large  |      21.6      |      21.4        |     22.9      |\n|                                                   | Llama2 7B         |    **24.3**   |    **26.3**    |   **32.8**   |\n| Multi-class Pairwise Causality  | DeBERTa-large  |    **29.4**   |    **35.8**    |   **60.8**   |\n|                                                   | Llama2 7B         |      23.2       |      23.1       |     32.7       |\n| Binary Pairwise Causality         | DeBERTa-large  |    **62.5**   |    **74.2**    |   **76.6**   |\n|                                                   | Llama2 7B         |      51.1       |      51.9       |     58.5       |\n\nThe results show that fine-tuned DeBERTa-large (encoder-only) models fail to perform well on CRAB, showing that our benchmark challenges the current SoTA methods. Compared to decoder-only models in a few-shots setting, DeBERTa-large tends to underperform when splitting by story, except for the easier binary pairwise causality prediction task. Additionally, as expected, the experiments with the random data split have higher scores, which validate the information leakage of the context from the train to test set and verify that models rely on the context (articles) when assessing the causal relationship of the two events. A subsequent study on how fine-tuning improves pre-trained LLMs causal reasoning abilities would be interesting, and we hope that our paper provides a strong and necessary benchmark for pursuing this research direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and their labeling methods and data statistics are thoroughly explained and analyzed.\n- They are releasing a new dataset. \n- Labeling tasks, qualification tasks, GPT prompts and other related details are explained well.\n", "reasons_to_reject": "Annotation agreement values are low (although causality level scoring is inherently a hard task as mentioned in the paper, annotation guidelines or task design could be improved to increase the agreement).\n\n", "questions_to_authors": "1. What were the qualification task or acceptance criteria for expert annotators mentioned on the paper? What are their annotator agreement values? How are they different from Amazon mTurkers in terms of qualification?\n2. Why did not use encoder models? Causality predictions can be mapped as a QA model task?\n3. Why didn't you try fine-tuning the models? This can provide more insights about the difficulty of the task ", "paper_topic_and_main_contributions": "This paper introduces a new dataset “Causal Reasoning Assessment Benchmark” which consists of 2,730 pairs of real-world events. The performances of GPT models on the new dataset suggest that these causality tasks are difficult for even state-of-the-art large language models.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Figure 2: The top left side first two figures ‘Dirent’ -> ‘Direct’.\n- L869: Missing a closing parenthesis.\n- Table 11: You are a journalist who ‘whats’ -> ‘wants’\n"}, "extra_text": {}}, {"review_id": "KpG0AlVprj", "reviewer": "Reviewer_g6sm", "summary": "", "questions": "[Q1] Question on Figure 1. Based on this Figure, one realizes that E1 (that is Musk offered to by Twitter for dollars 44 billion in April) causes in 'low causality' fashion E2 (that is 'Elon Musk closes dollars 44 billion deal to buy Twitter'). Provided that I am not an expert of causal reasoning, this does not seem a real cause. Maybe having a sound notion of what is causal and what is not may be helpful in selecting the appropriate examples, and spread beneficial effects on resources and on LMs training steps.\n[Q2] Section 3.5 Causality Strength Validation. \"we perform a validation of the collected causal scores by asking three expert annotators to further annotate event pairs with high variance\". Does this mean that final figures were obtained based only on expert annotators ratings? or how were ratings merged?\n[Q2.2] How many events are featured by high variance?\n[Q3] Figure 2 is referred many times, but there is no Figure 2. At times, Figure 3 seems compatible with information referred to as available in Figure 2, but not always. Please correct this mistake.\n[Q4] lines 454:458. \"Both models tend to predict causation between events with a gold causal score  above 50. Interestingly, if we increase this gold  score threshold and consider only causal events with a high causal connection, the performance in all models drops\". How do the Authors interpret this trend? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our proposed benchmark as a challenging and interesting resource. \n\n**How causal knowledge can be integrated into LLMs?**: Our proposed CRAB can be viewed as a causal graph that can be used by LLMs either as external knowledge (Zhang et al., Bosselut et al.) or as structured information incorporated in the model’s prompt (Yao et al., Schick et al.) as previous literature suggests. We look forward to future works on how they can integrate neural-symbolic methods that perhaps improve the causal reasoning performance on our benchmark. We thank the reviewer for suggesting further elaboration on the future work, and we will update the final version of the paper accordingly.\n\n**Details on Figure 1**: The reviewer discussed the relationship between E1 (“Musk offered to buy Twitter for 44 billion dollars in April”) and E2 (“Elon Musk closes 44 billion dollars deal to buy Twitter.”). We clarify that Figure 1 should be read as all events E1-5 leading to E6. The causality levels mentioned in the boxes are respective to the events only towards E6: E1 and E3 have low causality on E6; E2 has medium causality on E6;  E4 and E5 have high causality on E6.  The additional arrows represent temporal dependency (not necessarily causality) between events. We thank the reviewer for noticing that a better presentation could improve the message of the figure, and we will update it accordingly. \n\n**Expert validation process**: The reviewer asks for more details on the process we followed for validating mTurk annotations by experts. After crowdsourcing the causal scores from Mechanical Turk, we selected scores that fell on the boundary of different causal classes (high, medium, low, no causality) and also had high variance. This subset of event pairs consists of ~26.7% of the total benchmark. The experts were asked to validate these high-variance edge cases by choosing which of the neighboring classes was a better class for the annotations. We will update the final version of the paper by including this information in the main section of the benchmark creation.\n\n**Figure 2 can be found at the top of page 2.** We assume that maybe there was a rendering issue for that figure. We will try to optimize figure sizes and ensure proper image rendering for the final version of the paper.\n\n**Interpretation of causality threshold on models’ performance**: The reviewer suggested that further justification should be added regarding the performance curve of the models when the binary causality threshold changes. The binary causality threshold is the threshold we put on the causality score (0-100) to consider it causal. The more we increase the score threshold, the more precision drops (as the model’s prediction threshold remains the same), and the overall models’ performance (F1 score) drops. We thank the reviewer for this comment, and we will additionally include the precision and recall curves in the final version of the paper. A possible interpretation behind these results is that large language models are typically quite good at identifying the distributional similarity between concepts. When it comes to making causal judgments, they may identify multiple related events as causally associated with an outcome rather than a single consequential event. This means that they may misclassify events weakly related to the event in question as causes. Ideally, when asked about a binarized causal relationship, we would like models to provide only the main causes (strong relation between two events).  \n\n**References**:\n\nZhang, Xikun, et al. \"Greaselm: Graph reasoning enhanced language models for question answering.\" arXiv preprint arXiv:2201.08860 (2022).\n\nYao, Shunyu, et al. \"React: Synergizing reasoning and acting in language models.\" arXiv preprint arXiv:2210.03629 (2022).\n\nSchick, Timo, et al. \"Toolformer: Language models can teach themselves to use tools.\" arXiv preprint arXiv:2302.04761 (2023).\n\nBosselut, Antoine, et al. \"Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering.\" Proceedings of the AAAI conference on Artificial Intelligence. Vol. 35. No. 6. (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper proposes an interesting and useful resource, which seems to be challenging for NLP systems.\n", "reasons_to_reject": "- The approaches employed to deal with the tracking of causal relations seem to be far from reaching satisfactory results; Authors were solicited to elaborate on how they expect that LMs can be employed to track causal relations.", "questions_to_authors": "[Q1] Question on Figure 1. Based on this Figure, one realizes that E1 (that is Musk offered to by Twitter for dollars 44 billion in April) causes in 'low causality' fashion E2 (that is 'Elon Musk closes dollars 44 billion deal to buy Twitter'). Provided that I am not an expert of causal reasoning, this does not seem a real cause. Maybe having a sound notion of what is causal and what is not may be helpful in selecting the appropriate examples, and spread beneficial effects on resources and on LMs training steps.\n[Q2] Section 3.5 Causality Strength Validation. \"we perform a validation of the collected causal scores by asking three expert annotators to further annotate event pairs with high variance\". Does this mean that final figures were obtained based only on expert annotators ratings? or how were ratings merged?\n[Q2.2] How many events are featured by high variance?\n[Q3] Figure 2 is referred many times, but there is no Figure 2. At times, Figure 3 seems compatible with information referred to as available in Figure 2, but not always. Please correct this mistake.\n[Q4] lines 454:458. \"Both models tend to predict causation between events with a gold causal score  above 50. Interestingly, if we increase this gold  score threshold and consider only causal events with a high causal connection, the performance in all models drops\". How do the Authors interpret this trend? \n", "paper_topic_and_main_contributions": "This paper introduces CRAB, so named after 'Causal Reasoning Assessment Benchmark', that has been devised to track causal understanding of events in narratives. CRAB contains about 2.7k event pairs\nCRAB collects causal subgraphs, where events are linked with their potential causes, and labeled (by humans) with causal scores. The paper proposes another view on events and their causes (causal chains), that is chains of temporarily ordered events that led to a further event. 20 stories about major recent events were selected, and news articles collected; the events underwent a pipeline whereby they were extracted, verified, timelines were identified and event disambiguated. Finally, causal links between all (2730) event pairs were individuated, with involved events belonging to either same document or different documents. \n\nAll models perform poorly on CRAB, with GPT-4 showing a higher performance in most tasks compared to GPT-3 and Flan-Alpaca. However, the Authors admit that models perform worse on causal reasoning when events are derived from complex causal structures compared to simple linear causal chains, and that overall models fail to capture complex causal structures.\n\nThe paper is overall clear and I really enjoyed reading it. However, I feel that Authors should discuss one main issue. LM are basically probability distributions over text sequences: as such, LMs are unfit to deal with causal reasoning or matters such as acquiring any sort of propositional knowledge (what is needed to determine whether an event did influence another one).\nIt is not surprising that the models at stake obtained poor performance in the evaluation. Authors should then explain how a LM can be expected to track causal relations. This point should be discussed in the Introduction and/or discussed after having presented the experimental results. \n\n\n\nQuestions & Concerns\n\n- Question on Figure 1. Based on this Figure, one realizes that E1 (that is Musk offered to by Twitter for dollars 44 billion in April) causes in 'low causality' fashion E2 (that is 'Elon Musk closes dollars 44 billion deal to buy Twitter'). Provided that I am not an expert of causal reasoning, this does not seem a real cause. Maybe having a sound notion of what is causal and what is not may be helpful in selecting the appropriate examples, and spread beneficial effects on resources and on LMs training steps.\n- Section 3.5 Causality Strength Validation. \"we perform a validation of the collected causal scores by asking three expert annotators to further annotate event pairs with high variance\". Does this mean that final figures were obtained based only on expert annotators ratings? or how were ratings merged?\n- How many events are featured by high variance?\n- Figure 2 is referred many times, but there is no Figure 2. At times, Figure 3 seems compatible with information referred to as available in Figure 2, but not always. Please correct this mistake.\n- lines 454:458. \"Both models tend to predict causation between events with a gold causal score  above 50. Interestingly, if we increase this gold  score threshold and consider only causal events with a high causal connection, the performance in all models drops\". How do the Authors interpret this trend? \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cMQbtHg9pu", "reviewer": "Reviewer_GU2u", "summary": "", "questions": "Q1: What was the criteria used by the annotators to score the causal relations ?\n\nQ2: Considering that the number of documents is not very important (173 short news articles), why go to the trouble of having experts filter events extracted by a generative model while you could have the experts read the documents and extracted the events themselves ?\n\nQ3: why use an ill-suited generative model for evaluation while you could have trained even a simple model using the data you created ? Maybe even a CNN would have done a better job than the models you evaluated.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer describing our proposed dataset as interesting for the research community and highlighting that our annotation process is well implemented.\n\n**Depth over breadth:** We appreciate the reviewer highlighting that the strength of the proposed benchmark is focusing on testing LLMs in various complex causality structures on real-world events expressed in separate documents. To that note, the main goal of this benchmark is to focus on depth over breadth by capturing detailed and robust annotations (7-10 annotators per pair, 21300 annotations in total, for 2730 pairs of events coming from 173 documents) with high diversity in causal structures (multi-hop causal reasoning chains, complex causal graph structures).\n\n**Additional clarifications for the dataset creation method:** The reviewer asks for further clarifications on the dataset construction. Stories were selected to cover a variety of categories like business, economy, environment, politics, etc. A detailed breakdown of the category corresponding to each story can be found in Table 16 of our paper. We manually selected top stories based on the Yearly Google Trends for the past 10 years. Due to limited resources, we selected 20 stories for annotation, aiming at a high diversity among story categories. After the automated event extraction, as described in Section 3.3 of our paper, three NLP experts on causal reasoning tasks manually filtered out events that were not valid. Then, we asked mTurk workers to read one or two articles containing the events in the causal relationship that they needed to assess. Based only on the context (articles) presented to them, they provided a score from 0 to 100 of their judgment regarding the causal strength between the events. To ensure the understanding of the tasks by the annotators, we provided a detailed explanation and breakdown of the score and various annotated examples that can be found in the Appendix [A1] of our paper. Additional details on each step of the dataset creation process can be found in our paper's Appendix [A]. As suggested by the reviewer, some of these details should be present in the main paper as well, and therefore, we will incorporate these updates for the final version. \n\n**Semi-automated event extraction:** The reviewer asks why we did not rely on humans to extract events from stories, and used GPT3 instead. We noticed that manual event extraction took significant time for each article (around 20-30 mins per article- 57 hours for 1 person). Additionally, the level of abstraction of events and their format was not uniform across different annotators. Due to the limited experts we had, we decided that filtering the automatically generated events (annotation time ~3 mins per article) was more efficient, especially upon scaling it to the full benchmark size (9 hours for 1 person vs. 57 initially).\n\n**Fine-tuning on benchmark and going beyond prompting decoder-only models.** We initially evaluated our proposed dataset on decoder-only models because decoder-only models (especially API-based ones such as GPT-3 / 3.5 / 4) have become important pillars of AI products, motivating researchers to benchmark their capabilities and identify their biases and limitations. However, as the reviewer reasonably pointed out, it is important to additionally evaluate our causal benchmark on different architectures and inference techniques, providing additional insight on the difficulty of the task. \n\nOn that note, we fine-tune the following SoTA models:\n\n- DeBERTa-v3-large (encoder-only)\n- Llama2-7B (decoder-only)\n\nWe fine-tune both models on the 3 different pairwise causality tasks presented in our paper. For each task, we create 3 different train/test splits (75%/25% ratio), to study the generalization ability of the models after fine-tuning: \n\n- Date: We select 5 out of the 20 most recent stories for the test set and the rest for the train.\n- Story: We randomly select 5 stories for the test set and the rest for the train.\n- Random: We randomly split the event pairs dataset, regardless of the story or the date. \n\nThe F1 scores of the 18 new experiments are included in the table below:\n\n| CAUSALITY TASKS                | MODEL         | Split by date | Split by Story | Random split |\n|--------------------------------|---------------|:-------------:|:--------------:|:------------:|\n| Pairwise Causality Score          | DeBERTa-large  |      21.6      |      21.4        |     22.9      |\n|                                                   | Llama2 7B         |    **24.3**   |    **26.3**    |   **32.8**   |\n| Multi-class Pairwise Causality  | DeBERTa-large  |    **29.4**   |    **35.8**    |   **60.8**   |\n|                                                   | Llama2 7B         |      23.2       |      23.1       |     32.7       |\n| Binary Pairwise Causality         | DeBERTa-large  |    **62.5**   |    **74.2**    |   **76.6**   |\n|                                                   | Llama2 7B         |      51.1       |      51.9       |     58.5       |\n\nThe results show that fine-tuned DeBERTa-large (encoder-only) models fail to perform well on CRAB, showing that our benchmark challenges the current SoTA methods. Compared to decoder-only models in a few-shots setting, DeBERTa-large tends to underperform when splitting by story, except for the easier binary pairwise causality prediction task. Additionally, as expected, the experiments with the random data split have higher scores, which validate the information leakage of the context from the train to test set and verify that models rely on the context (articles) when assessing the causal relationship of the two events. A subsequent study on how fine-tuning improves pre-trained LLMs causal reasoning abilities would be interesting, and we hope that our paper provides a strong and necessary benchmark for pursuing this research direction.\n\n\n**References:**\n\nChung, Hyung Won, et al. \"Scaling instruction-finetuned language models.\" arXiv preprint arXiv:2210.11416 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The principal contribution of this paper is obviously the proposed dataset. While it it of relatively small size (2700 pairs), it deals with real-world events and with causal relation expressed in separate documents, which makes it interesting for research. The annotation seems to be well done.", "reasons_to_reject": "The way the dataset was created is limited: it consists of event causalities related to only 20 \"stories\" (like the buying of Twitter by Elon Musk), and all events were extracted from a set of 173 documents.\n\nMany details are also missing about the dataset creation method: how were the stories selected ? Who chose the events after the automated generation step ? And the most important, what was the criteria for the annotators to annotate the causal relations, i.e. on what basis did they score the causal relations ? Some of these details are described in the appendix but should be present in the paper itself.\n\nFinally, the conclusions drawn, that \"most systems achieve poor performance on the task\", are based on prompting existing GPT-based models to score event causalities input to them. As the models were not fine-tuned for this task, the chance that they would perform poorly under this evaluation is quite high. Why not train a simple model with the data created for this paper ?", "questions_to_authors": "Q1: What was the criteria used by the annotators to score the causal relations ?\n\nQ2: Considering that the number of documents is not very important (173 short news articles), why go to the trouble of having experts filter events extracted by a generative model while you could have the experts read the documents and extracted the events themselves ?\n\nQ3: why use an ill-suited generative model for evaluation while you could have trained even a simple model using the data you created ? Maybe even a CNN would have done a better job than the models you evaluated.", "paper_topic_and_main_contributions": "The main contribution of this paper is the dataset described in it. The authors explain how they created a dataset consisting of real world events and causal relations between them. These events were extracted from web pages using a combination of automated extraction and hand-labeling, and causal relations were further annotated by annotators.\n\nThe authors also provide an analysis of the dataset created as well as simple experiments using generative models, and from the results obtained, conclude that those have difficulties recognizing actual causalities in the dataset,", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The following questions are not to be answered, and should serve as guide to add missing information and details to the paper.\n\nl218: how were the stories selected ?\n\nl223: how were the documents clipped ?\n\nl237: higher than what ?\n\nHow were the events ordered in a timeline ?\n\nl270: how did you end up with 2370+360=2730 pairs ? Given about 19 events per story, this means that you can make 19*18/2=171 event pairs per story, for at least 3420 pairs total.\n\nl315,319: how were the causal frames/chains extracted and/or annotated ?\n\nl349: what is the prompt given to the model ?\n\nl380: what is the score-label mapping ?\n\nl437: is that really \"various\" models ? The three models tested are all variations of the same (GPT)."}}]

Author rebuttal:

Meta-review: The paper introduces a new benchmark "CRAB" -- Causal Reasoning Assessment Benchmark, which is used to evaluate causal understanding of events in real-world narratives. The dataset contains 2,730 pairs of real-world events, which are based on a set of 20 stories (while it seems small, the annotations are non-trivial and 7-10 annotators have annotated each instance). The performances of GPT models on this dataset suggest that these causality tasks are not very good, highlighting that this is an open task. 
The paper also contains analyses to provide insights into what makes the task difficult.
Decision: Accept-Main