Venue: NAACL 2025

# Cascading Large Language Models for Salient Event Graph Generation

Xingwei Tan\({}^{1,2}\), Yuxiang Zhou\({}^{2}\), Gabriele Pergola\({}^{1}\), Yulan He\({}^{1,2,3}\)

\({}^{1}\)Department of Computer Science, University of Warwick, UK

\({}^{2}\)Department of Informatics, King's College London, UK

\({}^{3}\)The Alan Turing Institute, UK

{Xingwei.Tan, Gabriele.Pergola.1}@warwick.ac.uk {Yuxiang.Zhou, Yulan.He}@kcl.ac.uk

###### Abstract

Generating event graphs from long documents is challenging due to the inherent complexity of multiple tasks involved such as detecting events, identifying their relationships, and reconciling unstructured input with structured graphs. Recent studies typically consider all events with equal importance, failing to distinguish salient events crucial for understanding narratives. This paper presents CALLMSAE, a CAScarding Large Language Model framework for SAlient Event graph generation, which leverages the capabilities of LLMs and eliminates the need for costly human annotations. We first identify salient events by prompting LLMs to generate summaries, from which salient events are identified. Next, we develop an iterative code refinement prompting strategy to generate event relation graphs, removing hallucinated relations and recovering missing edges. Powered by CALLMSAE, we present _NYT-SEG_, a large-scale automatically annotated event graph dataset which can serve as distant supervision signals. Fine-tuning contextualised graph generation models on _NYT-SEG_ outperforms the models trained on CAEVO data. Results on a human-annotated test set show that the proposed method generates salient and more accurate graphs, outperforming competitive baselines1.

Footnote 1: Code and data: [https://github.com/Kingwei-Tan/CALLMSAE](https://github.com/Kingwei-Tan/CALLMSAE).

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 2223–2245

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 2223–2245

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 2223–2245

## 1 Introduction

Events are fundamental discourse units which form the backbone of human communication. They are interconnected through various event relations such as hierarchical, temporal, or causal relations. Event relation graphs are vital for representing and understanding complex event narratives, with nodes representing events and edges denoting relationships between them. High-quality event relation graphs can enhance numerous downstream tasks, such as question answering Lu et al. (2022) and reasoning Melnyk et al. (2022).

Existing studies on contextualized event graph generation primarily focus on fine-tuning language models for end-to-end generation of linearized graphs from documents Madaan et al. (2021); Tan et al. (2024). These methods rely on distant supervision, such as events and event temporal relations detected using CAEVO McDowell et al. (2017), due to the data-intensive nature of language models and the significant manual effort required for annotating event graphs. However, CAEVO's bottom-up, predicate-centric extraction often results in sparse, low-quality event graphs populated with non-salient events. For example, CAEVO frequently identifies all verbs as events, including trivial ones like "_say_" and "_think_," which have minimal connections to other events and contribute little to narrative understanding, thus introducing noise. In recent research involving human annotation Dunietz et al. (2014); Liu et al. (2018), a _summarisation test_ is leveraged to identify salient events or entities, where an event or entity is considered salient

Figure 1: An example of salient event relation graph (top) generated from the NYT article (bottom).



if a human-written summary is likely to include it.

Motivated by these insights, we aim to improve the quality of distant supervision and mitigate noise in event graph generation by incorporating event saliency. Our hypothesis is that effective graph generation benefits from a top-down strategy, where the main events are identified first, rather than extracted solely in a bottom-up manner based on the predicate. Capitalizing on the powerful summarization capabilities of large language models (LLMs), we propose a method that first instructs LLMs to summarize documents before identifying salient events. Specifically, we introduce a codeprompting strategy for constructing event relation graphs encompassing hierarchical, temporal, and causal relations2 (see Figure 1). Unlike vanilla prompting, which queries each potential event pair individually, our code prompt format generates each relation type in a single pass. Furthermore, we incorporate an iterative refinement process using a _hallucination grader_ to filter spurious edges and iterative generation to recover missing ones. Finally, the abstractive nature of salient events presents an evaluation challenge, as they rarely match gold standard events exactly, even when semantically equivalent. To address this, we propose evaluation metrics based on semantic text embeddings for assessing the generated event relation graphs.

Footnote 2: We extend beyond the CAEVO’s temporal-only relations to encompass multiple relation types.

Our experimental results on the New York Times corpus Sandhaus and Sandhaus (2008) show that our proposed CALLMSAE, a novel CAscading Large Language Model framework for SAlient Event graph generation, outperforms the baselines in terms of event saliency and edge quality. The fine-tuned model surpasses previous models trained with CAEVO-generated graphs. Our contributions are summarised as follows:

* We propose CALLMSAE, a CAscading Large Language Model framework for SAlient Event graph generation, serving as a distant signal generator for contextualised graph generation models.
* We propose a novel contextualised evaluation metric for comparing salient event graphs. Our extensive experimental evaluation on LLM-generated event relation graphs in terms of event saliency and event relation on the NYT corpus, demonstrating how higher quality salient event graphs can improve context-alised graph generation.
* We provide a large-scale LLM-generated salient event graph dataset _NYT-SEG_ with three major event relation types for distant supervision (\(10,231\) documents), along with a human-annotated test set (\(100\) documents).

## 2 Related Work

Event Relation Graph ConstructionThe early idea of event relation graph construction comes from UzZaman et al. (2013), which introduces a dataset for evaluating an end-to-end system which takes raw text as input and output TimeML annotations (i.e., temporal relations). CAEVO McDowell et al. (2017) and Cogcomptime Ning et al. (2018) both utilise a wide range of manually designed features to train MaxEnt and averaged perception for extracting events and relations. Han et al. (2019) proposed a joint event and relation extraction model based on BERT Devlin et al. (2019) and BiLSTM Panchendrarajan et al. (2018). Other researchers focus on developing specialised sub-systems to classify extracted event pairs for relations Ning et al. (2019); Han et al. (2019); Wang et al. (2020); Pergola et al. (2021a, b); Tan et al. (2021, 2023). ATOMIC Sap et al. (2019) is a large-scale commonsense knowledge graph containing the causes and effects of events. MAVEN-ERE Wang et al. (2022) is built with event coreference, temporal, causal and subevent relations. However, ATOMIC and MAVEN-ERE completely rely on crowdsourcing and thus are difficult to extend. MAVEN-ERE is less than half the size of our dataset and does not consider the saliency of events.

Madaan et al. (2021) fine-tune GPT-2 to generate linearised graphs from documents in an end-to-end manner. Their temporal relation graphs used for training are produced by CAEVO. Following this direction, Tan et al. (2024) instead view the task as set generation and propose a framework based on set property regularisation and data augmentation. In this paper, we focus on generating multi-relation graphs via in-context learning, prompt interaction, and iterative refinement.

Salient Event IdentificationSeveral existing papers investigate the problem of identifying salient events. Choubey et al. (2018) build a rule-based classifier to identify central events by exploiting human-annotated event coreference relations. They find the central events either have large numbers of coreferential event mentions or have large stretch 

sizes. Jindal et al. (2020) propose a contextual model to identify salient events based on BERT and BiLSTM. They also mention several features, such as event trigger frequency, which are essential features to identify the salient events. Liu et al. (2018) propose a feature-based method using LeToR Liu et al. (2009) and a neural-based method called Kernel-based Centrality Estimation. To train and evaluate their methods, they build a dataset based on the _summarisation test_: an event is considered salient if a summary written by a human is likely to include it. Zhang et al. (2021) combine the Kernel-based Centrality Estimation with the event and temporal relation extraction model of Han et al. (2019b) to build a salience-aware event chain modelling system. However, they only focus on single-dimensional chains and only model temporal relations.

## 3 Cascading LLMs to Generate Salient Event Graphs

CALLMSAE combines various prompts in a pipelined manner to generate salient event graphs. In this section, we introduce the prompts for generating salient events, and then the method for generating relation graphs based on the salient events. Lastly, we define an evaluation metric for comparing event graphs: _Hungarian Graph Similarity_.

### Generate Salient Events

The _summarisation test_ (Section 1) is often used to guide the annotation of salient events or entities Dunietz et al. (2014); Liu et al. (2018); Lu et al. (2023); Lyu et al. (2024). These studies identify events or entities included in human-written summaries as salient. Similarly, we instruct LLMs to generate a summary first and then extract events from it. Examples of the summarisation and salient event generation prompts are shown in the Appendix (Table 11 and 12).

### Generate Graphs as Code Completion

While LLMs can extract salient events, they often struggle with identifying event relations Chan et al. (2023); Sun et al. (2022, 2024); Tan et al. (2024). Prompt engineering for extracting event relations is complex due to the need to incorporate various terminologies and graph constraints. Moreover, prompt efficiency is crucial as generating a large-scale dataset with LLMs can still incur significant computational costs, albeit less than crowdsourcing.

In our method, the main prompt for generating the event relation graph is formulated as a Python code completion task. The graph is defined using the NetworkX3 package in Python, with nodes representing the salient events generated in Section 3.1. LLMs are instructed to complete the code by adding relation edges using NetworkX's APIs.

Footnote 3: [https://networkx.org/documentation/stable/](https://networkx.org/documentation/stable/)

Recent research suggests that formulating prompts as code can enhance LLMs' reasoning abilities Wang et al. (2023); Zhang et al. (2023). In our task, the Python code format effectively incorporates all necessary terminologies, enabling LLMs to understand them without confusion. The Python code format also allows for the inclusion of constraints (e.g., ensuring the graph is a directed acyclic graph) and additional instructions (e.g., ask for explanations) as comments. LLMs can generate explanations as comments without disrupting the main content of the graph, which is difficult to achieve in other formats (e.g., JSON). Moreover, the code template simplifies parsing the response, as LLMs are directed to use the "\(\_\)add\(\_\)edge()" function to add the relations.

Since hierarchical, temporal, and causal relations are asymmetric, each can be represented by a Directed Acyclic Graph (DAG). We formulate three distinct prompts to guide LLMs in generating three DAGs, each representing one of these relation types. This approach avoids the complexity of a multi-label graph, and LLMs can focus on a single relation type and carefully consider the topological structure of the graph. We can also use the "\(\_\)find\(\_\)cycle()" function from NetworkX to detect constraint violations reliably. In addition, if relation types are interconnected, the initially generated graphs can help the generation of subsequent graphs (as will be explained in Section 3.4). We provide an example of the code prompt in Appendix (Table 13).

### Iterative Refinement

Hallucination GraderThe code prompt efficiently guides LLMs to generate graphs, but it often generates hallucinated relations. Based on our preliminary experiments, these hallucinations stem from the models' overconfidence in their relation predictions. Specifically, LLMs tend to infer event relations without explicit linguistic cues or strong evidence for logical inference. Consequently, LLMs predict far more relations than the gold standards, leading to low precision.

 

Recent studies show that LLMs can evaluate and correct their own outputs (Madaan et al., 2023; Asai et al., 2024). Thus, we introduce a hallucination grader to address hallucination. For each relation edge generated, we pose a question to the LLMs to determine whether the relation is grounded in the given document. If the LLMs respond with a "yes", the edge is retained; otherwise, it is discarded. An example of the hallucination grader prompt is shown in Appendix (Table 14).

Recover Missing EdgesThe main benefit of the hallucination grader is that it increases precision by removing low-confident edges. However, this process inevitably reduces recall. To mitigate this side effect, we introduce an iterative refinement process. After discarding hallucinated edges, we reinsert the code block containing the relation edges into the graph generation prompt and ask the LLMs to complete the code again. In this way, the LLMs can reconsider whether there are any missing relations in the document, thereby improving recall.

Once the LLMs generate a new graph, the hallucination grader checks the relation edges again. This process is repeated for a fixed number of times. We set the maximum number of iterations to \(5\) in our experiments, as the LLMs stop discovering new edges after \(2\) or \(3\) iterations in most documents.

### Complement Relation Types

Hierarchical, temporal, and causal relations are not independent of each other. In this case, we found that providing the graph for the first relation can benefit the generation of the dependent relation's graph. Specifically, we predict the hierarchical relation graph first. Then, we provide this graph to the LLMs and ask them to generate the temporal relation graph. Lastly, with both the hierarchical and temporal relation graphs available, the LLMs predict the causal relation graph.

The hierarchical relation describes two closely related events at different granularity levels. It focuses on the inherent semantics of the events and does not depend on other relation types. For example, "_writing a dissertation_" is a subevent of "_doing a PhD_". Therefore, we choose to predict the hierarchical relations first. Temporal relations can depend on hierarchical relations. For example, knowing "_doing a PhD_" happened before "_being prompted to Professor_" allows us to deduce that "_writing a thesis_" also happened before "_being prompted to Professor_". Thus, we predict temporal relations after hierarchical relations. Lastly, causal relations depend on both hierarchical and temporal relations, as the antecedent event in a causal relation must occur before the consequence. Therefore, the causal relation is predicted in the last step. For more details about the entire prompting process, please refer to the pseudocode in Appendix D.

### Hungarian Graph Similarity

It is challenging to compare event relation graphs generated by LLMs due to the abstractive nature of generation, making it difficult to align the generated events with the gold standard events (Li et al., 2023). Moreover, salient events are often high-level and abstract rather than fine-grained and concrete, which means some variations in wording are not only acceptable but also expected. Instead of using exact matching (Zhao et al., 2024) or rule-based

Figure 2: The proposed CALLMSAE framework.

 

token matching (Tan et al., 2024) on events and relations to calculate \(F_{1}\), adopting semantic-based evaluation metrics is more reasonable and fair. As more tasks adopt text generation frameworks, many researchers are also turning to metrics based on language models rather than traditional token matching metrics like ROUGE and BLUE (Goyal et al., 2022; Pratapa et al., 2023; Lyu et al., 2024a).

In this study, we propose a novel metric for evaluating LLM-generated event graphs, called **Hungarian Graph Similarity (HGS)**. The metric is based on the Hungarian assignment algorithm (Kuhn and Kuhn, 1955), which is widely used in object detection to match generated objects and target objects (Carion et al., 2020). It can find the optimal assignment given a cost matrix containing the distance between elements in two lists of objects. We adapt this algorithm to match predicted edges with edges in the gold standard graphs as follows:

1. Encode the events using SFR-Embedding-Mistral (Meng et al., 2024).
2. Given two edges of the same relation type, let \(\bar{e}^{h}_{1},\bar{e}^{t}_{1}\) be the embeddings of the head event and the tail event in the first edge. Let \(\bar{e}^{h}_{2},\bar{e}^{t}_{2}\) be the embeddings of the head and tail events in the second edges. We define the distance between the edges as \(\max\bigl{(}D_{cos}(\bar{e}^{h}_{1},\bar{e}^{h}_{2}),D_{cos}(\bar{e}^{t}_{1}, \bar{e}^{t}_{2})\bigr{)}\), where \(D_{cos}(\cdot,\cdot)\) is the cosine distance.
3. Build a cost matrix by computing the distance between every edge pair in the gold and predicted edge sets. Pad the matrix to a square matrix with the maximum cost value of \(1\).
4. Apply the Hungarian algorithm to the cost matrix to get the minimal cost value. The final score is \(1-\text{cost}\) value, making the value more intuitive (higher is better). To compute the HGS over all the documents, we weight the scores by the number of gold edges to obtain an average value.

In step \(2\), we take the maximum value of the distances between head and tail events because relation edges are considered matched only if both the head and tail events match.

For more detailed analysis, we define precision-oriented HGS and recall-oriented HGS. We match edges without padding the cost matrix in step \(3\) to obtain the total cost values of all matched edge pairs. Then, the total matched similarity is the number of matched edges minus the total cost. **Precision-oriented HGS** is computed by dividing the total matched similarity by the total number of predicted edges. **Recall-oriented HGS** is computed by dividing the total matched similarity by the total number of edges in the target graph.

## 4 Dataset

In this section, we describe how we construct the _NYT-SEG_ which includes LLM-generated distant training pairs and a human-annotated dataset based on the New York Times (NYT) corpus.

### Document Selection

We follow the same procedures as in Tan et al. (2024) to select documents from the NYT corpus, one of the largest news datasets, with additional filtering based on document length. We select \(10,347\) documents based on their descriptors indicating they are related to event narratives instead of opinions and discussions, such as sports and international politics. Among them, \(100\) documents are randomly sampled as the test set to be annotated by humans. More details about data selection are shown in Appendix A.1.

### Annotation Settings

We recruited annotators from Prolific4. There are two subtasks: _salient event identification_ and _event relation identification_. In the first subtask, the participants are asked to identify the salient event triplets: _actor_, _trigger_, and _object_ (optional). We provide the definition of an event and several examples in the guidelines. They are instructed to do the summarisation test: the salient events should be the events they would include in the summary of the given article. Moreover, we provide some prominent features for helping annotators to identify salient events Choubey et al. (2018); Jindal et al. (2020):

Footnote 4: [http://www.prolific.com](http://www.prolific.com)

* Frequency: salient events are frequently mentioned in the articles.
* First appearance: salient events are often mentioned at the beginning of the article.
* Stretch size: salient events are often mentioned throughout the articles. Stretch size is the distance between the location where the event is first mentioned and last mentioned. A salient event usually has a large stretch size.

 

In the second stage, we ask participants to identify relation triplets: _a source event_, _a relation type_, and _a target event_. Both the source and target events should be among the salient events identified in the first stage. In the guideline, we define three relation types: _happened\(\_\)before_, _caused\(\_\)by_, and _is\(\_\)subevent\(\_\)of_. _happened\(\_\)before_ indicates that the source event happened earlier than the target event. _caused\(\_\)by_ means the source event would not have happened if the target event did not happen. _is\(\_\)subevent\(\_\)of_ signifies that the source event is a subevent of the target event. Annotators were informed that relations would be either explicitly mentioned in the article or inferred based on evidence within the article. Further details about the guidelines and user interface can be found in Appendix A.4.

### Inter Annotator Agreement

Identifying salient events and event relations is complicated and time-consuming. We found it challenging to educate participants about these concepts because, in daily life, the meanings of events and relations differ from their definitions in the field of information extraction. Moreover, the technical definitions are much less intuitive to those outside the academic field. As a result, thorough training of participants is important to obtain high-quality annotations.

In total, we recruited \(3\) annotators to annotate \(100\) documents. Due to their varying availability, annotator \(1\) and \(2\) each annotated \(45\) documents, while annotator \(3\) annotated \(20\) documents. Among these, \(5\) documents were annotated by all three annotators. Following prior research in information extraction Gurulingappa et al. (2012); Zhao et al. (2024), we used \(F_{1}\) to measure the inter-annotator agreement on these \(5\) documents. To compute inter-annotator agreement, events or relations identified by one annotator are represented as set \(S_{1}\). Another annotator's annotation \(S_{2}\) serves as a pseudo-reference to compute precision \(=\frac{|S_{1}\cap S_{2}|}{|S_{1}|}\), recall \(=\frac{|S_{1}\cap S_{2}|}{|S_{2}|}\), and the \(F_{1}\) score \(=\frac{2|S_{1}\cap S_{2}|}{|S_{1}|+|S_{2}|}\).

Table 1 shows the agreement scores for stages \(1\) and \(2\). Identifying salient events is subjective, which makes it difficult to reach a complete agreement. Moreover, event relation identification is even more subjective and dependent on the previous stage, leading to less unanimous agreement.

### Dataset Statistics

Table 2 shows the distributions of the relation types after applying the transitive closure to the annotated graphs. _happened\(\_\)before_ emerges as the most frequent relation type, reflecting the predominant focus on temporal sequences in news articles, and they are relatively straightforward to identify. Conversely, _caused\(\_\)by_ is the least frequent as it is the most challenging to identify.

## 5 Experiments

### Model Settings

We compare against the following baselines:

* CAEVO McDowell et al. (2017) is a pipeline system based on a MaxEnt classifier and manual features for extracting events and relations.
* Madaan et al. (2021) trained language models on CAEVO-generated linearised graphs with LM objective. We implemented their method to train a Flan-T5 model.
* Tan et al. (2024) also trained language models on CAEVO-generated graphs, but applied data augmentations and regularisations to mitigate the set element misalignment issue. We applied their method to train a Flan-T5.
* Han et al. (2019b) proposed a joint event and temporal relation extraction model. We adapted the model to hierarchical and causal relations by training on MAVEN-ERE. We also replaced BERT with Longformer Beltagy et al. (2020) to make it suitable for long documents.
* GPT-4 OpenAI (2024) and GPT-3.5 are based on the generative pre-train framework.

\begin{table}
\begin{tabular}{c c c} \hline \hline Annotation & Stage \(1\) & Stage \(2\) \\ \hline
1 \& 2 & \(0.838\) & \(0.676\) \\
1 \& 3 & \(0.771\) & \(0.645\) \\
2 \& 3 & \(0.847\) & \(0.710\) \\ Average & 0.819 & \(0.677\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Inter-annotator agreement measured by \(F_{1}\).

\begin{table}
\begin{tabular}{l l} \hline \hline Relation Type & Number \\ \hline happened\(\_\)before & \(310\) \\ caused\(\_\)by & \(202\) \\ is\(\_\)subevent\(\_\)of & \(245\) \\ Total & \(757\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: The distributions of the relation types.

 

We used "gpt-4-1106-preview" and "gpt-3.5-turbo" respectively.
* Mixtral is an LLM based on the Mistral model and the mixture of expert framework. We used the 8x7B instruct Jiang et al. (2024).
* Llama3 is an LLM based on the Llama framework. We used the 70B-instruct 8bit version for a balance between speed and performance.

We fine-tuned a Flan-T5-base (\(250\)M) with the relation graphs generated by CALLMSAE, following the same method as in Tan et al. (2024). The baseline prompt evaluates whether each event pair is supported by the document, akin to the hallucination grader described in Section 3.3. Thus, it serves as an ablation of our method without incorporating the code prompt. Another baseline, which asks for the relation type given an event pair, is also tested and included in our experiment.

CALLMSAE is designed to be model-agnostic. Due to budget constraints and the preliminary test results, we chose Llama3 as the backbone of all the prompt-based methods detailed in Table 5.

### Event Saliency Evaluation

Table 3 shows the salient features (defined in Section 4.2, computation formulas in Appendix B) extracted from various backbone LLMs using summarisation prompts, alongside comparison with CAEVO and human annotations. The LLM-generated events are much more salient than CAEVO-generated events and exhibit similarity to human annotations.

We also use human annotations to evaluate the saliency. In the salient event identification annotation, we provide the events generated by CAEVO and Miktral as candidate salient events. Note that only the top CAEVO events ranked in saliency features are shown. Half of the candidates are from CAEVO and the other half are from Mixtral. They are randomly shuffled and then shown to the annotators. We compute the precision, recall, and based on how the annotators select them. We also compute HGS using human-annotated salient events as references (Table 4). It is clear that although CAEVO extracted more events than Mixtral, many of them are not salient. Mixtral outperforms CAEVO significantly across all evaluation metrics.

### Salient Event Relation Graph Evaluation

The salient event relation graph evaluation results are shown in Table 5. The compared methods (row \(1\) - \(4\)) are outperformed by _Baseline Prompt_ (row \(5\)) on all relation types. _Baseline Prompt (rel type)_ (row \(6\)), which asks the model to generate all possible relation types given the event pair, performs slightly better than _Baseline Prompt_. However, _Baseline Prompt_ and _Baseline Prompt (relation type)_ are slow and costly because the number of prompts they need for building one graph is \(O(n^{2})\), where \(n\) is the number of events in the document. On the other hand, the time complexity of _Code Prompt_ is \(O(1)\). Moreover, _Code Prompt_'s overall HGS is significantly higher than _Baseline Prompt_ and _Baseline Prompt (relation type)_ on all relation types. _Baseline Prompt_ check the event pairs more thoroughly and thus have higher recall but its precision is much lower. The complete CALLMSAE combines the code prompt and hallucination grader for iterative refinement, checking missing relations and verifying them to prevent hallucination. _Code Prompt (dependent rels)_ (the \(8\)th row) is an ablation of _CALLMSAE_ (the \(9\) row), differing only in the

\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & \multicolumn{1}{c}{\(P\)} & \multicolumn{1}{c}{\(R\)} & \multicolumn{1}{c}{\(F_{1}\)} & \multicolumn{1}{c}{HGS} \\ \hline CAEVO & \(3.29\) & \(3.72\) & \(3.49\) & \(18.18\) \\ Miztral & \(48.97\) & \(56.77\) & \(52.59\) & \(67.15\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Precision, recall, and \(F_{1}\) based on the choices of the annotators. Hungarian graph similarity (HGS) is defined in Section 3.5. The values are in percentage.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & Mean event number & Event Frequency \(\uparrow\) & First Appearance \(\downarrow\) & Stretch Size \(\uparrow\) \\ \hline CAEVO & \(34.71\) & \(0.05\) & \(0.46\) & \(0.07\) \\ Human & \(8.26\) & \(\mathbf{0.11}\) & \(0.31\) & \(\mathbf{0.20}\) \\ GPT-4 & \(6.49\) & \(0.09\) & \(0.37\) & \(0.18\) \\ Llama3 & \(5.17\) & \(0.09\) & \(\mathbf{0.30}\) & \(0.19\) \\ Mixtral & \(10.60\) & \(0.10\) & \(0.33\) & \(\mathbf{0.20}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: The average number of extracted events and the saliency features (in percentage values).

 

absence of iterative refinement. These results highlight the effectiveness of the hallucination grading approach, which effectively increases the precision and strikes a balance with recall.

In the temporal category, the results of _Code Prompt (dependent rels)_ are obtained when provided with hierarchical graphs generated by CALLMSAE to LLMs. It has much higher overall HGS and precision than _Code Prompt_ without hierarchical information, showing that hierarchical information can mitigate hallucinations during the temporal graph generation. In the casual category, the results of _Code Prompt (dependent rels)_ are obtained when given both hierarchical and temporal graphs generated by CALLMSAE. The additional information also increases precision.

_Fine-tuned T5_ outperform all the methods based on CAEVO [106, 237, 48], showing that the high-quality graphs generated by CALLMSAE can boost the contextualised graph generation. Interestingly, the performance of the _Fine-tuned T5_, fine-tuned on CALLMSAE-generated data, exceeds that of CALLMSAE itself, implying that the fine-tuned model can effectively adapt the reasoning patterns provided by Llama3 and generalise them.

### Human Evaluation of Event Graph

We recruited additional annotators to evaluate the generated graph on \(50\) of the test documents. We asked them whether the relation edges are correct. These scores (\(\frac{\text{correct edges}}{\text{generated edges}}\)) can be viewed as precision (Table 6). The human evaluation correlates with the HGS scores, verifying our conclusion.

### Format Error and Cycles in the Graphs

We specified the relation graphs as directed acyclic graphs in the prompt. If there is a cycle in the generated graph, it means that the LLM failed to follow the instructions. A cycle also indicates constraint violations because all the relations in the graphs are asymmetric. We use the APIs in _NetworkX_ package to detect cycles in the transitive closure of the graphs. If the Python interpreter returns an error, it is classified as a format error. We prompt each LLM three times on the test set. Table 7 shows the average number of documents encountering format errors or cycles. All LLMs have low rates of format errors which shows that state-of-the-art LLMs can understand the instruction well and generate executable Python code. Among them, GPT-3.5 and Llama3 have no errors. About \(10\%\) of graphs generated by GPT-3.5 have cycles, suggesting that

\begin{table}
\begin{tabular}{l r c c} \hline \hline  & Hier & Temp & Causal & Overall \\ \hline Baseline (rel type) & \(0.58\) & \(0.66\) & \(0.48\) & \(0.60\) \\ CALLMSAE & \(0.71\) & \(0.71\) & **0.65** & \(0.69\) \\ Fine-tuned T5 & **0.72** & **0.74** & \(0.62\) & **0.70** \\ \hline \hline \end{tabular}
\end{table}
Table 6: The human evaluation scores.

\begin{table}
\begin{tabular}{l r c} \hline \hline  & Format Error \(\downarrow\) & Cycle \(\downarrow\) \\ \hline GPT-3.5 & \(0\)\% & \(10.67\)\% \\ GPT-4 & \(3.67\)\% & \(1.67\)\% \\ Mixtral & \(3.33\)\% & \(2.33\)\% \\ Llama3 & \(0\)\% & \(0\)\% \\ \hline \hline \end{tabular}
\end{table}
Table 7: The average number of CALLMSAE-generated graphs out of \(100\) with format errors or cycles.

\begin{table}
\begin{tabular}{l r r r r r r r r r} \hline \hline  & \multicolumn{3}{c}{Hierarchical} & \multicolumn{3}{c}{Temporal} & \multicolumn{3}{c}{Causal} \\ \cline{2-10}  & \(PHGS\) & \(RHGS\) & \(HSG\) & \(PHGS\) & \(RHGS\) & \(HGS\) & \(PHGS\) & \(RHGS\) & \(HGS\) \\ \hline Han et al. [101] & \(0.158\) & \(0.247\) & \(0.098\) & \(0.092\) & \(0.352\) & \(0.148\) & \(0.084\) & \(0.316\) & \(0.116\) \\ CAEVO & - & - & - & - & \(0.030\) & \(0.558\) & \(0.092\) & - & - & - \\ Madaan et al. [101] & - & - & - & \(0.061\) & \(0.439\) & \(0.116\) & - & - & - \\ Tan et al. [104] & - & - & - & \(0.126\) & \(0.335\) & \(0.187\) & - & - & - \\ \hline Baseline Prompt & \(0.076\) & **0.651** & \(0.248\) & \(0.085\) & \(0.627\) & \(0.195\) & \(0.062\) & **0.657** & \(0.207\) \\ Baseline Prompt (\(acl\) type) & \(0.288\) & \(0.375\) & \(0.268\) & \(0.135\) & \(0.604\) & \(0.261\) & \(0.185\) & \(0.513\) & \(0.256\) \\ Code Prompt & \(0.174\) & \(0.559\) & \(0.315\) & \(0.153\) & **0.678** & \(0.283\) & \(0.121\) & \(0.632\) & \(0.272\) \\ Code Prompt (dependent rels) & N.A. & N.A. & N.A. & \(0.211\) & \(0.601\) & \(0.341\) & \(0.135\) & \(0.599\) & \(0.272\) \\ CALLMSAE (ours) & \(0.196\) & \(0.544\) & \(0.334\) & **0.294** & \(0.509\) & \(0.327\) & \(0.198\) & \(0.529\) & \(0.295\) \\ Fine-tuned T5 (calLMSAE) & **0.314** & \(0.434\) & **0.339** & \(0.244\) & \(0.544\) & **0.362** & **0.366** & \(0.397\) & **0.343** \\ \hline \hline \end{tabular}
\end{table}
Table 5: The Hungarian Graph Similarity (HGS) of the LLM-generated graphs on the human-annotated NYT dataset. PHGS is precision-oriented HGS. RHGS is recall-oriented HGS. _Code Prompt (dependent rels)_ means adding hierarchical graphs in the prompts for temporal graphs; and adding hierarchical and temporal for causal graphs. _Fine-tuned TS (CALLMSAE)_ means fine-tuning a flan-T5 using the graphs generated by CALLMSAE. All prompt-based methods (row \(5\cdot 9\)) are based on _Llama3-70B-instruct_.

 

GPT-3.5 may have inferior reasoning ability compared to other LLMs. GPT-4 and Mixtral both have low rates of cycle occurrence, but they are beaten by Llama3 which has no cycle in all generations, showing its remarkable understanding of the transitive and asymmetric constraints in the complex event relation graphs.

## 6 Conclusion

This study explored utilising LLMs to generate salient event relation graphs from news documents without relying on human annotations. We studied how the events generated by LLMs are compared to the traditional methods in terms of event saliency. We further demonstrated that CALLMSAE-generated graphs can serve as distant signals to fine-tune smaller models and outperform those based on CAEVO. The CALLMSAE-generated event graphs, together with the human-annotated test set are collected as _NYT-SEG_.

### Limitations

CALLMSAE is more demanding than CAEVO in terms of computational power and time. To generate the _NYT-SEG_, we spent \(2,200\) wall clock hours in inference. More details about resource cost are disclosed in Appendix D.

Although we have tested many prompting methods and included several of the most effective ones in this paper, we have not explored all possible combinations due to the extensive volume of recent literature on prompt engineering. There might still exist combinations of prompts that could further improve performance Alzaid et al. (2024); Zhou et al. (2024). However, we are almost certain that any potential combinations, if they exist, are likely to be more complex and thus less efficient for building large-scale datasets. For example, we did not add demonstrations in graph generation because the code template is already quite lengthy. Adding more documents could potentially exceed the context windows of some LLMs, making it challenging for them to interpret the instructions effectively. The main goal of this work is to demonstrate the potential of LLM-based generation can help the data-demanding event graph generation task.

### Ethics Statement

Event relation graph generation is a powerful tool for understanding text. A potential misuse of the proposed method is mining user behaviours on their private data. For example, salient event relation graphs can be extracted from users' tweets to analyse their potential reactions to advertisements and scams. That could be a huge risk to social media users.

Another potential risk is that the saliency may introduce bias. LLMs may have their preferences in selecting a specific group of events as important events due to the data they were trained on. This is a question which requires further large-scale investigation. However, we think this risk is negligible in this study because we work on document-level information. There is little room for selection given that the news articles are already the products of choice and distillation. If the system is used to extract information from a border information source, such as social media, the risk must be carefully assessed.

## Acknowledgements

This work was supported in part by the UK Engineering and Physical Sciences Research Council (EPSRC) through a Turing AI Fellowship (grant no. EP/V020579/1, EP/V020579/2). Xingwei Tan was supported by the Warwick Chancellor's International Scholarship. This work was conducted on the Sulis Tier-2 HPC platform hosted by the Scientific Computing Research Technology Platform at the University of Warwick. Sulis is funded by EPSRC Grant EP/T022108/1 and the HPC Midlands+ consortium.

## References

* Elzaid et al. (2024) Ethar Alzaid, Gabriele Pergola, Harriet Evans, David Snead, and Fayyaz Minhas. 2024. Large multimodal model-based standardisation of pathology reports with confidence and its prognostic significance. _The Journal of Pathology: Clinical Research_, 10(6):e70010.
* Asai et al. (2024) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In _The Twelfth International Conference on Learning Representations_.
* Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. _CoRR_, abs/2004.05150.
* Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In _European conference on computer vision_, pages 213-229. Springer 

Chunkit Chan, Jiayang Cheng, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2023. Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations.
* Choubey et al. (2018) Prafulla Kumar Choubey, Kaushik Raju, and Ruihong Huang. 2018. Identifying the most dominant event in a news article by mining event coreference relations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 340-345, New Orleans, Louisiana. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dunietz and Gillick (2014) Jesse Dunietz and Daniel Gillick. 2014. A new entity salience task with millions of training examples. In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers_, pages 205-209, Gothenburg, Sweden. Association for Computational Linguistics.
* Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. _arXiv preprint arXiv:2209.12356_.
* Gurulingappa et al. (2012) Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. _Journal of biomedical informatics_, 45(5):885-892.
* Han et al. (2019a) Rujun Han, I-Hung Hsu, Mu Yang, Aram Galstyan, Ralph Weischedel, and Nanyun Peng. 2019a. Deep structured neural network for event temporal relation extraction. In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_, pages 666-106, Hong Kong, China. Association for Computational Linguistics.
* Han et al. (2019b) Rujun Han, Qiang Ning, and Nanyun Peng. 2019b. Joint event and temporal relation extraction with shared representations and structured prediction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 434-444, Hong Kong, China. Association for Computational Linguistics.
* Jiang et al. (2019c) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Laimple, Leio Renard Lavaud, Lucile Saulnier, Mariecne Lanchaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Theophile Gervet, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2024. Mixtral of experts.
* Jindal et al. (2020) Disha Jindal, Daniel Deutsch, and Dan Roth. 2020. Is killed more significant than fled? a contextual model for salient event detection. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 114-124, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Kuhn (1955) Harold W Kuhn. 1955. The hungarian method for the assignment problem. _Naval research logistics quarterly_, 2(1-2):83-97.
* Li et al. (2023) Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023. Evaluating chatgpt's information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness.
* Liu et al. (2009) Tie-Yan Liu et al. 2009. Learning to rank for information retrieval. _Foundations and Trends(r) in Information Retrieval_, 3(3):225-331.
* Liu et al. (2018) Zhengzhong Liu, Chenyan Xiong, Teruko Mitamura, and Eduard Hovy. 2018. Automatic event salience identification. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1226-1236, Brussels, Belgium. Association for Computational Linguistics.
* Lu et al. (2023) Junru Lu, Jiazheng Li, Byron Wallace, Yulan He, and Gabriele Pergola. 2023. NapSS: Paragraph-level medical text simplification via narrative prompting and sentence-matching summarization. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 1079-1091, Dubrovnik, Croatia. Association for Computational Linguistics.
* Lu et al. (2022) Junru Lu, Xingwei Tan, Gabriele Pergola, Lin Gui, and Yulan He. 2022. Event-centric question answering via contrastive learning and invertible event transformation. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 2377-2389, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Lyu and Pergola (2024a) Chen Lyu and Gabriele Pergola. 2024a. SciGisPy: a novel metric for biomedical text simplification via gist inference score. In _Proceedings of the Third Workshop on Text Simplification, Accessibility and Readability (TSAR 2024)_, pages 95-106, Miami, Florida, USA. Association for Computational Linguistics.
* Lyu and Pergola (2024b) Chen Lyu and Gabriele Pergola. 2024b. Society of medical simplifiers. In _Proceedings of the Third Workshop on Text Simplification, Accessibility and Readability (TSAR 2024)_, pages 61-68, Miami, Florida, USA. Association for Computational Linguistics 

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shahsah Gupta, Bodishatwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Zadambakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Madaan and Yang (2021) Aman Madaan and Yiming Yang. 2021. Neural language modeling for contextualized temporal graph generation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 864-881, Online. Association for Computational Linguistics.
* McDowell et al. (2017) Bill McDowell, Nathanael Chambers, Alexander Ororbia II, and David Reitter. 2017. Event ordering with a generalized model for sieve prediction ranking. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 843-853, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Melnyk et al. (2022) Igor Melnyk, Pierre Dobgin, and Payel Das. 2022. Knowledge graph generation from text. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 1610-1622, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Meng et al. (2024) Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. 2024. Sfr-embedding-mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog.
* Ning et al. (2019) Qiang Ning, Sanjay Subramanian, and Dan Roth. 2019. An improved neural baseline for temporal relation extraction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6203-6209, Hong Kong, China. Association for Computational Linguistics.
* Ning et al. (2018) Qiang Ning, Ben Zhou, Zhili Feng, Haoruo Peng, and Dan Roth. 2018. CogCompTime: A tool for understanding time in natural language. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 72-77, Brussels, Belgium. Association for Computational Linguistics.
* OpenAI (2024) OpenAI, 2024. 2024. Gpt-4 technical report. OpenAI.
* Panchendrarajan and Amaresan (2018) RrubaaPanchendranajan and Arvanindh Amaresan. 2018. Bidirectional LSTM-CRF for named entity recognition. In _Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation_, Hong Kong. Association for Computational Linguistics.
* Pergola et al. (2021a) Gabriele Pergola, Lin Gui, and Yulan He. 2021a. A disentangled adversarial neural topic model for separating opinions from plots in user reviews. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2870-2883, Online. Association for Computational Linguistics.
* Pergola et al. (2021b) Gabriele Pergola, Elena Kochkine, Lin Gui, Maria Liakata, and Yulan He. 2021b. Boosting low-resource biomedical QA via entity-aware masking strategies. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1977-1985, Online. Association for Computational Linguistics.
* Pratapa et al. (2023) Adithya Pratapa, Kevin Small, and Markus Dreyer. 2023. Background summarization of event timelines. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 8111-8136, Singapore. Association for Computational Linguistics.
* Sandhaus (2008) Evan Sandhaus. 2008. The New York Times Annotated Corpus.
* Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence_, AAAI'19/IAAI'19/EAAI'19. AAAI Press.
* Sun et al. (2022) Zhaoyue Sun, Jiazheng Li, Gabriele Pergola, Byron Wallace, Bino John, Nigel Greene, Joseph Kim, and Yulan He. 2022. PHEE: A dataset for pharmacovigilance event extraction from text. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5571-5587, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Sun et al. (2024) Zhaoyue Sun, Gabriele Pergola, Byron Wallace, and Yulan He. 2024. Leveraging ChatGPT in pharmacovigilance event extraction: An empirical study. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 344-357, St. Julian's, Malta. Association for Computational Linguistics.
* Tan et al. (2021) Xingwei Tan, Gabriele Pergola, and Yulan He. 2021. Extracting event temporal relations via hyperbolic geometry. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8065-8077, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Tan et al. (2023) Xingwei Tan, Gabriele Pergola, and Yulan He. 2023. Event temporal relation extraction with Bayesian 

translational model. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, pp. 1125-1138. External Links: Link Cited by: SS4.
* X. Tan, Y. Zhou, G. Pergola, and Y. He (2024)Set-aligning framework for autoregressive event temporal graph generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico, pp. 3872-3892. External Links: Link Cited by: SS4.
* N. UzZaman, H. Llorens, L. Derczynski, J. Allen, M. Verhagen, and J. Pustejovsky (2013)SemEval-2013 task 1: TempEval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), Atlanta, Georgia, USA, pp. 1-9. External Links: ISBN 978-0-647-11-19-1, Document, Link Cited by: SS4.
* H. Wang, M. Chen, H. Zhang, and D. Roth (2020)Joint constrained learning for event-event relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 696-706. External Links: Link Cited by: SS4.
* X. Wang, Y. Chen, N. Ding, H. Peng, Z. Wang, Y. Lin, X. Han, L. Hou, J. Li, Z. Liu, P. Li, and J. Zhou (2022)MAVENERE: a unified large-scale dataset for event coreference, temporal, causal, and subevent relation extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 926-941. External Links: Link Cited by: SS4.
* X. Wang, S. Li, and H. Ji (2023)Code4Struct: code generation for few-shot event structure prediction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 3640-3663. External Links: Link Cited by: SS4.
* L. Zhang, H. Xu, Y. Yang, S. Zhou, W. You, M. Arora, and C. Callison-Burch (2023)Causal reasoning of entities and events in procedural texts. In Findings of the Association for Computational Linguistics: EACL 2023, Dubrovnik, Croatia, pp. 415-431. External Links: Link Cited by: SS4.
* X. Zhang, M. Chen, and J. May (2021)Salience-aware event chain modeling for narrative understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 1418-1428. External Links: Link Cited by: SS4.
* R. Zhao, Q. Zhu, H. Xu, J. Li, Y. Zhou, Y. He, and L. Gui (2024)Large language models fall short: understanding complex relationships in inductive narratives. arXiv preprint arXiv:2402.11051. Cited by: SS4.
* Y. Zhou, J. Li, Y. Xiang, H. Yan, L. Gui, and Y. He (2024)The mystery of in-context learning: A comprehensive survey on interpretation and analysis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, pp. 14365-14378. External Links: Link Cited by: SS4.

 

Additional Details of Dataset Construction

### Document Selection

We select news documents from the NYT corpus based on the descriptors available. With regards to the generation of salient even graphs, the most relevant documents tend to be centered around event narratives, so that they could be rich in event relations. Tan et al. (2024) investigated which descriptors are rich in event narrative using event frequency \(\times\) inverse-descriptor frequency. We chose the documents using the same descriptors as them (e.g., "_airlines and airplanes_", "_united states international relations_", "_civil war and guerrilla warfare_", "_track and field_", "_soccer_", etc.).

We applied additional filtering based on the number of words in the documents. Documents with more than \(8500\) words or less than \(100\) words are excluded. Based on our preliminary observations, the extremely long documents are not typically news articles (only takes \(0.02\%\) in the entire NYT). They tend to be collections of articles over longer time spans, making them not suitable as focus of this study. Additionally, very long articles may affect the performance of open-source LLMs only due to limitations in the context length rather than their reasoning abilities. On the other hand, articles that are too short are less likely to contain complex event relation graphs, so we also exclude them. The final average word count of the selected \(10347\) documents is \(780\).

### Frequent words and descriptors in the annotated dataset

Table 8 reports the most frequent trigger words among the human-identified salient events and LLM-generated salient events after filtering out the light words (words that have no semantic meaning). We could see that "_win_", "_play_", and "_defeat_" are prominent triggers due to the sports topics within the dataset. These articles usually mention multiple events with these triggers. Triggers like "_express_", "_include_", and "_make_" are instead common across different scenarios.

Table 9 shows the most frequent descriptors in the human-annotated test set and the distant train set. These are the typical event-rich topics and are full of narratives.

### Disclaimers of Risks

Consider that a large portion of the new articles in the New York Times corpus are about violent incidences, such as terrorist attacks and war. To prevent inflicting harm to traumatised victims, we show the information clearly in the recruitment description on the Prolific platform (Figure 3).

### Guidelines and User Interface

A well-designed user interface is essential for collecting high-quality data efficiently. We fully cooperate with participants to improve the user interface iteratively based on their feedback.

In the salient event identification stage (Figure 4), we show the title, abstract, and content of the article on the right side. We show candidate events, which are extracted through CAEVO and Mixtral, on the left sidebar. The shown CAEVO events are the top events ranked based on the saliency feature score. The participants can choose the candidate events which they think are accurate and salient. The guideline also informs them that if multiple options refer to the same event, they can only choose the most accurate and informative one. If a salient event is not present among the candidates, they could write it in the text input box and add it.

In the event relation identification stage (Figure 5), they could choose a source event, a relation type, and a target event to add a relation triplet. The source event and the target event need to be chosen from the salient event list from the first stage. We automatically detect and prevent any new event that will lead to duplication and contradiction.

Figure 3: The recruitment descriptions.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline  & \multicolumn{2}{c}{Test} & \multicolumn{2}{c}{Train} \\ \cline{2-6} Rank & Word & Count & Word & Count \\ \hline
1 & win & \(41\) & win & \(2,964\) \\
2 & express & \(15\) & make & \(1,591\) \\
3 & play & \(14\) & face & \(1,564\) \\
4 & make & \(13\) & express & \(1,411\) \\
5 & defeat & \(12\) & include & \(1,307\) \\ \hline \hline \end{tabular}
\end{table}
Table 8: The top \(5\) most frequent trigger words in the human-annotated test set and the distant train set.

 

* [10] M. C. Bader, A. K. Goyal, and R. J. S. Sutton, "Theoretical evaluation of the human-level system: a review," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pp. 2359-2368, 2007.
* [11] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in _Advances in neural information processing systems_, vol. 42, no. 1, pp. 1329-1339, 2014.
* [12] L. Chen, Y. Li, X. Zhang, Z. Wang, and H. Yu, "Learning deep convolutional networks with large scale data," _IEEE Transactions on Neural Networks_**5**(1), pp. 1-10, Jan 2015.
* [13] T. Chawla, W. Dong, and F. He, "A unified framework for image recognition," in _Computer Vision and Pattern Recognition_, vol. 10, no. 1, pp. 1-1, Jan 2015.
* [14] N. DeVito, V. I. Goodfellow, E. Farhadi, and A. Courville, "Deep residual learning for image recognition," _arXiv preprint arXiv:1512.03092_ (2015).

[MISSING_PAGE_POST]

 

The participants can also deselect the added event if they change their minds. The participants were asked to finish the first stage first, and then annotate the second stage based on their own annotations in the first stage.

In the following are reported the screenshots of the guideline pages (Figure 6).

### More details about the annotation

We started the annotation process by releasing several trial rounds, during which we chose participants based on their dedication and understanding of the terminologies. It required considerable communication efforts to ensure they had an accurate understanding of the task definition.

During training, we found a common mistake among the annotators was that they tended to overestimate the _is_subevent_of_ relation. They often confused it with the _caused_by_ relation or temporal inclusion.

We advised them that _is_subevent_of_ pertains to two events on different granularity levels but referring to the same subject. To distinguish _is_subevent_of_ from temporal overlap, they could check whether the actor in the subevent is the same as or a part of the actor or object in the parent event. For example, if a parent event is "_a team did something_" the subevent can be "_a member of the team did something_".

### Information about the Annotators

The annotators were paid at the rate of 8E/h. We screened native English speakers from all over the world to ensure they could read English articles fluently. We also selected participants based on their previous submissions and approval rates to ensure they were familiar with the platform and were high-quality annotators.

Two of the final annotators are identified as male, and they both come from the UK. One of the final annotators is identified as female, and she comes from Canada. They all identified as white.

### Dataset Licensing

The original NYT corpus is available for noncommercial research license. One of our authors has obtained the license. Based on the license, we could not include the original text in our dataset. Thus, we will only release the generated/annotated graphs. Our dataset will also be in noncommercial research license.

## Appendix B Saliency Features

Inspired by Choubey et al. (2018), we calculate the saliency features to show how our proposed method differs from previous methods in terms of event saliency. Unlike conventional computation methods, these saliency features are calculated on the sentence level to be comparable across documents of various lengths. These saliency features are:

**Event frequency**: A salient event tends to appear frequently in the document. Let \(D=\{s_{0},s_{1},...,s_{n-1},s_{n}\}\) be the document and the list of sentences in the document. Let \(e\) be the event. Let \(M(e)=\{s_{i},s_{j},...,s_{k}\},0\leqslant i<j<k\leqslant n\) be the list of sentences which mention the event \(e\). The event frequency is calculated as:

\[frequency(D,e)=\frac{|M(e)|}{n+1}. \tag{1}\]

**First appearance**: News writers usually mention the salient event as early as possible to attract readers' attention. The first appearance of the event \(e\) is computed as:

\[first\_appearance(D,e)=\frac{i}{n}. \tag{2}\]

**Stretch size**: Salient events tend to be mentioned all across the document. The stretch size of event \(e\) is calculated as:

\[stretch\_size(D,e)=\frac{k-i}{n}. \tag{3}\]

To detect which sentences mention the event \(e\), we first lemmatise the words in the document and

\begin{table}
\begin{tabular}{l c l r r} \hline \hline  & \multicolumn{2}{c}{Test} & \multicolumn{2}{c}{Train} \\ \cline{2-5} Rank & Descriptor & Count & Descriptor & Count \\ \hline
1 & U.S. International Relations & \(27\) & Terrorism & \(2,885\) \\
2 & Terrorism & \(21\) & U.S. International Relations & \(2,574\) \\
3 & Bombs and Explosives & \(17\) & Bombs and Explosives & \(1,727\) \\
4 & U.S. Armament and Defense & \(15\) & U.S. Armament and Defense & \(1,717\) \\
5 & Politics and Government & \(15\) & Politics and Government & \(1,649\) \\ \hline \hline \end{tabular}
\end{table}
Table 9: The top \(5\) most frequent descriptors in the human-annotated test set and the distant train set.

 

the given event. Then, detect whether there is a matched substring the same as the given event in each sentence. However, the abstractive nature of LLM-based salient event generation makes exact matching not viable. To detect the event mention of LLM-generated events, we formulate a series of prompts. We first ask: "_Which sentence in the document below mentions the event [event]"? Please enclose that sentence in () and show it. Document: """"[doc_content]"""_. Then, we employ iterative refinement in case the LLM misses any other sentences: "_Is there any other sentence in the document directly mentioning the event [event]"? Please enclose that sentence in () and show it."_ Lastly, we collect the sentences from the responses.

We run the methods on the human-annotated dataset (\(100\) documents). We compute the saliency features of the events in each document and take the average across the events. Lastly, all the values are averaged across all the documents.

## Appendix C Comparison of Formats

In our preliminary experiments, we tested asking LLMs to generate event relations in JSON format (Table 10). When generating JSON response, we found LLMs often make format errors (e.g., using double quotation marks in the wrong place) which makes the responses difficult to parse, resulting in lower HGS. We also found the JSON format is not as flexible as the Python format which can integrate more information (e.g., explanations and definitions) as comments.

## Appendix D Prompting Details

Due to limited resources, the sampling of the distant supervision dataset was conducted on multiple machines with different specifications, including one with \(6\times\) RTX \(3090\), one with an A\(100\), and one with \(2\times\) A\(40\). The total time cost for prompting Llama3 to construct the training data (\(10,231\) documents) is about \(2,200\) hours (the wall-clock time of all the machines combined).

Table 13 shows an example of the code prompt for hierarchical graph generation and the response from Llama3. Table 14 shows an example of the hallucination prompt and the response.

Algorithm 1 is the pseudo-code of the entire salient event graph generation process.

In the summarization prompt, we use a temperature of \(0.8\) and a top_p of \(0.9\). For the salient event generation prompt, we use a temperature of \(0.5\) and a top_p of \(0.9\). The relation graph generation prompt also uses a temperature of \(0.5\) and a top_p of \(0.9\). The hallucination grader prompt uses a temperature of \(0\).

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & Hier & Temp & Causal \\ \hline JSON & \(0.310\) & \(0.277\) & \(0.251\) \\ Python & **0.315** & **0.283** & **0.272** \\ \hline \hline \end{tabular}
\end{table}
Table 10: The comparison between Python format prompt and JSON format prompt in Hungarian Graph Similarity.

 



\begin{table}
\begin{tabular}{|p{142.3pt}|p{170.7pt}|} \hline Input to the LLMs & Response from the LLMs \\ \hline Write a summary of the document below using one paragraph. Document: ”” Residents of British Columbia will receive a big cut in their income taxes on New Year’s Day, their second in six months. But far from winning applause, the province’s governing Liberal Party is experiencing a drop in public support. Tax cuts were a major promise in the campaign that brought the Liberals to office in the province in a landslide election win last May. The Liberals, led by Gordon Campbell, a former secondary school teacher, real-estate executive and three-term mayor of Vancouver, won all but two of the 79 seats in the western province’s legislative assembly. They defeated the left-leaning New Democrats, whose 10 years in office were marked by a growing public role in the economy and numerous tales of economic mismanagement. With the Jan. 1 reductions, personal income tax rates for provincial taxes – which constitute a much larger share of overall taxes for Canadians than state taxes do for Americans – will have been chopped by an average of 25 percent, turning British Columbia from one of Canada’s most heavily taxed provinces to one with some of the lowest tax rates. ... .... ........ Summary: \\ \hline \end{tabular}
\end{table}
Table 11: An example of the hallucination prompt input and the response from Llama\(3\).





\begin{table}
\begin{tabular}{|p{142.3pt}|p{142.3pt}|} \hline Input to the LLMs & Response from the LLMs \\ \hline Could you please finish the following code? import networkx as nx document = {document} event_list = {event_list_strr} & Here is the completed code: import networkxx as nx hierarchical_graph = nx.DiGraph() \\
This is a graph representing the hierarchical relation between the events in the document & for event in event_list: hierarchical_graph.add_node(event) \\
Each edge in the graph represents a subevent relation between the head and tail nodes which are events & \# Add hierarchical relations as edges to the graph hierarchical_graph.add_edge(“the government responded by 28%”, “liberals cut spending”) \\
An edge means the head event is a subevent of the tail event. They are closely related but on different granularity levels. hierarchical_graph = nx.DiGraph() \# This is a directed acyclic graph. There should not be any cycle in the graph. \\
# Add events as nodes for event in event_list: hierarchical_graph.add_node(event) \\
Add hierarchical relations as edges to the graph using .add_edge() function. The function takes two strings as inputs. Each string represents an event. \\
There should not be any cycle in the graph. Explain the reason for each added edge as a comment after each function call \\ \hline \end{tabular}
\end{table}
Table 13: An example of the relation graph generation prompt input and the response from Llama3.





\begin{table}
\begin{tabular}{|p{142.3pt}|p{142.3pt}|} \hline Input to the LLMs & Response from the LLMs \\ \hline You are a grader assessing whether an answer is grounded in / supported by a set of facts. Give a binary score ‘yes’ or ‘no’ score to indicate whether the answer is grounded in or supported by a set of facts. Then, provide a short explanation. Here are the facts: \(\{\)document\(\}\) & Score: Yes Explanation: The answer is grounded in the facts because it accurately identifies a specific action taken by the government (reducing the provincial civil service by 28%) as a subevent of the broader event of cutting spending, which is mentioned in the text. \\ \end{tabular}
\end{table}
Table 14: An example of the hallucination prompt input and the response from Llama\(3\).



```
1:Input: Document \(d\), Max Refinement Round \(k\)
2:Output: An Event Relation Graph \(g\)
3:\(summary\leftarrow\) Summary_Generation\((d)\)
4:\(salient\_events\leftarrow\) Event_Generation\((summary)\)
5:\(hierarchical\_graph\leftarrow\) null
6:\(current\_round\gets 0\)
7:while\(current\_round<n\)do
8:\(hierarchical\_graph\leftarrow\) Hierarchical_Graph_Generation\((d,salient\_events,\\ hierarchical\_graph)\)
9:\(hierarchical\_edges\leftarrow\) Get_Edges\((hierarchical\_graph)\)
10:for\(edge_{i}\) in \(hierarchical\_edges\)do
11:\(remove\_edge\leftarrow\) Hallucination_Grader\((d,edge_{i})\)
12:if\(remove\_edge\)then
13:\(hierarchical\_graph\leftarrow\) Remove_edge\((hierarchical\_graph,edge_{i})\)
14:endif
15:endfor
16:\(current\_round\gets current\_round+1\)
17:endwhile
18:\(temporal\_graph\leftarrow\)null
19:\(current\_round\gets 0\)
20:while\(current\_round<n\)do
21:\(temporal\_graph\leftarrow\) Temporal_Graph_Generation\((d,salient\_events,temporal\_graph,\\ hierarchical\_graph)\)
22:\(temporal\_edges\leftarrow\)Get_Edges\((temporal\_graph)\)
23:for\(edge_{i}\) in \(temporal\_edges\)do
24:\(remove\_edge\leftarrow\)Hallucination_Grader\((d,edge_{i})\)
25:if\(remove\_edge\)then
26:\(temporal\_graph\leftarrow\)Remove_edge\((temporal\_graph,edge_{i})\)
27:endif
28:endfor
29:\(current\_round\gets current\_round+1\)
30:endwhile
31:\(causal\_graph\leftarrow\)null
32:\(current\_round\gets 0\)
33:while\(current\_round<n\)do
34:\(casal\_graph\leftarrow\)Causal_Graph_Generation\((d,salient\_events,causal\_graph,\\ temporal\_graph,hierarchical\_graph)\)
35:\(causal\_edges\leftarrow\)Get_Edges\((causal\_graph)\)
36:for\(edge_{i}\) in \(causal\_edges\)do
37:\(remove\_edge\leftarrow\)Hallucination_Grader\((d,edge_{i})\)
38:if\(remove\_edge\)then
39:\(causal\_graph\leftarrow\)Remove_edge\((causal\_graph,edge_{i})\)
40:endif
41:endfor
42:\(current\_round\gets current\_round+1\)
43:endwhile
44:\(g\leftarrow\{hierarchical\_graph,temporal\_graph,causal\_ graph\}\)
```

**Algorithm 1** CALLMSAE: CASceding Large Language Models for SAlient Event graph generation 

2. If you think there is an event that isn't listed, you can add it by entering the event in the text box.The **event should at least contain a subject and a trigger**.
3. If you think a ticked event makes no sense, untick it. when two options are referring to the same event, untick the one you think is less informative.

Every modification will be saved automatically.

## Definition of Event

An event is anything that happens as described in the article. We represent the events in a structured format: **actor; trigger; target**. The actor of the event is usually the subject of a sentence. The trigger can be seen as the predicate of a sentence. The target is usually the object in the sentence which is optional.

### Example

In example 1, _New York is one of the four candidate cities competing to be presented to the IOC_ should not be chosen because the predicate isn's something that can be considered as an event. An event is essentially a change of state. Predicates like "is" is only describing one state. On the other hand, _New York is competing; with three cities to be presented to the IOC_ should be chosen because the predicate can indicate an event.

\begin{tabular}{l l}  & Officials Will Tour A Changed New \\  & York \\  & Include the whole world. This means that the person has been asked to know whether they have different positions or locations. Therefore \\  & Include the whole world. This means that the person has been asked to know whether they have different positions or location. Therefore \\  & Include the whole world.

 

In example 3, _MetroStars; won; Major League_ isn't selected because it is less accurate and informative than the second option. They are referring to the same event and we don't want duplication. The fourth and fifth options are not selected due to they are more about a description of state than a change of state.

_Game; was marked; by physical play_ isn't selected because there is no direct reference in the article that the game is marked by physical play.

Below are more annotated examples.

Figure 6: Annotation guidelines of salient event identification shown to the annotators.

 


Figure 7: Annotation guidelines of relation identification shown to the annotators.