Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models (2024)

Abstract: Cross-document event coreference resolution (CDECR) involves clustering event mentions across multiple documents that refer to the same real-world events. Existing approaches utilize fine-tuning of small language models (SLMs) like BERT to address the compatibility among the contexts of event mentions. However, due to the complexity and diversity of contexts, these models are prone to learning simple co-occurrences. Recently, large language models (LLMs) like ChatGPT have demonstrated impressive contextual understanding, yet they encounter challenges in adapting to specific information extraction (IE) tasks. In this paper, we propose a collaborative approach for CDECR, leveraging the capabilities of both a universally capable LLM and a task-specific SLM. The collaborative strategy begins with the LLM accurately and comprehensively summarizing events through prompting. Then, the SLM refines its learning of event representations based on these insights during fine-tuning. Experimental results demonstrate that our approach surpasses the performance of both the large and small language models individually, forming a complementary advantage. Across various datasets, our approach achieves stateof-the-art performance, underscoring its effectiveness in diverse scenarios.

heading: 1 Introduction
text: Event coreference resolution is a useful task in information extraction (Lu and Ng, 2018). This is crucial for achieving a more comprehensive understanding of intricate narratives and facilitating knowledge extraction from diverse textual sources. The coreference of events typically relies on a thorough understanding of document-level context (Minh Tran et al., 2021; Kriman and Ji, 2021;
*Corresponding author
Xu et al., 2022). Cross-document event coreference (Lee et al., 2012), involving the comparison of event mentions from different documents, presents additional challenges. On one hand, distinct events in different documents may be portrayed in a very similar manner, especially for events of the same type (challenge 1). On the other hand, the portrayal of the identical event may vary significantly across different documents (challenge 2). The model is required to grasp comparable coreference evidence from varied contexts and make judgments based on it (refer to the examples in Table 4 and 15 for better illustration).
Existing work (Held et al., 2021; Yu et al., 2022) attempts to address CDECR based on fine-tuning
2985
small language models (SLMs)1, as shown in Figure 1a. However, the complexity and diversity of the context make it prone to learning pseudofeatures by capturing simple co-occurrences rather than genuinely coreference-related terms, including contextual words, entity mentions and other event mentions associated with the given event mention. Supporting this observation, CDECR remains a significant challenge for SLMs, as evidenced by achieving only around 70% CoNLL F1 score on the FCC dataset (Bugert et al., 2021).
Recent advancements in LLMs have significantly advanced the field of NLP, enabling the effective resolution for tasks like machine translation (Jiao et al., 2023) and text summarization (Bang et al., 2023), with just a few demonstrations. However, when it comes to information extraction (IE) tasks, LLMs encounter challenges in task-specific adaptation. Specifically, LLMs struggle to achieve the same level of accuracy as supervised SLMs because a small number of demonstrations cannot comprehensively cover the complex annotation guidelines of these tasks (Han et al., 2023; Li et al., 2023a). Moreover, the inherent nature of the CDECR task, which involves processing multiple documents, imposes enhanced demands on understanding the lengthy context in the demonstrations.2 Instead of directly predicting CDECR structures, the relative strength of LLMs can enhance the generic understanding of individual documents, particularly the inherent meaning of diverse event mentions, which is complementary to the advantage of SLMs in understanding structures with thorough fine-tuning.
To leverage the relative strengths of LLMs and SLMs, we propose a collaborative approach, as shown in Figure 1b. First, we use the LLM to summarize event mentions from different documents. Then we feed these insights to the SLM to enhance its understanding of event mentions, enabling it to make coreference judgments based on more focused contexts. For the LLM summarization, we design a two-step workflow with separate generic prompts to guide its comprehension of the context of each mention, instead of task-specific in-context learning or fine-tuning. For the SLM, we employ joint representation learning to integrate the origi-
1In this work, SLM refers to pre-trained language models with relatively fewer parameters, which are more cost-effective for fine-tuning on specific tasks, such as BERT and RoBERTa.
2On average, each instance of demonstration in the ECB+ dataset contains close to 15k tokens.
nal document and the generated summary. We conduct experiments on three datasets of CDECR, and the results demonstrate that our collaborative approach, as compared to methods solely relying on the LLM or SLM, exhibits significant improvements. Across all three datasets (ECB+, GVC, and FCC), our approach achieves state-ofthe-art results, with increases of 1%, 2.7%, and 7% in CoNLL F1, respectively (averaged over three independent experiments for each dataset). Through analysis, it is demonstrated that our approach more thoroughly addresses the aforementioned challenge 1 of similarly portrayed contexts, making a substantial contribution to performance improvement.
To the best of our knowledge, we are the first to propose a collaborative approach that leverages the universal capabilities of LLMs to address CDECR, achieving superior performance compared to the state-of-the-art baseline.3

heading: 2 Related Work
text: CDECR Early work addresses CDECR by employing machine learning methods with manually designed features (Bejan and Harabagiu, 2010; Yang et al., 2015; Vossen and Cybulska, 2018; Bugert et al., 2021). Recent neural approaches have utilized SLMs to encode event mentions, obtaining their embeddings for supervised coreference resolution. Initial efforts involve encoding at sentence level and fusing the embeddings of mentions and the incomplete arguments extracted by SRL as the representation of mentions (Barhom et al., 2019; Zeng et al., 2020; Allaway et al., 2021; Yu et al., 2022). Subsequent work incorporates extensive context directly into encoding, leading to noticeable improvements (Caciularu et al., 2021; Cattan et al., 2021a; Held et al., 2021; Hsu and Horwood, 2022; Ahmed et al., 2023). More recently, Chen et al. (2023) and Ravi et al. (2023) establish connections between event mentions using a discourse rhetorical structure constructor and a GPT-3 model fine-tuned with additional data for temporal reasoning, respectively. In comparison to existing work, we are the first to establish comprehensive connections between event mentions and their corresponding contextual elements, including contextual words, entity mentions, and other event mentions, by leveraging the intrinsic knowledge and out-ofthe-box context comprehension ability of LLMs.
3The code and data are publicly available at https://github.com/taolusi/SECURE.
LLM for IE Several recent studies (Ma et al., 2023; Li et al., 2023a; Han et al., 2023; Yuan et al., 2023; Gao et al., 2023; Wei et al., 2023; Xie et al., 2023; Li and Zhang, 2023; Xu et al., 2023; Wadhwa et al., 2023; Qi et al., 2023; Ling et al., 2023) have evaluated the performance of LLMs, predominantly ChatGPT, using in-context learning methods on various IE tasks. These investigations universally demonstrate that LLMs exhibit commendable performance in zero-shot and few-shot settings, yet there remains a substantial gap when compared to state-of-the-art supervised SLMs, with the performance gap widening for more complex tasks.
In addition, there are also methods directly using labeled data from IE tasks to fine-tune LLMs(Lu et al., 2022; Zhou et al., 2023; Wang et al., 2023; Sainz et al., 2023). In general, training on these larger-scale models, such as Code-LLaMA and Flan-T5, has yielded results comparable to supervised baselines and demonstrated improvements in zero-shot settings. However, when the training of LLMs does not result in significant performance gains, the training cost, compared to training SLMs, becomes less cost-effective.
Integration of LLM and SLM The integration of LLM and SLM is an emerging approach, with only a few explorations in complex IE tasks. Ma et al. (2023) prompts the LLM to rerank a few difficult samples filtered by the supervised SLM and achieves improvements on various few-shot IE tasks. Their method is based on the observation that LLMs excel only at a small number of hard samples. Wan et al. (2023) first utilizes the LLM to generate reasoning logic for demonstrations retrieved by a fine-tuned SLM, then feeds this combined input back to the LLM for relation extraction, surpassing supervised baselines on some datasets. An inherent challenge lies in finding reasonable demonstrations for NULL-type triples, leading to poor performance on complex tasks such as ACE05. Additionally, inducing complex reasoning logic for each of the k-demonstrations is costly, leading them to sample only a subset of ACE05 and TACRED test sets. Xu et al. (2023) and Li et al. (2023b) leverage LLMs for data enhancement in sentence and document-level relation extraction tasks, respectively. The gap between triples recognized by LLMs and those annotated under manually crafted rules introduces potential shifts in data distribution, making the effectiveness in applications unclear.
Overall, the aforementioned integration methods have exhaustively attempted to adapt LLMs to spe-
cific tasks by prompting them to establish accurate connections with artificially defined labels. In contrast, our approach only requires LLMs to perform generic tasks, leveraging their inherent capabilities to assist specific tasks.
Concurrently, akin to our approach, Ding et al. (2024) and Nath et al. (2024) also leverage LLM generation to assist SLM on CDECR. While Ding et al. (2024) prompts LLM with task instructions to generate multiple counterfactual instances for original mention pairs, Nath et al. (2024) employs similar task prompts to guide LLM in generating coreference reasoning processes for mention pairs. Unlike our approach, which involves a general task of having LLM process each mention individually, their methods require LLM to directly handle the relationships between mentions given coreference labels. In terms of efficiency, their methods are less effective than ours as they need to handle combinations of mention pairs, resulting in a quadratic increase in the number of processing entries.

heading: 3 Method
text: We adopt the state-of-the-art method proposed by Held et al. (2021) as our baseline (Section 3.1), then summarize events using generic prompts for LLM (Section 3.2), and finally integrate the representations of events from both the summary and the original context into baseline system (Section 3.3).

heading: 3.1 Task and Baseline
text: The goal of the CDECR task is to group coreferential event mentions across multiple documents into clusters. We formalize the task as follows:
Input: A corpus comprising multiple documents denoted by D, where D = {D1, D2, ..., D|D|}, with |D| representing the number of documents in the dataset. Let M represent all event mentions in the corpus, such that M = {m11,m12, . . . ,mij , . . . ,m|D|,k}, where k denotes the number of event mentions in each document, and mij signifies the j-th event mention in document Di.
Output: A set of clusters, denoted as C, where C = {C1, C2, ..., Cn}. For each cluster Ck, Ek represents all the event mentions contained in the cluster Ck, such that Ek = {ek1, ek2, . . . , ekj , . . . , ekM}, where M is the total number of event mentions in cluster Ck, and ekj is the jth event mention in the cluster Ck.
Our baseline consists of two key modules for
coreference clustering: candidate retrieval and pairwise classification. Both of these modules primarily involve using a RoBERTa (Liu et al., 2019) encoder to encode the context and obtain vector representations of event mentions, which can be seamlessly replaced by our collaborative approach. We formalize the encoding process as follows:
For each event mention mij , its vector representation can be obtained as:
hij = fenc(mij , Di) (1)
Here fenc is an encoder network used for encoding Di and concatenating the representations of the boundary tokens of mij . The resulting representation hij is fed into the subsequent neural network.

heading: 3.2 LLM Summarization
text: Summarizing events for CDECR poses a nontrivial challenge. Existing summarization methods are typically designed to provide a general overview of documents, making it difficult to extract information specific to certain types of events. This not only provides limited assistance for coreference but may also lead to the omission of crucial details. Furthermore, designing a summary template for each type of event is not only impractical in real-world applications4 but also introduces bias, potentially causing LLMs to misinterpret or hallucinate information due to the inherent incompleteness of event information in documents.
To address various types of events and gather specific details from complex contexts, we design a two-step workflow to prompt the LLM, as shown in Table 1. The first step is responsible for extracting tailored information for different types of
4Based on the rough statistics in our experiment, the ECB+ dataset contains over 400 event types.
events in the context of the document. The second step aims to expand the details of the entities mentioned in the output of the first step, as entity details are often scattered throughout the document. In each step, we employ a straightforward prompt to accomplish a primary task. Our prompts adhere to the basic principle of faithfulness, avoiding additional interpretations to prevent semantic shifts. Compared to a synthesized single-step workflow, our two-step workflow guarantees that each step remains focused on its main objective, thereby preventing interference between the two steps, as illustrated by the analysis in Section 4.4.
In the first step, we instruct the LLM agent to “elaborate” an event mention, rather than the conventional instruction of “summarize”. The term “elaborate” implies an explanatory behavior based on the concept of the mention words themselves, emphasizing the support of details from the document context. This suggests that LLMs can automatically select any relevant details from the context to support this explanation, including contextual words, entity mentions, and event mentions. This provides a standardized and feasible way to understand events, leveraging the LLM’s intrinsic knowledge and contextual understanding capabilities without imposing complex rules for the LLM to adhere to.
In the second step, we prompt the LLM agent to use coreference resolution to aggregate detailed information about entities, as entity coreference is a more standardized task compared with event and performing it within a document reduces complexity. Additionally, we require the LLM to perform temporal reasoning based on the publication date of the document, further reducing ambiguity in coreference evidence comparison.
In both steps, we specify the generation format to ensure the consistency between the mention spans in summary and original document. This not only reduces the generation difficulty of LLM but also facilitates SLM in establishing the connection between the two during joint representation learning.

heading: 3.3 Integration into Final SLM
text: The SLM takes the original document and the generated summary as inputs. Through a direct joint representation learning technique, the new mention vector representation can be seamlessly integrated into the baseline.
Specifically, for the mention mij , let Sij represents the generated summary, and m(s)ij signifies the mention within it. By concatenating the original document Di and the summary Sij , a new document D′i is formed. Let f ′ enc denotes the new encoder network. It first encodes the new document D′i, obtaining vector representations hij and h (s) ij for mij and m (s) ij respectively. These vectors are then concatenated to form the fused mention vector representation h′ij , which can seamlessly substitute hij in the baseline for subsequent operations. The joint representation learning process can be represented as:
h′ij = f ′ enc({eij , e(s)ij }, D′i)
= concat ( fenc({eij , e(s)ij }, D′i) )
= concat(hij , h (s) ij )
Here {eij , e(s)ij } denotes a set containing two elements, implying that vector representations for both eij and e (s) ij can be derived using the same process as for a single element. This integration method, which involves concatenating the original context and generated summary for joint representation learning, enables mutual learning of each other’s context in the same attention space, thereby enhancing the understanding of genuinely coreference-related terms.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Dataset We conduct experiments on three CDECR datasets: Event Coreference Bank Plus (ECB+) (Cybulska and Vossen, 2014), Gun Violence Corpus (GVC) (Vossen et al., 2018), and Football Coreference Corpus (FCC) (Bugert et al., 2021). The widely-used ECB+ dataset consists of
news articles from various topics, including earthquakes, murders, acquisitions, etc. Each topic includes two similar subtopics, such as “6.1 earthquake Indonesia 2009” and “6.1 earthquake Indonesia 2013”. This setup aligns with the challenge 1 mentioned in introduction, asking the model to distinguish similar events. Similarly, GVC and FCC, focusing on news incidents of gun violence and football tournaments, respectively, also have multiple subtopics under one overarching topic. More details can be found in Table 7 (Appendix A.1).
Evaluation Metrics Following previous work (Barhom et al., 2019; Held et al., 2021), we conduct a comprehensive comparison using metrics including MUC, B3, CEAFe, CoNLL, and LEA. The CoNLL F1 is a composite metric representing the average of the first three. B3 is chosen for analysis, following Held et al. (2021).
Hyper Parameters For LLM summarization, we use the “GPT-4-0613” model via OpenAI API, setting the sampling temperature t = 0 to reduce the impact of randomness. In the first step of the generation workflow, we introduce a pre-step of instructing the LLM to perform dependency parsing on the sentence containing the event mention. Based on the parsing results, the LLM then elaborates on the mention. For SLM integration, we employ the pre-trained RoBERTaLARGE model (Liu et al., 2019) to embed event mentions, following our baseline (Held et al., 2021). For all three datasets, we apply a consistent set of hyper-parameters for finetuning, as detailed in Table 8 (Appendix A.2). In all experiments, be it primary results or analyses, we ensure reliability by conducting three independent experiments and averaging the outcomes.

heading: Directly Using LLM to Predict the Structure
text: of CDECR We test the performance of GPT-4 using different in-context learning methods, including few-shot and zero-shot learning, with varied contexts, such as full context and mention-inclusive sentences5. The inherent nature of the CDECR task poses a challenge for LLMs in dealing with inputs (comprising hundreds of documents as context) and outputs (consisting of coreference structures formed by thousands of event mentions) that exceed manageable lengths. To tackle this, we first opt for the “GPT-4-Turbo-Preview” model from
5“Mention-inclusive sentences” indicates that we retain only those sentences containing mentions, reducing the complexity of contextual understanding.
OpenAI, which supports input up to 120k tokens and output to 4096 tokens. Second, we partition the data by topic and process it sequentially. Each time, all documents within a single topic are used as input, and the outputs from all topics are simply merged for testing. Note that this procedure only applies to the multi-topic ECB+ dataset, as there are no cross-topic links. For the single-topic GVC and FCC datasets, inputs and outputs exceeding the length limit are directly truncated. More implementation details, including prompt design as well as the selection and parameter settings of the GPT-4 models, can be found in Appendix A.3.

heading: 4.2 Results
text: The main results are presented in Table 2. Our method achieves new state-of-the-art results on all
6For fairness, results obtained under their custom oracle setting, which utilizes gold coreference information from the dev and test sets, are not included in the comparison.
three datasets, outperforming both the previously reported best results and the improved results obtained by our reproduced baseline.
ECB+ On this widely studied dataset, our method demonstrates improvements of 1.5% in CoNLL F1, compared to our baseline. In comparison to Held et al. (2021), upon which our baseline is built, our method also exhibits a 1% increase in CoNLL F1.7 This improvement stands out notably in recent research, accompanied by significance testing to demonstrate its robustness. Compared to Chen et al. (2023), who also employs RoBERTaLARGE for encoding while proposing a different method to leverage broader contexts, we also achieve a 1% improvement, showcasing the effectiveness of our method in utilizing context. More experiments and discussions under additional evaluation principles, including without singletons and
7Our baseline is slight lower in CoNLL F1 than reported by Held et al. (2021), potentially be attributed to randomness.
at the topic level, can be found in the Appendix B.2. GPT-4 utilizing few-shot learning significantly lags behind our method, with nearly a 10% gap in CoNLL F1, indicating that GPT-4 still faces substantial adaptability challenges in directly predicting cross-document event coreference structures. This also demonstrates the effectiveness of our method in leveraging the inherent general capabilities of LLMs. Further analysis of GPT-4’s performance can be found in Section 4.5. We also compare the efficiency of LLM utilization (in terms of number of API calls and token consumption) between summarization and structure prediction, as detailed in Appendix B.1.
GVC & FCC Our method demonstrates improvements of 2.7% and 7.0% in CoNLL F1 on the GVC and FCC datasets, respectively, compared to our baseline. The significant improvement on the challenging FCC dataset further underscores the effectiveness of our collaborative approach in leveraging LLM. Additionally, our baseline also shows improvements of 1.0% and 7.3% compared to Held et al. (2021), highlighting our comprehensive exploration on these two less-studied datasets.
GPT-4 exhibits abnormal performance on the GVC and FCC datasets, primarily due to truncation issues stemming from its length constraints, as mentioned in Section 4.1. This is more pronounced on FCC, where longer multi-document contexts are encountered in the test set compared to GVC (4274 vs 1360 sentences). Further elaboration on the truncation problem can be found in Appendix B.1.

heading: 4.3 The Impact of LLM Summarization
text: Error Analysis To gain a deeper understanding of the improvements achieved through LLM summarization, we perform a quantitative analysis on the false links within the clusters (see Table 3).
Similar to Yu et al. (2022), we categorize link errors into two types: false positive (FP) and false negative (FN). FP links (incorrect links) occur when two non-coreferential mentions are clustered together, while FN links (missing links) occur when two coreferential mentions are not clustered together. Additionally, we further categorize FP links into two sub-types based on whether two mentions share the same event type.8 FPA (false positives caused by arguments) indicates that two mentions of the same type differ in argument information. FPT (false positives caused by types) implies that
8Details are provided in Appendix A.4.
two mentions actually belong to different event types, eliminating the need to consider arguments.
FPA Our method demonstrates the most substantial reduction in FPA errors across all three datasets, making the greatest contribution to the overall improvement. The reduction is approximately 30% for both ECB+ and GVC, and nearly 90% for FCC. The significant reduction on FCC is primary attributed to its nature, comprising multiple consecutive events from a large tournament, resulting in more pronounced contextual similarities. This underscores the effectiveness of our method in distinguishing events with similar contextual narratives (aligning with the challenge 1 from introduction). In Table 4, we present instances illustrating two highly similar earthquakes. The original context includes details about the earthquake occurrence, earthquake casualties, media coverage, and historical events. Our generated summaries primarily focus on the core details of the earthquakes, such as date and specific location, thus facilitating their differentiation. It can be observed that our LLM summarization is capable of identifying specific information for particular events and aggregating sufficient details from the entire context.
FPT Compared to FPA, there are significantly fewer FPT errors, only appearing in the ECB+ dataset. The few occurrences on the single-topic GVC and FCC datasets can likely be disregarded, possibly due to random factors. This is because in the multi-topic ECB+ dataset, there may be topics in the test set that were not encountered in the training set, leading to unseen event types. By reducing half of the FPT errors on the ECB+ dataset, it signifies that our summarization also assists in distinguishing unseen event types.
FN Our method shows less improvement in reducing FN errors compared to FP. The challenges arise from two primary factors. Firstly, mentions of the same event can vary greatly in expression styles. Secondly, some event mentions naturally lack sufficient details as the authors assume that
readers already possess necessary background information. We illustrate these issues with instances in Appendix B.3. For these cases (aligning with the challenge 2 in introduction), additional training data or external information retrieval may be necessary, as our faithful summarization based on the original context struggles to cope.
Overall, LLM summarization excels in consolidating information for specific events, facilitating the differentiation of similar yet non-coreferential events. Relatively, its effectiveness is limited for events with significant expression differences or those lacking essential details.

heading: LLM Summarization VS LLM Paraphrase To
text: validate that the performance improvement brought by our summarization is due to genuinely extract-
ing crucial information rather than introducing diversity in context, we conduct a comparison with paraphrases generated by the LLM. We prompt the LLM to paraphrase the context of mentions instead of the sentences they belong to, and use the same hyper-parameters for fine-tuning the SLM. As shown in Figure 2, compared to our baseline, LLM paraphrase exhibits a slight improvement on GVC and FCC, with a more pronounced decline on ECB+. More importantly, it significantly lags behind our summarization on all datasets. This comparison demonstrates the capability of our summarization method in selecting and aggregating relevant information. The prompt for LLM paraphrase is provided in Table 11 (Appendix A.5).

heading: 4.4 Ablation Study on the Two-step Workflow
text: We conduct an ablation study to specifically illustrate the effect of Step 1 and Step 2 in LLM summarization (Table 1). As shown in Figure 3, both steps contribute to the overall improvement, with the second step being more pronounced, especially on the FCC dataset. This is attributed to the longer documents in FCC, with nearly double the number of sentences in each document compared to the other two datasets. This demonstrates that the information provided in Step 1 establishes a solid foundation but is relatively localized. Step 2, involving global information expansion, plays a crucial role in overall enhancement.
To examine the benefits of decomposed execution, we further integrate the two-step workflow
into a single-step one through simple concatenation. Despite demonstrating comparable performance on GVC, the integrated workflow shows a noticeable lag, being 1.2% and 2% behind on ECB+ and FCC, respectively, in terms of B3 F1. This indicates that even with straightforward instructions, decomposing the multi-objective task into multiple independent steps is necessary, as evidenced by the recent LLM agent studies (Aksitov et al., 2023).
We perform error analysis and compare the lengths of the generated summaries to provide a detailed explanation of the impact of each step in the workflow and its decomposition. Further details can be found in Appendix B.4.

heading: 4.5 Analysis of GPT-4 Performance on CDECR
text: Table 5 presents the results of different in-context learning methods. It shows that GPT-4 achieves its optimal performance using few-shot learning with mention-inclusive sentences as context (FewMIS), yet it only achieves results comparable to the lemma matching-based method. Table 6 further compares different types of errors. Compared to our baseline and our method, Few-MIS has a slight reduction in FPT errors but a significant increase in FPA and FN errors. This indicates that GPT-4 has limited ability to differentiate between similar but non-coreferential events based on arguments, and struggles to link coreferential events with significant narrative differences based on semantics. The reduction in FPT errors may also be attributed to its limited comprehension ability, thereby avoiding errors caused by excessive interpretation of event types. This aligns with our observation that GPT-4 relies on a simplistic approach of clustering based on the literal meaning of mentions without con-
sidering their contexts. Additionally, the role of demonstrations appears limited to expanding the scope of matching for synonymous mentions.
From Table 5, it is also evident that incorporating the full context, compared to solely utilizing mention-inclusive sentences as context, results in a significant performance decline. This indicates that with more context, GPT-4 not only has limited ability to extract effective cues but also suffers from disrupted comprehension of the local context. Additionally, compared to few-shot learning, zero-shot learning demonstrates higher recall but significantly lower precision. This is because many completely unrelated mentions are clustered into a single cluster. This highlights the complexity of the CDECR task, indicating that GPT-4 struggles to perform basic clustering when relying solely on the task description.
We further investigate the impact of the number of in-context demonstrations on GPT-4’s performance. Details can be found in Appendix B.5.

heading: 5 Conclusion
text: We design generic tasks to leverage the potential of LLMs for CDECR, effectively bridging the gap between the general capabilities of LLMs and the complex annotation guidelines of specific IE tasks. Results show that by harnessing the inherent knowledge and comprehension abilities of LLMs to gain a deeper understanding of events, our collaborative approach can alleviate the challenge of SLMs for complex contextual understanding, ultimately enhancing performance.

heading: Limitations
text: The LLM we use for our collaborative approach is GPT-4-0613. Moving forward, we plan to assess the performance of additional LLMs, such as LLaMa (Touvron et al., 2023).
For CDECR, where internal information within the given document might be insufficient, there arises a need for external information retrieval. We are considering further leveraging the capabilities of LLMs to explore how to retrieve supplementary information from external corpora such as news articles. Our aim is to combine this additional information with the given documents to enhance performance.

heading: Ethics Statement
text: We adhere to the ACL Code of Ethics.

heading: Acknowledgement
text: We acknowledge the assistance of Pai Liu from University of Rochester in conducting experiments related to GPT-4. This work was supported by the STI 2030—Major Projects (Grant No. 2022ZD0208800) and the Alibaba Innovative Research (Grant No. 10313H022101).

heading: A.1 Dataset Statistics
text: As shown in Table 7.

heading: A.2 SLM Fine-tuning Hyper-parameters
text: As shown in Table 8.

heading: A.3 Prompt Design and Model Details of
text: GPT-4 Evaluation
The prompt is shown in Table 9. For ECB+, we introduce only one randomly selected topic from the training data as the demonstration, which comprises 39 documents, accounting for 6.6% of the entire training set. For GVC and FCC, we use the same number of randomly selected documents as in ECB+ for demonstration. It is important to note that we have conducted multiple rounds of prompt optimization to ensure GPT-4’s performance, including:
• Designing a reasonable format to tag each mention in the document with a unique mention_id to avoid literal confusion.
• Designing the output format as mention_id: cluster_index instead of cluster_index: [mention_id1, ..., mention_idn] to ensure that no mention is omitted.
• Avoiding declaring specific conditions for event coreference in the task description, including coreferential participants, locations, and times. It is demonstrated that these conditions do not improve performance; instead,
they lead GPT-4 to make coreference judgments based solely on individual conditions.
We set the model parameters, including seed and temperature, to 0 to minimize randomness. Additionally, we specify the output format to be in JSON for better post-processing.
During our experimentation, there were changes in the GPT-4 model provided by OpenAI. The introduction of “GPT-4-turbo-preview”, which can handle longer texts compared to “GPT-4-0613”, offers conditions for lenghy context composed by multiple documents (although it still faces length limitations in our actual testing). Consequently, our direct evaluation of the GPT-4 model was moved to “GPT-4-turbo-preview”.
Since most of our summary-based experiments were completed on “GPT-4-0613”, we did not migrate our experiments to “GPT-4-turbo-preview” due to cost considerations. Additionally, based on our observation with minimal use cases and external leaderboard9, “GPT-4-turbo-preview” (currently pointing to “GPT-4-0125-preview”) exhibits performance that is not inferior to “GPT-4-0613”.

heading: A.4 Event Type Categorization
text: To categorize event types, we establish a three-layer hierarchical structure of (mention->cluster->type), linking types between mentions. Specifically, if there are synonymous mentions between any two clusters, they belong to the same event type, and all mentions within the clusters belong to a synonymous event type. Drawing inspiration from (Ahmed et al., 2023), we determine mention synonymity by matching their span words. Table 10 illustrates that the contents of FCC and GVC belong to the same topic, resulting in a concentrated set of event types. Conversely, ECB+ involves various topics such as quake, murder, acquisition, etc., leading to a diverse set of event types.

heading: A.5 LLM Paraphrase Prompt
text: As shown in Table 11.

heading: B Experimental Results and Analysis

heading: B.1 API Efficiency and Truncation Issues
text: Compared to directly prompting GPT-4 for structured predictions of event coreference, our two-step
9https://huggingface.co/spaces/lmsys/chatbot-arenaleaderboard
prompting for summarizing each document’s mentions does incur more API calls and token consumption, as shown in Table 12. The primary additional overhead comes from generating more natural summaries for each mention rather than a final cluster label, which is the core of our collaborative approach.
Based on our approach, we also strive to enhance the efficiency of GPT-4 utilization, including:
• Processing all mentions within the same document simultaneously: this avoids assigning a separate document input for each mention, thereby reducing the number of API calls and token consumption, thus improving efficiency. To ensure the accuracy of parallel processing, we employ a concise pre-step (e.g., dependency parsing) integrated into step 1, as described in Section 4.1.
• We strive to summarize event mentions through designing concise prompts, thereby avoiding the additional comsumption of complex inference chains and in-context learning methods.
• Some recent work aims to improve the performance of LLMs by having them generate complex reasoning logic, as mentioned in the part of Integration of LLM and SLM in related work. This approach typically involves
dealing with a large number of combinations of mention pairs . In comparison, our collaborative approach only requires processing each document’s mentions once, thus offering a relative efficiency advantage while enhancing performance.
We can also illustrate the truncation issue through the statistics in Table 12. It shows that processing all test data from ECB+ consumed over 166k tokens for input and 25k tokens for output. With GPT-4’s output limited to 4096 tokens per instance, processing all test data in one go would allow us to get results for only 15% of the total mentions. The similar issue primarily results in poor performance on the GVC and FCC datasets. In the future, we will explore ways to address the length issues caused by multi-document scenarios, possibly through multiple processing iterations.

heading: B.2 Evaluation Under the Conditions of Without Singletons and at the Topic Level
text: The experimental results under the condition of with/without singletons are presented in Table 13. The results demonstrate that our method achieves state-of-the-art performance, surpassing Chen et al. (2023) by 3.5% and our baseline by 2.7% in CoNLL F1 under the without singletons condition. Additionally, our method demonstrates a relatively smaller performance gap between with and without singletons compared to Chen et al. (2023) (6.8% vs 9.3%) and our baseline (6.8% vs 8.0%). This further emphasizes the effectiveness of our method.
For topic level evaluation, it advocates not using subtopic-level document clustering, forcing models to confront the lexical ambiguity challenge. Our method, based on a baseline that performs can-
didate coreferential mention retrieval at a global level, avoids leveraging topic structure information and achieves better results than subtopic clustering methods. Therefore, we do not perform additional comparisons at the topic level.

heading: B.3 False Negative Cases
text: Given the context where mentions of the same event can vary greatly in expression styles, we provide an illustrative example in Table 14.
In cases where event mentions naturally lack sufficient details, we illustrate this phenomenon through Table 15, which presents two mentions of the same earthquake. The context for the first mention contains essential information such as time, location, magnitude, casualties, etc. In contrast, the context for the second mention primarily describes the subjective experiences of the individuals involved, lacking details related to the event itself. Despite our summarization extracting key information from the original context, it encounters difficulties in supporting coreference judgments.

heading: B.4 Two-step Workflow Analysis
text: Error Analysis We conduct error analysis for the workflow with only Step 1, the complete two-step workflow (Step 2), and the integrated single-step workflow.
As shown in Figure 4, Step 1 exhibits a significant reduction in FPA errors across all three datasets, indicating its effectiveness in extracting tailored information. However, an increase in FN errors is observed across all three datasets, suggesting that while Step 1 provides sufficiently distinctive information, it lacks the details needed to link
mentions of the same event. This issue was notably addressed by the introduction of Step 2, resulting in a substantial decrease in FN errors across all datasets. FPA errors are also largely maintained at the level achieved in Step 1, leading to a significant improvement in coreference results. This emphasizes the indispensable roles of both Step 1 and Step 2 in the final outcomes. In Table 16, we provide examples to compare summaries generated by Step 1 and Step 2.
Compared to the two-step workflow, the integrated single-step workflow shows differing degrees of increase in both FPA and FN errors, further underscoring the necessity of decomposed execution.
Summarization length comparison We further compare the lengths of summaries generated in Step 1 and Step 2. As illustrated by the green line in Figure 5, it is evident that Step 2, building upon Step 1, results in approximately double the length. The additional detailed content contributes to the reduction of FN errors, effectively linking mentions of the same event. Furthermore, as indicated by the red and blue lines, our generated summaries remain within approximately 20% of the original document starting from a document length of 200 words. Moreover, with the increase in document length, this proportion further diminishes. This reflects the conciseness our summarization.

heading: B.5 The Impact of the Number of In-Context Demonstrations on GPT-4 Performance on CDECR
text: We test the peak performance by increasing the number of documents for demonstration. The results are shown in Figure 6, and it can be observed that:
• Under the condition of utilizing only mentioninclusive sentences as context, with the introduction of more documents (even exceeding
the quantity in the test set), there is still no significant improvement in the performance of GPT-4. And there remains a considerable gap compared to the F1 score of our method (77.2% vs 87.8%).
• Under the condition of utilizing full context, an increase in the number of documents can even degrade performance. Since the complete context is crucial for event coreference resolution, it indicates that understanding and utilizing more context is a significant bottleneck limiting the performance of GPT-4.

heading: Future Work
text: In the future, we will explore ways to address the length issues caused by multi-document scenarios, possibly through multiple processing iterations