Generating Harder Cross-document Event Coreference Resolution Datasets using Metaphoric Paraphrasing (2024)

Abstract: The most widely used Cross-Document Event Coreference Resolution (CDEC) datasets fail to convey the true difficulty of the task, due to the lack of lexical diversity between coreferring event triggers (words or phrases that refer to an event). Furthermore, there is a dearth of event datasets for figurative language, limiting a crucial avenue of research in event comprehension. We address these two issues by introducing ECB+META, a lexically rich variant of Event Coref Bank Plus (ECB+) for CDEC on figurative and metaphoric language. We use GPT-4 as a tool for the metaphoric transformation of sentences in the documents of ECB+, then tag the original event triggers in the transformed sentences in a semi-automated manner. In this way, we avoid the re-annotation of expensive coreference links. We present results that show existing methods that work well on ECB+ struggle with ECB+META, thereby paving the way for CDEC research on a much more challenging dataset.1

heading: 1 Introduction
text: Cross-Document Event Coreference Resolution (CDEC) involves identifying mentions of the same event within and across documents. An issue with CDEC is that the widely used dataset, Event Coref Bank plus (ECB+; Cybulska and Vossen (2014)), is biased towards lexical similarities, both for triggers and associated event arguments, and therefore has a very strong baseline (Cybulska and Vossen, 2015; Kenyon-Dean et al., 2018; Ahmed et al., 2023a). To see this, consider the excerpts from ECB+ shown in Figure 1(a). This consists of three killing events selected from separate articles sharing a common trigger. An algorithm capable of matching the triggers and tokens within the sentences, such as "Vancouver" and "office," can readily discern that Event 2 is coreferent with Event 3, and not Event 1. This
1Code/data: github.com/ahmeshaf/llms_coref
leads to the question of whether the state-of-the-art methods using this corpus (Held et al., 2021) learn the semantics of event coreference, or are merely exploiting surface triggers.
Figurative language, encompassing metaphors, similes, idioms, and other non-literal expressions, is an effective tool for assessing comprehension across cognitive, linguistic, and social dimensions (Lakoff and Johnson, 1980; Winner, 1988; Gibbs, 1994; Palmer and Brooks, 2004; Palmer et al., 2006). Figurative language, by its nature, draws on a wide array of cultural, contextual, and imaginative resources to convey meanings in nuanced and often novel ways. Consequently, it employs a broader vocabulary and more unique word combinations than literal language (Stefanowitsch, 2006). Most recent work on metaphors has been focused on generation (Stowe et al., 2020, 2021b; Chakrabarty et al., 2021a), interpretation (Chakrabarty et al., 2022, 2023), and detection (Li et al., 2023; Joseph et al., 2023; Wachowiak and Gromann, 2023). Yet, there is a dearth of event datasets for figurative language which limits an important research direction of event comprehension.
In this paper, we address these two challenges by leveraging GPT-4 in constrained metaphoric paraphrasing of ECB+documents. We introduce a novel dataset named ECB+META , which we generate using a semi-automatic approach. This involves applying metaphoric transformations to the event triggers within ECB+ and then hand-correcting the tagged triggers in the new corpus. As depicted in Figure 1(b), the trigger word killing in Events 2 and 3 of ECB+ become slaying and snuffing out the flame of life of in ECB+META, respectively.
This approach preserves the coreference annotations from ECB+, thereby avoiding an expensive coreference re-annotation task. Thus, we create several versions of “tougher” CDEC benchmark datasets with enhanced lexical diversity with varying levels of metaphoricity. We present baseline
276
results using previous methods—Held et al. (2021) and Ahmed et al. (2023a) (described in §3.2), and show the limitation of these approaches on this dataset. Finally, we correlate lexical diversity and text complexity with CDEC and test the hypothesis that CDEC gets more difficult as the lexical diversity/complexity of the corpus increases.

heading: 2 Related Work

heading: 2.1 CDEC Datasets
text: ECB+2 is the most widely used dataset for CDEC, yet it has limited utility in realistic applications because of how simple the dataset is. The Gun Violence Corpus (GVC; Vossen et al. (2018)), for instance, was introduced as a way of adding ambiguity to the task. Yet, both these datasets lack lexical diversity in terms of coreferent event triggers. Ravenscroft et al. (2021) is one such work that addresses the diversity question through crossdomain coreference, however, a dataset focusing CDEC on figurative language does not exist to our best knowledge.
Even with the use of modern annotation tools (Klie et al., 2018; Ahmed et al., 2023b), annotating CDEC datasets is expensive. Works such as Bugert and Gurevych (2021); Eirew et al. (2021) use Wikipedia as a way of bootstrapping ECR annotations automatically. In a similar vein, we bootstrap CDEC annotations for figurative language in a synthetic way using GPT-4.

heading: 2.2 Metaphoric Paraphrasing
text: The task of metaphoric paraphrasing has been explored through a variety of methods. A primary theme is sentential paraphrasing by replacing literal words with metaphors (Stowe et al., 2021a,b; Chakrabarty et al., 2021b). These approaches fine tune language models with control codes to indicate
2Corpus detailed in §A
metaphors, exploiting available metaphoric data to facilitate transformations from literal language to metaphoric. However, they rely on extensive data, and there is evidence that modern large language models excel at metaphor generation (Chakrabarty et al., 2023) and paraphrasing (Kojima et al., 2023; OpenAI, 2023). For this reason, we leverage GPT-4 via ChatGPT functionality for our experiments.

heading: 2.3 CDEC Methods
text: Non-filtering Methods: Previous works (Meged et al., 2020; Zeng et al., 2020; Cattan et al., 2021; Allaway et al., 2021; Caciularu et al., 2021; Yu et al., 2022) in CDEC have been successful using pairwise mention representation learning models, a method popularly known as cross-encoding. These methods use distributed and contextuallyenriched “non-static” vector representations of mentions from Transformer-based (Vaswani et al., 2017) language models like various BERT-variants (Devlin et al., 2019; Beltagy et al., 2020) to calculate supervised pairwise scores for those event mentions. While these methods demonstrate SoTA performance, their applicability is hindered by their quadratic complexity at inference.
Filtering Methods: Keeping usability and tractability in mind, we experiment only with the recent work that adds a low-compute mention pair filtering step before crossencoding. These approaches aid in the removal of numerous irrelevant mention pairs, thereby directing focus toward the most pertinent pairs with resource-intensive models. For instance, in their work, Held et al. (2021) propose a retrieval, vector-based K-nearest neighbor method, that helps find and focus only on the hard negatives in the corpus. In contrast, Ahmed et al. (2023a) employ simplified lexical similarity metrics to filter out a substantial number of truly non-coreferent pairs in the corpus.

heading: 3 Methodology
text: We first synthetically create ECB+META by employing metaphoric paraphrasing of the original corpus. Then we tag the event triggers of the original corpus in ECB+META in a semi-automated manner. Finally, we adopt two existing CDEC methods to test this new dataset. We describe each of these steps:

heading: 3.1 Metaphoric Paraphrasing using GPT-4
text: We paraphrase ECB+’s sentences in a constrained manner in which we convert only the event triggers in a sentence into metaphors. We first extract the event mentions from each sentence of the documents in the corpus, then prompt GPT-4 to convert only the trigger words in the sentence to metaphors. We adopt a chain of thought prompting approach (Kojima et al., 2022), where we provide the steps that need to be followed in the conversion (see §B).
To enhance diversity and sample appropriate metaphors, we generate five metaphors for every trigger word in the sentence and then task GPT-4 to select the most coherent one from the list. We diversify metaphoricity levels by using both singleword and multi-word metaphors. As illustrated in Figure 3, the conversion of "killing" into a singleword metaphor is "slaying," while its transformation into a multi-word phrasal metaphor is "extinguishing the candle of life." We develop two versions of ECB+META, designated as ECB+META1 for single-word transformations and ECB+METAm for multi-word transformations, respectively.
Using the generated conversions, we first automatically tag the original events in the transformed sentences. Then, we hand-correct cases where the conversion is ambiguous. In the end, we are left with two versions of the validation and the test sets of ECB+META preserving the original coreference annotations of ECB+.

heading: 3.2 CDEC Methods
text: Filtering Step for CDEC: The BiEncoder K-NN (KNN) approach, introduced by Held et al. (2021) involves a novel approach to mention pair retrieval before doing CDEC. This method focuses on selecting mentions that are most similar to a given target mention using their static vector representations and a Vector Store (like FAISS Johnson et al. (2019)). To achieve this, they fine-tune the RoBERTa-Large (Liu et al., 2019) pre-trained model using a contrastive Categorical Loss function, with categories corresponding to event clusters within the corpus. This fine-tuning process uti-
lizes token embeddings generated by the language model and trains on the centroid representations of gold standard event clusters. Due to computation constraints, we use RoBERTa-Base instead of RoBERTa-Large in this work. For the same reason, we use triplet-loss with mention pairs instead of the centroid of clusters.
The Lemma Heuristic (LH; Ahmed et al. (2023a)) leverages lexical features to pre-filter noncoreferent pairs before CDEC. This way, they eliminate the need for an additional fine-tuning step as required in the KNN approach. LH focuses on creating a balanced set of coreferent and non-coreferent pairs while minimizing the inadvertent exclusion of coreferent pairs (false negatives) by the heuristic. It accomplishes this by first generating a set of synonymous lemma pairs from the training corpus and then applying a sentence-level word overlap ratio to prune pairs that don’t meet the threshold or lack synonymy. In this work, we use the LH method for filtering and also as a baseline lexical method following Ahmed et al. (2023a).
Cross-encoder3: The Cross-Encoder (CE) functions within CDEC as a pairwise classifier, leveraging joint representations of a mention pair (ei, ej). First, it combines the two event mentions with their respective contexts into a single unified string to facilitate cross-attention. Next, it derives the tokenlevel representations of each mention after encoding this unified string. Finally, the joint representation is the concatenation of the context-enhanced token representations (vei , vej ) along with their element-wise product, as illustrated below:
v(ei,ej) = [vei , vej , vei ⊙ vej ] (1) The resulting vector v(ei,ej) is then refined through a binary cross-entropy loss function using logistic regression that learns coreference. In our work, we use the learned weights of the CELH4. For the KNN cross-encoder (CEKNN), we trained the weights of RoBERTA-Base using the KNN to generate focused mention pairs. We carry out our experiments in a transfer learning format where we train the crossencoders only on the training set of ECB+ and use the test sets of ECB+META. This is motivated by the work of Ortony et al. (1978), which argues the human processes required for comprehension of figurative and literal uses of language are essentially similar.
3Described in more detail in §C 4Provided by the authors
GPT-4 as Pairwise Classifier: Yang et al. (2022) demonstrated the viability of a prompt-based binary coreference classifier using GPT-2, though the results were sub-par. Building on their work, we employ a similar prompting technique with GPT-4 to develop an enhanced classifier. This classifier determines whether a pair of events, identified by marked triggers in sentences, are coreferent by responding with “Yes” or “No”. Similar to CE, we vary this method by incorporating the two fitering techniques (GPTLH, GPTKNN)

heading: 4 Results

heading: 4.1 Metaphor Quality Control
text: To assess the quality of the generated metaphors, an annotator familiar with the events in the ECB+ dataset manually examines the Devsmall sets. We chose a familiarized annotator because metaphors often abstract away many of the details that make coreference obvious, and we are interested in whether or not the generated paraphrases would (by any stretch of the imagination) reasonably be interpreted as referring to the original event.
The annotator examines each of the original event mentions alongside their paraphrased versions and makes a binary judgment as to whether the two can be reasonably interpreted as referring to the same event. We estimate based on the results that approximately 99% of ECB+META1 and 95% of ECB+METAm could be reasonably interpreted by a human as being coreferent to the original event mentions from which they are derived.

heading: 4.2 Coreference & Lexical Diversity
text: We use B3 (Bagga and Baldwin, 1998) and CoNLL (Denis and Baldridge, 2009; Pradhan et al., 2012) clustering metrics, in which we use the B3R for estimating recall, CoNLL as the overall metric (evaluated using CoVal (Moosavi et al., 2019)). For the methods that use LH as the filtering step, we follow Ahmed et al. (2023a)’s clustering with connected components. For KNN as the filtering step, we use Held et al. (2021)’s greedy agglomeration.
Filtering Scores: Following previous work, we first assess the B3R score on oracle results. This tests how well the filtering methods perform in minimizing false negatives (coreferent pairs that are eliminated inadvertently). From Table 1 we observe a substantial difference in the recall measures of ECB+ and ECB+META versions. The LH approach particularly takes a toll because
it relies on synonymous lemma pairs from the train set. Interestingly, KNN does well on the ECB+META versions, with only a minor drop in recall for ECB+META1 and about 10% drop for ECB+METAm. Between ECB+META1 and ECB+METAm, as expected, the recall drops more in ECB+METAm as more complex metaphors are used here.
CDEC Scores: We present the overall CoNLL F1 scores in Table 2 for the baseline (LH), the two fine-tuned cross-encoders (CELH, CEKNN), and the methods that use GPT-4 (GPTLH, GPTKNN). From the table, it is evident that LH is no longer a strong baseline for ECB+META versions with a drop in 20% score. Both CELH and CEKNN show a pattern of reducing score from ECB+META1 to ECB+METAm, with CELH performing considerably worse. Interestingly, the drop in scores for CEKNN is not substantial for ECB+META1 but there is a dramatic drop of 20% for ECB+METAm. GPTLH achieves the highest scores on ECB+ and ECB+METAm, demonstrating that GPT-4’s performance aligns with the state-of-the-art, unlike its predecessor GPT-2. However, the financial implications of using GPTLH and GPTKNN are noteworthy; running CDEC with these methods incurred approximately $75 in API costs to OpenAI.
From these results, we can conclude three things: a) ECB+ is an easy dataset, b) datasets with complex metaphors are harder benchmarks, and c) GPT-4 is only as good as the CE methods with a significant amount of added costs.
Lexical Diversity: We estimate the lexical diversity (MLTD; McCarthy and Jarvis (2010)) of the mention triggers of event clusters. We first eliminate singleton clusters. Then we calculate a weighted average (by cluster size) of the MLTD score for each cluster. The scores we achieved for the test sets of each version of ECB+ are as follows: ECB+: 7.33. ECB+META1: 11.92, ECB+METAm: 26.48. From the lower CDEC scores from Table 2 and the increasing diversity scores of the more complex corpus, we can establish a negative correlation between CDEC scores and MLTD.
Overall, the results confirm our hypothesis that when a dataset a) moves away from strong lexical overlap and b) has figurative language usage, the CDEC scores drop.

heading: 5 Analysis

heading: 5.1 Coreference Resolution Difficulty
text: We evaluate whether the paraphrased versions are more difficult for humans to determine as coreferent. On the Devsmall splits of ECB+META1, ECB+METAm, and ECB+, a human annotator reaches the same coreference verdict regardless of the degree of figurative language approximately 98% of the time. Cases in which the human annotator did not reach the same verdict generally involved convergent metaphorical language, for example:
Event a: The Indian navy unfurled the words that it had ensnared 23 pirates in the law’s net who cast ominous shadows over a merchant vessel in the Gulf of Aden on Saturday, the latest in a series of recent violent ballets with Somali pirates. Event b: Indian Naval Ship throws a net over three pirate vessels in a single orchestrated symphony .
were incorrectly identified as coreferent; in actuality the former refers to the arrest of the pirates but the latter refers to the interception of their ships. This analysis supports the findings of Ortony et al. (1978): that, for humans, figurative language use and literal language do not substantially affect comprehension.

heading: 5.2 Qualitative Error Analysis
text: We examined the coreference predictions of CEKNN on 142 common mention pairs between ECB+, ECB+META1, and ECB+METAm, as CEKNN achieved the best overall performance. For mention pairs that CEKNN correctly predicted as coreferent across all versions, we noticed a pattern: the same event trigger was shared in each (see Figure 4).
In cases where CEKNN got the prediction right on ECB+ but wrong on the META versions, the event triggers in ECB+ were changed to different ones in the META versions (see Figure 5). When CEKNN incorrectly predicted coreference on ECB+ but correctly predicted it in the META versions, it was because the same triggers in ECB+ were altered to different ones (see Figure 6). This further affirms that the model heavily relies on surface triggers for making coreference decisions.

heading: 6 Future Work
text: Future research could explore applying more recent CDEC techniques on ECB+META. These techniques could include symbolic grounding, as discussed in Ahmed et al. (2024b,a), and event type categorical cross-encoding, as proposed by Otmazgin et al. (2023). Another outcome of this research is to use CDEC as a text complexity metric (Hale, 2016) of a corpus. We argue that a corpus is more complex if a CDEC algorithm is not able to identify that different explanations of the same event are the same. An interesting line of future work would be to automatically generate an optimally complex CDEC corpus, i.e., a corpus that yields the lowest coreference score.
In this work, we rely on the GPT-4’s metaphor list and substitution choice. The only control we have is to make a coherent choice, however, we find ourselves subjected to the unpredictable outputs, colloquially referred to as “hallucinations”, generated by GPT-4. In the future, we aim to integrate human feedback into the process of metaphor selection and to employ annotated metaphor databases from studies such as Joseph et al. (2023).

heading: 7 Conclusion
text: In this paper, we introduced ECB+META a lexically rich variant of ECB+ using constrained metaphoric paraphrasing of the original corpus. We provide hand-corrected event trigger annotations of two versions of ECB+META differing in the kind of metaphoric transformation using either single words or phrases. We finally provide baseline results using existing SoTA methods on this dataset and show their limitations when there is substantial lexical diversity in the corpus. Through the provided data and methodology, we lay a path forward for future research in Cross-Document Event Coreference Resolution on more challenging datasets.
Limitations
The study faced several limitations, including its focus on a single language-English. Some experiments were conducted within a small sample space, especially for Devsmall, potentially leading to biased results and limiting the generalizability of the findings. Finally, while the study utilized variations within a single dataset, the reliance on this sole dataset could introduce inherent biases, affecting the broader applicability of the research outcomes.
Reproducibility Concern: All the coreferencing experiments are reproducible, but the generation of ECB+META is not. So we may have vastly different results if a new version of ECB+META is created with the methodology. However, we released all the generated text that came out of our work and the code to run the experiments.
LLMs on ECB+. Contamination Concern The GPT-4 has likely been contaminated by the test sets of ECB+, i.e., GPT-4 has been pretained on this benchmark. With the recent work involving GPT and ECB+ (Yang et al., 2022; Ravi et al., 2023a,b), it seems likely the test set is also been used in the instruction fine-tuning of GPT-4. But we stress the synthesizing of datasets to battle contamination as we do in our work.
Ethics Statement
AI-generated text should always be thoroughly scrutinized before being used for any application. In our work, we provide methods to synthesize new versions of the same real articles. This can have unintentional usage in the propagation of disinformation. This work is only intended to be applied to research in broadening the field of event comprehension. Our work carries with it the inherent biases in news articles of ECB+ corpus and has the potential of exaggerating it with the use of GPT-4, which in itself has its own set of risks and biases.

heading: Acknowledgements
text: We thank the anonymous reviewers for their helpful suggestions that improved our paper. We are also grateful to Susan Brown, Alexis Palmer, and Martha Palmer from the BoulderNLP group for their valuable feedback before submission. Thanks also to William Held and Vilém Zouhar for their insightful comments. We gratefully acknowledge the support of DARPA FA8750-18-2-0016-AIDA – RAMFIS: Representations of vectors and Abstract Meanings for Information Synthesis and a
sub-award from RPI on DARPA KAIROS Program No. FA8750-19-2-1004. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA or the U.S. government.

heading: A ECB+ Corpus
text: The ECB+ corpus (Cybulska and Vossen, 2014) is a popular English corpus used to train and evaluate systems for event coreference resolution. It extends the Event Coref Bank corpus (ECB; Bejan and Harabagiu (2010)), with annotations from
around 500 additional documents. The corpus includes annotations of text spans that represent events, as well as information about how those events are related through coreference. We divide the documents from topics 1 to 35 into the training and validation sets5, and those from 36 to 45 into the test set, following the approach of Cybulska and Vossen (2015). We further break the documents of the validation set into two subsets: Dev and Devsmall for our error analysis. Full corpus statistics can be found in Table 3.
5Validation set includes documents from the topics 2, 5, 12, 18, 21, 34, and 35

heading: B Metaphoric Paraphrase Prompt
text: We present the prompt used with GPT-4 in Figure 2 for generating the Metaphoric Paraphrasing of ECB+ documents. We use two separate prompts for generating single-word metaphors and multi-word metaphors. We ran this prompt on the validation and test sets of ECB+ using GPT-4 as the LLM and a temperature value of 0.7. We force GPT-4 to produce JSON-style output to avoid parsing issues. It costs about $16 to generate ECB+META1 and $18 to generate ECB+METAm with GPT-4 API calls. In the future, we plan to provide this conversion of the training set of ECB+ as well.

heading: C Experiment Setup
text: LH details: we set the sentence-level word overlap ratio threshold at 0.005. We employ spaCy 3.7.4 as the lemmatizer to extract the root forms of words. KNN details: we adopt the RoBERTa-Base model, enhanced with a triplet loss function calculated by F.triplet_margin_loss with a 10 margin, L2 norm (p = 2), and ϵ = 1e− 6 for stability, without swapping and mean reduction. Our optimization uses AdamW, targeting bi-encoder parameters with a 1 × 10−5 learning rate across 20 iterations and batches of 4. CELH details: We utilize the RoBERTa-Base model with the AdamW optimizer. Learning rates are set to 1× 10−5 for BERT class parameters and 1 × 10−4 for the classifier. The model is trained over 20 epochs, using the sentences in which the
two mentions occur as context, and mention pairs generated by LH. CEKNN details: It mirrors the CELHconfiguration but it is trained on mention pairs from KNNexclusively.
All Non-GPT experiments are conducted on a single NVIDIA RTX 3090 with 24GB of VRAM. For generating the META datasets, we utilized GPT-4 (model version: gpt4-0613), setting the temperature parameter to 0.7.
D ECB+METAm Complete Results
We provide the baseline results for validation sets of ECB+METAm. As shown in Table 4, the results are consistent even for the development sets, where we see significantly low coreference scores with the used methods. Interestingly, LH performs better than the cross-encoder methods on these splits.

heading: E Error Analysis
text: For more examples, please checkout the provided excel file in data repository.

heading: Future Work
text: In the future, we plan to provide this conversion of the training set of ECB+ as well