Argument-Aware Approach To Event Linking (2024)

Abstract: Event linking connects event mentions in text with relevant nodes in a knowledge base (KB). Prior research in event linking has mainly borrowed methods from entity linking, overlooking the distinct features of events. Compared to the extensively explored entity linking task, events have more complex structures and can be more effectively distinguished by examining their associated arguments. Moreover, the information-rich nature of events leads to the scarcity of event KBs. This emphasizes the need for event linking models to identify and classify event mentions not in the KB as “outof-KB,” an area that has received limited attention. In this work, we tackle these challenges by introducing an argument-aware approach. First, we improve event linking models by augmenting input text with tagged event argument information, which facilitates the recognition of key information about event mentions. Subsequently, to help the model handle “out-of-KB” scenarios, we synthesize out-of-KB training examples from in-KB instances through controlled manipulation of event arguments. Our experiments across two test datasets showed significant enhancements in both in-KB and out-of-KB scenarios, with a notable 22% improvement in out-of-KB evaluations.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 12769–12781 August 11-16, 2024 ©2024 Association for Computational Linguistics
Event linking connects event mentions in text with relevant nodes in a knowledge base (KB). Prior research in event linking has mainly borrowed methods from entity linking, overlooking the distinct features of events. Compared to the extensively explored entity linking task, events have more complex structures and can be more effectively distinguished by examining their associated arguments. Moreover, the information-rich nature of events leads to the scarcity of event KBs. This emphasizes the need for event linking models to identify and classify event mentions not in the KB as “outof-KB,” an area that has received limited attention. In this work, we tackle these challenges by introducing an argument-aware approach. First, we improve event linking models by augmenting input text with tagged event argument information, which facilitates the recognition of key information about event mentions. Subsequently, to help the model handle “out-of-KB” scenarios, we synthesize out-of-KB training examples from in-KB instances through controlled manipulation of event arguments. Our experiments across two test datasets showed significant enhancements in both in-KB and out-of-KB scenarios, with a notable 22% improvement in out-of-KB evaluations.

heading: 1 Introduction
text: Event Linking (Nothman et al., 2012; Ou et al., 2023) involves associating mentions of events in text with corresponding nodes in a knowledge base (KB). This process of linking nodes can enhance text comprehension to facilitate various downstream applications, such as question-answering and recommendation systems (Yu et al., 2023b; Li et al., 2020; Jacucci et al., 2021).
Despite its significance, event linking remains a challenging and relatively under-explored task (Yu
∗The authors contribute equally.
et al., 2023a; Pratapa et al., 2022). In contrast to entities that generally maintain consistent attributes over time, events can vary based on nuances in event arguments, such as time, location, and their participants, leading to increased complexity and ambiguity. For instance, Fig. 1 illustrates that the two event mentions of “war” should be differentiated and linked to distinct Wikipedia entries by recognizing their unique occurrence times and involved leaders, despite the similarity in event names and combatants.
Previous studies on event linking predominantly employed methods for entity linking. Yu et al. (2023a) and Pratapa et al. (2022) both reuse the framework from Wu et al. (2020), which first uses a bi-encoder to perform efficient retrieval, followed by precise re-ranking of top candidates using a cross-encoder. Yu et al. (2023a) further enhanced this approach by incorporating adjacent named entities into the query text, emphasizing the role of entities in event linking. However, not all adjacent entities are relevant to the event, and even the relevant ones can play different roles in an event.
Furthermore, given the limited entries KBs usually possess, it is always a practical challenge for linking models to deal with out-of-KB queries (Dong et al., 2023). This challenge is more acute in event linking due to the vast number of newly occurring events, of which only a fraction
12769
are recorded in KBs. However, this issue has often been overlooked in existing event linking research.
In this work, we enhance event linking systems by capitalizing on the role of event arguments in distinguishing events. We use established event extraction models (Huang et al., 2022; Hsu et al., 2023a) to capture the participants, time, and locations of the query event. As illustrated in Fig. 1, this argument data is crucial for differentiating events. To address out-of-KB challenges, we train models to predict “out-of-KB” labels using synthetic out-of-KB query data, which is created by manipulating the event arguments of existing queries. For example, our system will replace British East India Company and the Maratha Empire in Fig. 1 with alternative fictional combatant pairs to form the training data for “out-of-KB” prediction.
We apply our design to a model architecture akin to Yu et al. (2023a); Pratapa et al. (2022) and conduct experiments on the two event linking datasets introduced by Yu et al. (2023a). Our approach yields a 22% accuracy improvement over previous baselines for out-of-KB testing and an over 1% increase for in-KB testing. Additionally, by comparing various methods for generating synthetic out-of-KB examples, we demonstrate that our data synthesis approach successfully balances in-KB and out-of-KB usage for event linking. Our code can be found in https://github.com/ PlusLabNLP/Argu_Event_Linking.

heading: 2 Related Work
text: Entity Linking, which associates entity mentions with KB entries, has been studied for years (Bunescu and Pasca, 2006; Mihalcea and
Csomai, 2007; Gupta et al., 2017; Botha et al., 2020). Common approaches include using neural networks to represent queries and KB entries for discriminative prediction (Francis-Landau et al., 2016; Wu et al., 2020; Zhang et al., 2022) or using generation-based methods (Cao et al., 2021; Mrini et al., 2022; Xiao et al., 2023). While these techniques can be adapted for event linking, they are not tailored to incorporate the structured information within events, which, as we will demonstrate in §5, is vital for disambiguating events for grounding.
Event Linking are first introduced by Nothman et al. (2012). Recently, Yu et al. (2023a) and Pratapa et al. (2022) make efforts on introducing English and multilingual datasets on the task. However, their approaches overlook the influence of event arguments to the task and neglect the discussion of handling out-of-KB cases.

heading: 3 Problem Statement
text: Event linking connects a mention of an event in the text to relevant entries within a knowledge base. If no matching entry exists, models predict “NIL,” as established in prior research (Zhu et al., 2023; Yu et al., 2023a). The event-linking model analyzes a sequence of tokens x = x1, x2, ..., xeve, ..., xl, where xeve signifies the target event mention. Its objective is to output y, which can be either an entry ei from the knowledge base (size n), or “NIL” if no suitable match is found.

heading: 4 Method
text: We hereby introduce our event linking model with two key innovations, as illustrated in Fig. 2. First,
to help the model distinguish event attribute details, we propose to first tag the event argument information of the input query (§4.1). Second, to improve the model’s capability to deal with out-ofKB cases, we introduce a negative data generation method that synthesizes potential out-of-KB examples to train our model (§4.2). Finally, we train the model with these data changes (§4.3).

heading: 4.1 Event Argument Tagging
text: To make event linking models better capture event argument information, we use the UniST model (Huang et al., 2022) trained on the MAVEN dataset (Wang et al., 2020) to first identify the event types of query events. Given predicted event types, we extract event arguments using the TagPrime model (Hsu et al., 2023a) trained on the GENEVA dataset (Parekh et al., 2023). Text with event argument tags is used by our model. Take the passage in Fig. 3 as an example, “Germany” will be extracted as the “Assailant” of the invasion, “the Soviet Union” will be highlighted as the “Victim”. More relevant details about event trigger and argument extraction can be found in Appx. §B.

heading: 4.2 Negative Data Generation
text: Prior research on event linking largely overlooked out-of-KB issues, mainly due to the limited availability of diverse training data for such scenarios. To address this gap, we design a pipeline to generate synthetic training data, enhancing the ability of event linking systems to make robust predictions for both in-KB and out-of-KB queries.
Creating out-of-KB event queries is non-trivial because randomly altering the query text does not guarantee that the event falls outside the KB or at least stops referencing the original event. Di-
rectly altering the event mention word may lead to text that is incoherent or still references the same event, such as changing “invaded” to “grew” or “attacked” in Fig. 3. We address the challenge by leveraging our ob-
servation that events differ when argument configurations change. To generate a data point, we first sample an in-KB query from the training set, along with its tagged event mention and arguments. We then instruct a large language model (LLM) to adjust this example through a two-step process: first modifying the tagged event arguments and then making edits to ensure coherence and fluency, as demonstrated in Fig. 3. 1
To increase the likelihood of generating more realistic out-of-KB query cases, we instruct the LLM to create contexts that violates its own knowledge. However, there remains a possibility that the generated context may reference other events within the KB. To minimize this impact, in actual data usage, we treat our generated event query as a “negative” training data point when paired with top KB entries for the original sampled in-KB mention. Further details are provided in §4.3.

heading: 4.3 Model
text: We apply our proposed techniques to the same retrieve-and-rerank model architecture (Wu et al., 2020) used in prior works (Yu et al., 2023a).
The retrieve stage involves a bi-encoder model. A candidate encoder first encodes each entry in the KB into a dense space. A text query q with event information tags is then fed into the other encoder
1This two step generation is generated through a single prompt. We use GPT-3.5-Turbo (Ouyang et al., 2022) with 2-shot examples (Brown et al., 2020) to instruct the model. More details about the prompt are listed in Appx. §C.
(query encoder) and projected into the same dense space. Top-k candidates will be extracted by measuring the dot product similarities between q and every KB entry.
After obtaining top-k KB candidates c1, c2, ..., ck ∀ci ∈ {e1, e2, ...en}, a crossencoder is employed to encode every pair (q, ci) to a score S(q, ci). The best candidate is selected by ranking the scores c = argmaxci S(q, ci).
To handle out-of-KB scenarios, prior work (Yu et al., 2023a), lacking out-of-KB training examples, generates the final output cfinal by setting an arbitrary threshold θ:
cfinal = { c, if S(q, c) < θ NIL, otherwise.
Unlike this approach, our method introduces a learned “NIL” class trained with our synthetic negative data. During the re-ranking phase, we expand the candidate pool to include k + 1 options, the extra one being a randomly initialized embedding that represents “NIL” :
cfinal =argmax ci S(q, ci)
, where i ∈ {0, 1, ..., k}, c0 = NIL.
Our cross-encoder is trained to predict “NIL” when the input query q is replaced with the synthetic negative query we generated, illustrated as the “Negative Query” in Fig. 2.

heading: 5 Experiments

heading: 5.1 Experimental Settings
text: Datasets from the prior event linking work (Yu et al., 2023a) are used. The data is constructed by Yu et al. (2023a). The event KB used is the collection of Wikipedia pages with English titles. The datasets include the Wikipedia dataset, which contains training and in-domain testing data, and the New York Times (NYT) dataset, which contains out-of-domain and out-of-KB testing data. We introduce the details of the datasets below and list their statistics in Tab. 1. • The Wikipedia dataset contains the training,
validation, and test splits. The Wikipedia dataset is collected automatically from hyperlinks in Wikipedia. A hyperlink text is considered an event mention if the linked Wikipedia title has its mapped FIGER type (Ling and Weld, 2012), a fine-grained set of entity tags, being “Event.” By construction, the Wikipedia dataset contains in-KB event mentions only.
• The NYT dataset is a smaller, manually annotated test set. 2,500 lead paragraphs are sampled from The New York Times Annotated Corpus and then annotated through Amazon Mechanical Turk. The dataset comes from real-life news articles and contains out-of-KB event mentions that is not covered by Wikipedia.
Evaluation Metrics. We follow Yu et al. (2023a) to evaluate models using accuracy.
Compared Methods: • BM25: a term-based ranking algorithm for infor-
mation retrieval. • GENRE: a generation-based entity linking
model retrieves entities by generating their unique names. We follow the analysis from (Yu et al., 2023a), who train GENRE on the eventlinking training set and perform inference. • BLINK (Wu et al., 2020): the retrieve-andrerank model architecture introduced for entity linking. We adopt its code but train it on the event linking training set. BLINK processes the entire token sequence, enriched with special tokens to mark the beginning and end of the event mention, i.e., x1, x2, ..., [Ms], xeve, [Me], ..., xl, and apply retrieve-and-rerank training and inference. Here, [Ms] and [Me] are the special markers. • EveLINK (Yu et al., 2023a): the current SOTA event linking model that also follows retrieve-and-rerank framework. It adopts BLINK but enhances the text query by adding local name entity information. Specifically, EveLink initially performs named entity recognition on x, identifying named entities, ne1, ne2, ... and their type t1, t2, .... It then combines the token and entity sequences as input to models, i.e., x1, x2, ..., [Ms], xeve, [Me], ..., xl, [SEP ], [t1s], ne1, [t1e], [t2s], ne2, [t2e], ..., where [SEP ] is the BERT separator, and [t1s], [t1e], ... are the special tokens indicating the start and end of named entity types, respectively. • Our method also employs the retrieve-andrerank framework. Yet, our method differs from BLINK in that we incorporate event argument
information of x and apply negative data generation to augment the training data for better support on “out-of-KB” cases. Specifically, our model’s input format can be represented as x1, x2, ..., [r1s], xarg1, [r1e], ..., [Ms], xeve, [Me], ..., [r2s], xarg2, [r2e], ..., where xarg denotes the event argument token, and [r1s], [r1e], ..., are special tokens indicating the corresponding argument roles. All reported results in this section are the average of three random runs. Appx. §D covers all implementation details of the compared methods.

heading: 5.2 Main Results
text: Tab. 2 presents our main results, categorized by the type of event mention (All/Verb/Noun). In the in-domain Wikipedia evaluation, our approach surpasses all baseline methods across all categories. For the out-of-domain, out-of-KB evaluation using the NYT dataset, our method demonstrates its robustness with an over 20% absolute improvement.

heading: 5.3 Analysis
text: In this section, we present studies to verify our two innovations. The exploration of the possibilities of using LLMs in event-linking modeling can be found in Appx. §A.

heading: 5.3.1 Bi-Encoder Results
text: Bi-encoder results are shown in Tab. 3. Directly analyzing the bi-encoder performance allows us to assess the impact of integrating event argument data into the text. Our approach surpasses all baseline methods, showing greater enhancements in the harder cases as the number of candidates decreases.

heading: 5.3.2 Effectiveness of Negative Data Generation
text: We benchmark our approach against two alternative methods for generating negative data to train
the cross-encoder: (1) Non-argument-aware Data Generation, which also employs GPT-3.5-Turbo but does not incorporate event information into the prompts, as detailed in Appx. §C; (2) KB Pruning, a strategy introduced by Dong et al. (2023) in the entity linking field. This method creates negative samples by randomly eliminating 10% of KB entries and marking the associated training data as negative examples external to the KB. Tab. 4 shows the comparison. While KB Pruning ensures high-quality negative examples outside the KB, it negatively affects performance on in-KB tests. In contrast, our method, designed to emphasize event information, effectively balances the use of in-KB and out-of-KB cases.

heading: 6 Conclusion
text: In this work, we introduce an argument-aware method designed to improve event linking models. This approach aids in disambiguating events and generating out-of-KB training examples. Experimental results demonstrate that our method enhances the accuracy for both in-KB and out-of-KB queries. Our findings reveal that the system, trained on flattened data, struggles to process structured textual information effectively. Therefore, implementing our guidance about event arguments can improve its understanding of structured events.

heading: Acknowledgements
text: We gratefully acknowledge the insightful feedback provided by both the anonymous reviewers and the UCLA PLUS Lab and UCLA-NLP group members on earlier versions of this paper. This research was funded by a Cisco Sponsored Research Award.
Limitation
Although our incorporation of event argument information is shown to be highly effective, it does introduce additional pipelines to extract and tag event information, bringing extra cost to model training and inference. Additionally, since we demonstrated that tagging event information in the query helps improve performance, it would be interesting to explore the possibility of also leveraging structured event information in the KB entries as well. We leave this investigation for future work.
Broader Consideration
We employ LLMs to generate training data for our model. Consequently, the model may inherit biases from the LLM, potentially leading to ethical concerns. Despite the low likelihood and our use of the data for negative examples, we recommend a thorough evaluation of these potential issues prior to deploying the model in real-world applications.

heading: A Experiments with Large Language Models as Ranking Model
text: To the best of our knowledge, current event linking systems lack the integration of Large Language Models (LLMs). Inspired by recent information retrieval techniques leveraging LLMs (Pradeep et al., 2023), we explore their effectiveness by incorporating an instructed LLM for candidate re-ranking within the existing event linking pipeline, replacing the traditional cross-encoder.
We establish a baseline using GPT-3.5-Turbo and compare our proposed method with this LLM baseline 2. To ensure a fair comparison, we utilize the same bi-encoder system and sample a subset of 1,000 test examples.
The results presented in Table 5 demonstrate that the method solely employing LLM for re-ranking significantly underperforms our approach. This finding highlights the key advantage of our method, which leverages LLMs for generating negative data, leading to superior performance.

heading: B Event Extraction Details
text: Our system employs a two-step approach for comprehensive event extraction from text. First, we perform event typing to categorize the event described in the input text. We leverage UniST (Huang et al., 2022) to achieve this, trained on the MAVEN dataset, which offers broad coverage with 168 event types. This ensures our system can handle a wide range of potential input events.
Next, upon identifying the event type, we proceed with event argument extraction. We utilize TagPrime (Hsu et al., 2023a), the state-of-the-art model in this domain. To maintain consistency with MAVEN’s ontology, we employ the GENEVA dataset, which aligns closely with MAVEN while encompassing 115 event types due to minor modifications. Consequently, our system offers comprehensive support for 115 event types, encompassing
2Implementation details can be found in Appx. §D.
220 distinct argument roles, effectively covering diverse event types within queries on our experimental dataset.
While alternative event extraction models exist, such as DEGREE (Hsu et al., 2022), UniIE (Lu et al., 2022), AMPERE (Hsu et al., 2023b), and PAIE (Ma et al., 2022), each trained on various datasets, our approach stands out as the most comprehensive event extraction pipeline to date. It captures the vast majority of event types, providing a robust solution for diverse user needs. For a more in-depth exploration of alternative methodologies, we recommend referring to the works of Peng et al. (2023); Huang et al. (2023).

heading: C Data Generation Details
text: Our negative data generation begins by sampling from our tagged training and validation sets. To ensure the quality of the generated data, we filter out examples where the labeled event mentions are proper nouns or numeric values. Additionally, to better leverage our observations, we exclude examples with fewer than two tagged event arguments.
Following this filtering step, we employ a twostage process to generate negative data, as detailed in the prompt provided in Fig. 4.
Due to the high cost of GPT-3.5-Turbo, we only generated a final set of 6,600 examples from the training set and 1,600 examples from the validation set to train and develop our method.
We employed the same data generation method for the non-argument-aware baseline, but without tagging event arguments. The prompt was adjusted to reflect the absence of this information (see Fig. 5 for the specific prompt used). Notably, the text content of the few-shot examples remained identical in both cases; the only distinction lies in the presence of event argument tags.
D Implementation Details
Model Training and Inference Pipeline Our system follows a two-stage training approach. First, we train a bi-encoder to encode queries and candidate event mentions. We then use the queries and their top 10 candidates to train the cross-encoder. During inference, we also first retrieve the top 10 candidates using the trained bi-encoder. Then, we re-rank the retrieved candidates using the trained cross-encoder and select the top 1 candidate to compare with the ground truth. To ensure a fair comparison, all baselines and our methods follow
this recipe. Training is conducted on Nvidia A100 40G GPUs. The bi-encoder typically requires approximately 6 hours to train, while the cross-encoder takes around 30 hours.
Model Architectures Both the bi-encoder and cross-encoder leverage the pre-trained BERT-baseuncased model. For the bi-encoder, we set a maximum query and candidate length of 300 tokens. The training utilizes a learning rate of 1e − 5, a batch size of 48, and runs for 15 epochs to optimize GPU memory usage.
The cross-encoder also employs BERT-baseuncased, with a maximum query and candidate length of 256 tokens. Here, the training process utilizes a learning rate of 2e− 5, a batch size of 6, and runs for 20 epochs.
Baseline Configurations For the threshold θ used for baseline NIL prediction (see §4.3), we follow the description in (Yu et al., 2023a)’s paper and set it as 0.5 after normalization.
For the BM25 baseline, we use a query (context) window size of 16, which follows the practice from Pratapa et al. (2022).
For the analysis of negative data generation methods, details on data generation are covered Appx. §C. Additionally, the KB pruning baseline is implemented by randomly pruning 10% of the unique labels present in the training set. We then label the corresponding samples (5984 samples) as out-of-KB.
For the LLM baseline, on the Wikipedia test set, the model is asked to re-rank the top-k documents given the query. On the NYT test set, the model is given an additional option to simply label the query as “NIL” if none of the documents describes the event in the query. The two prompts for the Wikipedia and NYT test sets are shown in Fig. 6 and Fig. 7, respectively.

heading: E Full Bi-encoder Evaluation Results
text: We present the full bi-encoder evaluation results, which include more recall values, in Tab. 6.

heading: Future Work
text: We leave this investigation for future work. Broader Consideration
We employ LLMs to generate training data for our model. Consequently, the model may inherit biases from the LLM, potentially leading to ethical concerns. Despite the low likelihood and our use of the data for negative examples, we recommend a thorough evaluation of these potential issues prior to deploying the model in real-world applications