Event-Arguments Extraction Corpus and Modeling using BERT for Arabic (2024)

Abstract: Event-argument extraction is a challenging task, particularly in Arabic due to sparse linguistic resources. To fill this gap, we introduce the Wojood corpus (550k tokens) as an extension of Wojood, enriched with eventargument annotations. We used three types of event arguments: agent, location, and date, which we annotated as relation types. Our inter-annotator agreement evaluation resulted in 82.23% Kappa score and 87.2% F1-score. Additionally, we propose a novel method for event relation extraction using BERT, in which we treat the task as text entailment. This method achieves an F1-score of 94.01%. To further evaluate the generalization of our proposed method, we collected and annotated another out-of-domain corpus (about 80k tokens) called Wojood and used it as a second test set, on which our approach achieved promising results (83.59% F1-score). Last but not least, we propose an end-to-end system for event-arguments extraction. This system is implemented as part of SinaTools, and both corpora are publicly available at https: //sina.birzeit.edu/wojood

heading: 1 Introduction
text: Understanding and extracting events from text is crucial in natural language understanding (Khalilia et al., 2024) for applications like disaster monitoring (Hernandez-Suarez et al., 2019), emergency response (Simon et al., 2015), insurance decision support, and fostering community resilience (Ahmad et al., 2019). Events, a type of named entity mentions (Jarrar et al., 2023a), are connected to other entities through their arguments. This connection forms the foundation of the event-argument extraction task, which is closely related to relation extraction. By identifying events and linking them with arguments like agents, locations, and dates, we establish meaningful relationships that enhance applications such
as information retrieval systems (Singh, 2018), word sense disambiguation (Jarrar et al., 2023b; Al-Hajj and Jarrar, 2021), and knowledge graph construction (Ye et al., 2022). Figure 1 shows how the event "The Economic Forum Meeting" is connected with its three arguments.
Despite the significance of event-argument extraction, there is a notable gap in the availability of comprehensive, annotated corpora for this purpose, especially for under-resourced languages like Arabic (Darwish et al., 2021; Haff et al., 2022). To address this gap, we have developed an event-argument extraction corpus specifically for Arabic. We extended the Wojood corpus (Jarrar et al., 2022a), which is the largest and most recent NER corpus for Arabic (Liqreina et al., 2023; Jarrar et al., 2024, 2023a). We annotated arguments of event entities in Wojood corpus by manually linking each event entity with its corresponding argument entities. As shown in Figure 2, three types of arguments are introduced (agent, location, date) for each event. Our Wojood extension (i.e., the event-arguments annotations) is called WojoodHadath.
Furthermore, we introduce a novel method for event-argument extraction using BERT and achieved 94.01% F1-score. Based on WojoodHadath, we generated a dataset of premisehypotheses sentence pairs (we call it HadathNLI). We used this dataset to fine-tune BERT, as a natural language inference (NLI) task. To test the generalization of our approach, we have constructed an additional out-of-domain dataset (about 80k tokens) called WojoodOutOfDomain and used it as a second test set. Our model achieves,
309
again, promising results (83.59% F1-score). Finally, to streamline the event-argument extraction, we propose an end-to-end system specifically designed for the Arabic event-argument extraction task. In summary, the contributions of this paper are:
1. WojoodHadath corpus (550k tokens) manually annotated with event argument relations. This corpus is used to generate an NLI dataset HadathNLI (25k premise-hypotheses pairs).
2. WojoodOutOfDomain, an additional annotated corpus (80k tokens) for out-of-domain evaluation.
3. Novel methodology for event-argument extraction by framing the task as an NLI problem, achieving high performance.
4. Novel end-to-end system for event-argument relation extraction.
All datasets are available online1, and the endto-end system is implemented and can be used as part of the SinaTools (Hammouda et al., 2024)2.
This paper is organized as follows: Section 2 reviews previous research. Section 3 discusses corpus annotations. Section 4 presents the interannotator agreement analysis. Section 5 outlines the methodology. Section 6 details the dataset construction, while Section 7 covers the experimental setup. Section 8 introduces the out-of-domain dataset. Section 9 elaborates on the end-to-end system, and Section 10 explores ablation studies. Finally, Section 11 concludes the paper and discusses future directions.

heading: 2 Related works
text: Events are occurrences or actions that happen over time, involving specific participants and locations. They have temporal components and rely on physical entities to take place (Jarrar, 2021; Jarrar and Ceusters, 2017). Extensive research has focused on event extraction and event argument extraction (EAE), especially in English. However, EAE in Arabic is limited, leaving a gap in the literature (Chouigui et al., 2018; Alomari et al., 2020).
Automated methods, utilizing either statistical algorithms or NLP techniques, are employed to identify relationships between words, using large
1Datasets: https://sina.birzeit.edu/wojood/ 2SinaTools: https://sina.birzeit.edu/sinatools
corpora to detect related terms through their cooccurrence patterns (Khallaf et al., 2023). For instance, Hkiri et al. (2016) proposed a model to extract event entities from Arabic news articles using the GATE tool, employing a five-stage entity identification process that establishes links between events and their corresponding arguments. However, their dataset is small as it consists of only 1, 650 sentences.
In a similar context, AL-Smadi and Qawasmeh (2016) proposed to use an unsupervised rule-based technique to extract events and the relationship with their associated entities from 1, 000 Arabic tweets covering Time, Agent, Location, Target, Trigger, and Product. They linked extracted events and entities to a knowledge base, achieving 75.9% accuracy. This accuracy pertains to the textual representation within the tweet corresponding to the event expression, event type identification (97.7% accuracy), and event time extraction (87.5% accuracy). However, despite these achievements, there remains a need for larger and more diverse datasets to further validate these findings.
Multilinguality has increasingly attracted attention across various fields due to its potential to enhance the understanding and processing of diverse linguistic data (Jarrar and Amayreh, 2019; Jørgensen et al., 2023; Duaibes et al., 2024). This direction has notably influenced relation extraction tasks, as seen with SMILER (Seganti et al., 2021), which aims to improve entity and relation extraction, including Arabic. Seganti et al. (2021) fine-tuned HERBERTa on the Arabic subset of SMILER, achieving high performance in identifying relations and entities despite its smaller dataset of 9k sentences. This suggests that the unique linguistic features of Arabic may contribute to the model’s robustness. Furthermore, Cabot et al. (2023) introduced valuable resources, namely, SREDFM and REDFM, designed for multilingual relation extraction. SREDFM encompasses over 40 million triplet instances across 18 languages, featuring 400 relation types and 13 entity types. Their mREBEL model, pre-trained on SREDFM, exhibited a remarkable improvement of 15 points in Micro-F1 compared to HERBERTa. However, their qualitative error analysis excludes Arabic and Chinese because the authors are not proficient in these languages.
Joint extraction models have become increasingly important in NLP. Addressing the need for
improved event extraction in Arabic text, El Khbir et al. (2022) introduced a joint model for event extraction in Arabic text using the ACE 2005 dataset. They used a graph-based representation to extract entities, relationships, and event triggers, along with their arguments. This approach led to improved Arabic NER accuracy through different experiments and tokenization schemes.
In the realm of biomedicine, Xu et al. (2022) introduced the NBR (NLI improved Biomedical Relation Extraction) method, which verbalizes relations in natural language hypotheses, allowing the model to utilize semantic information effectively in prediction, even with limited data. In contrast, Cao et al. (2023) addresses cross-lingual EAE challenges through their innovative LanguAge-oriented Prefix-tunIng Network (LAPIN) approach. LAPIN utilizes a languageoriented prefix generator module to handle language variations and a language-agnostic template constructor module to create adaptable templates. Experimental results demonstrate LAPIN’s outperforming, achieving an average F1 improvement of 4.8% and 2.3% on two multilingual EAE datasets compared to the previous state-of-the-art models. These efforts collectively contribute to advancing the field of EAE in various languages, including Arabic.

heading: 3 Dataset and annotation

heading: 3.1 Corpus Preparation
text: To construct an event-argument corpus, we extended the existing Wojood corpus (Jarrar et al., 2022b). This is because it is a rich Arabic nested named entity corpus comprising 550k tokens, supporting 21 distinct entity types, including categories such as person, organization, location, event, and date. Notably, it incorporates 2, 772 annotated events. According to Wojood guidelines, a mention is annotated as an event if it represents an occurrence of general interest, like battles, wars, sports events, demonstrations, disasters, elections, and national or religious holidays.
Our objective in this paper is to identify event arguments and establish relationships between these arguments and the respective event entities, as illustrated in Figure 2.

heading: 3.2 Annotation Process
text: Initially, we assigned a unique identifier to each entity in Wojood, see Figure 2(a). Then, we linked
these entities with relationships, see Figure 2(b).

heading: 3.2.1 Relationship Types
text: We propose to use these relations:
• hasAgent: specifies participant(s) involved in the event, which can be a PERS, ORG, OCC, or NORP named entities.
• hasLocation: indicates where the event occurred, which can be GPE, LOC, and FAC named entities.
• hasDate: points when the event occurred, which can be TIME or DATE.

heading: 3.3 Annotation Guidelines
text: We propose the following guidelines to annotate the corpus:
1. Event arguments are recognized only within the same sentence.
2. Entities with different entity IDs are considered distinct entities. For example, in the sentence ( H@XA Ë@ Pñ K @ ø Qå ÖÏ @
KQË @ É J ®Ó / The killing of the Egyptian president Anwar al-Sadat), the entity (ø Qå ÖÏ @
KQË @/the Egyptian president) and the entity ( H@XA Ë@ Pñ K @ / Anwar al-Sadat) are regarded as separate entities, thus two agents in the event. However, in reality, they refer to the same individual.
3. The same event may have multiple agents, as in (ø Q»QÖÏ @ ½
JJ. Ë @ ð éJ K A JJ. ÊË @ éÓñºmÌ'@ á K. àðAª K éJ ̄ A ® K @ ©J ̄ñ K / Signing a cooperation agreement between the Lebanese government and the Central Bank). In Figure 3, ( éJ K A JJ. ÊË @ éÓñºmÌ'@ / the Lebanese government) and (ø Q»QÖÏ @ ½
JJ. Ë @ / the Central Bank) are both agents of the same event.
4. In the same sentence, two different event entities can share the same argument. For example, in the sentence (( é º JË @) 1967 ÐA« H. Qk YªK. Qå Ó ú ̄ Q Kñ JÓ ú æ AJ Ë@ © ñË@/The
political situation in Egypt is tense after the 1967 War (Al Naksa)), there are two event entities, according to Wojood guidelines. The first event is (1967 ÐA« H. Qk/The 1967 war), and the second event is (( é º JË @)/Al Naksa). In this way, the entity (1967 ÐA«/the year 1967) is considered the argument for both events in the sentence.

heading: 3.4 Corpus Statistics
text: Wojood comprises 550k tokens with 2, 772 event entities. Among these entities, we annotated 1, 974 events with event-arguments relations - the other 798 do not have arguments. Notably, there are 355 (18%) events that are annotated with at least two arguments, and 77 (18% of agent relations) annotated with multiple agents. This underscores the corpus’s rich and diverse interconnection of events and their roles. The number of instances for each relation type is shown in Table 1.

heading: 4 Inter-annotator Agreement
text: To evaluate the quality of our annotations, we randomly selected 5% of the annotations and asked
our two annotators to annotate in parallel. We computed the Inter-annotator Agreement (IAA) using Cohen’s kappa and F1-score. The results in Table 2 illustrate high agreement.
4.1 Calculating kappa We assessed annotator agreement for three relations. Agreement happens when both annotators assign the same relationship type. The kappa coefficient (Eugenio and Glass, 2004) is defined as:
K = P0 − Pe 1− Pe
(1)
where Po is the observed agreement, and Pe is the expected agreement. Pe is calculated as :
Pe = 1
N2
∑
T
nT1 × nT2 (2)
4.2 Calculating F1-score The F1-score for a specific relation (e.g., hasAgent) is calculated using Equation 3. True positives (TP) are those when annotators agree, while false negatives (FN) and false positives (FP) arise from disagreements. FN arises from the first annotator’s disagreement, while FP arises from the second annotator’s disagreement.
F1 − Score = 2TP 2TP + FN + FP
(3)

heading: 4.3 Discussion and Annotation Challenges
text: We encountered several challenges, including:
1. Although event arguments should be encompassed within the sentence referencing the event, it was not always easy for the annotators to decide if entities within the same sentence as the event should be annotated as event entities.
2. In the hasAgent relation, it can sometimes be challenging to determine whether an entity serves as an agent for the event. See the example in Figure 4.
3. The annotators have different levels of experience; one is skilled at defining event arguments, while the other is not. Consequently, one annotator agreed to designate a specific entity as the agent, while the second did not.

heading: 5 Even-Argument Extraction (EAE)
text: In this section, we provide an in-depth explanation of our approach to addressing the EAE task through the NLI task. Figure 5 illustrates our framing of the EAE problem.

heading: 5.1 Problem Formulation
text: The objective of the EAE task is to identify the relationships between the event(s) in a sentence and the named entities mentioned in the same sentence. That is, given a sentence s annotated with a set of event entities E = {ei}ni=1 and a set of other named entities N = {nj}mj=1, the goal of EAE is to identify the relation r ∈ R for each pair (ei, nj) in s, where R ∈ {hasAgent, hasLocation, hasDate}.

heading: 5.2 Event Relation Extraction as NLI
text: We propose to solve the EAE by framing it as a Natural Language Inference (NLI) task. In NLI, we assess whether one sentence (the premise) entails another (the hypothesis). That is, a pair of sentences is classified as True or False. To extract an event argument relation from a sentence (See Figure 5), we treat the original sentence as a premise and generate the hypothesis automatically. The hypothesis is another sentence generated using a template, to represent a possible relationship. In other words, we propose to treat EAE as a binary NLI task focusing on entailment. The input sentence s is the premise, while the hypothesis is a verbalized template representing a relation r between event ei and a named entity nj mentioned in the same sentence. The model then determines if the premise "entails" the hypothesis (as classification True/False), indicating the existence of the relation r between ei and nj .
Template Construction For each of the three relations, we designed a template parameterized over the named entities mentioned within the same sentence. Each template has two placeholders: Event Placeholder Pevent and Entity Placeholder Pentity. These placeholders are filled with event mention ei and named entity mention nj , respectively. Figure 5 illustrates how a template can be used to generate a hypothesis for the hasDate relation between the ( è Q « ¨A¢̄ úÎ« ÉÓA Ë@ PA mÌ'@ / Total siege on Gaza strip) event and the (2023 QK. ñ J» @ 9 /October 9, 2023) entity. The three templates used are shown in Table 3
Sentence Encoder The sentence encoder is used to extract representations for input text. In our approach, the input text is a pair consisting of a premise (the input sentence s) and a hypothesis (the filled-in template h). A transformer encoder, denoted by T , is used to process the input sequence and derive its representation, T → Rd. This encoder effectively captures contextual information and semantic relationships within the text pair generating representation H for the sentence pairs. Formally,
H = T ([CLS]s[SEP ]h) (4)
where [CLS] is the special classification token, [SEP ] is a separator between the premise and hypothesis, and d is the encoding dimension.
Relation Classifier The feature vector H from the sentence encoder is input into a fully connected layer. This layer calculates the predicted probability that the premise entails the hypothesis, with a True result indicating that the relationship r specified in h exists between the event e and the entity n. Formally,
ŷi = σ (HW + b) (5)
where ŷ indicates whether the hypothesis holds a positive relation, while W and b represent a weight matrix and a bias term, respectively.
Training Objective Our training objective is to prioritize the accurate identification of positive instances over negatives. To accomplish this, we utilize a weighted crossentropy loss LWCE (Eq. 6), which penalizes misclassifications of positive instances more heavily. Additionally, we use Noise Contrastive Estimation Loss LNCE (Eq. 7) to improve the discrimination between positive and negative instances (Robinson et al., 2020). The final loss function is expressed as Loss = LWCE + LNCE .
LWCE = − 1 N
N∑
i=1
( wpyi log(ŷi)
+ wn(1− yi) log(1− ŷi) ) (6)
LNCE = − 1 N
N∑
i=1
log ( exp(s(yi)/τ)∑ j exp(s(yj)/τ) ) (7)
6 Construct HadathNLI Dataset
Based on our annotated WojoodHadath corpus, we created an NLI dataset (called HadathNLI), which is a dataset of premise-hypotheses sentence pairs that we use to train the EAE model. To create this dataset, we carried out the following steps:

heading: 1. Premise Sentences Preparation
text: We extracted sentences from the WojoodHadath corpus and utilized them as premises. These sentences were then split into training (70%) and testing (30%) sets. This split is important to avoid overlap between premises in the two splits during the generation of positive and negative pairs.

heading: 2. Hypothesis Sentences Preparation
text: Multiple hypotheses are generated for each premise sentence, utilizing the entities mentioned within the sentence. These entities are annotated as arguments of a specific event in the WojoodHadath corpus. That is, based on the eventargument relation annotations (see section 3.2.1), each event and its arguments are used to generate a hypothesis using the template for relation r. This hypothesis is then paired with the premise sentence to form a positive pair.

heading: 3. Generating Negative Pairs
text: In premise sentences, entities that are not linked with events (i.e., not event arguments) are used to generate negative pairs. That is, we link events with entities that are not their correct arguments (i.e., not annotated as a relation in WojoodHadath), and generate them as hypotheses using templates. Each generated hypothesis is paired with its premise and used as a negative pair.
HadathNLI Dataset Statistics For the training phase, four templates are designed for each relation, each with a particular verbalizer (see Table 5). This step aims at data augmentation and enhancing the contextual diversity of the training set. For the test phase, one template per relation was selected from the training templates to generate the hypothesis (see Table 3). The final dataset consists of 25, 473 pairs, comprising 10, 478 positive and 14, 995 negative pairs. Table 4 presents more statistics.

heading: 7 EAE Modeling Experiments
text: The HadathNLI is used to train an EAE model.

heading: 7.1 Training Hyperparameters
text: During training, we employed a k-fold strategy with k = 5 to ensure the robustness of model evaluation. For all experiments, we utilized the UBC-NLP/ARBERTv2 (Abdul-Mageed et al., 2021) as the text encoder model, with a learning rate of 2e−5 and the AdamW optimizer with a weight decay of 1e−8. In the LWCE loss function, wpos = 1 and wneg = .5, while τ = 1 in LNCE loss function.

heading: 7.2 Results and Discussion
text: Table 6 shows the obtained results using the aforementioned experimental setups. We average the scores across five folds and report the model with the best average F1score. Table 7 provides insight into the NLI model performance in each relation. Note that this NLI performance represents the accuracy of our model in classifying sentence pairs, which is not the EAE accuracy. The accuracy of event-argument relation extraction (EAE) shall be presented in section 9.
Notably, our NLI model achieves high F1scores across all event relations, with the hasDate relation exhibiting the highest score (96.44%). Additionally, the negative relations also achieved a notably high F1 score of 95.29%, indicating the model’s effectiveness in recognizing entities unrelated to the event.
To validate this remarkable performance of the model and to test its generalization, we constructed a new corpus and conducted additional out-of-domain experiments.
8 Additional WojoodOutOfDomain Dataset
To evaluate the model’s generalization and its robustness to contexts beyond the WojoodHadath do-
mains, we constructed a new out-of-domain corpus (referred to as the WojoodOutOfDomain corpus). We then extracted an NLI dataset and used it for model testing.

heading: 8.1 Corpus Preparation
text: This corpus covers 10 distinct domains: economics, finance, politics, science, technology, art, law, agriculture, history, and sports each containing nearly 8k tokens. The corpus covers events from 2010 to 2022, manually collected from news websites such as Aljazeera, and Alarabiya, totaling 80k tokens.
We used the same Wojood annotation guidelines to maintain consistency. Table 8 shows the number of instances for each relationship type.
8.2 Construct WojoodOutOfDomain Dataset An NLI dataset was generated from WojoodOutOfDomain using the same methodology as HadathNLI, employing only the templates designated for testing, resulting in a total of 1124 pairs. Detailed statistics are provided in Table 9.

heading: 8.3 Experiments and Results
text: The WojoodOutOfDomain is used to evaluate the EAE model, which was trained on the HadathNLI. The results, presented in Table 10, highlight the model’s robust generalization across diverse domains, despite encountering challenges like domain-specific vocabulary and linguistic nuances. Despite a slight performance decline compared to the HadathNLI test set, the model maintained a high overall average F1−score of 83.38% on the WojoodOutOfDomain.

heading: 9 End-to-End System for EAE

heading: 9.1 System Architecture
text: This section introduces our novel end-to-end EAE system, which efficiently extracts event-related information from text by seamlessly identifying entity boundaries, determining their types, and recognizing argument entities and their relations to an event entity. We utilized the EAE NLI model to construct the system. As illustrated in Figure 6, the overall steps are: Named Entity Recognition Module: The system starts by extracting entities and their types from the input sentence. The process is carried out through the online Wojood web service3. Then,
3https://sina.birzeit.edu/wojood
the system evaluates the extracted entities. If an event entity is recognized in a sentence, then other
entities in this sentence are considered candidate arguments for this event. Template Selection Module: For each entity, a template is selected based on its category and used to construct the hypothesis. Argument Extraction Module: The input sentence is paired with the template and sent to the EAE NLI model to identify the argument entities for an event and their corresponding relationships. The type of template serves as the basis for establishing the relationship between the event and the entity. Specifically, if the EAE NLI model indicates a positive connection between the input sentence and the template, then the entity and the event linked within the template are considered to have a relationship indicated by the template label.

heading: 9.2 Computing EAE baselines
text: The performance of our proposed EAE system is evaluated using HadathNLI and WojoodOutOfDomain test sets. Results are shown in Table 11. Note that the relation classifiers share parameters, and the evaluation assumes named entities to be correctly recognized.

heading: 10 Ablation Studeis
text: Best Template: to choose the best template to implement in the EAE system for each relation (Figure 5), we evaluated them using the test set. Table 12 shows that t2 performed slightly better.
Best loss function: We compared two loss functions: cross-entropy loss (LCE) and Noise Contrastive Estimation Loss (LNCE). Table 13 shows that while LNCE slightly outperforms LCE on the HadathNLI, its performance significantly improves on WojoodOutOfDomain, showing its efficacy across diverse contexts.
Additionally, we improved the model’s performance by combining weighted cross-entropy loss (LWCE) with LNCE , using weights wp = 1 and wn = 0.5. This approach, with higher weights for positive relations and slightly lower weights for negative ones, yielded the best results.

heading: 11 Conclusion and Future work
text: Our introduction of the WojoodHadath and WojoodOutOfDomain corpora significantly advances Arabic event-argument extraction by providing a rich dataset with high inter-annotator agreement. Our novel BERT-based method for event relation extraction demonstrates exceptional performance, achieving high F1 scores on both the HadathNLI dataset and on WojoodOutOfDomain dataset. Additionally, our implementation of the EAE end-to-end system as part of the open-source SinaTools will enrich the Arabic NLP industry.
Large language models (LLMs) can further enhance our work by extracting information, including named entities and relationships, from text to populate knowledge graphs and improve other knowledge graph tasks like embedding and completion (Barbon Junior et al., 2024). In future work, we will explore the capabilities of LLMs to enhance event-argument extraction. Integrating LLMs into our framework could potentially improve the accuracy and scalability of our event extraction system.
Limitations
The constructed WojoodHadath and WojoodOutOfDomain corpora primarily focus on MSA data and do not cover dialectal variations. Furthermore, even though we included out-of-domain tests to assess performance, our results are constrained to the specific domains used in our study.
Ethics Statement
The corpora provided for this research are derived from public sources, eliminating specific privacy concerns. The results of our research will be made publicly available to enable the research community to build upon them for the public good and peaceful purposes. Our data, tools, and ideas are strictly intended for non-malicious, peaceful, and non-military purposes.

heading: Acknowledgements
text: This research is partially funded by the research committee at Birzeit University. We extend our gratitude to Taymaa Hammouda for the technical support and to the students who helped and supported us during the annotation process, especially Haneen Liqreina and Sana Ghanim.

heading: Future Work
text: In future work, we will explore the capabilities of LLMs to enhance event-argument extraction. Integrating LLMs into our framework could potentially improve the accuracy and scalability of our event extraction system. Limitations
The constructed WojoodHadath and WojoodOutOfDomain corpora primarily focus on MSA data and do not cover dialectal variations. Furthermore, even though we included out-of-domain tests to assess performance, our results are constrained to the specific domains used in our study