AREEj: Arabic Relation Extraction with Evidence (2024)

Abstract: Relational entity extraction is key in building knowledge graphs. A relational entity has a source, a tail and a type. In this paper, we consider Arabic text and introduce evidence enrichment which intuitively informs models for better predictions. Relational evidence is an expression in the text that explains how sources and targets relate. This paper augments the existing SRED relational extraction dataset with evidence annotation to its 2.9-million Arabic relations. We leverage the augmented dataset to build AREEj, a relation extraction with evidence model from Arabic documents. The evidence augmentation model we constructed to complete the dataset achieved .82 F1-score (.93 precision, .73 recall). The target AREEj outperformed SOTA mREBEL with .72 F1-score (.78 precision, .66 recall).

heading: 1 Introduction
text: We define relational extraction with evidence (REE) as the task of extracting related entities (source and target), identifying the relation type between them, and providing evidence from text to support the relations. Relation Extracting (RE) is an important task to build knowledge graphs. Enriching knowledge graph edges with evidence labels helps in several ways: (i) it supports explainable AI tasks potentially by large language models (Pruthi et al., 2020), and (ii) it improves the performance of the RE models as evidence plays the role of a hint for an existing relation.
Evidence is important as it helps understanding Machine Learning (ML) decisions (Pruthi et al., 2020) and provides decision-makers with more confidence in ML model predictions. In turn, detecting evidence boosts the performance of ML models as it is a step in a chain-of-thought to infer the relation and its type (Wei et al., 2022).
A relational entity with evidence ree = ⟨s, t, sne, tne, rt, e⟩ where s is the source, t is the
target, sne and rne are named entity types of the source and the target, rt is the relation type, and e is the evidence. The elements s, t, and e are extracted from the text itself, and the rest are predefined categories and classes. One document may contain multiple REEs and REEs may have elements in common.
Figure 1 shows a sentence with four relations where the main entity s1 entity is connected as source with three other entities: t1 inception date (2010), t2 headquarters location (Doha), and t3 country (Qatar). Doha is also a source s2 = t2 connected to t3 indicating it is in Qatar. The evidence shows between brackets explaining why the the model reported these relations and their types.
Researchers developed systems capable of extracting relations from Arabic text including a morphology-based and regular expression based approach (Jaber and Zaraket, 2017), and a crosslanguage learning approach (Taghizadeh et al., 2018) that leveraged knowledge transfer for the lack of large Arabic relational datasets.
Research leveraged Wikipedia data to build a dataset with 19 relation types (Zakria et al., 2019). Multilingual data from Wikipedia was also utilized to construct a larger dataset, across 14 languages and with 36 relation types (Seganti et al., 2021), where at most one relation was assigned per
67
text. BERT (Devlin et al., 2019) was used to train two models:
• A sequence classification that classifies the type of the relation in the text, and • An entity extraction that identifies the source and target entity of the relation.
The claim is that multilingual models achieve better results than single language models. The Arabic subset of the data contains 9,000 rows with 9 unique relation types.
Later work presents SREDFM with 40-Million relations in 18 different languages to be the largest RE dataset up to our knowledge. Its Arabic subset covers 393 out of a total of 400 relation types with 2.9-Million relations. SREDFM is extracted from Wikipedia using the cRocoDiLe tool (Huguet Cabot and Navigli, 2021). The mREBEL model was trained on SREDFM to perform relational extraction (Huguet Cabot et al., 2023).
This paper presents work that augments the Arabic subset of SREDFM with relational evidence, and then to use the resulting dataset to train AREEj, a relational extraction with evidence model. We perform the augmentation via finetuning an open-source large language model to extract evidence from the Arabic text given a relational sequence. The evidence augmentation task performed well with .82 F1-score, .93 precision and .73 recall AREEj extracted relation with evidence with .72 F1-score, .78 precision and .66 recall. We make the annotated dataset available online for the research community.

heading: 2 Related Work
text: RE started as a cascaded task where it first classifies source and target entities, and then identifies relation types between them. Recent research approached RE as a sequence-to-sequence (Seq2Seq) transformation. The raw text constitutes the input sequence, and the relation constitutes the output sequence.
REBEL extracts relational entities from English text excluding evidence (Huguet Cabot and Navigli, 2021). It is a seq2seq Bidirectional and Auto-Regressive Transformers (BART) model (Lewis et al., 2020) trained on linearized relations extracted automatically from Wikipedia using cRocoDiLe.
The mREBEL tool (Huguet Cabot et al., 2023) is trained and tested on the (SREDFM ) and
(REDFM ) multilingual RE datasets extracted from Wikipedia data. Each relation is specified as a tuple wit source, target, entity types for source and target, and relation type. The training dataset was automatically generated, while the testing dataset was manually revised. mBART (Liu et al., 2020), which is typically used for translation, was used to map and normalize the relational types across all included 18 languages.
The work in (El Khbir et al., 2022) extracts named entities, relation types, and event triggers jointly. It covers 7 named entity types, 6 relation types, and 8 event types. The data is encoded by concatenating the last and third last BERT embeddings and extracting entities and events to identify the spans.
A rule-based model extracts the source, target, and relation spans depending on Part-ofSpeech tags (Saber et al., 2022). The Stanford Arabic Parser (Green and Manning, 2010) and WordNet (Miller, 1994) were leveraged to extract triplets, which works well for sentences with simple structures.
In (Huang et al., 2021), an approach was introduced to extract relations from documents and use attention to extract evidence. Documents are multiple sentences and the evidence is several sentences that support a detected relation. This is different from our target evidence as we target finegrained relations at the sentence level. We also extract the relation and the evidence simultaneously.
The work in (Ma et al., 2023) improves on memory efficiency and lack of annotations. Our work solves the annotations problem by using ChatGPT-4 to annotate the data with evidence, fine-tune an open-source LLM using the generated data, and then use the fine-tuned LLM to annotate 2.9 million relations with evidence.
To do that we leverage the chain of thought (CoT) technique in our prompts (Wei et al., 2022) where requiring intermediate steps in inference improves the quality of the final result and avoids inaccuracies and hallucinations.
The work in (Wadhwa et al., 2023) compares several LLM approaches for RE. This includes GPT-3 few-shot learning, a fine-tuned Flan-T5, and improved versions of both with CoT. The results recommend utilizing LLMs with CoT for RE tasks.

heading: 3 Relational Data with Evidence
text: We constructed ArSRED via augmenting the 2.9- Million relations Arabic subset of SREDFM with evidence as illustrated in Figure 2. We did this in several steps.
Construct representative documents and relations subset (RDRS). We selected a subset of Arabic SREDFM to be augmented with seed evidence annotations. Intuitively, we targeted extraction of relations from smaller text to boost the GPT context performance and limit the number of tokens used later on with ChatGPT-4. We selected the documents with length between five and 80 words inclusive. This happened to be around 89% of the 2.9 million documents.
Then we split the set based on the relation type and the source and target entity types resulting in 2,526 bins that represent all typology combinations present in the set. Then we selected a representative sample (20 relations) from each bin starting from the smaller bins. We ended up with 68,183 relations selected from their corresponding 13,180 documents. Note that the total number of relations is higher than 20 ∗ 2, 526 as when we include a document, we also keep all its relations and not only the ones from the selecting bin.
Construct Base Evidence Annotations. We used ChatGPT-4 to find evidence for the RDRS relations. We optimized ChatGPT-4 performance with a CoT based prompt which improved its per-
formance.
We tried several prompts with a variety of examples. The final prompt consisted of two examples and four relations. We provided an explanation of the intermediate steps to get the evidence for each relation. We passed only one document per explanation. The output is the text of the evidence or “none”.
Documents with plenty of relations resulted in less evidence elements returned. So we added the specific number of elements to the prompt. This partially fixed the problem. This augmented 55,078 RDRS relations with evidence and the remaining 13,105 relations with “none’ forming the base evidence annotations.
We repeated the same process with the whole Arabic subset of the testing REDFM and humanly revised the results for 345 documents, and 864 relations with 32 relation types. We consider this as our testing set.
Fine-tune LLM for evidence annotations. We used the base evidence annotations to finetune ArBartEv from the mBART (Liu et al., 2020) model. ArBartEv transforms an Arabic input text with a relation tuple, into an evidence of the relation in a sequence-to-sequence manner.
ArSRED Annotation with evidence. We passed the whole 499,638 documents with 2.9 million relations Arabic subset of SREDFM to ArBartEv. to obtain ArSRED set augmented with evidence. A small 6% of the predictions came with a corrupted output sequence structure. We noticed upon inspection that these are mostly large documents, and we marked the evidence as “none”.
The following is an example of how ArBartEv takes a sentence with a relation and produces a relation with evidence to construct the training dataset.
Input:
Q Ôg B@ ÐQ êË @ Õæ A K. A @ ¬ðQª ÖÏ @ , ÐQ êË @ @ Yë è Q m.Ì'@ ú ̄ Pñ ëX ú ̄ © ®K ÐQë Q . » @ ñë , ú ÍAÒ
Ë@ . . . Qå Ó ú ̄
s: Q Ôg B@ ÐQ êË @ ,location; t: Pñ ëX ,loca-
tion;rt:location
Output:
s: Q Ôg B@ ÐQ êË @ ,location; t: Pñ ëX ,loca-
tion;rt:location; e:ú ̄ © ®K

heading: 4 Training ArBartEv and AREEj
text: ArBartEv requires relations to produce evidence. ArBartEv takes x = ⟨t, r⟩ composed of text t and relation r and produces y = ⟨r, e⟩ where e is the evidence.
p(y = r, e|x) = len(y)∏
i=1
pmBART(yi|y<i, x) (1)
This produced dataset ArSRED as discussed in Section 3. The intuition behind including r in both input and output is that transformers tend to work better when asked to perform tasks in steps and with hints.
We trained AREEj based on the mBART seed model and ArSRED to take an Arabic text t and produce a relation r with evidence e.
p(y = r, e|t) = len(y)∏
i=1
pmBART(yi|y<i, t) (2)
We target only Arabic text so we limited the tokenizer to Arabic and kept the named and relation entity types as special tokens as specified in REBEL and mREBEL.
Model fine-tuning was executed on 1x NVIDIA RTX A6000 48 GB VRAM GPU. The hyperparameters used are .00005 learning rate, 5000 warm-up steps, 16 batch size, and 156140 max steps. The rest are the default hyperparameters of mBART.

heading: 5 Results and Evaluation
text: For evidence equivalence for ArBartEv, we computed the number of word-based intersections between its evidence results and those produced by ChatGPT-4 on REDFM. For entries with no intersection, we count that as a miss. For entries with intersections, we performed a manual check and a human expert deemed the extracted evidence as true or not. ArBartEv achieved an F1-score of .82 on evidence extraction (P=.93,R=.73).
For AREEj, we performed a beam search at the output level and obtained six predictions. Note that these predictions may have relations that share relational elements. We noticed that some extracted relations were correct as evaluated by a human expert, yet the testing set missed them. AREEj correctly identifies 72 relations that mREBEL misses, while mREBEL identifies 47 relations AREEj misses. AREEj achieves .72 F1score (p=.78, r=.66) with correct evidence .86% of the time as detailed in the Extarction row of Table 1. AREEj was trained on 2.9-Million relations with evidence annotations while mREBEL was trained on 40-Million relations in 18 languages without evidence annotations. Table 1 also shows that AREEj outperforms mREBEL in relation classification which measures whether any relation (regardless of type) exists between two detected entities.
Fourteen relation types are detected by AREEj but not by mREBEL. Examples include ’employer’, ’use’, ’member of’, ’industry’, ’main subject’, and ’date of death’. Four relation types are detected by mREBEL and not by AREEj including ’characters’, and ’member of sports team’. Lower rows of Table 1 compare AREEj and mREBEL on selected relation types from ArSRED.

heading: 6 Conclusion
text: Adding evidence annotations resulted in extracting more relations and relation types. Recall increased significantly and precision suffered slightly. In future work, we plan to use the extracted evidence to improve precision via relating to relation and entity types and reducing the “none” samples to improve recall.

heading: 7 Limitations
text: The testing dataset contained only 32 relation types. This is an initial limitation of REDFM. Relations in text exist that are not reported in REDFM
requiring manual checks. Augmenting REDFM with instances covering the 393 relations should be considered in future work.

heading: Future Work
text: Augmenting REDFM with instances covering the 393 relations should be considered in future work