Knowledge-Driven Cross-Document Relation Extraction (2024)

Abstract: Relation extraction (RE) is a well-known NLP application often treated as a sentence or document-level task. However, a handful of recent efforts explore it across documents or in the cross-document setting (CrossDocRE). This is distinct from the single document case because different documents often focus on disparate themes, while text within a document tends to have a single goal. Current CrossDocRE efforts do not consider domain knowledge, which are often assumed to be known to the reader when documents are authored. Here, we propose a novel approach, KXDocRE, that embed domain knowledge of entities with input text for cross-document RE. Our proposed framework has three main benefits over baselines: 1) it incorporates domain knowledge of entities along with documents’ text; 2) it offers interpretability by producing explanatory text for predicted relations between entities 3) it improves performance over the prior methods. Code and models are available at https://github.com/kracr/ cross-doc-relation-extraction.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 3787–3797 August 11-16, 2024 ©2024 Association for Computational Linguistics
Relation extraction (RE) is a well-known NLP application often treated as a sentence or document-level task. However, a handful of recent efforts explore it across documents or in the cross-document setting (CrossDocRE). This is distinct from the single document case because different documents often focus on disparate themes, while text within a document tends to have a single goal. Current CrossDocRE efforts do not consider domain knowledge, which are often assumed to be known to the reader when documents are authored. Here, we propose a novel approach, KXDocRE, that embed domain knowledge of entities with input text for cross-document RE. Our proposed framework has three main benefits over baselines: 1) it incorporates domain knowledge of entities along with documents’ text; 2) it offers interpretability by producing explanatory text for predicted relations between entities 3) it improves performance over the prior methods. Code and models are available at https://github.com/kracr/ cross-doc-relation-extraction.

heading: 1 Introduction
text: Identifying relations between entity pairs within unstructured text is known as relation extraction (RE). Earlier studies concentrated on uncovering relations between entities within a sentence (Zhang et al., 2017; Hsu et al., 2022; Zeng et al., 2014; dos Santos et al., 2015; Cai et al., 2016; Jain et al., 2023). However, an entity pair need not necessarily occur in a sentence. It can also be part of a document. There is substantial literature on extracting relations between entities based on the document itself (Liu et al., 2023; Xu et al., 2020; Christopoulou et al., 2019; Nan et al., 2020; Zeng et al., 2020; Li et al., 2021; Jain et al., 2024). Beyond a single sentence or a document, entity pairs can also be present in different documents (Yao et al., 2021).
An analysis on Wikipedia shows that more than 57.6 relational facts do not co-occur in a single document (Vrandečić and Krötzsch, 2014). This indicates that many facts occur across the documents. Considering that, little work has been performed in cross-document relation extraction (CrossDocRE). CodRED (cross-document relation extraction dataset) is the first dataset published in this line of work (Yao et al., 2021), which serves as a starting point to solve CrossDocRE. Documents containing the source entity are identified and retrieved from multiple documents; the same is done for the target entity. Various text paths be-
3787
tween entity pairs (source and target entity) across documents are recognized using the mentioned entity (other than the source and target entity). Text paths refer to the paths that connect the source and target entities via the mentioned entities. These paths are retrieved from both the source and target documents.
Figure 1 shows an example of CrossDocRE. Between entity pairs GCompris and GNU Project, multiple text paths via the mentioned entities, such as Linux, Qt and GNU, can be used to get the correct relation label as developer. CodRED uses these text paths in a bag and performs reasoning (Yao et al., 2021). Although this approach seems reasonable, along with relevant text, these text paths also contain noisy data. To overcome these issues, ECRIM (entity-centered crossdocument relation extraction) proposed filtering text paths using a mentioned entity (Wang et al., 2022). However, ECRIM only works in one of the settings of CrossDocRE where text paths are provided for reasoning. To address this issues, MR.COD has been proposed, which is a multi-hop reasoning framework based on path mining and ranking (Lu et al., 2023). These models rely on the knowledge between entities in a text and do not consider the domain knowledge associated with entities. Past work along these lines uses entity types and entity aliases to predict the relation (FernàndezCañellas et al., 2020). RECON (Bastos et al., 2021) encoded attribute and relation triples in the Knowledge Graph. KB-Both (Wu and He, 2019) uses entity details from hyperlinked text documents of Wikipedia and Knowledge Graph (KG) from Wikidata to enhance performance. Given input text as a sentence or a document, these models use basic details of entities to improve the performance. In this paper, we explore whether incorporating domain knowledge can enhance the performance of CrossDocRE tasks. The main contributions of this work are as follows.
• We propose a novel model that integrates domain knowledge with a cross-document relation extraction model.
• Our validation demonstrates the effectiveness of our model, with nontrivial performance gains in cross-document RE.
• We enhance the predicted relationships through textual explanations, offering insights into how the relations were predicted.

heading: 2 Related work
text: The relation between entities can extend across multiple documents, and researchers have investigated the extraction of entities, events, and relationships in a cross-document context with unlabeled data (Yao et al., 2010). CodRED (Yao et al., 2021) is the first open source human-annotated cross-document dataset. In this work, the authors address the problem using two approaches. The first method involves a pipeline approach in which they construct a relational graph for each document and reason over these graphs to extract the desired relation. The second method, referred as the joint approach, combines various text path representations through a selective attention mechanism to predict relations. Although this method is effective, it does not consider the mentioned entity-based sentences. ECRIM uses an entity-based document context filter to retain useful information in the given documents by using the mentioned entities in the text paths. Secondly, they solve CrossDocRE using cross-path entity relation attention, allowing entity relations across text paths to interact with each other (Wang et al., 2022). Nevertheless, this work focuses on a closed setting where evidential context has been given instead of all documents. A multi-hop evidence retrieval method based on evidence path mining and ranking has been proposed in MR.COD (Lu et al., 2023). In evidence path mining, a multi-document passage graph is constructed, where passages are linked by edges corresponding to shared entities. A graph traversal algorithm mines the passage paths from source-to-target entities. In evidence path ranking, paths are ranked based on relevance and top-K evidence paths are selected as input for downstream relation extraction models. Alternatively, a causality-guided global reasoning algorithm is also used to filter confusing information and achieve global reasoning to solve cross-document relation extraction (Wu et al., 2023). Proposed models for CrossDocRE until now does not consider background knowledge.

heading: 3 Methodology

heading: 3.1 Problem statement
text: For a given entity pair, ⟨es, eo⟩ our task is to predict the target relation rc ∈ R that holds between es and eo within a given corpus of documents Cni=1, where R is the relation set and n is total number of documents. If no relation is inferred, it returns the NA label. Besides es and eo, we introduce the
notion of potential bridge entities, which are other mentioned entities in a document that could act as intermediate concepts that help link es and eo via multi-hop connections. The documents provided in CrossDocRE are also annotated with these mentioned entities. We denote Ei = {e1i , . . . , emi } to be the set of m mentioned entities in the i-th document, i = 1, . . . ,M . CrossDocRE works in two settings, closed and open. In the closed setting, only the related documents are provided to the models, and the relations are inferred from the provided documents. In the more challenging and realistic open setting, the whole corpus of documents is provided, and the model needs to efficiently and precisely retrieve related evidence from the corpus. We now discuss the architecture of KXDocRE (Figure 2) in the subsequent sections.

heading: 3.2 Domain Knowledge
text: We consider three forms of domain knowledge knowledge for cross-document relation extraction (Nadgeri et al., 2021): 1) The type information of the source and target entity; 2) The connecting path between the source and target entities, extracted using Wikidata; and 3) A combination of 1) and 2). We use the term context in the rest of the paper to capture these three forms of knowledge.
Entity type context (EC): Predicting the relationship between two entities relies heavily on their respective entity types. For instance, if both entities belong to the Person category, certain relationships such as has organization and has location are not viable. However, relationships such as child and spouse become possible. Therefore, incorporating entity type information can assist the model in excluding obvious relationships, thereby improving its performance. Consider the example given in Figure 3. The type of the source entity Jim Lynagh is Person and the type of the target entity Irish Republic, is Geo Political Entity, which is used to identify locations, countries, cities, or geopolitical regions. The entity type context for this example is, EC{Q6196505, Q1140152}={Person, GeoPoliticalEntity}
Connecting path context (CC): The contextual path pertains to an entity pair ⟨es, eo⟩. We consider context paths of length up to Nh, the hop distance (a tunable parameter) between the entity pair. In Figure 3, the entity Jim Lynagh is five hops away from the entity Irish Republic. The four nodes between the entity pair on the path are intermediary entities, and the five edges are the intermediary properties. The contextual path (CPei,ej ) is formed using the intermediary enti-
ties and properties. So, the connecting path context, CC{Q6196505,Q1140152}={instance of, Human, model item, Douglas Adams, country of citizenship, United Kingdom, replaces, United Kingdom of Great Britain and Ireland, followed by}.
Entity type with connecting path context (ECC): ECC combines the entity type and the connecting path context. For Figure 3, ECC{Q6196505,Q1140152} = EC{Q6196505,Q1140152} ∪ CC{Q6196505,Q1140152}.
The steps to generate EC, CC, and ECC for a given entity pair are outlined in Algorithm 1. The ContextGeneration function (lines 3-12) computes the EC, CC, and ECC for the given entity pair. CC is constructed in steps of one hop. In lines 14-17, the adjacent edge and node information is retrieved and added to the path. This continues for Nh hops (lines 18-20). The EC of the source and the target entity is obtained using the named entity recognition technique (lines 27-35).

heading: 3.3 Entity-based filter
text: In this step, we select sentences that contain meaningful connecting information about the source and target entity and remove irrelevant sentences. The sentences are filtered using the mentioned entity, em. Similar to the baseline (Wang et al., 2022), our underlying premise is that a sentence holds significance if it contains either a source or a target entity. Additionally, a sentence is relatively significant if it mentions another entity in conjunction with the entity pair found in both documents. We calculate each entity’s score based on three conditions. 1) The source and target entities co-occur in some sentence with em (Θ1), 2) An entity, eo, co-occurring with em, also co-occurs with the source entity in a sentence and the target entity in a different sentence (Θ2), 3) em is part of a text path connecting source and target entities (Θ3). Figure 4 depicts these conditions using the example from Figure 1. The red color represents direct occurrence with the source or target entity in the same sentence, the
Algorithm 1: ContextGeneration Input : Entity pair (vs,vo), Number of hops (Nh) Output : Context
1 Initialization: i← 1, source← vs, pathi−1, finalcontext, finalentitytype, adjacent_node, hop_pathi← None 2 Context← ContextGeneration(vs, vo, Nh) 3 Function ContextGeneration(vs, vo, Nh): 4 contextpath, entitytype← {} 5 foreach entity pair vs, vo ∈ KnowledgeBase do 6 contextpath.append(ExplorePath(vs, vo, Nh)) 7 end 8 foreach entity pair vs, vo do 9 entitytype.append(ExploreEntityType(vs,
vo) 10 end 11 finalcontext← contextpath ∪ entitytype 12 return {finalcontext} 13 Function ExplorePath(vs, vo, Nh): 14 edgei, adjacent nodei = GetOneHopFromSource(vs) 15 pathi = {edgei, adjacent nodei} 16 i← 1 17 while vo ̸= adjacent node and i ≤ Nh do 18 pathi = pathi ∪ GetOneHopFromSource(adjacent node) 19 i← i + 1 20 end 21 if adjacent node == vo then 22 return {pathi} 23 end 24 else 25 return {} 26 end 27 Function ExploreEntityType(vs, vo): 28 entity_types, entity_typeo← {} 29 if EntityTypeExist(vs) then 30 entity_types← GetEntityType(vs) 31 end 32 if EntityTypeExist(vo) then 33 entity_typeo← GetEntityType(vo) 34 end 35 finalentitytype← entity_types ∪ entity_typeo
return {finalentitytype}
black line represents indirect co-occurrence, and the green line represents potential co-occurrence.
For a mentioned entity em in each text path pi, the score of each mentioned entity em is calculated by the following equation,
score(em) = λS1(em) + ηS2(em) + κS3(em)
S1(e m) = { 1, if Θ1(em) 0, otherwise
S2(e m) = { |{eo | (Θ1(eo) ∧ I(eo)) = 1}| , if Θ2(em) 0, otherwise
S3(e m) = { |{pi | em ∈ Emi }| , if Θ3(em) 0, otherwise
where λ, η, κ are hyper parameters and I(eo) = 1 when eo and em are co-occuring in the same
sentence, where eo ∈ Emi \{em}. S2 sums the number of occurrences of eo and S3 sums the occurrence of mentioned entity in path pi. Next, we calculate the importance score Imps of each sentence by aggregating the scores of each mentioned entity, Imps=
∑ em∈Ems score(em), where Ems is men-
tioned entity (bridge) in sentence s. The sentences are then ranked based on their importance score, and the top K sentences form the candidate set, S = {s1, s2, ..., sK}, where K is set to 16 based on the experiment results. We reused the entity based filter from ECRIM model (Wang et al., 2022).

heading: 3.4 Relevance-based filter
text: We aggregate the candidate set sentences from the previous step with the context selected using entity type with the connecting context. Let Stotal = S ∪ {SECC}, where SECC is the string based domain knowledge obtained from ECC. We then apply a relevance-based filter that considers the semantic relevance of the sentence. Here, we assume that if a sentence is semantically similar to another sentence, including the target entity, this sentence should be more informative than other sentences. Our objective is to get the most informative context I∗ from the candidate set Stotal for reasoning about the relation between entities.

heading: 3.5 Encoder
text: We mark the start and end of every entity and context using special tokens in sentences. Following the baseline (Yao et al., 2021), we have used the BERT model (Devlin et al., 2019) to encode the tokens selected from the previous step. Here, start and end are the start and end positions of the j-th mention, ej is the j-th mentioned entity, and n is the total number of sentences.
ej = BERT ({(Stotalni=1})endstart

heading: 3.6 Relation matrix
text: Besides the mentioned entity, common relations also exist across various text paths. To capture these details from the text path, we used the crosspath entity relation attention module based on Transformer (Vaswani et al., 2017). We collect all entity mentioned representation in a bag and then generate relation representations for entity pairs (eu, ev). Here, Er, Eu, Ev are learnable parameters and eu, ev are combinations of all entities present, including the mentioned entity.
ru,v = ReLU (Er (ReLU (Eueu +Evev)))
To model the relation interaction across paths, we build a relation matrix M ∈ R|E|×|E|×d, where E = ⋃N i=1Ei denotes all the entities in the entity set Ei of text path pi and Ei = {es, eo} ∪ Emi and N is total number of entities.

heading: 3.7 Transformer
text: For capturing the intra and inter path dependencies, we apply a multi-layer Transformer to perform self attention on the flattened relation matrix M̂ ∈ R|E|2×d
M̂ (t+1|1) = Transformer(M̂ (t) )
We obtain the target relation representation rhi,ti for each path pi from the last layer of the Transformer.

heading: 3.8 Classifier
text: After getting the relation representation rhi,ti for each text path pi for each pair of entities, rhi,ti is used as a classification feature. We feed these features to the MLP classifier to get the score for each relation. The relation that gets the maximum score is the predicted relation.
ŷi = MLP (rhi,ti)
A max pooling operation is applied to each relation label to obtain the final score for each relation type r.
ŷ(r) = Max { ŷ (r) i }N i=1
After obtaining the scores for all relations, a global threshold θ is applied to filter out the categories lower than the threshold. A additional threshold is introduced by baseline paper (Wang et al., 2022) to control which class should be output. The scores of target classes should be greater than threshold and scores of non target class are less than threshold. Formally, for each Bag B, loss is defined as:
L = log  eθ + ∑
r∈ΩBneg
eŷ (r)
 
+ log  e−θ + ∑
r∈ΩBpos
eŷ (r)
 
Here, ŷr is score for relation r , θ represent threshold and is set to zero, ΩBpos and Ω B neg are positive and negative classes between target entity pair.

heading: 3.9 Explanation
text: To improve the interpretability of our model, we incorporated an explainable module that can explain the predicted relationship by providing the filtered sentences given to the model. This represents a notable advancement, as contemporary state-ofthe-art models cannot often furnish explanations alongside their predictions. In CrossDocRE, the most challenging issue lies in the length of the documents because it affects the amount of noise the model has to handle. So having an explanation module also helps in getting to know how well the model is able to handle noise. Along with that, it also helps in facilitating error analysis. We retrieve the tokens that are fed to the model (I∗), converting them into strings. This way, we get the exact token data that was used to make the relation prediction. This process enables us to obtain the precise token data, which drives predictions.

heading: 4 Experimental setup
text: Metrics. We used F1 and area under curve (AUC) scores following the baseline (Yao et al., 2021) for a fair comparison. For the open setting, we retrieve the top k document paths from the Wikipedia corpus1 and use the models trained in the closed setting to predict a relation. These paths are scored using three conditions. 1) Entity count: the number of occurrences of the source entity in the document containing the source and likewise with the target entity. 2) Shared entity: the number of shared entities that appear in source and target documents 3) TF-IDF: The TF-IDF similarity between the two documents. Similar to baseline, we selected the top 16 paths with the highest scores. Hyper-parameters used in our model are shown in Table 1.
Baseline model for comparison. We used all the baseline models available in CrossDocRE for comparison. 1) CodRED (Yao et al., 2021) is
1We follow the baseline paper and set k to 16 (Yao et al., 2021)
used, which extracts text snippets surrounding the two entities in the document as input and feeds it into a BERT-based model 2) ECRIM (Wang et al., 2022); an entity-based document filter is constructed and then fed into a BERT-based model. 3) MR.COD (Lu et al., 2022) is a multi-hop evidence retrieval method based on evidence path mining and ranking. 4) LGCR (Wu et al., 2023) discusses local to global reasoning method which enables efficient distinguishing, filtering and global reasoning on complex information from a causal perspective.
We conducted our evaluation in response to the following research questions. RQ1: What is the effectiveness of KXDocRE in combining context knowledge with reasoning in solving CrossDocRE? RQ2: Does the explanation generated by our approach provide sufficient grounds to support the inferred relation?
Dataset. We used the CodRED dataset for evaluation. CodRED contains 11,971 entities and 276 relation types in this dataset. The details of CoDRED dataset are given in Table 2. Bags represent relational facts of dataset, 2733 are positive labeled facts and 16,668 are labeled as NA in training set.

heading: 5 Results
text: Our model works in the open and closed settings of CrossDocRE. We evaluated our model with three variations: using entity context (EC), connecting context (CC), and both (ECC). We used an NVIDIA A100-SXM4 tensor core GPU with 40GB of memory on Linux 5.4.0-125 with Python version 3.8.5. The results for closed and open settings are available in Table 3 and Table 4. The results on the test
set were obtained from CodaLab2. In contrast to a closed setting, performance declines in an open setting due to the retrieval of paths, not all of which significantly contribute to reasoning. Compared to the baseline model, KXDocRE improves by ≈ 3% in the F1 score (closed setting) and ≈ 4% in the F1 score (open setting). In all settings of KXDocRE, it was able to reason over text better than baseline. Our findings demonstrated the significant role domain knowledge can play in the process of reasoning. Hence, our to answer RQ1 is affirmative. In addition to BERT, we did attempt using RoBERTa3 and GPT24 in our evaluations. However, there was no improvement in the F1 score. Hence, we decided to report BERT scores.

heading: 5.1 Ablation Study
text: Effectiveness of relevance and entity-based filter: We studied the impact of relevance and entity-based filters on the performance of KXDocRE. Figure 5 shows the F1 score obtained
2https://codalab.lisn.upsaclay.fr/ competitions/3770
3https://huggingface.co/docs/transformers/en/ model_doc/roberta
4https://huggingface.co/docs/transformers/en/ model_doc/gpt2
using the Dev dataset. After removing the relevance filter, the performance dropped by 7.7% and if the entity-based filter is removed, the performance dropped by 4.9%. After removing both filters, the F1 score drops significantly by 12%. This indicates that the two filters play an important role in KXDocRE.
Effectiveness of explanations: In crossdocument setting, predicting the relationship between entity pairs involves considering multiple text paths (averaging 3646) found across various documents. The explanation module aids in surfacing the most relevant paths that influence the prediction. This module is expected to help us to understand the prediction using a single paragraph containing the relevant paths, rather than scrutinizing each document individually. We discuss the explanation module using a case study (Figure 6). The labeled relation between Oichi (Q635214) and Ohatsu (Q1050395) is Child (P40). The explanation text provides text paths between Oichu and Ohatsu via Azai, Toyotomi and Yodo-dono (Oichi is the spouse of Azai Nagamasa, who is the child of Yodo-dono, who is the sibling of Ohatsu); hence, Oichi is the child of Ohatshu. Therefore, the explanation text provides enough evidence along with the context to reason over the data to understand the predicted relationship. Hence, we answer RQ2.
Effectiveness of number of hops on KXDocRE: We also studied the impact of the number of hops considered for extracting context in a CC setting on the F1 score (Figure 7). The F1 score increases with the number of hops until a point,
and after that, the F1 score starts decreasing and saturates. Increasing the hops do not add much new and relevant information after a certain point. Due to this reason, we considered the number of hops (Nh) up to 5.
Error analysis: We studied the successful and failed cases of KXDocRE based on the domain knowledge of the entity pair. From Table 5, we can say that the likelihood of a correct prediction is higher if context is available for the given entity pair.
Complexity analysis: In the CodRED dataset, the text path consists of 129,548 instances in the training set, 40,740 instances in the development set, and 40,524 instances in the test set. The average time of a single epoch’s execution in the baseline and our model is given in Table 6. The execution time for KXDocRE is longer (linear increment) due to the addition of context in the module. The time taken to create domain knowledge is available in Table 7.
Case study: We discuss two successful cases and one failed case of KXDocRE and compared them with the baseline model (Table 8). Case1: To identify the relation between Dreamlover (Q909801) and If it’s Over (Q1095958) from Documents 1 and 2, we use domain knowledge of both entities. EC does not exist for the given entity pair, hence we use CC. CC for both entities is {part of, Emotions, tracklist, If It’s Over, followed by}. Adding CC helps KXDocRE to predict the correct relation compared to our baseline model, ECRIM. Case2: In second case study, for entity pairs Airbus A320neo family (Q6488) and Airbus (Q67), we add the ECC context as {owned by, ORG, ORG}, which helps KXDocRE help in predicting the relation. Case3: We studied a failed case of KXDocRE for entity pairs Adium (Q58058) and x86_64 (Q272629). CC for this entity pair is {instance of, free software, subclass of, software, model item, Mozilla Firefox model item}. This context does not contribute significantly in predicting the relationship.

heading: 6 Conclusion
text: In this paper, we introduced a novel model KXDocRE with three settings (EC, CC, ECC) that incorporate domain knowledge in CrossDocRE. Our results suggest that model performance is improved by integrating diverse domain knowledge. Also, we provide an explanation text for each prediction that makes our model interpretable. As a future work, researchers can create a versatile model that can work on diverse document sets, accumulating domain knowledge.

heading: 7 Limitations
text: Our work has limitations in terms of creating a connecting context — such a context will only be created if there is some path between two entities in Wikidata. We plan to overcome this limitation by incorporating other external knowledge bases for creating context. With the increase in the number of text paths and mentioned entities, the GPU memory consumption increases, and the speed decreases.

heading: 8 Acknowledgement
text: We express our sincere gratitude to the Infosys Centre for Artificial Intelligence (CAI) at IIIT-Delhi for their support. RK’s effort has been supported by the U.S. National Library of Medicine (through grant R01LM013240.)

heading: Future Work
text: As a future work, researchers can create a versatile model that can work on diverse document sets, accumulating domain knowledge