Towards Better Graph-based Cross-document Relation Extraction via Non-bridge Entity Enhancement and Prediction Debiasing (2024)

Abstract: Cross-document Relation Extraction aims to predict the relation between target entities located in different documents. In this regard, the dominant models commonly retain useful information for relation prediction via bridge entities, which allows the model to elaborately capture the intrinsic interdependence between target entities. However, these studies ignore the non-bridge entities, each of which co-occurs with only one target entity and offers the semantic association between target entities for relation prediction. Besides, the commonly-used dataset–CodRED contains substantial NA instances, leading to the prediction bias during inference. To address these issues, in this paper, we propose a novel graphbased cross-document RE model with nonbridge entity enhancement and prediction debiasing. Specifically, we use a unified entity graph to integrate numerous non-bridge entities with target entities and bridge entities, modeling various associations between them, and then use a graph recurrent network to encode this graph. Finally, we introduce a novel debiasing strategy to calibrate the original prediction distribution. Experimental results on the closed and open settings show that our model significantly outperforms all baselines, including the GPT-3.5-turbo and InstructUIE, achieving state-of-the-art performance. Particularly, our model obtains 66.23% and 55.87% AUC points in the official leaderboard1 under the two settings, respectively, ranking the first place in all submissions since December 2023. Our code is available at https: //github.com/DeepLearnXMU/CoRE-NEPD.

heading: 1 Introduction
text: Relation Extraction (RE) is a fundamental natural language processing (NLP) task, which aims to predict the relationship between two entities in
∗*Corresponding author. 1https://codalab.lisn.upsaclay.fr/
competitions/3770#results
a given context. Usually, conventional RE studies limit the context within a single sentence or document (Zeng et al., 2014; Santos et al., 2015; Cai et al., 2016; Zhang et al., 2018; Wang et al., 2022). However, since a large number of relational facts are not described in the same document (Yao et al., 2021), many researchers have begun to concentrate on cross-document RE (CoRE), where the given target entities (head entity and tail entity) are located in different documents (Yao et al., 2021; Wang et al., 2022; Wu et al., 2023).
In this regard, Yao et al. (2021) first explore this task. They not only release the dataset CodRED, where relevant documents of target entities connected by bridge entities are organized as text paths, but also propose BERT-based models for this task. However, their models suffer from the negative effect of irrelevant context in the model input and do not fully leverage the connections across text paths. To solve these issues, Wang et al. (2022) propose an Entity-based Cross-path Relation Inference Method (Ecrim). They employ an entitycentered noise filter to refine the model input and incorporate a cross-path entity relation attention to capture the connections across different text paths. Unlike the Ecrim, Wu et al. (2023) present a localto-global causal reasoning model (LGCR), which is a graph-based model using a local causality estimation algorithm to filter the noisy information. Despite their success, their models still suffer from two issues.
First, previous methods only consider the target entities and bridge entities, while ignoring the non-bridge entities that solely co-occur with only one target entity. According to the statistics of the CodRED dataset (Yao et al., 2021), we observe that on average each text path contains 18.8 non-target entities: 2.6 bridge entities while 16.2 non-bridge entities that may also help the relation prediction. For example, in the text path 1 of Figure 1, the target entities “Europa Plus” and “Soviet Union”
680
only occur with the non-bridge entities “Russian radio station” and “Russian Soviet Republic”, respectively. Thus, it is difficult to correctly predict their relation due to the absence of bridge entities. If non-target entities “Russian radio station” and “Russian Soviet Republic” are semantically related, then there may also be some relationship between the target entities “Europa Plus” and “Soviet Union”. Therefore, intuitively, such semantic correlation can be exploited to benefit the relation prediction between target entities. Second, 85% of document bags are labeled as NA relation in the CodRED training set. As a result, the training dataset can only offer limited supervision signals for the model to learn non-NA relations, which leads to prediction bias.
To address these issues, in this paper, we propose a novel graph-based CoRE model with non-bridge entity enhancement and prediction debiasing. As illustrated in the right part of Figure 1, we represent each input document bag with a unified entity graph, where entity nodes are initialized with BERT representations. In this graph, each node indicates a target entity, bridge entity, or non-bridge entity, and two types of edges are introduced: 1) cooccurrence edges, each of which connects a target entity and a non-target entity that co-occurs with it. 2) semantic-related edges, which are used to connect any two semantic-related non-target entities. Then, we utilize Graph Recurrent Network (GRN) (Zhang et al., 2018) to encode this graph, where the interdependency among connected nodes is captured based on the recurrent gating mechanism. Afterward, we aggregate the entity representations
learned from GRN and utilize a cross-path entity relation attention module to capture the connections across text paths.
Besides, inspired by the studies on other NLP tasks (Utama et al., 2020; Xiong et al., 2021; Wang et al., 2023a), we propose a simple yet effective prediction debiasing strategy to calibrate the relation prediction. Specifically, we introduce two prediction distributions: 1) yrela. To obtain this distribution, we fix the parameters of the original model and retrain a new classifier only using nonNA instances. Notably, the new classifier can avoid the negative impact of excessive NA instances and achieve better prediction performance on non-NA instances. 2) ybias. We mask the most important non-target entities of our entity graph as input to derive this distribution. Compared with the original prediction distribution, this distribution is more biased and thus can be used for debiasing in a manner of subtraction. Finally, we integrate the two newlyintroduced prediction distributions to calibrate the original ones. To the best of our knowledge, our work is the first attempt to study the prediction debiasing in this task.
To investigate the effectiveness of our model, we conduct extensive experiments on both the closed and open settings of CodRED. Experimental results and in-depth analysis show that our model significantly outperforms all competitive baselines, including the LLMs. Particularly, compared with all submitted results since December 2023, our model ranks the first place in the official leaderboard.

heading: 2 Our Model
text: In this section, we introduce our proposed model in detail. As shown in Figure 2, our model contains three important components: a BERT encoder, a GRN encoder, and a classifier with a cross-path entity relation attention module, and then we detail the prediction debiasing strategy used in inference.

heading: 2.1 BERT Encoder
text: Given a target entity pair, we first obtain relevant documents and conduct data preprocessing2 to filter the noisy information over multiple documents. As a common practice, we use BERT3 to learn the representations of entities, which will provide useful initial information for the subsequent GRN encoding. Concretely, for each entity ei, we first insert a special symbol “*” at the start and end of entity mentions and obtain the mention representations by max-pooling operation. Then we collect all the mentions in a text path and follow Robin et al. (2019) to obtain its path-level representation hei , using hei = log ∑Nei j=1 exp ( hmij ) , where Nei denotes the mention number of ei in the text path, and mij is the representation of its j-th entity mention.

heading: 2.2 GRN Encoder
text: To construct our GRN encoder, we first represent the whole input document bag as a unified entity graph, which facilitates the introduction of nonbridge entities to strengthen the semantic association between target entities. Then, we introduce a GRN to encode the graph, where the learned entity representations will provide information for the subsequent relation prediction.
2The detailed data preprocessing we used can be found in Appendix A
3We analyze the effect of the BERT version in Appendix C

heading: 2.2.1 Entity Graph
text: To facilitate the subsequent description, we take the document bag shown in Figure 1 as an example and describe how to use a unified entity graph to represent an undirected one G=(V,E).
In the node set V , each node is a target entity, a bridge entity, or a non-bridge entity. Apparently, our entity graph considers more information than previous studies. Let us revisit the example in Figure 1, where we first identify the bridge entity “Russian”, the non-bridge entities “Russian radio station” and “Russian Soviet Republic”, and then include them with the target entities “Europa Plus” and “Soviet Union” into the entity graph. Note that we use different nodes to represent the same head entity for different text paths, which facilitates learning the relation between target entities specific to the text path.
To capture various kinds of interdependences between entities for cross-document RE, we consider two kinds of edges in the edge set E: (1) each target entity and each non-target entity (bridge or non-bridge entities) within the same document are connected via a co-occurrence edge; (2) any two semantic-related non-target entities are connected by a semantic-related edge. During this process, we calculate the cosine similarity between the representations of any two non-target entities, and determine that they are semantically related if their similarity is greater than a threshold η. Back to Figure 1, “Europa Plus” and “Russian radio station” are connected by a co-occurrence edge, and “Russian radio station” and “Russian Soviet Republic” are connected by a semantic-related edge.

heading: 2.2.2 Encoding with GRN
text: We then introduce GRN (Zhang et al., 2018) to encode this graph. Typically, it updates node representations using recurrent gating mechanisms,
and thus has been widely used in many NLP tasks (Yin et al., 2019, 2020; Lai et al., 2021). Here, we choose GRU to update node representations since it has fewer parameters and better efficiency.
In Figure 3, we show the procedure of updating node representations in our graph at the t-th timestep. Specifically, for the node of entity ei, we first gather the information from its connected entity nodes of the graph:
c (t) i =
∑
j∈A(ei) e (t−1) j , (1)
where c(t)i is the collected context information used to update the node representation of entity ei, A(ei) represents the set of neighboring nodes of entity ei, e (t−1) j is the node representation of entity ej at the (t−1)-th timestep. Besides, we initialize the entity nodes with their representations learned from BERT encoder: e(0)i =hei , where 1≤i≤|V |. Afterward, we update the node representation of entity ei in the following way:
r (t) i = σ(W rc (t) i + U re (t−1) i ), (2) z (t) i = σ(W zc (t) i + U ze (t−1) i ), (3) u (t) i = tanh(W uc (t) i + U u(r (t) i ⊙ e (t−1) i )), (4) e (t) i = (1− z (t) i )⊙ u (t) i + z (t) i ⊙ e (t−1) i , (5) where r(t)i and z (t) i are reset and update gates, ⊙ is the Hadamard product, u(t)i denotes the temporary node representations of ei at t-th step, W ∗ and U∗ are trainable parameter matrixes.

heading: 2.3 Classifier with Cross-path Entity Relation Attention
text: Following Wang et al. (2022), on the basis of the entity representations learned from GRN, we utilize a cross-path entity relation attention to model
the connections across different text paths, aggregating better relation representations of the target entity pair. Then, we feed these aggregated representations into an MLP classifier for prediction.
Specifically, we first collect all the entity representations learned from the GRN encoder and then construct a entity relation matrix M ∈ R|V |×|V |, with each element ri,j denoting the representation of the relation between the entities ei and ej :
ri,j = ReLU(W (ReLU(W uei +W vej)), (6)
where ei, ej are the representations of entity ei, ej respectively, W ∗ are trainable parameter matrixes.
Then we flatten the entity relation matrix M and perform self-attention on it, to capture the intraand inter-path dependencies. By doing so, we can obtain the representation rk of the relation between the target entities of each text path k. Subsequently, we feed rk into the MLP classifier for prediction:
yk = MLP(rk), (7)
where yk is the predicted relation distribution for text path k.
Finally, to obtain the bag-level prediction y, we follow Wang et al. (2022) to perform a max-pooling operation as follows:
y = Max({yk}Nk=1), (8)
where N denotes the number of text paths in a document bag.

heading: 2.4 Model Inference with Prediction Debiasing
text: Following common practice, we train our model by minimizing the cross-entropy loss of training data. However, as previous analysis, the training data contains numerous target entity pairs with NA relation, which leads to the prediction biasing during inference. To address this issue, we propose a simple yet effective debiasing strategy that introduces two prediction distributions to calibrate the original prediction ones in the following way:
y = y + λ (yrela − ybias), (9)
where λ is a hyper-parameter controlling the effects of newly-introduced prediction distributions. Here, we give detailed descriptions to yrela and ybias:
• yrela. This prediction distribution is used to refine the prediction of the model on the in-
stances with non-NA relations. To obtain this distribution, we stack a new classifier on the original model. During the training process, we fix all parameters of the original model and only tune this classifier using the non-NA instances. Notably, unlike the original classifier, the training of this classifier avoids the negative impact of excessive NA instances. Thus, it can achieve better prediction performance on non-NA instances.
• ybias. Compared with the original prediction distribution, this distribution is more biased and thus can be used to debias the original prediction distribution in a manner of subtraction. Notably, unlike the above yrela, we do not retrain the model or classifier to obtain ybias. To obtain this distribution, we first quantify the importance of each non-target entity with the average weight of target entities attending to the non-target entity. Subsequently, we mask the most important 50% non-target entities and then feed the remaining sub-graph into the GRN to derive ybias. Apparently, due to this sub-graph lacking some important nontarget entities, ybias prefers NA relation and thus is more biased than y.

heading: 3 Experiments

heading: 3.1 Setup
text: Dataset. To evaluate our model, we use the commonly-used dataset–CodRED (Yao et al., 2021) to conduct experiments under two settings: 1) closed setting, where the related documents of target entities are given in advance for constructing the text path. 2) open setting, where we have to first retrieve related documents from Wikipedia and then evaluate the model performance. Table 1 shows the detailed statistics of CodRED. Note that the training data of CodRED contains 16,668 document bags with NA relation and 2,733 document
bags with non-NA relations, where the significant number difference between NA instances and nonNA instances leads to the prediction bias.
Settings. When constructing our model, we set the similarity threshold η for semantic-related edges to 0.6.4 As for the debiasing strategy, we set the mask rate of important non-target entities to 0.5 and λ to 0.1, which will be analyzed in Section 3.3. Besides, we employ a 3-layer GRN to encode our unified entity graph and a 2-layer Transformer for cross-path entity relation attention, where the embedding size and hidden state dimension are both set to 768. To effectively train our model, we employ AdamW with a learning rate of 3e-5. To ensure a fair comparison, we follow Yao et al. (2021) to train the model on the closed setting, where extra document-level data are involved.5
As implemented in previous studies (Yao et al., 2021; Wang et al., 2022), we use four evaluation metrics for the development set: F1, AUC, P@500, and P@1000, and two evaluation metrics for the test set: F1 and AUC. Additionally, we compare our model with the LLMs using micro-F1. Finally, following Yao et al. (2021), we obtain the evaluation scores on the test set by submitting prediction results into Codalab.6

heading: 3.2 Baselines
text: We compare our model with the following baselines:
• End-to-end (Yao et al., 2021). This model employs BERT to obtain the representations of text paths. Then, a selective attention module is used to obtain the aggregated representations of target entities. Finally, the aggregated representations are fed into a classifier to predict the relation between target entities.
• Ecrim (Wang et al., 2022). It is our most important baseline. This model first uses an entity-based document-context filter to retain useful information in the given documents by using the bridge entities in the text paths. Then, it is equipped with a cross-path entity relation attention, which allows entity relations across text paths to interact with each other.
• LGCR (Wu et al., 2023). This model first estimates the causal effect for each semantic
4We analyze the effect of η in Appendix B 5We investigate the effect of extra document-level data in
Section 3.7 6https://codalab.lisn.upsaclay.fr/ competitions/3770
unit (text path, head entity, tail entity, and bridge entity). Then, it constructs a global reasoning graph based on the co-occurrence of entity mentions and the structure of text paths, and uses the relative causal association calculated by local causal effect to control the message propagation ability between nodes.
The number of trainable parameters for our model and the above baselines are 1.30×108, 1.08×108, 1.23× 108, and 1.19× 108, respectively.
To further verify the performance of our model, we compare it with powerful LLMs. In addition to the commonly-used GPT-3.5-turbo (Ouyang et al., 2022), we consider a variant of InstructUIE (Wang et al., 2023b), termed as InstructUIE-FT, which is fine-tuned on CodRED to enhance its cross-document RE ability. Note that InstructUIE is a strong information extraction framework that captures the inter-task dependency to uniformly model various information extraction tasks.
To investigate the few-shot ability of LLMs in classification tasks, the standard practice based on LLMs introduces an exemplar for each candidate label to investigate the few-shot ability of LLMs (Yoo et al., 2022; Dong et al., 2023; Fan et al., 2023). However, in the CoRE task, the number of candidate labels is 276, making it impractical to introduce an exemplar for each label due to the maximum context length of the LLMs. Therefore, we only test the zero-shot performance of LLMs. The detailed settings for these LLMs are shown in Appendix D.

heading: 3.3 Effect of the Adaptive Parameter λ
text: We first investigate the impact of the hyperparameter λ (See Equation 9) on the development set under the closed setting. To this end, we gradually vary λ from 0.05 to 0.25 with an increment of 0.05 in each step.
As shown in Figure 4, we find that our model
achieves the best performance when λ=0.1. Therefore, we set λ=0.1 for all experiments thereafter.

heading: 3.4 Main Results
text: Table 2 shows the experimental results under two settings. Note that the performance of our reproduced LGCR and Ecrim model are comparable to that of their original paper, proving that our experimental comparison is convincing. Overall, under both settings, our model exhibits better performance in most metrics than all baselines, except for P@500. Most importantly, we submit the test results to the official competition leaderboard, where our model obtains 66.23% and 55.87% AUC points under the closed and open settings, respectively, ranking the first place in all submitted results since December 2023.
Moreover, we compare our model with LLMs in Table 3. Under both settings, we can find that our model significantly outperforms the LLMs, including InstructUIE-FT that has also been fine-tuned on CodRED. These results once again demonstrate the effectiveness of our model.

heading: 3.5 Impact of Non-bridge Entity Number
text: To assess the impact of non-bridge entities on our model, we first sort the development set according to the number of non-bridge entities in ascending order, and then equally split it into four subsets. After that, we compare the model performance on these subsets. To clearly display the experimental results, we only compare ours with Ecrim, LGCR and InstructUIE-FT, which are competitive baselines according to the results reported in Table 3.
As shown in Figure 5, regardless of the setting, as the number of non-bridge entities increases, the performance advantage of our model becomes in-
creasingly apparent. These results strongly demonstrate the generality and effectiveness of our model.

heading: 3.6 Ablation Study
text: To investigate the effectiveness of different components of our model, we further compare our model with the following variants in Table 4:
• w/o yrela. In this variant, we only use ybias for prediction debiasing.
• w/o ybias. We only use yrela for prediction debiasing in this variant.
• w/o yrela, ybias. In this variant, we do not calibrate the model prediction.
• w/o NBE. In this variant, we remove all nonbridge entities from our entity graph.
• w/o GRN. In this variant, we remove the GRN Encoder from our model.
• w/o yrela, ybias, NBE(Non-bridge entities). In this variant, we directly remove all non-bridge entities from our entity graph. In other words, we only consider target entities and bridge entities in this variant. Notice that the main difference between this variant and Ecrim is that it adds an additional GRN on the top of BERT encoder.
• w/o yrela, ybias, SRE(Semantic-related edges). When constructing this variant, we remove the debiasing module and all semantic-related edges from our entity graph.
Table 4 lists the experimental results, where, all variants are inferior to our model, verifying the effectiveness of each proposed module.

heading: 3.7 Effect of Extra Document-level Data
text: To investigate the effect of extra document-level data in training, we only use the cross-document data to train various models.
From Table 5, we observe that the performance of various models on the CoRE task sharply decreases, which demonstrates that the extra document-level data can help models to capture more useful information. Moreover, we can observe that our model still significantly outperforms all baselines in this setting, confirming the capability of our model.

heading: 3.8 Case Study
text: Table 9 in the Appendix displays the prediction results of different models on two text paths sampled from the test set. In text path 1, due to the lack of bridge entities, Ecrim is unable to connect the head and tail entities, leading to the incorrect relation prediction. By contrast, our model leverages substantial non-bridge entities of text path 1 to build connections between target entities. Thus, both Ours and w/o yrela, ybias successfully predict the relation of text path 1, demonstrating that non-bridge entities can indeed provide useful supplementary information for the cross-document RE. Meanwhile, in text path 2, both Ecrim and w/o yrela, ybias mistakenly predict the relation between target entities as NA. Only when the prediction debiasing strategy is used, our model can calibrate the prediction, thereby correctly predicting the relation as continent. This is consistent with the results reported in Table 4, further verifying the effectiveness of our strategy.

heading: 4 Related Work
text: Most studies on RE mainly focus on the sentencelevel RE (Cai et al., 2016; Zhang et al., 2018, 2019; Ikuya et al., 2020; Zhang et al., 2023b) and document-level RE (Zeng et al., 2020; Wang et al., 2021; Zhang et al., 2022, 2023a), which are committed to identifying the relation between target entities from a sentence and document, respectively. Unlike these studies, in this work, we concentrate
on cross-document RE that aims at predicting the relation between target entities located in different documents. Yao et al. (2021) comprehensively investigated this task and released the first humanannotated CoRE dataset, CodRED. Besides, they explore an end-to-end model that jointly considers documents in text paths to predict the relation. However, it not only suffers from the negative effect of irrelevant context in text paths but also does not fully leverage the interconnections across text paths. To address the above-mentioned issues, Wang et al. (2022) propose Entity-based Cross-path Relation InferenceMethod (Ecrim). Typically, it uses an attention mechanism to capture the connection among different text paths through bridge entities, which is important to predict the final relation. Recently, Wu et al. (2023) put forward localto-global causal reasoning (LGCR). To aggregate information over multiple text paths, they construct a global heterogeneous graph, where a local causality estimation algorithm is proposed to assess the importance of different nodes in the graph.
However, their methods suffer from two limitations: 1) ignore the non-bridge entities, which exist broadly in each text path and can offer semantic associations between target entities, especially in the absence of bridge entities. 2) ignore the bias caused by the prominent number difference between NA and non-NA instances. In this work, we propose a graph-based model to fully exploit non-bridge entities for cross-document RE. Besides, along the research line of debiasing in NLP (Joshua R et al., 2021; Xiong et al., 2021; Wang et al., 2023a), we propose a simple yet effective prediction debiasing strategy to refine the prediction of our model.

heading: 5 Conclusion and Future work
text: In this paper, we propose a novel graph-based crossdocument RE model. Concretely, we first represent the input bag as a unified entity graph, where abun-
dant non-bridge entities are introduced to provide useful information. Then, we use GRN to encode this graph, where the learned entity representations form the basis for subsequent relationship prediction. Besides, we propose a simple yet effective debiasing strategy to refine the original prediction distribution. To the best of our knowledge, the graph-based model with non-bridge entities and our debiasing strategy has not been explored before. Extensive experiments on the commonly-used dataset CodRED demonstrate the superiority of our model. In the future, we will study how to introduce more external knowledge to refine our model.
Limitations
One limitation of the present work lies in that we only rely on the attention score to measure the importance of nodes in the graph, and do not consider dynamically adjusting the importance score. Additionally, in the entity-based graph, the edges are connected using a heuristic method, which may overlook useful information.

heading: Acknowledgements
text: The project was supported by National Natural Science Foundation of China (No. 62276219), and the Public Technology Service Platform Project of Xiamen (No. 3502Z20231043). We also thank the reviewers for their insightful comments.

heading: A Data preprocessing
text: Strictly following Wang et al. (2022), we preprocess our experimental datasets. As shown in Fig 7, we first retrieve relevant text paths for the target entities from the CodRED dataset, forming a document bag. Note that, since the length of each document in a text path may exceed the limit of BERT, we then use an entity-based document-context filter (Wang et al., 2022) to select salient sentences for each document and ensure the total length of a text path is less than 512. Finally, we concatenate the head document and tail document from each text path and then obtain the input of BERT, where the number of text paths in a document bag is the batch size.

heading: B Effect of the threshold η for semantic-related edge
text: We also investigate the impact of the hyperparameter η on the development set under the closed setting. To this end, we gradually vary η from 0.4 to 0.8 with an increment of 0.1 in each step. As shown in Figure 6, we find that our model achieves the best performance when η=0.6. Therefore, we set η=0.6 for all experiments.

heading: C Effect of the BERT version
text: GPT-3.5-turbo InstructUIE
Prompt 1 Given the list of relations: [“highway system”, “country”, “place of birth”, ...], read the given text path [text path] and predict the relation between head entity: [h] and tail entity: [t]. Answer in the format [“relation label”, “confidence score(Decimal between 0-1)”] without any explanation. Text: [text path]. In the above text, what is the relationship between [head entity] and [tail entity]? Option: [“highway system”, “country”, “place of birth”, ...]. Answer:
Prompt 2 Analyze the information provided in the text path [text path], which includes mentions of the head entity [h] and the tail entity [t]. Infer the relation from the given relation set: [“highway system”, “country”, “place of birth”, ...] between the target entity pair. Please consider the entire path and answer in the format [“relation”, “confidence score(Decimal between 0-1)”] without any explanation. Text: [text path]. Find the relationship between [head entity] and [tail entity] in the above text. Option: [“highway system”, “country”, “place of birth”, ...]. Answer:
Prompt 3 Given a text path [text path] containing mentions of the head entity [h] and the tail entity [t] in separate sentences, your goal is to identify and infer the relation from the pre-defined set [“highway system”, “country”, “place of birth”, ...] between the target entity pair. Pay close attention to the specific information provided in the path. Answer in the format [“relation”, “confidence score(Decimal between 0-1)”] without any explanation. Text: [text path]. Given the above text, please tell me the relationship between [head entity] and [tail entity]. Option: [“highway system”, “country”, “place of birth”, ...]. Answer:
Table 8: Zero-shot prompts for GPT-3.5-turbo and InstructUIE on CoRE task
We also investigate the impact of the BERT version. As shown in Table 6 and Table 7, when using BERTlarge, our model still significantly outperforms both the previous SOTA: LGCR and LLMbased methods: GPT-3.5-turbo and InstructUIE-FT, demonstrating the effectiveness of our model.

heading: D Experimental settings for LLMs
text: The detailed experimental settings for the LLMs are as follows:
• GPT-3.5-turbo. As implemented in previous studies (Han et al., 2023; Wang et al., 2023b; Yuan et al., 2023), we devise three diverse prompts as shown in Table 8, which prompt LLM to generate both the relation label and the corresponding confidence scores. Subse-
quently, to obtain the bag-level relation prediction, we select the highest confidence score among all text paths within a document bag. We conduct experiment under each prompt separately and then average the results under all prompts as the final experimental result.
• InstructUIE and InstructUIE-FT. We first use LoRA (Hu et al., 2021) to obtain InstructUIEFT. During this process, we set the learning rate to 5e-5 and the batch size to 8. As for the prompt, we follow Wang et al. (2023b) to design three prompts as shown in Table 8. Note that, to obtain the bag-level prediction, we first get the prediction score of the relation label generated by the model. After that, we take the relation label with the highest prediction score among all text paths as the final prediction of a document bag. Like GPT-3.5turbo, we also average the results under all prompts as the final experimental result.
to mark the text paths, where the target entities , bridge entities , and non-bridge entities are marked in green, blue, and orange respectively. In text path 1, with the help of non-bridge entities, both our model and ours w/o yrela, ybias can predict the correct relation. In text path 2, we can find the proposed debiasing strategy helps our model for relation prediction.

heading: Future Work
text: In the future, we will study how to introduce more external knowledge to refine our model. Limitations
One limitation of the present work lies in that we only rely on the attention score to measure the importance of nodes in the graph, and do not consider dynamically adjusting the importance score. Additionally, in the entity-based graph, the edges are connected using a heuristic method, which may overlook useful information