Reward-based Input Construction for Cross-document Relation Extraction (2024)

Abstract: Relation extraction (RE) is a fundamental task in natural language processing, aiming to identify relations between target entities in text. While many RE methods are designed for a single sentence or document, cross-document RE has emerged to address relations across multiple long documents. Given the nature of long documents in cross-document RE, extracting document embeddings is challenging due to the length constraints of pre-trained language models. Therefore, we propose REward-based Input Construction (REIC), the first learningbased sentence selector for cross-document RE. REIC extracts sentences based on relational evidence, enabling the RE module to effectively infer relations. Since supervision of evidence sentences is generally unavailable, we train REIC using reinforcement learning with RE prediction scores as rewards. Experimental results demonstrate the superiority of our method over heuristic methods for different RE structures and backbones in crossdocument RE. Our code is publicly available at https://github.com/aailabkaist/REIC.

heading: 1 Introduction
text: The task of relation extraction (RE) aims to identify relations between a pair of target entities in a given text (Zeng et al., 2014; Zhang et al., 2017; Sahu et al., 2019; Xu et al., 2021). This task is fundamental in natural language processing (NLP) and provides crucial information for applications such as information retrieval (Kadry and Dietz, 2017) and question answering (Li et al., 2019; Chen et al., 2019). Most existing RE methods are limited to scenarios where the entity pair is within a single sentence (Wang et al., 2016; Zhu et al., 2019) or a single document (Nan et al., 2020; Huang et al., 2021; Xu et al., 2021). However, many scenarios require extracting relations across multiple documents, where the target entity pair may not coexist
*Equal contribution
within the same document. Therefore, recent efforts to address these challenges have led to the proposal of cross-document RE by Yao et al. (2021), and research in this area has received considerable attention (Wang et al., 2022; Wu et al., 2023; Lu et al., 2023; Son et al., 2023).
Unlike traditional RE research, cross-document RE involves extracting relational facts from largescale long documents. For example, the DocRED dataset (Yao et al., 2019), developed for documentlevel RE, contains an average of 198 words per document. In contrast, the CodRED dataset (Yao et al., 2021), designed for cross-document RE, has an average of 2,416 words per document. Given the substantial length of documents in cross-document RE tasks, extracting document embeddings from pre-trained language models, a step common to all methods, poses considerable challenges. This is because pre-trained language models are limited by the maximum number of tokens they can process; for example, BERT (Devlin et al., 2019) typically has a limit of 512 tokens.
A simple approach adopted in several studies is to use text snippets around the target entities as input to pre-trained language models, as shown in the dashed boxes in the middle panel of Figure 1. Often, this text snippet is determined by either word proximity or document structure without any adjustment. However, if only a subset of sentences around the entities is extracted from long documents, there is a risk of missing important sentences that are crucial for RE. For example, as seen in the left panel of Figure 1, using snippets around the entities fails to capture the relation ‘platform’ between the entity pair (Kubuntu, x86-64).
In this paper, we introduce REward-based Input Construction (REIC), the first learning-based input sentence selection module specifically designed for cross-document RE. The right panel of Figure 1 displays the sentences selected by our model from the example. By identifying relational evidence
9254
sentences through the selection module, the RE module infers the relation between the target entities using the reasoning chain illustrated by the gradient-colored arrows in Figure 1. Our approach is to develop a sentence selection module that computes the probability of selecting sentences based on the currently selected sentences. We show that this sentence selection process can be modeled as a Markov decision process (MDP). We specifically choose to use MDP because there is no supervision for sentence selection from corpus, so RE models have to perform iterative learning by trials. Subsequently, we train the sentence selection module using reinforcement learning (RL), where the relation prediction score obtained from the selected sentences serves as the reward. Through experimental validation, REIC outperforms other heuristic methods across various RE modules.

heading: 2 Related Work

heading: 2.1 Cross-document Relation Extraction
text: Traditionally, research in RE has primarily focused on the sentence- or document-level, typically dealing with documents no longer than a single paragraph, as reviewed in Appendix A. However, there has been a growing interest in cross-document RE, which aims to identify relations between entities across multiple long documents. Yao et al. (2021) laid the groundwork for cross-document RE by introducing the CodRED benchmark dataset. They
also proposed a simple baseline model that utilizes document embeddings obtained from pre-trained language models like BERT (Devlin et al., 2019). Building on this work, ECRIM (Wang et al., 2022) proposed an entity-based input filtering approach and an entity-based RE module with an attention mechanism, leading to significant performance improvements. Additionally, Wu et al. (2023) introduced a local causal estimation algorithm and a globally guided reasoning algorithm based on causality. In addition, Lu et al. (2023) presented a method for multi-hop evidence retrieval through evidence path mining and ranking, and Son et al. (2023) proposed a method for constructing explicit reasoning paths using bridging entities. In our approach, we focus on selecting essential sentences to effectively extract document information from long documents for cross-document RE. Unlike other methods, our approach involves learning-based input construction, and these surveyed models can utilize our input construction method.

heading: 2.2 Sentence Selection in NLP
text: Numerous studies have addressed the challenge of selecting relevant sentences from extensive and noisy textual datasets while preserving contextual coherence. These approaches include deep learning-based binary classification for each sentence (Cheng and Lapata, 2016), auto-regressive techniques to capture sentence correlations (Zhou et al., 2018; Zhong et al., 2019; Zhang et al.,
2023b), methods that consider summary-level context (Zhong et al., 2020), and the use of diffusion models (Zhang et al., 2023a).
Other lines of research utilize RL for sentence selection (Narayan et al., 2018). This approach naturally reflects the correlation between sentences, as the selection process is optimized by considering the current state, which includes the sentences already selected. Furthermore, RL offers flexibility in reward shaping, making it suitable for specific downstream tasks. For example, Feng et al. (2018); Qin et al. (2018); Qu et al. (2019); Takanobu et al. (2019) used RL to filter noisy datasets, thereby improving the performance of sentence-level RE. Also, Xu et al. (2022) utilized RL to evidence extraction in document-level RE, and Man et al. (2022) applied RL to find input tokens for BERT embedding within documents for event-event RE. Our approach builds on previous methods for sentence selection and represents the first application of RL to cross-document RE.

heading: 3 Methods

heading: 3.1 Problem Formulation
text: The cross-document RE (Yao et al., 2021) aims to infer a relation r ∈ R between a pair of target entities (eh, et), where eh is the head entity and et is the tail entity, from a bag of text paths {pn}Nn=1. Each text path pn consists of two documents (dhn, d t n) that satisfy the following conditions: 1) dhn and d t n contain the head and tail entities, respectively; 2) there exists at least one entity, called a bridge entity, which is shared by both documents.
Our goal is to develop a method that effectively extracts sentences from an input document to facilitate the relation extraction between two entities. Given a document d consisting of M sentences, de-
noted by d = {sm}Mm=1, our approach is to select important sentences and obtain document embeddings using pre-trained language models such as BERT (Devlin et al., 2019). Given the maximum input length constraint of pre-trained language models, collecting significant sentences enables effective extraction of document information.

heading: 3.2 Overall Framework
text: We begin by describing our proposed framework as depicted in Figure 2. We divide the cross-document RE module into the input sentence selection module and the RE module. The goal of the sentence selection module is to identify important sentences for RE in each document. Next, the RE module outputs relation prediction scores based on the sentences identified by the sentence selection module.
Specifically, for each text path pn = (dhn, d t n), we utilize the sentence selection module to extract a summarized text path p̄n = (Shn, S t n), containing essential sentences to recognize relations between entity pairs. Afterwards, these summarized text paths become an input to the RE module, which employs a pre-trained language model as its backbone, to produce relation prediction scores ŷ(Shn, S t n) for each path. The RE module is trained using relation prediction loss. For example, ECRIM (Wang et al., 2022) adopts a multi-label global-threshold loss:
LRE = log(eθ + ∑
r∈R\Ωh,t eŷr)
+ log(e−θ + ∑
r∈Ωh,t e−ŷr), (1)
where θ is the threshold and Ωh,t denotes the set of relations that match the relation of the target entity pair. On the other hand, we compute the selection reward to train the sentence selection module using RL, as explained in Section 3.4. It is important
to note that our sentence selection module can be integrated into any existing RE modules.

heading: 3.3 REIC: Reward-based Input Construction
text: We present the proposed sentence selection module, called REward-based Input Construction (REIC), for cross-document RE. We build the structure of the REIC module inspired by Man et al. (2022), as shown in Figure 3. First, we obtain the representation zm of each sentence sm. We denote the sentence containing the target entity as stgt. We are trying to extract information relevant to the target entities in this document. Therefore, it would be helpful to include the information from the sentence that contains the target entity. To achieve this, we concatenate stgt and sm and input them into the pre-trained BERT encoder to generate a sentence embedding zm for the mth sentence:
zm = BERT([stgt, sm]). (2)
Note that we maintain the positions in the document when concatenating.
Next, we propose an iterative sentence selection process. We introduce the policy network Gϕ, with trainable parameters ϕ, for generating the sentence selection probabilities. The term ‘policy’ is used because we treat the sentence selection process as a policy and apply policy gradient methods, as we will explain in Section 3.4. The policy network takes the sentence embeddings as input and outputs the probability of selecting each sentence based on the previously selected sentences.
Specifically, we denote the set of selected sentences after iteration t as St, and we initialize it as S0 = {stgt}, meaning that the target sentence is selected by default. At iteration t + 1, we aim to select the next important sentence smt+1 from the given set of t + 1 selected sentences, St = {stgt, sm1 , ..., smt}. To provide information about the sentences already selected, we introduce the Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) network to the sentence embedding zm. By feeding the embedding zmt of the selected sentence at time t into the LSTM, the hidden vector hmt of the LSTM serves as the representation of the previously selected sentences:
hmt = LSTM(zmt , hmt−1). (3)
Then, we concatenate the embedding hmt of the selected sentences with the embedding zm of each unselected sentence and feed them into the policy network Gϕ to obtain the selection probabilities:
π̂(sm|St) = Gϕ([zm, hmt ]). (4)
We select the sentence from the sampling with the selection probability as the next selected sentence:
smt+1 ∼ Categorical(π̂(sm|St)). (5)
This process is iterated for a maximum number of iterations T and is outlined in Algorithm 1.
Algorithm 1: Algorithm of REIC Input: Input sentences {sm}Mi=1 Output: Selected sentences S
1 M← {1, ...,M} 2 zm ← BERT([stgt, sm]) for m ∈M 3 S ← {stgt} 4 z ← ztgt 5 M←M\ {tgt} 6 h← ∅ 7 for t ∈ {1, ..., T} do 8 h← LSTM(z, h) 9 π̂m ← Gϕ([zm, h]) for m ∈M
10 m̂ ∼ Categorical(π̂m) 11 S ← S ∪ {sm̂} 12 z ← zm̂ 13 M←M\ {m̂} 14 end

heading: 3.4 Training REIC with RL
text: We discuss the method of training the REIC module based on RL. We represent the sentence selection process as a MDP. The state space consists of combinations of selected sentences, which is represented by a set of selected sentences, S. Selecting one sentence at each iteration is considered as an action. Thus, when the action of selecting sentence sm from the current state S is taken, it transitions to the state S∪{sm}. In this case, the policy, which is the distribution of actions given a state, can be represented by the selection probability from Eq. (4). Once the action is determined; the next state is also determined, so is the transition probability.
The proposed sentence selection module ultimately needs to choose sentences that perform well for the RE task. Therefore, it is necessary to set the reward function to provide signals for RE. An intuitive approach is to identify the important sentences necessary for RE and provide rewards when these sentences are selected. However, obtaining the important sentences from the large dataset requires specialized knowledge and a considerable amount of time, making it extremely challenging. Therefore, we intend to determine the reward as the relation prediction results obtained by performing RE using the selected sentences.
Specifically, for a given text path, considering the selected sentences Sh for the head document and St for the tail document, the reward R is,
R(Sh) = R(St) = λrŷr(S h, St), (6)
where ŷr(Sh, St) is the prediction score of the true relation r obtained by inputting the selected sentence pairs (Sh, St) into the RE module. Here, λr is a reward hyperparameter adjusted based on the relation type. Empirically, due to the dataset imbalance problem where there are more ‘no relation’ (N/A) cases than positive cases, we find that it is beneficial to assign a higher value to positive relations compared to ‘no relation’. In our experiments, we set λr = 10 for positive relations and 1 otherwise, as discussed in Section 4.4. This hyperparameter λ can be considered as a prior of Bayesian RL. Additionally, depending on the loss structure, we also consider replacing it with rewards that are proportional to the prediction scores, which is detailed in Section 4.1.
Based on the defined MDP, we update the policy network directly using one of the policy gradient methods, REINFORCE (Williams, 1992):
ϕ← ϕ+ αR(ST )∇ log π̂(ST ), (7) where log π̂(ST ) = ∑T t=1 log π̂(smt |St−1). (8)
Here, α is the learning rate. In the implementation, for the sake of time and memory efficiency, we fix the BERT encoder for sentence embedding extraction and only optimize the parameters of the policy network and the LSTM.
We train the REIC module and the RE module simultaneously, as described in Figure 2. The REIC module extracts important sentences from documents in a batch, and these selected sentences are used to predict relations through the RE module. Based on the predicted outputs, we compute the reward and the RE loss. Then, we optimize the REIC module and the RE module using policy gradient and backpropagation, respectively.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: We evaluate the proposed method on the crossdocument RE task. We use the CodRED dataset (Yao et al., 2021), which consists of 276 relation types derived from English Wikipedia and Wikidata. Documents in CodRED contain an average of 4,939 tokens. Given our focus on evaluating cross-document reasoning, we conduct experiments in a closed setting where text paths are provided. We adopt End-to-end (Yao et al., 2021) and ECRIM (Wang et al., 2022) for the RE modules. We follow the settings provided by each model. We
consider both BERT-base and RoBERTa-large (Liu et al., 2019) models as the embedding backbone.
Regarding the REIC module, we use a fixed BERT-base (Devlin et al., 2019) model as the embedding backbone. We employ the AdamW (Loshchilov and Hutter, 2019) optimizer with a learning rate of 3e-3. Sentence embeddings are set to a dimension of 768. We use a 1-layer LSTM with a hidden vector dimension of 512. The policy network comprises a 2-layer feed-forward neural network with a hidden dimension of 512.
We use Snippet (Yao et al., 2021) and ECRIMIC (Wang et al., 2022), which are accessible implementations, as baselines for comparing the input construction modules. Snippet uses tokens near the target entity as input, while ECRIM-IC conducts heuristic sentence filtering based on the bridge entity. In line with previous work (Yao et al., 2021), we use the AUC, F1, Precision@500 (P@500), and Precision@1000 (P@1000) as evaluation metrics. Further settings are specified in Appendix B.
Reward Function We set the reward function to be proportional to the prediction score of the given relation, but different for each RE module due to the difference in their loss structures. For the End-to-End RE module, we set the reward function by the difference between the given relation score and the highest score excluding the relation:
R = λr(ŷr −maxi ̸=r ŷi)/ŷr. We set λr = 10 for r ̸= N/A and λr = 1 for r = N/A, and clip rewards below 0. For the ECRIM RE module, we set the reward function by the difference between the given relation score and threshold θ in the RE loss: R = λr(ŷr − θ), where λr is the same as before.

heading: 4.2 Quantitative Results
text: Table 1 presents the performance results of different input construction modules with various RE modules. We run experiments with all combinations of RE module backbones, their structures, and input construction module types. As shown in Table 1, REIC demonstrates the best performance across all scenarios. In particular, the model combining the REIC module with ECRIM RE module using the RoBERTa backbone exhibits state-of-theart performance on the development set with an AUC of 66.41 and F1 of 63.47, and on the test set with an AUC of 65.88 and F1 of 65.02.
Our proposed approach outperforms previous methods: Snippet + End-to-end (Yao et al., 2021), with an increase of 1.97 in AUC and 1.34 in F1; and ECRIM-IC + ECRIM (Wang et al., 2022), with an increase of 1.54 in AUC and 2.34 in F1, on the development set when using the BERT backbone. In the case of the input construction module of ECRIM, due to its heuristic nature, its performance even decreased compared to the Snippet approach
on the End-to-end backbone that does not leverage bridge entity information (a decrease of 2.45/2.27 in AUC for BERT/RoBERTa backbones, respectively). Conversely, our method consistently improves performance in all scenarios because the selection module is trained based on the rewards received from each RE module.

heading: 4.3 Analysis of Input Construction
text: Effect Analysis for Document Length To analyze the effect of document length, we partition the CodRED development set based on the average number of tokens in the text paths for each entity pair. Figure 4 presents the F1 score corresponding to the average number of tokens. We observe that our sentence selection method shows comparable performance in most cases, and especially greater effectiveness in cases with very long document lengths. This improvement is attributed to the dispersion of important sentences in longer documents, which allows our learning-based approach to select relevant sentences more effectively.
Effect Analysis for Number of Bridge Entities in Text Paths To analyze the effect of the number of bridge entities in text paths, we partition the CodRED development set based on the number of mentions of bridge entities in the text paths for each entity pair. Figure 5 illustrates the variation in F1 scores according to the average number of mentions of bridge entities. Our model exhibits superior performance to ECRIM-IC, a heuristic construction method based on bridge entities, until the number of mentions reaches 16, which accounts for 64% of the entity pairs in the dataset. However, we observe that the performance of REIC is slightly lower than ECRIM-IC when the number of mentions is very high. These results suggest that while ECRIM-IC struggles to distinguish important sentences when the number of mentions is insufficient; REIC, which learns directly from sentences, is more effective. However, when the number of bridge entities is large enough, the heuristic method based on bridge entities is slightly better. Therefore, we believe that incorporating bridge entity information into the reward function of our method can provide synergistic benefits, which we identify as a topic for future work.
Analysis for Number of Bridge Entities in Selected Sentences For the analysis of selected sentences, we examine the number of mentions of bridge entities in sentences selected by ECRIM-IC
and REIC. Table 2 displays the average number of bridge entity mentions for each sentence selection module based on the relation type of entity pairs. Both modules extract more bridge entities from text paths of entity pairs with positive relations. ECRIM-IC extracts more bridge entities in all cases because it extracts sentences associated with bridge entities. Although our model is trained based on prediction scores without directly considering the number of bridge entities, it still extracts more bridge entities in positive relations. This observation highlights the importance of bridge entities in cross-document RE and suggests that our model effectively learns these bridge entities.
For entity pairs with positive relations, paths can either represent positive relations (positive paths) or not express the relation (N/A paths). We conduct an analysis of the number of bridge entities for each of these path types. Figure 6 plots the number of bridge entity mentions for each sentence selection module in histograms. ECRIM-IC tends to extract more bridge entities for all path types. Conversely, REIC extracts fewer bridge entities in N/A paths. We interpret this phenomenon as a preference to extract fewer unnecessary or ambiguous entities from N/A paths, thereby focusing on extracting meaningful entities to infer relations.
Runtime Analysis Figure 7 compares the training wall-clock time per epoch for each input construction method. Unlike other methods, REIC introduces the sentence selection module, which takes more time. REIC takes about 1.40 times as long as the ECRIM-IC training time. Additionally, we observed that REIC takes about 1.75 times as long as the ECRIM-IC inference time. From these observations, the introduction of the REIC module leads to an increased time consumption. Currently, the selection process operates at the sentence level, involving the extraction of sentence-level embeddings and the computation of selection scores. However, we believe that extending this process to larger structural units, such as paragraphs or paths, could potentially reduce the number of selections and thus the overall time required.

heading: 4.4 Ablation Studies
text: Multi-step vs. One-step Selection We employ a strategy of selecting one sentence at a time over T multi-steps in the sentence selection process, inspired by Man et al. (2022). We explore the option of selecting T sentences all at once in a one-step manner. In this case, we sample a sentence T times based on the selection probability obtained in the first iteration. The evaluation time of the multi-step and one-step selections is almost the same because the policy network and the LSTM are much shallower compared to other networks such as BERT. As shown in the ‘One-step’ row of Table 3, we observe an increase in performance of 0.92 in AUC with multi-step selection. We attribute this improvement to the LSTM providing information about the currently selected sentence, thus helping to select better sentences at each iteration.
Reward Hyperparameter λr We set the reward hyperparameter to λr = 10 for the positive relations to ensure that paths with positive relations receive higher rewards. This adjustment aims to address the problem that N/A paths occur about 14 times as often as positive paths, as reported in Yao et al. (2021). To assess the effectiveness of this adjustment, we train a model with λr set to 1 for all relations. As shown in the ‘λr = 1∀r’ row of Table 3, we observe a performance improvement with REIC when assigning larger rewards to positive relations compared to when rewards are equally distributed. This suggests the necessity of adjusting rewards to account for the imbalance between positive and N/A relations, which is common in RE datasets. Moreover, considering potential imbalances among positive relations, further adjustments to reward hyperparameters may be necessary, which we identify as a direction for future work.
Number of Selected Sentences T In Figure 8, we conduct an ablation study on the hyperparameter T , representing the number of selected sentences. We observe consistently better performance compared to the Snippet across different numbers of sentences. Increasing the number of sentences tends to improve performance up to a certain level, as it provides more information to the RE module. However, including more sentences than necessary may introduce noisy information, potentially affecting the ability to effectively distinguish relations.

heading: 4.5 Case Study
text: Table 4 presents a case study of the key sentences selected by REIC to infer the entity relation, where the sentence missed by the baselines is highlighted in color. This sentence is crucial for inferring the relation between the entity pair. We also include the complete sets of sentences extracted by each input construction method in Tables 9 to 11 of
Appendix C.2. Although ECRIM-IC extracts more bridge entities, REIC selects important sentences necessary to elucidate entity relations.

heading: 5 Conclusion
text: We present REward-based Input Construction (REIC), a learning-based sentence selection module tailored for cross-document RE. We tackle the limitation of document embedding extraction from long documents. REIC selects important sentences based on relational evidence, enabling the RE module to effectively infer relations. Given the unavailability of supervision for evidence sentences, we use RL to train REIC. We show the superiority of our method over heuristic methods. This highlights the potential of learning-based approaches to improve the performance of cross-document RE.
Limitations
Limitations of our method include that the introduction of a network for input construction increases the model runtime by a factor of 1.40 for training and 1.75 for inference in the ECRIM-IC runtime. To alleviate this, we precompute and store sentence embeddings from BERT to avoid repetitive BERT evaluations during training. One possible solution is to apply model compression methods, such as pruning and weight factorization, which have been used in many NLP studies (Lan et al., 2020; Gordon et al., 2020; Wang et al., 2020).

heading: Acknowledgements
text: This research was supported by Research and Development on Key Supply Network Identification, Threat Analyses, Alternative Recommendation from Natural Language Data (NRF) funded by the Ministry of Education (2021R1A2C200981612).

heading: A Further Review of Related Work
text: Early relation extraction models were proposed to predict relationships between entities within a single sentence (Zeng et al., 2014; Wang et al., 2016; Zhang et al., 2017; Zhu et al., 2019). Some research on sentence-level relation extraction introduced a novel convolutional neural network (CNN) architecture to extract sentence-level features from word tokens (Zeng et al., 2014; Wang et al., 2016). Additionally, Zhang et al. (2017) not only presented an LSTM sequence model with entity position-aware attention, but also proposed TACRED, a large supervised dataset for sentence-level RE, serving as a benchmark dataset. GP-GNNs (Zhu et al., 2019) propose a novel graph neural network where its parameters are generated through a propagated message passing module taking sentences as input.
Yao et al. (2019) introduced DocRED, a comprehensive dataset with human-annotated documentlevel relation extraction, aiming to advance from sentence-level to document-level relation extraction, offering a wealth of relation facts. Several methods for document-level RE utilize documentlevel graphs constructed from entities or mentions, employing graph neural networks (GNN) for path reasoning to infer relations between entities (Sahu et al., 2019; Nan et al., 2020; Zeng et al., 2020). On the other hand, many document-level RE methods have emerged that do not rely on graph structures. Huang et al. (2021) presents a straightforward method to heuristically select evidence sentences. SSAN (Xu et al., 2021) integrates structural dependencies into self-attention, enhancing attention flow for relation extraction. Recent challenges for document-level RE include high memory usage and limited annotations. DREEAM (Ma et al., 2023) and S2ynRE (Xu et al., 2023) offer a memory-efficient solution and self-training to generate evidence and training data.

heading: B Additional Experimental Settings
text: B.1 Datasets The CodRED dataset, sourced from its official Github repository, comprises relation triplets, evidence, and documents, licensed under MIT. CodRED also includes processed Wikidata (CC0 license) and Wikipedia (licensed under CC BY-SA and GFDL). We processed the raw data as per repository instructions, transforming and storing documents in a Redis server for downstream RE methods. In particular, for efficiency, we stored
BERT embeddings for each sentence of every document in the server environment for operational efficiency. CodRED comprises 276 relations, and its statistics are presented in Table 5.
B.2 Implementation Details
B.2.1 Model Parameters Table 6 displays the number of parameters for different models. Each row represents a RE module’s backbone, structure, an input construction module and the corresponding number of parameters. For BERT, the parameter count is 108,310,272, while for RoBERTa, it is 355,462,144. Our REIC model introduces a selector module, increasing the parameter count by 3,281,921 for each RE module.
B.2.2 Code Implementation During the implementation process, we utilized the official codes of End-to-end1 (Yao et al., 2021) and ECRIM2 (Wang et al., 2022). We trained both the end-to-end and ECRIM models using a learning rate of 3e-5. The training process involved running 2 epochs for the End-to-end and 10 epochs for the ECRIM. One change is to set the batch size to 4 and the gradient accumulation step to 4 for ECRIM.
End-to-end model underwent the modification during implementation that had a significant impact on performance. In the original code, the entity marking token ‘[UNUSED@]’, where ‘@’
1https://github.com/thunlp/CodRED 2https://github.com/MakiseKuurisu/ecrim
is a natural number, was utilized. However, we found that this token was functionally meaningless and replaced it with ‘[unused@]’. Then, the performance of the modification, as shown in the row of ‘Snippet + End-to-end’ in Table 1, exhibited a significant improvement compared to the results reported in Yao et al. (2021).
Our REIC model implementation is also based on the official End-to-end and ECRIM codebases. Additionally, we incorporated the sentence selection module from SCS-EERE3 (Man et al., 2022) as a reference. If the selected T sentences exceed the token limit, we use only the first 512 tokens in the order they were chosen. Our REIC model was trained according to the settings specified for each RE structure, and detailed training configurations for our REIC are provided in Section 4.1. We utilized a single NVIDIA A100 80GB model for training both the baseline and our model. We performed a single run for each of the various RE backbones and structures to produce the results.
B.3 Details of Input Construction Analysis
Tables 7 and 8 provide the number of bags in each partition for each analysis.

heading: C Additional Experimental Results
text: C.1 Reward Analysis Figure 9 shows the selection reward curve during training. As training progresses, the reward gradually increases. After a certain initial step, the reward stabilizes at 1, which is a reward value for correct predictions of the N/A relations. This is probably influenced by the large proportion of N/A relations in the dataset and the model correctly predicts the N/A relations. Subsequently, in later stages, the reward surpasses 1, indicating that the model is increasingly focused on correctly predicting positive relations.
C.2 Case Study Tables 9 to 11 provide the complete set of sentences selected by each input construction module for the case study analyzed in Section 4.5.

heading: D Potential Risks
text: Our method may select incorrect or irrelevant sentences, resulting in inaccurate relation extraction. This risk could occur if there are limitations in the training data. In addition, there could be cases of noisy or harmful labels in the training data, which could affect the extraction of harmful sentences.
3https://github.com/hieumdt/SCS-EERE

heading: Future Work
text: Moreover, considering potential imbalances among positive relations, further adjustments to reward hyperparameters may be necessary, which we identify as a direction for future work. Number of Selected Sentences T In Figure 8, we conduct an ablation study on the hyperparameter T , representing the number of selected sentences. We observe consistently better performance compared to the Snippet across different numbers of sentences. Increasing the number of sentences tends to improve performance up to a certain level, as it provides more information to the RE module. However, including more sentences than necessary may introduce noisy information, potentially affecting the ability to effectively distinguish relations