Venue: NAACL 2022

# Document-Level Relation Extraction with Sentences Importance Estimation and Focusing

Wang Xu1, Kehai Chen1, Lili Mou2, Tiejun Zhao1

###### Abstract

Document-level relation extraction (DocRE) aims to determine the relation between two entities from a document of multiple sentences. Recent studies typically represent the entire document by sequence- or graph-based models to predict the relations of all entity pairs. However, we find that such a model is not robust and exhibits bizarre behaviors: it predicts correctly when an entire test document is fed as input, but errs when non-evidence sentences are removed. To this end, we propose a Sentence Importance Estimation and Focusing (SIEF) framework for DocRE, where we design a sentence importance score and a sentence focusing loss, encouraging DocRE models to focus on evidence sentences. Experimental results on two domains show that our SIEF not only improves overall performance, but also makes DocRE models more robust. Moreover, SIEF is a general framework, shown to be effective when combined with a variety of base DocRE models.1

Footnote 1: The code is publicly available at [https://github.com/xwjim/SIZE](https://github.com/xwjim/SIZE)

+Harbin Institute of Technology, China

\({}^{2}\)Dept. Computing Science, Alberta Machine Intelligence Institute (Amii) University of Alberta, Canada xuwang@hit-mtlab.net, {chenkehai,tjzhao}@hit.edu.cn, doublepower.mou@gmail.com

## 1 Introduction

Document-level relation extraction (DocRE) aims to predict entity relations across multiple sentences. It plays a crucial role in a variety of knowledge-based applications, such as question answering Sorokin and Gurevych (2017) and large-scale knowledge graph construction Baldini Soares et al. (2019). Different from sentence-level relation extraction Zeng et al. (2014); Xiao and Liu (2016); Song et al. (2019), the supporting evidence in the DocRE setting may involve multiple sentences scattering in the document. Thus, DocRE is more a realistic setting, attracting increasing attention in the field of information extraction.

Most recent DocRE studies use the entire document as a clue to predict the relations of all entity pairs without concerning where the evidence is located Nan et al. (2020); Zeng et al. (2020); Xu et al. (2021a,b). However, one can identify the relation of a specific entity pair from a few sentences. Huang et al. (2021) show that irrelevant sentences in the document would hinder the performance of the model.

Moreover, we observe that a DocRE model, trained on the entire document, may err when non-evidence sentences are removed. In Figure 1, for example, we need to identify the relation "MemberOf" between the entities _Brad Wilk_ and _Rage Against the Machine_. The evidence sentences are {1,2}, and humans can easily identify such a relation when reading sentences {1,2} only. However, the recent DocRE model GAIN Zeng et al. (2020) identifies the relation "MemberOf" correctly from the entire document {1,2,3}, but predicts "not MemberOf" from sentences {1,2}. Intuitively, removing sentence {3} should not change the result, as this sentence does not provide information regarding whether "MemberOf" holds or not for the two entities. Such model behaviors are undesired, because it shows that the model is not robust and lacks interpretability.

To this end, we propose a novel **S**entence **I**or **E**stimation and **F**ocusing **(**SIEF**) framework to encourage the model to focus on evidence sentences for predicting the relation of

Figure 1: A DocRE model predicts correctly for an entire document, but errs when a non-evidence sentence is removed.



an entity pair. Specifically, we first evaluate the importance of each sentence by the difference between the output probabilities of the document with and without this sentence. If the predicted probability of a relation does not change, or even increases, when a sentence is removed, it typically indicates that the sentence is _non-evidence_. Then, we propose an auxiliary loss to encourage the model to produce the same output distribution, when the entire document is fed as input and when a non-evidence sentence is removed. In this way, the model pays more attention to the evidence sentences for the classification. Our SIEF method is a general framework that can be combined with different underlying DocRE models.

We evaluated the generality and effectiveness of our approach on the large-scale DocRED dataset Yao et al. (2019). Experimental results show that the proposed approach combines well with various recent DocRE models and significantly improves the performance. We further evaluated our approach on a dialogue relation extraction dataset, DialogRE Yu et al. (2020); our SIEF yields consistent improvement, showing the generality of our approach in different domains.

## 2 Related Work

Relation extraction (RE) can be categorized by its granularity, such as sentence-level Doddington et al. (2004); Xu et al. (2016); Wei et al. (2020) and document-level Gupta et al. (2019); Zhu et al. (2019). Early work mainly focuses on sentence-level relation extraction. Pantel and Pennacchiotti (2006) propose a rule-based approach, and Mintz et al. (2009) manually design features for classifying relations. In the past several years, neural networks have become a prevailing approach for relation extraction Xu et al. (2015); Song et al. (2019).

Document-level relation extraction (DocRE) is attracting increasing attention in the community, as it considers the interactions of entity mentions expressed in different sentences Li et al. (2016); Yao et al. (2019). Compared with the sentence level, DocRE requires the model collecting and integrating inter-sentence information effectively. Recent efforts design sequence-based and graph-based models to address such a problem.

Sequence-based DocRE models encode a document by the sequence of words and/or sentences, for example, using the Transformer architecture Devlin et al. (2019). Zhou et al. (2021) argue that the Transformer attentions are able to extract useful contextual features across sentences for DocRE, and they adopt an adaptive threshold for each entity pair. Zhang et al. (2021) model DocRE as a semantic segmentation task and predict an entity-level relation matrix to capture local and global information.

Graph-based DocRE models abstract a document by graphical structures. For example, a node can be a sentence, a mention, and/or an entity; their co-occurrence is modeled by an edge. Then graph neural networks are applied to aggregate inter-sentence information Quirk and Poon (2017); Christopoulou et al. (2019); Zeng et al. (2020) construct double graphs, applying graph neural networks to mention-document graphs and performing path reasoning over entity graphs. Xu et al. (2021a) explicitly incorporate logical reasoning, commonsense reasoning, and coreference reasoning into DocRE, based on both sequence and graph features.

Different from previous work, our paper proposes SIEF as a general framework that can be combined with various sequence-based and graph-based DocRE models. In our approach, we propose a sentence importance score and a sentence focusing loss to encourage the model to focus on evidence sentences, improving the robustness and the overall performance of DocRE models.

## 3 Problem Definition

In this section, we present the formulation of document relation extraction (DocRE). Consider an unstructured document comprising \(N\) sentences, \({\mathbb{D}}=\left\{\mathbf{s}_{1},\mathbf{s}_{2},\cdots,\mathbf{s}_{N}\right\}\), where each sentence \(\mathbf{s}_{n}\) is a sequence words. In a DocRE dataset, the document \({\mathbb{D}}\) is typically annotated with entity mentions, each mention (e.g., U.S. and USA) labeled by its conceptual entity \(e\) and its entity type (e.g., location).

A DocRE model \(F\) is usually formulated as multi-label classification Yao et al. (2019). \(F_{j}\) predicts whether the \(j\)th relation holds for the \(i\)th marked entity pair in a document, given by

\[P_{ij}=\mathbf{F}_{j}({\mathbb{D}},e_{i_{h}},e_{i_{t}})=\Pr[r_{ij}=1|{\mathbb{D}}, e_{i_{h}},e_{i_{t}}] \tag{1}\]

where \(e_{i_{h}}\) is the head entity and \(e_{i_{t}}\) is the tail entity; \(r_{ij}\in\left\{0,1\right\}\) is the groundtruth label regarding entity pair \(i\) and relation \(j\) 

To train the model, the binary cross-entropy loss is used as the objective for parameter estimation:

\[\begin{split}\mathcal{L}_{\text{rel}}=-\sum_{\mathbb{D}\in \mathcal{C}}&\sum_{i_{h}\neq i_{t}}\sum_{j\in\mathcal{R}}\{r_{ij} \log P_{ij}\\ +(1-r_{ij})\log(1-P_{ij})\}\end{split} \tag{2}\]

where \(\mathcal{C}\) denotes the entire corpus and \(\mathcal{R}\) denotes the set of relation types.

During inference, we obtain the relation(s) of a given entity pair by thresholding the predicted probabilities, following most previous work Yao et al. (2019); Zhou et al. (2021).

## 4 Methodology

In this section, we will describe our approach in detail. The overview of our framework is shown in Figure 2. First, we describe the estimation of sentence importance in Section 4.1. Sentences with low importance scores are treated as non-evidence. Then, Sections 4.2 and 4.3 present our approach that encourages the model to produce the same output distribution, when the entire document is fed as input and when non-evidence sentences are removed. Section 4.4 further presents the architectures of DocRE models.

### Sentence Importance Estimation

We estimate the importance of each sentence for a specific entity pair. Low-scored sentences will be treated as non-evidence, and in principle, can be removed without changing DocRE predictions.

We propose a sentence importance score based on the DocRE predictions with and without the sentence in question. Our observation is that the relation extraction task is usually monotonic to evidence, i.e., (non-strictly) more relations will be predicted with more sentences. If we remove a sentence and the predicted probability of a relation decreases, then the sentence is likely to be the evidence. If the predicted probability does not change, then the sentence is likely to be non-evidence. Moreover, the predicted probability may sometimes increase when a sentence is removed, in which case the DocRE model is not robust, as this violates monotonicity.

Formally, we consider removing one sentence at a time, and the document with the \(n\)th sentence removed is denoted by \(\hat{\mathbb{D}}^{(-n)}=\{\mathbf{s}_{1},\cdots,\mathbf{s}_{n-1},\mathbf{s}_{ n+1},\cdots,\mathbf{s}_{N}\}\). For a DocRE model \(F\), we obtain the classification probabilities \(P_{ij}=\mathbf{F}_{j}(\mathbb{D},e_{i_{h}},e_{i_{t}})\) based on the original document, and \(\hat{P}_{ij}^{(-n)}=\mathbf{F}_{j}(\hat{\mathbb{D}}^{(-n)},e_{i_{h}},e_{i_{t}})\) with sentence \(n\) removed.

We propose the importance score as

\[g_{ij}^{(-n)}=P_{ij}\log\frac{P_{ij}}{\hat{P}_{ij}^{(-n)}} \tag{3}\]

The formula appears similar to Kullback-Leibler (KL) divergence. However, we only take one term in the KL summation, because the KL divergence, albeit asymmetric in its two arguments, cannot model the increase or decrease of \(\hat{P}_{ij}^{(-n)}\), whereas our \(g_{ij}^{(-n)}\) is monotonically decreasing with \(\hat{P}_{ij}^{(-n)}\). Compared with a naive difference or ratio between \(P_{ij}\) and \(\hat{P}_{ij}^{(-n)}\), we find that our KL-like score is more robust in the scale of \(P_{ij}\) when determining non-evidence sentences.

We treat a sentence \(n\) as _non-evidence_ if \(g_{ij}^{(-n)}<\beta\) for a thresholding hyperparameter \(\beta\). The resulting set of non-evidence sentences is denoted by \(\mathbb{K}_{ij}\) for the an entity pair \((e_{i_{h}},e_{i_{t}})\) and relation \(j\)

Figure 2: We estimate the sentence importance (for a specific entity pair and relation) by the difference of the classification probabilities with and without the sentence. Then, we encourage the DocRE model to predict the same probability when the entire document is fed as input and when a non-evidence sentence is removed.

 



### Sentence Focusing Loss

We propose a sentence focusing loss to encourage the model to produce the same output distribution when the entire document is fed as input and when non-evidence sentences are removed.

Ideally, the predicted probability should remain the same if we remove any combination of the sentences in \(\mathbb{K}_{ij}\). Therefore, we penalize the extent to which the predicted probability is changed.

We propose the sentence focusing loss as:

\[\mathcal{L}_{\text{sf}}=-\sum_{\mathbb{D}\in\mathcal{C}}\sum_{i_{ h}\neq i_{t}}\sum_{j\in\mathcal{R}}\sum_{\mathbb{J}_{ij}\subseteq\mathbb{ K}_{ij}}\{P_{ij}\log(\hat{P}^{(-\mathbb{J}_{ij})}_{ij}) \tag{4}\] \[+(1-P_{ij})\log(1-\hat{P}^{(-\mathbb{J}_{i})}_{ij})\}\]

where \(\mathbb{J}_{ij}\) is a subset of \(\mathbb{K}_{ij}\) and \(\hat{P}^{(-\mathbb{J}_{ij})}_{ij}=\mathbf{F}_{j}(\mathbb{D})\mathbb{J}_{ij},e_{i_{ n}},e_{i_{t}})\) is the predicted probability with \(\mathbb{J}_{ij}\) removed from \(\mathbb{D}\), and the total loss is \(\mathcal{L}=(\mathcal{L}_{\text{rel}}+\mathcal{L}_{\text{st}})/2\).

Essentially, our sentence focusing loss ensures \(P_{ij}\) is close to \(\hat{P}^{(-\mathbb{J}_{ij})}_{ij}\), which intuitively makes sense because non-evidence sentences should not affect the prediction. Our approach can also be thought of as a way of data augmentation. However, compared with one-hot groundtruth labels, our sentence focusing loss works with soft labels \(P_{ij}\) and \(\hat{P}^{(-\mathbb{J}_{ij})}_{ij}\), which are believed to contain more information Hinton et al. (2015), and our gradient propagates to both \(P_{ij}\) and \(\hat{P}^{(-\mathbb{J}_{ij})}_{ij}\) for training.

The calculation of Eqn. (4) is time- and resource-consuming, because the number of the subsets \(\mathbb{J}_{ij}\) grows combinatorially with the number of non-evidence sentences. Moreover it should be calculated repeatedly once the parameter of the model is updated. To this end, we propose a simplified training strategy to approximate Eqn. (4) in the next subsection.

### Training Strategy

We propose a strategy to simplify the calculation and the training procedure. Concretely, we only remove one non-evidence sentence in \(\mathbb{K}_{ij}\) at a time instead of a subset of \(\mathbb{J}_{ij}\subseteq\mathbb{K}_{ij}\), and we aggregate the effect of different non-evidence sentences by:

\[\begin{split}&\mathcal{L}_{\text{sf}}=-\sum_{\mathbb{D}\in \mathcal{C}}\sum_{n=1}^{N}\sum_{i_{h}\neq i_{t}}\sum_{j\in\mathcal{R}}\mathbb{ I}(g^{(-n)}_{ij}<\beta)\\ &\{P_{ij}\log(\hat{P}^{(-n)}_{ij})+(1-P_{ij})\log(1-\hat{P}^{(-n )}_{ij})\}\end{split} \tag{5}\]

where \(\mathbb{I}\) is the indicator function. Essentially, we linearly approximate the combination of multiple non-evidence sentences in (4) by an outer summation. In this way, the number of terms does not grow combinatorially, but linearly w.r.t. \(N\).

In implementation, we further simply the summation over \(n\) by Monte Carlo sampling of a randomly selected sentence \(n\) in each gradient update. The loss is reformulated as follows:

\[\begin{gathered}\mathcal{L}_{\text{sf}}=-\sum_{\mathbb{D}\in \mathcal{C}}\sum_{i_{h}\neq i_{t}}\sum_{j\in\mathcal{R}}\mathbb{I}(g^{(-n)}_{ ij}<\beta)\\ &\{P_{ij}\log(\hat{P}^{(-n)}_{ij})+(1-P_{ij})\log(1-\hat{P}^{(-n )}_{ij})\}\end{gathered} \tag{6}\]

As seen, we need to forward the base models twice in each update, with and without the sentence \(n\). Huang et al. (2021) propose a similar idea but train different entity pairs in a document based on different sets of sentences; all sentence are processed repeatedly among entity pairs in a document. Their approach is much slower than ours.

To sum up, the proposed SIEF framework identifies non-evidence sentences and penalizes the difference of predicted probabilities when a non-evidence sentence is removed. Our approach is a generic framework and can be adapted to various DocRE model easily, without introducing extra parameters into the model.

### DocRE Model Architectures

Our SIEF can be applied to various base DocRE models. To evaluate its generality, we consider the following recent models.

**BiLSTM**Yao et al. (2019)2. A bi-directional long short term memory (BiLSTM) encodes the document, and an entity is representated by BiLSTM's hidden states, averaged over entity mentions. The head and tail entity representations are fed to a multi-layer perceptron (MLP) for relation extraction.

Footnote 2: [https://github.com/thunlp/DocRED](https://github.com/thunlp/DocRED)

**BERTbase**Devlin et al. (2019)3. A pre-trained language model is used for document encoding.

Footnote 3: [https://github.com/](https://github.com/) DreamInvoker/GAIN

**HeterGSAN**Xu et al. (2021b)4. HeterGSAN is a recent graph-based DocRED model, which constructs a heterogeneous graph of sentence, mention, and entity nodes; it uses graph neural networks for relation extraction.





**GAIN**(Zeng et al., 2020)3. GAIN constructs two graphs: mention-document graphs and entity graphs, and performs graph and path reasoning over the two graphs separately. When combining our SIEF with GAIN, we achieve the best performance among all the base models with SIEF on DocRED. Thus, we will explain this model in more detail.

Footnote 3: [https://github.com/nlpdata/dialogre](https://github.com/nlpdata/dialogre)

Essentially, a node in the mention-document graph is either a mention or a document. The mentions are connected to its document, and two mentions are connected if they co-occur in one sentence. In the entity graph, two entities are connected if they are mentioned in one sentence. To classify the relation, GNN is applied to the mention-document graph, enhanced with path information in the entity graph, shown in Figure 3.

When combining SIEF with GAIN, we randomly remove one sentence from the document. The corresponding nodes and edges are removed in the GAIN's graphs. Then we obtain the output probabilities with and without the sentence, \(P_{ij}\) and \(\hat{P}^{(-n)}_{ij}\), separately. If the sentence important score \(g^{(-n)}_{ij}\) in Eqn. (3) is below a threshold \(\beta\), the sentence is treated as non-evidence for the entity pair \((e_{i_{h}},e_{i_{t}})\) and relation \(j\). We apply the sentence focusing loss Eqn. (4) to improve the robustness.

For prediction, we apply the trained DocRE model to the entire document, because with our approach the model is already robust when non-evidence sentences are presented. Empirical results will show that our SIEF consistently improves the performance of base DocRE models.

## 5 Experiments

### Setup

**Datasets.** DocRED is a large-scale human-annotated dataset for document-level relation extraction Yao et al. (2019). The dataset is constructed from Wikipedia and Wikidata, containing 3053 documents for training, 1000 for development, and 1000 for test. In total, it has 132,375 entities and 56,354 relational facts in 96 relation types. More than 40% of the relational facts require reasoning over multiple sentences. The standard evaluation metrics are F1 and Ign F1 Yao et al. (2019); Zeng et al. (2020), where Ign F1 refers to the F1 score excluding the relational facts in the training set.

We also evaluated our approach on DialogRE V2, Yu et al. (2020), which contains 36 relation types, 17 of which are interpersonal. We followed the standard split with 1073 training dialogues, 358 validation, and 357 test. Following Yu et al. (2020), we report macro F1 scores in both the standard and conversational settings; the latter is denoted by F1\({}_{c}\).

**Competing Methods.** We experimented our SIEF on a number of base models, namely, BiLSTM, BERTbase, HeterGSAN, and GAIN Section 4.4). These base models are all considered for comparison.

For DocRED, we consider additional competing methods: **Two Phase**Wang et al. (2019), which first predicts whether the entity pair has a relation and then predicts the relation type; **LSR**Nan et al. (2020), which constructs the graph by inducing a latent document-level graph; **Reconstructor**Xu et al. (2021b), which encourages the model to reconstruct a reasoning path during training: **DRN**Xu et al. (2021a), which considers different reasoning skills explicitly and uses graph representation and context representation to model the reasoning skills; **ATLOP**Zhou et al. (2021), which aggregates contextual information by the Transformer attentions and adopts an adaptive threshold for different entity pairs; and **DocNet**Zhang et al. (2021), which models DocRE as a semantic segmentation task.

For DialogRE, we followed Yu et al. (2020) and considered **BERT** and **BERT\({}_{s}\)** for comparison,5 where **BERT\({}_{s}\)** prevents a model from overfitting by replacing of the interpersonal augment with a special token.

Figure 3: The model architecture of GAIN with SIEF. A sentence is randomly removed from the document. The corresponding nodes and edges are removed from the mention–document graph and the entity graph.





**Implementation Details**. We use the repositories [2, 3, 4, 5] of base models to implement our approach. We mostly followed the standard hyperparameters used in the base models. Our SIEF has one hyperparameter \(\beta\) in Eqn. (5). It was set to 0.8, and Section 5.2 presents the effect of tuning \(\beta\).

### Results and Analyses

**Main results.** Table 1 presents the detailed results on the development and test sets of the DocRED dataset. We first compare DocRE systems with GloVe embeddings Yao et al. (2019). We see that the proposed SIEF method significantly improves the performance of all base models, including the sequence model (i.e., BiLSTM) and graph models (i.e., HeterGSAN and GAIN); the average improvement is 2.05 points in terms of test F1. This shows that SIEF is compatible with both sequence and graph models, indicating the generality and effectiveness of the proposed method.

For the DocRE system with BERTbase, SIEF also consistently improves the base models, showing that SIEF is complementary to the modern BERT architecture. Especially, combining SIEF and GAIN Zeng et al. (2020) with BERTbase encoding yields state-of-the-art performance in terms of F1.

We further conducted experiments on the DialogRE dataset, and compare our approach with the BERT baselines in Yu et al. (2020). As seen, the results are consistent with the improvement on DocRED, as our SIEF largely improves F1 and Fl1\(c\) for both base models. This further confirms the generality of our approach in different domains.

In the rest of this section, we present in-depth analyses to better understand our model with DocRED as the testbed. All base models use GloVe embeddings as opposed to BERT due to efficiency concerns.

**Intra- and Inter-Sentence Performance.** We breakdown the relation classification performance into intra-sentence reasoning and inter-sentence reasoning. Ideally, if only one sentence is needed to determine the relation of an entity pair, then it belongs to the intra-sentence category; if two or more sentences are needed, then it belongs to the inter-sentence category. We follow Nan et al. (2020) and approximate it by checking whether two entities are mentioned in one sentence.

The results are shown in Table 3. SIEF again consistently improves base models in terms of both Intra-F1 and Inter-F1. However, the improvement on Intra-F1 is larger than that on Inter-F1. This is because our SIEF encourages the model to focus on evidence by removing one sentence at a time, but does not explicitly model sentence relations. Based on this analysis, we plan to extend the SIEF framework with multi-sentence DocRE reasoning in our future work.

**Performance of predicting evidence sentences.** In our paper, we propose a sentence importance score to measure how much a sentence contributes to the classificat

\begin{table}
\begin{tabular}{l|c c|c c} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c|}{Dev} & \multicolumn{2}{c}{Test} \\ \cline{2-6}  & Igna F1 & F1 & Igna F1 & F1 \\ \hline \multicolumn{6}{c}{_DocSE Systems with GoVe_} \\ \hline LSR Nan et al. (2020) & 48.82 & 55.17 & 52.15 & 54.18 \\ Reconstructor Xu et al. (2021b) & 54.25 & 55.70 & 53.25 & 55.13 \\ DRN Xu et al. (2021) & 54.61 & 56.49 & 54.35 & 56.33 \\ \hline BiLSTM Yao et al. (2019) & 48.87 & 50.94 & 48.78 & 51.06 \\ +**SIEF** & 52.08 & 54.20 & 51.03 & 53.22 \\ HeerGSAN Xu et al. (2021b) & 52.17 & 54.40 & 52.07 & 53.52 \\ +**SIEF** & 54.49 & 56.30 & 53.94 & 55.85 \\ GAN Xing et al. (2020) & 53.05 & 55.29 & 52.66 & 55.08 \\ +**SIZE** & 55.07 & 56.96 & 54.72 & 56.75 \\ \hline \hline \multicolumn{6}{c}{_DocRE Systems with BERTBase_} \\ \hline Two-Phase Wang et al. (2019) & – & 54.42 & – & 53.92 \\ LSR Nan et al. (2020) & 52.43 & 59.00 & 56.97 & 59.05 \\ Reconstructor Xu et al. (2021b) & 58.13 & 60.18 & 57.12 & 59.45 \\ DRN Xu et al. (2021a) & 59.33 & 61.39 & 59.15 & 61.37 \\ ATDO Zhou et al. (2021) & 59.22 & 61.09 & 59.31 & 61.30 \\ DoneNet Zhang et al. (2021) & 59.86 & 61.83 & **59.93** & 61.86 \\ \hline BERT\textbackslash{} (Yie et al., 2020) & 54.63 & 56.77 & 53.93 & 56.27 \\
**+SIEF** & 57.13 & 59.11 & 57.87 & 58.93 \\ HeerGGAN Xu et al. (2021b) & 57.00 & 59.13 & 56.21 & 58.54 \\ +**SIEF** & 57.99 & 60.04 & 57.93 & 60.02 \\ GAIN Zeng et al. (2020) & 59.14 & 61.22 & 59.00 & 61.24 \\
**+SIEF** & **59.82** & **62.24** & 59.87 & **62.29** \\ \hline \end{tabular}
\end{table}
Table 1: Results on the development and test sets of the DocREG dataset. Bold indicates the best performance.

\begin{table}
\begin{tabular}{l|c c|c c} \hline Model & \multicolumn{2}{c|}{Dev} & \multicolumn{2}{c}{Test} \\ \cline{2-5}  & F1 & Fl1c & F1 & F1c \\ \hline BERT Yu et al. (2020) & 60.6 & 55.4 & 58.5 & 53.2 \\
**+SIEF** & 61.4 & 57.6 & 59.9 & 56.1 \\ \hline BERTs Yu et al. (2020) & 63.0 & 57.3 & 61.2 & 55.4 \\
**+SIEF** & **64.3** & **60.6** & **61.8** & **58.4** \\ \hline \end{tabular}
\end{table}
Table 2: Results on DialogRE.

\begin{table}
\begin{tabular}{l|c c|c c} \hline Model & Intra-F1 & Inter-F1 \\ \hline BiLSTM STM & 57.05 & 43.49 \\ +**SIEF** & 60.56 (\(\Delta_{n+3.51}\)) & 45.96(\(\Delta_{m+2.47}\)) \\ \hline HeteroGAN & 61.79 & 47.06 \\ +**SIEF** & 63.01 (\(\Delta_{n+1.22}\)) & 48.11(\(\Delta_{n+1.05}\)) \\ \hline GAIN & 61.67 & 48.77 \\
**+SIEF** & **63.21 (\(\Delta_{n+1.54}\))** & **48.98(\(\Delta_{n+0.21}\))** \\ \hline \end{tabular}
\end{table}
Table 3: Results of Intra-F1 results and Infer-F1 on development set of DocRED. The difference is compared between SIEF and the respective base model.



annotation. We evaluate such performance in Table 4 by Precision, Recall, and F1 scores against manually annotated evidence sentences that are provided in the dataset. In this analysis, we do not perform relation prediction, but concern about entity pairs knowingly having certain relations. Specifically, for entity pair \((e_{i_{h}},e_{i_{t}})\) with relation \(j\), we calculate the importance score \(g^{(-n)}_{ij}\) for each sentence and cut off evidence/non-evidence sentences with a threshold based on the development F1 score.

As seen, all base models achieve above 60% F1, suggesting that the proposed importance score is indeed indicative for predicting evidence and non-evidence sentences.

With the proposed SIEF framework, the performance improves for all metrics, with an average improvement of 2.95 F1 points across three base models. This further verifies that our SIEF framework not only improves relation extraction performance, but also is able to better detect evidence and non-evidence sentences, which is important for the interpretability of machine learning models.

**Robustness of DocRE models.** We further investigate the robustness of DocRE models by showing the difference between the predicted distributions with and without non-evidence sentences. We show in Figure 5 the scatter plots of the probability \(P\) based on the entire document and the probability \(\hat{P}^{(-n)}_{ij}\) with a random non-evidence sentence removed.

As shown in the figure, the points of the base models (left magenta plots) scatters over a wider range, whereas our SIEF training (right cyan plots) makes them more concentrated on the diagonal, indicating that the prediction \(P_{ij}\) on the entire document is mostly the same as \(\hat{P}^{(-n)}_{ij}\) with a non-evidence removed. This shows the robustness of SIEF-trained models, as they are less sensitive to non- evidences sentences for DocRE.

**Analysis on hyperparameter \(\beta\).** Our SIEF framework has one hyperaparameter \(\beta\) that controls how strict we treat a sentence as evidence or non-evidence (Section 4.3). We analyze the effect of \(\beta\) in Figure 4.

Figure 4: Performances of the classification (in F1 scores) on the development set of different hyperparameter \(\beta\) in Eqn. (6) during the training.

Figure 5: Robustness of DocRE models.

 

As seen, our SIEF approach consistently benefits the base models with a large range of \(\beta\) values. Intuitively, if \(\beta\) is too small, very few sentences will be treated as non-evidence and our sentence focusing loss is less effective; if \(\beta\) is too large, it has a high false positive rate of non-evidence sentences. Empirically, a moderate \(\beta\) around (0.6-0.8) yields the highest performance. From the plots, we also see that our hyperparameter \(\beta\) is insensitive to the base models, justifying our design of Eqn. (3).

**Sentence importance score VS other heuristics.** To investigate the effectiveness of our sentence importance score in Eqn. (3), we compare it with several alternative heuristics: 1) We randomly select half of the sentences as the non-evidence set, denoted by **Rand**; and 2) We consider the non-evidence set as the sentences without entity mentions, denoted by **NoMention**.

The results of the performance in terms of F1 and Ign F1 on the development set are shown in Table 5. As seen, the simple heuristic Rand outperforms the base model, as Rand can be thought of as noisy data augmentation. The NoMention heuristic outperforms Rand, as sentences without entity mentions are more likely to be non-evidence. Moroever, SIEF is superior to both Rand and NoMention, showing that our sentence importance scores is a more effective indicator of evidence and non-evidence sentences.

**Our sentence focusing loss VS learning from groundtruth.** We encourage the DocRE models to generate consistent output probabilities with and without non-evidence (Section 4.2) by a cross-entropy loss between two soft distributions \(P_{ij}\) and \(\hat{P}^{(-n)}_{ij}\). To investigate the effect of such a sentence focusing loss, we compare it with an alternative choice: we learn \(\hat{P}^{(-n)}_{ij}\) directly from the groundtruth label \(r_{ij}\).

Table 6 shows the results on the development set in terms of F1 and Ign F1. As seen, both methods can improve the performance of the base models. This confirms that removing non-evidence sentences can serve as a way of data augmentation, boosting the performance of DocRE models. Moreover, we observe that our sentence focusing loss is better than learning from the groundtruth labels, showing that the soft predictions provide more information than one-hot labels, consistent with knowledge distillation literature Hinton et al. (2015).

**Case Study.** Figure 6 shows a case study of GAIN and GAIN+SIEF models. For the entity pair (_Brad Wilk_, _Rage Against the Machine_), both GAIN and GAIN+SIEF predicts the relation "MemberOf", which is consistent with the reference. We see that Sentence 3 is non-evidence, and in principle, it should not affect DocRE prediction in this case. However, the base GAIN model makes a wrong prediction "not MemberOf", as the predicted probability is below the threshold, which is determined by validation based on predicted binary probabilities of all relations. By contrast, our SIEF model is able to make correct predictions when different non-evidence sentences are removed, demonstrating its robustness.

## 6 Conclusion

In this paper, we propose a novel Sentence Information Estimation and Focusing (SIEF) approach to document relation extraction (DocRE).

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline \multirow{2}{*}{Method} & \multicolumn{2}{c|}{BiLSTM} & \multicolumn{2}{c|}{HeterGSAN} & \multicolumn{2}{c}{GAIN} \\ \cline{2-7}  & Ign F1 & F1 & Ign F1 & F1 & Ign F1 & F1 \\ \hline Base & 48.87 & 50.94 & 52.17 & 54.40 & 53.05 & 55.29 \\ +**SIEF** & **52.08** & **54.20** & **54.49** & **56.30** & **55.07** & **56.96** \\ +GT truth & 50.36 & 52.56 & 52.65 & 54.69 & 53.75 & 55.87 \\ \end{tabular}
\end{table}
Table 6: Comparing our sentence focusing loss with learning from groundtruth labels (denoted by GTtruth).

Figure 6: Case Study.

 

We design a sentence importance score and a sentence focusing loss to encourage the model to focus on evidence sentences. The proposed SIEF is a general framework, and can be combined with various base DocRE models. Experimental results show that SIEF consistently improves the performance of base models in different domains, and that it improves the robustness of DocRE models.

## Acknowledgments

We are grateful to the anonymous reviewers and meta reviewers for their insightful comments and suggestions. This work is funded by the National Key Research and Development Program of China (No. 2020AAA0108000). The corresponding authors are Kehai Chen and Tiejun Zhao. Lili Mou is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant No. RGPIN2020-04465, the Amii Fellow Program, the Canada CIFAR AI Chair Program, a UAHJIC project, a donation from DeepMind, and Compute Canada (www.computecanada.ca).

## References

* S. Baldini Soares, N. FitzGerald, J. Ling, and T. Kwiatkowski (2019)Matching the blanks: distributional similarity for relation learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 2895-2905. Cited by: SS3.1.
* F. Christopoulou, M. Miwa, and S. Ananiadou (2019)Connecting the dots: document-level neural relation extraction with edge-oriented graphs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, pp. 4925-4936. External Links: Link, Document Cited by: SS1, SS2.1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pp. 4171-4186. Cited by: SS1, SS2.1.
* tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation, pp. 837-840. Cited by: SS1, SS2.1.
* P. Gupta, S. Rajaram, H. Schutze, and T. A. Runkler (2019)Neural relation extraction within and across sentence boundaries. In Proceedings of the Association for the Advance of Artificial Intelligence, pp. 6513-6520. Cited by: SS1, SS2.1.
* G. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network. In Proceedings of the Annual Conference on Neural Information Processing Systems Deep Learning and Representation Learning Workshop, Cited by: SS1, SS2.1.
* Q. Huang, S. Zhu, Y. Feng, Y. Ye, Y. Lai, and D. Zhao (2021)Three sentences are all you need: local path enhanced document relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, pp. 998-1004. Cited by: SS1, SS2.1.
* J. Li, Y. Sun, R. J. Johnson, D. Scialky, C. Wei, R. Leaman, A. Peter Davis, C. J. Mattingly, T. C. Wiegers, and Z. Lu (2016)BioCreative v CDR task corpus: a resource for chemical disease relation extraction. Database: The Journal of Biological Databases and Curation. Cited by: SS1, SS2.1.
* M. Mintz, S. Bills, R. Snow, and D. Jurafsky (2009)Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, pp. 1003-1011. Cited by: SS1, SS2.1.
* G. Nan, Z. Guo, I. Sekulic, and W. Lu (2020)Reasoning with latent structure refinement for document-level relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 1546-1557. Cited by: SS1, SS2.1.
* P. Pantel and M. Pennacchiotti (2006)Espresso: leveraging generic patterns for automatically harvesting semantic relations. In Proceedings of the International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics, pp. 113-120. Cited by: SS1, SS2.1.
* C. Quirk and H. Poon (2017)DISTANT supervision for relation extraction beyond the sentence boundary. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, pp. 1171-1182. Cited by: SS1, SS2.1.
* L. Song, Y. Zhang, D. Gildea, M. Yu, Z. Wang, and J. Su (2019)Leveraging dependency forest for neural medical relation extraction. Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, Cited by: SS1, SS2.1.
* D. Sorokin and I. Gruevych (2017)Context-aware representations for knowledge base relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1784-1789.

 

Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William W. J. Wang. 2019. Fine-tune BERT for DocRED with two-step process. arXiv preprint arXiv:1909.11898. Cited by: SS4.
* Z. Wei, J. Su, Y. Wang, Y. Tian, and Y. Chang (2020)A novel cascade binary tagging framework for relational triple extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 1476-1488. Cited by: SS3.
* M. Xiao and C. Liu (2016)Semantic relation classification via hierarchical recurrent neural network with attention. In Proceedings of the International Conference on Computational Linguistics: Technical Papers, pp. 1254-1263. Cited by: SS1, SS2.
* W. Xu, K. Chen, and T. Zhao (2021)Discriminative reasoning for document-level relation extraction. In Findings of the Annual Meeting of the Association for Computational Linguistics, pp. 1653-1663. Cited by: SS1, SS2.
* W. Xu, K. Chen, and T. Zhao (2021)Document-level relation extraction with reconstruction. In Proceedings of the Association for the Advance of Artificial Intelligence, Vol. 16, pp. 14167-14175. Cited by: SS1, SS2.
* Y. Xu, R. Jia, L. Mou, G. Li, Y. Chen, Y. Lu, and Z. Jin (2016)Improved relation classification by deep recurrent neural networks with data augmentation. In Proceedings of the International Conference on Computational Linguistics: Technical Papers, pp. 1461-1470. Cited by: SS1, SS2.
* Y. Xu, D. Ye, P. Li, X. Han, Y. Lin, Z. Liu, Z. Liu, Z. Huang, J. Zhou, and M. Sun (2019)DocRED: a large-scale document-level relation extraction dataset. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 764-777. Cited by: SS1, SS2.
* D. Ye, Y. Lin, J. Du, Z. Liu, P. Li, Maosong Sun, and Zhiyuan Liu (2020)Coreferential reasoning learning for language representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 7170-7186. Cited by: SS1, SS2.
* D. Yu, K. Sun, C. Cardie, and D. Yu (2020)Dialogue-based relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 4927-4940. Cited by: SS1, SS2.
* D. Zeng, K. Liu, S. Lai, G. Zhou, and J. Zhao (2014)Relation classification via convolutional deep neural network. In Proceedings of the International Conference on Computational Linguistics: Technical Papers, pp. 2335-2344. Cited by: SS1, SS2.
* S. Zeng, R. Xu, B. Chang, and L. Li (2020)Double graph based reasoning for document-level relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1630-1640. Cited by: SS1, SS2.
* N. Zhang, X. Chen, X. Xie, S. Deng, C. Tan, M. Chen, F. Huang, L. Si, and H. Chen (2021)Document-level relation extraction as semantic segmentation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 3999-4006. Cited by: SS1, SS2.
* W. Zhou, K. Huang, T. Ma, and J. Huang (2021)Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of the Association for the Advance of Artificial Intelligence, Vol. 1462-14620. Cited by: SS1, SS2.
* H. Zhu, Y. Lin, Z. Liu, J. Fu, T. Chua, and M. Sun (2019)Graph neural networks with generated parameters for relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 1331-1339.