SlowGuess/NAACL_2022_OCR huggingface dataset:
Wang Xu<sup>1</sup>, Kehai Chen<sup>1</sup>, Lili Mou<sup>2</sup>, Tiejun Zhao<sup>1</sup>  
<sup>1</sup>Harbin Institute of Technology, China
$^{2}$ Dept. Computing Science, Alberta Machine Intelligence Institute (Amii) University of Alberta, Canada
xuwang@hit-mtlab.net, {chenkehai,tjzhao} @hit.edu.cn, doublepower.mou@gmail.com
Document-level relation extraction (DocRE) aims to determine the relation between two entities from a document of multiple sentences. Recent studies typically represent the entire document by sequence- or graph-based models to predict the relations of all entity pairs. However, we find that such a model is not robust and exhibits bizarre behaviors: it predicts correctly when an entire test document is fed as input, but errs when non-evidence sentences are removed. To this end, we propose a Sentence Importance Estimation and Focusing (SIEF) framework for DocRE, where we design a sentence importance score and a sentence focusing loss, encouraging DocRE models to focus on evidence sentences. Experimental results on two domains show that our SIEF not only improves overall performance, but also makes DocRE models more robust. Moreover, SIEF is a general framework, shown to be effective when combined with a variety of base DocRE models.
Document-level relation extraction (DocRE) aims to predict entity relations across multiple sentences. It plays a crucial role in a variety of knowledge-based applications, such as question answering (Sorokin and Gurevych, 2017) and large-scale knowledge graph construction (Baldini Soares et al., 2019). Different from sentence-level relation extraction (Zeng et al., 2014; Xiao and Liu, 2016; Song et al., 2019), the supporting evidence in the DocRE setting may involve multiple sentences scattering in the document. Thus, DocRE is more a realistic setting, attracting increasing attention in the field of information extraction.
Most recent DocRE studies use the entire document as a clue to predict the relations of
[1] Rage Against the Machine is an American rap metal band from Los Angeles, California. [2] Formed in 1991, the group consists of vocalist Zack de la Rocha, guitarist Tom Morello, bassist Tim Commerford and drummer Brad Wilk. [3] After a self-issued demo, the band signed with Epic Records and released its debut album Rage Against the Machine in 1992. ...
Relation: MemberOf	Supporting Evidence: {1,2}
Model Input	{1,2,3}	{1,2}
Ground Truth	MemberOf	MemberOf
GAIN Prediction	MemberOf	not MemberOf (undesired)
Figure 1: A DocRE model predicts correctly for an entire document, but errs when a non-evidence sentence is removed.
all entity pairs without concerning where the evidence is located (Nan et al., 2020; Zeng et al., 2020; Xu et al., 2021a,b). However, one can identify the relation of a specific entity pair from a few sentences. Huang et al. (2021) show that irrelevant sentences in the document would hinder the performance of the model.
Moreover, we observe that a DocRE model, trained on the entire document, may err when non-evidence sentences are removed. In Figure 1, for example, we need to identify the relation "MemberOf" between the entities Brad Wilk and Rage Against the Machine. The evidence sentences are  $\{1,2\}$ , and humans can easily identify such a relation when reading sentences  $\{1,2\}$  only. However, the recent DocRE model GAIN (Zeng et al., 2020) identifies the relation "MemberOf" correctly from the entire document  $\{1,2,3\}$ , but predicts "not MemberOf" from sentences  $\{1,2\}$ . Intuitively, removing sentence  $\{3\}$  should not change the result, as this sentence does not provide information regarding whether "MemberOf" holds or not for the two entities. Such model behaviors are undesired, because it shows that the model is not robust and lacks interpretability.
To this end, we propose a novel Sentence Importance Estimation and Focusing (SIEF) framework to encourage the model to focus on evidence sentences for predicting the relation of
The code is publicly available at https://github. com/xwjim/SIEF
Human Language Technologies, pages 2920 - 2929
an entity pair. Specifically, we first evaluate the importance of each sentence by the difference between the output probabilities of the document with and without this sentence. If the predicted probability of a relation does not change, or even increases, when a sentence is removed, it typically indicates that the sentence is non-evidence. Then, we propose an auxiliary loss to encourage the model to produce the same output distribution, when the entire document is fed as input and when a non-evidence sentence is removed. In this way, the model pays more attention to the evidence sentences for the classification. Our SIEF method is a general framework that can be combined with different underlying DocRE models.
We evaluated the generality and effectiveness of our approach on the large-scale DocRED dataset (Yao et al., 2019). Experimental results show that the proposed approach combines well with various recent DocRE models and significantly improves the performance. We further evaluated our approach on a dialogue relation extraction dataset, DialogRE (Yu et al., 2020); our SIEF yields consistent improvement, showing the generality of our approach in different domains.
Relation extraction (RE) can be categorized by its granularity, such as sentence-level (Doddington et al., 2004; Xu et al., 2016; Wei et al., 2020) and document-level (Gupta et al., 2019; Zhu et al., 2019). Early work mainly focuses on sentence-level relation extraction. Pantel and Pennacchiotti (2006) propose a rule-based approach, and Mintz et al. (2009) manually design features for classifying relations. In the past several years, neural networks have become a prevailing approach for relation extraction (Xu et al., 2015; Song et al., 2019).
Document-level relation extraction (DocRE) is attracting increasing attention in the community, as it considers the interactions of entity mentions expressed in different sentences (Li et al., 2016; Yao et al., 2019). Compared with the sentence level, DocRE requires the model collecting and integrating inter-sentence information effectively. Recent efforts design sequence-based and graph-based models to address such a problem.
Sequence-based DocRE models encode a document by the sequence of words and/or sentences, for example, using the Transformer
architecture (Devlin et al., 2019). Zhou et al. (2021) argue that the Transformer attentions are able to extract useful contextual features across sentences for DocRE, and they adopt an adaptive threshold for each entity pair. Zhang et al. (2021) model DocRE as a semantic segmentation task and predict an entity-level relation matrix to capture local and global information.
Graph-based DocRE models abstract a document by graphical structures. For example, a node can be a sentence, a mention, and/or an entity; their co-occurrence is modeled by an edge. Then graph neural networks are applied to aggregate inter-sentence information (Quirk and Poon, 2017; Christopoulou et al., 2019; Zeng et al., 2020). Zeng et al. (2020) construct double graphs, applying graph neural networks to mention-document graphs and performing path reasoning over entity graphs. Xu et al. (2021a) explicitly incorporate logical reasoning, commonsense reasoning, and coreference reasoning into DocRE, based on both sequence and graph features.
Different from previous work, our paper proposes SIEF as a general framework that can be combined with various sequence-based and graph-based DocRE models. In our approach, we propose a sentence importance score and a sentence focusing loss to encourage the model to focus on evidence sentences, improving the robustness and the overall performance of DocRE models.
In this section, we present the formulation of document relation extraction (DocRE). Consider an unstructured document comprising  $N$  sentences,  $\mathbb{D} = \{\mathbf{s}_1,\mathbf{s}_2,\dots ,\mathbf{s}_N\}$ , where each sentence  $\mathbf{s}_n$  is a sequence words. In a DocRE dataset, the document  $\mathbb{D}$  is typically annotated with entity mentions, each mention (e.g., U.S. and USA) labeled by its conceptual entity  $e$  and its entity type (e.g., location).
A DocRE model  $\pmb{F}$  is usually formulated as multi-label classification (Yao et al., 2019).  $\pmb{F}_{j}$  predicts whether the  $j$ th relation holds for the  $i$ th marked entity pair in a document, given by
$$
P _ {i j} = \boldsymbol {F} _ {j} (\mathbb {D}, e _ {i _ {h}}, e _ {i _ {t}}) = \Pr \left[ r _ {i j} = 1 | \mathbb {D}, e _ {i _ {h}}, e _ {i _ {t}} \right] \tag {1}
$$
where  $e_{i_h}$  is the head entity and  $e_{i_t}$  is the tail entity;  $r_{ij} \in \{0,1\}$  is the groundtruth label regarding entity pair  $i$  and relation  $j$ .
Figure 2: We estimate the sentence importance (for a specific entity pair and relation) by the difference of the classification probabilities with and without the sentence. Then, we encourage the DocRE model to predict the same probability when the entire document is fed as input and when a non-evidence sentence is removed.
To train the model, the binary cross-entropy loss is used as the objective for parameter estimation:
$$
\begin{array}{l} \mathcal {L} _ {\text {r e l}} = - \sum_ {\mathbb {D} \in \mathcal {C}} \sum_ {i _ {h} \neq i _ {t}} \sum_ {j \in \mathcal {R}} \left\{r _ {i j} \log P _ {i j} \right. \tag {2} \\ \left. + \left(1 - r _ {i j}\right) \log \left(1 - P _ {i j}\right) \right\} \\ \end{array}
$$
where  $\mathcal{C}$  denotes the entire corpus and  $\mathcal{R}$  denotes the set of relation types.
During inference, we obtain the relation(s) of a given entity pair by thresholding the predicted probabilities, following most previous work (Yao et al., 2019; Zhou et al., 2021).
In this section, we will describe our approach in detail. The overview of our framework is shown in Figure 2. First, we describe the estimation of sentence importance in Section 4.1. Sentences with low importance scores are treated as non-evidence. Then, Sections 4.2 and 4.3 present our approach that encourages the model to produce the same output distribution, when the entire document is fed as input and when non-evidence sentences are removed. Section 4.4 further presents the architectures of DocRE models.
4.1 Sentence Importance Estimation
We estimate the importance of each sentence for a specific entity pair. Low-scored sentences will be treated as non-evidence, and in principle, can be removed without changing DocRE predictions.
We propose a sentence importance score based on the DocRE predictions with and without the sentence in question. Our observation is that the relation extraction task is usually monotonic to evidence, i.e., (non-strictly) more relations will
be predicted with more sentences. If we remove a sentence and the predicted probability of a relation decreases, then the sentence is likely to be the evidence. If the predicted probability does not change, then the sentence is likely to be non-evidence. Moreover, the predicted probability may sometimes increase when a sentence is removed, in which case the DocRE model is not robust, as this violates monotonicity.
Formally, we consider removing one sentence at a time, and the document with the  $n$ th sentence removed is denoted by  $\hat{\mathbb{D}}^{(-n)} = \{\mathbf{s}_1,\dots ,\mathbf{s}_{n - 1},\mathbf{s}_{n + 1},\dots ,\mathbf{s}_N\}$ . For a DocRE model  $\pmb{F}$ , we obtain the classification probabilities  $P_{ij} = F_j(\mathbb{D},e_{i_h},e_{i_t})$  based on the original document, and  $\hat{P}_{ij}^{(-n)} = F_j(\hat{\mathbb{D}}^{(-n)},e_{i_h},e_{i_t})$  with sentence  $n$  removed.
We propose the importance score as
$$
g _ {i j} ^ {(- n)} = P _ {i j} \log \frac {P _ {i j}}{\hat {P} _ {i j} ^ {(- n)}} \tag {3}
$$
The formula appears similar to Kullback-Leibler (KL) divergence. However, we only take one term in the KL summation, because the KL divergence, albeit asymmetric in its two arguments, cannot model the increase or decrease of  $\hat{P}_{ij}^{(-n)}$ , whereas our  $g_{ij}^{(-n)}$  is monotonically decreasing with  $\hat{P}_{ij}^{(-n)}$ . Compared with a naive difference or ratio between  $P_{ij}$  and  $\hat{P}_{ij}^{(-n)}$ , we find that our KL-like score is more robust in the scale of  $P_{ij}$  when determining non-evidence sentences.
We treat a sentence  $n$  as non-evidence if  $g_{ij}^{(-n)} < \beta$  for a thresholding hyperparameter  $\beta$ . The resulting set of non-evidence sentences is denoted by  $\mathbb{K}_{ij}$  for the an entity pair  $(e_{i_h}, e_{it})$  and relation  $j$ .
4.2 Sentence Focusing Loss
We propose a sentence focusing loss to encourage the model to produce the same output distribution when the entire document is fed as input and when non-evidence sentences are removed.
Ideally, the predicted probability should remain the same if we remove any combination of the sentences in  $\mathbb{K}_{ij}$ . Therefore, we penalize the extent to which the predicted probability is changed.
We propose the sentence focusing loss as:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {s f}} = - \sum_ {\mathbb {D} \in \mathcal {C}} \sum_ {i _ {h} \neq i _ {t}} \sum_ {j \in \mathcal {R}} \sum_ {\mathbb {J} _ {i j} \subseteq \mathbb {K} _ {i j}} \left\{P _ {i j} \log \left(\hat {P} _ {i j} ^ {(- \mathbb {J} _ {i j})}\right) \right. \\ + (1 - P _ {i j}) \log (1 - \hat {P} _ {i j} ^ {(- \mathbb {J} _ {i})}) \} \tag {4} \\ \end{array}
$$
where  $\mathbb{J}_{ij}$  is a subset of  $\mathbb{K}_{ij}$  and  $\hat{P}_{ij}^{(-\mathbb{J}_{ij})} = F_{j}(\mathbb{D} \backslash \mathbb{J}_{ij}, e_{i_{h}}, e_{i_{t}})$  is the predicted probability with  $\mathbb{J}_{ij}$  removed from  $\mathbb{D}$ , and the total loss is  $\mathcal{L} = (\mathcal{L}_{\mathrm{rel}} + \mathcal{L}_{\mathrm{sf}}) / 2$ .
Essentially, our sentence focusing loss ensures  $P_{ij}$  is close to  $\hat{P}_{ij}^{(-\mathbb{J}_{ij})}$ , which intuitively makes sense because non-evidence sentences should not affect the prediction. Our approach can also be thought of as a way of data augmentation. However, compared with one-hot groundtruth labels, our sentence focusing loss works with soft labels  $P_{ij}$  and  $\hat{P}_{ij}^{(-\mathbb{J}_{ij})}$ , which are believed to contain more information (Hinton et al., 2015), and our gradient propagates to both  $P_{ij}$  and  $\hat{P}_{ij}^{(-\mathbb{J}_{ij})}$  for training.
The calculation of Eqn. (4) is time- and resource-consuming, because the number of the subsets  $\mathbb{J}_{ij}$  grows combinatorially with the number of non-evidence sentences. Moreover it should be calculated repeatedly once the parameter of the model is updated. To this end, we propose a simplified training strategy to approximate Eqn. (4) in the next subsection.
We propose a strategy to simplify the calculation and the training procedure. Concretely, we only remove one non-evidence sentence in  $\mathbb{K}_{ij}$  at a time instead of a subset of  $\mathbb{J}_{ij} \subseteq \mathbb{K}_{ij}$ , and we aggregate the effect of different non-evidence sentences by:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {s f}} = - \sum_ {\mathbb {D} \in \mathcal {C}} \sum_ {n = 1} ^ {N} \sum_ {i _ {h} \neq i _ {t}} \sum_ {j \in \mathcal {R}} \mathbb {I} \left(g _ {i j} ^ {(- n)} <   \beta\right) \\ \left\{P _ {i j} \log \left(\hat {P} _ {i j} ^ {(- n)}\right) + \left(1 - P _ {i j}\right) \log \left(1 - \hat {P} _ {i j} ^ {(- n)}\right) \right\} \tag {5} \\ \end{array}
$$
where  $\mathbb{I}$  is the indicator function. Essentially, we linearly approximate the combination of multiple non-evidence sentences in (4) by an outer summation. In this way, the number of terms does not grow combinatorially, but linearly w.r.t.  $N$ .
In implementation, we further simply the summation over  $n$  by Monte Carlo sampling of a randomly selected sentence  $n$  in each gradient update. The loss is reformulated as follows:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {s f}} = - \sum_ {\mathbb {D} \in \mathcal {C}} \sum_ {i _ {h} \neq i _ {t}} \sum_ {j \in \mathcal {R}} \mathbb {I} \left(g _ {i j} ^ {(- n)} <   \beta\right) \\ \left\{P _ {i j} \log \left(\hat {P} _ {i j} ^ {(- n)}\right) + \left(1 - P _ {i j}\right) \log \left(1 - \hat {P} _ {i j} ^ {(- n)}\right) \right\} \tag {6} \\ \end{array}
$$
As seen, we need to forward the base models twice in each update, with and without the sentence  $n$ . Huang et al. (2021) propose a similar idea but train different entity pairs in a document based on different sets of sentences; all sentence are processed repeatedly among entity pairs in a document. Their approach is much slower than ours.
To sum up, the proposed SIEF framework identifies non-evidence sentences and penalizes the difference of predicted probabilities when a nonevidence sentence is removed. Our approach is a generic framework and can be adapted to various DocRE model easily, without introducing extra parameters into the model.
4.4 DocRE Model Architectures
Our SIEF can be applied to various base DocRE models. To evaluate its generality, we consider the following recent models.
BiLSTM (Yao et al., 2019) $^{2}$ . A bi-directional long short term memory (BiLSTM) encodes the document, and an entity is represented by BiLSTM's hidden states, averaged over entity mentions. The head and tail entity representations are fed to a multi-layer perceptron (MLP) for relation extraction.
$\mathbf{BERT}_{\mathrm{base}}$  (Devlin et al., 2019) $^3$ . A pre-trained language model is used for document encoding.
HeterGSAN (Xu et al., 2021b). HeterGSAN is a recent graph-based DocRED model, which constructs a heterogeneous graph of sentence, mention, and entity nodes; it uses graph neural networks for relation extraction.
$^{2}$ https://github.com/thunlp/DocRED
<sup>3</sup>https://github.com/DreamInvoker/GAIN
<sup>4</sup>https://github.com/xwjm/DocRE-Rec
Figure 3: The model architecture of GAIN with SIEF. A sentence is randomly removed from the document. The corresponding nodes and edges are removed from the mention-document graph and the entity graph.
GAIN (Zeng et al., 2020) $^3$ . GAIN constructs two graphs: mention-document graphs and entity graphs, and performs graph and path reasoning over the two graphs separately. When combining our SIEF with GAIN, we achieve the best performance among all the base models with SIEF on DocRED. Thus, we will explain this model in more detail.
Essentially, a node in the mention-document graph is either a mention or a document. The mentions are connected to its document, and two mentions are connected if they co-occur in one sentence. In the entity graph, two entities are connected if they are mentioned in one sentence. To classify the relation, GNN is applied to the mention-document graph, enhanced with path information in the entity graph, shown in Figure 3.
When combining SIEF with GAIN, we randomly remove one sentence from the document. The corresponding nodes and edges are removed in the GAIN's graphs. Then we obtain the output probabilities with and without the sentence,  $P_{ij}$  and  $\hat{P}_{ij}^{(-n)}$ , separately. If the sentence important score  $g_{ij}^{(-n)}$  in Eqn. (3) is below a threshold  $\beta$ , the sentence is treated as non-evidence for the entity pair  $(e_{i_h}, e_{i_t})$  and relation  $j$ . We apply the sentence focusing loss Eqn. (4) to improve the robustness.
For prediction, we apply the trained DocRE model to the entire document, because with our approach the model is already robust when nonevidence sentences are presented. Empirical results will show that our SIEF consistently improves the performance of base DocRE models.
Datasets. DocRED is a large-scale human-annotated dataset for document-level relation extraction (Yao et al., 2019). The dataset is constructed from Wikipedia and Wikidata, containing 3053 documents for training, 1000 for development, and 1000 for test. In total, it has 132,375 entities and 56,354 relational facts in 96 relation types. More than  $40\%$  of the relational facts require reasoning over multiple sentences. The standard evaluation metrics are F1 and Ign F1 (Yao et al., 2019; Zeng et al., 2020), where Ign F1 refers to the F1 score excluding the relational facts in the training set.
We also evaluated our approach on DialogRE (V2, Yu et al., 2020), which contains 36 relation types, 17 of which are interpersonal. We followed the standard split with 1073 training dialogues, 358 validation, and 357 test. Following Yu et al. (2020), we report macro F1 scores in both the standard and conversational settings; the latter is denoted by  $\mathrm{F1}_c$ .
Competing Methods. We experimented our SIEF on a number of base models, namely, BiLSTM,  $\mathrm{BERT}_{\mathrm{base}}$ , HeterGSAN, and GAIN (Section 4.4). These base models are all considered for comparison.
For DocRED, we consider additional competing methods: Two Phase (Wang et al., 2019), which first predicts whether the entity pair has a relation and then predicts the relation type; LSR (Nan et al., 2020), which constructs the graph by inducing a latent document-level graph; Reconstructor (Xu et al., 2021b), which encourages the model to reconstruct a reasoning path during training; DRN (Xu et al., 2021a), which considers different reasoning skills explicitly and uses graph representation and context representation to model the reasoning skills; ATLOP (Zhou et al., 2021), which aggregates contextual information by the Transformer attentions and adopts an adaptive threshold for different entity pairs; and DocuNet (Zhang et al., 2021), which models DocRE as a semantic segmentation task.
For DialogRE, we followed Yu et al. (2020) and considered BERT and  $\mathbf{BERT}_{\mathrm{s}}$  for comparison, where  $\mathbf{BERT}_{\mathrm{s}}$  prevents a model from overfitting by replacing of the interpersonal augment with a special token.
$^{5}$ https://github.com/nlpdata/dialogre
Model	Dev	Test
Ign F1	F1	Ign F1	F1
DocRE Systems with GloVe
LSR (Nan et al., 2020)	48.82	55.17	52.15	54.18
Reconstructor (Xu et al., 2021b)	54.25	55.70	53.25	55.13
DRN (Xu et al., 2021a)	54.61	56.49	54.35	56.33
BiLSTM (Yao et al., 2019)	48.87	50.94	48.78	51.06
+SIEF	52.08	54.20	51.03	53.22
HeterGSan (Xu et al., 2021b)	52.17	54.40	52.07	53.52
+SIEF	54.49	56.30	53.94	55.85
GAIN (Zeng et al., 2020)	53.05	55.29	52.66	55.08
+SIEF	55.07	56.96	54.72	56.75
DocRE Systems with BERTbase
Two-Phase (Wang et al., 2019)	-	54.42	-	53.92
LSR (Nan et al., 2020)	52.43	59.00	56.97	59.05
Reconstructor (Xu et al., 2021b)	58.13	60.18	57.12	59.45
DRN (Xu et al., 2021a)	59.33	61.39	59.15	61.37
ATLOP (Zhou et al., 2021)	59.22	61.09	59.31	61.30
DocuNet (Zhang et al., 2021)	59.86	61.83	59.93	61.86
BERTbase (Ye et al., 2020)	54.63	56.77	53.93	56.27
+SIEF	57.13	59.11	57.87	58.93
HeterGSAN (Xu et al., 2021b)	57.00	59.13	56.21	58.54
+SIEF	57.99	60.04	57.93	60.02
GAIN (Zeng et al., 2020)	59.14	61.22	59.00	61.24
+SIEF	59.82	62.24	59.87	62.29
Table 1: Results on the development and test sets of the DocRE dataset. Bold indicates the best performance.
Model	Dev	Test
F1	F1c	F1	F1c
BERT (Yu et al., 2020)	60.6	55.4	58.5	53.2
+SIEF	61.4	57.6	59.9	56.1
BERTs (Yu et al., 2020)	63.0	57.3	61.2	55.4
+SIEF	64.3	60.6	61.8	58.4
Implementation Details. We use the repositories $^{2,3,4,5}$  of base models to implement our approach. We mostly followed the standard hyperparameters used in the base models. Our SIEF has one hyperparameter  $\beta$  in Eqn. (5). It was set to 0.8, and Section 5.2 presents the effect of tuning  $\beta$ .
5.2 Results and Analyses
Main results. Table 1 presents the detailed results on the development and test sets of the DocRED dataset. We first compare DocRE systems with GloVe embeddings (Yao et al., 2019). We see that the proposed SIEF method significantly improves the performance of all base models, including the sequence model (i.e., BiLSTM) and graph models (i.e., HeterGSAN and GAIN); the average improvement is 2.05 points in terms of test F1. This shows that SIEF is compatible with both sequence and graph models, indicating the generality and effectiveness of the proposed method.
For the DocRE system with  $\mathrm{BERT}_{\mathrm{base}}$ , SIEF also