SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2: Results on DialogRE.
Model	Intra-F1	Inter-F1
BiLSTM	57.05	43.49
+SIEF	60.56 (Δ=+3.51)	45.96 (Δ=+2.47)
HeterGSAN	61.79	47.06
+SIEF	63.01 (Δ=+1.22)	48.11 (Δ=+1.05)
GAIN	61.67	48.77
+SIEF	63.21 (Δ=+1.54)	48.98 (Δ=+0.21)
Table 3: Results of Intra-F1 results and Infer-F1 on development set of DocRED. The difference is compared between SIEF and the respective base model.
consistently improves the base models, showing that SIEF is complementary to the modern BERT architecture. Especially, combining SIEF and GAIN (Zeng et al., 2020) with  $\mathrm{BERT}_{\mathrm{base}}$  encoding yields state-of-the-art performance in terms of F1.
We further conducted experiments on the DialogRE dataset, and compare our approach with the BERT baselines in Yu et al. (2020). As seen, the results are consistent with the improvement on DocRED, as our SIEF largely improves F1 and  $\mathrm{F1}_c$  for both base models. This further confirms the generality of our approach in different domains.
In the rest of this section, we present in-depth analyses to better understand our model with DocRED as the testbed. All base models use GloVe embeddings as opposed to BERT due to efficiency concerns.
Intra- and Inter-Sentence Performance. We breakdown the relation classification performance into intra-sentence reasoning and inter-sentence reasoning. Ideally, if only one sentence is needed to determine the relation of an entity pair, then it belongs to the intra-sentence category; if two or more sentences are needed, then it belongs to the inter-sentence category. We follow Nan et al. (2020) and approximate it by checking whether two entities are mentioned in one sentence.
The results are shown in Table 3. SIEF again consistently improves base models in terms of both Intra-F1 and Inter-F1. However, the improvement on Intra-F1 is larger than that on Inter-F1. This is because our SIEF encourages the model to focus on evidence by removing one sentence at a time, but does not explicitly model sentence relations. Based on this analysis, we plan to extend the SIEF framework with multi-sentence DocRE reasoning in our future work.
Performance of predicting evidence sentences. In our paper, we propose a sentence importance score to measure how much a sentence contributes to the classification without using additional
Figure 4: Performances of the classification (in F1 scores) on the development set of different hyperparameter  $\beta$  in Eqn. (6) during the training.
Model	Precision	Recall	F1
BiLSTM	60.14	68.41	64.01
+SIEF	65.00	67.99	66.46(Δ=+2.45)
HeterGSAN	65.40	70.95	68.06
+SIEF	71.40	70.21	70.80(Δ=+2.74)
GAIN	65.28	71.17	68.10
+SIEF	71.94	71.60	71.77(Δ=+3.67)
Table 4: Results of the evidence prediction on the development set of DocRED.
annotation. We evaluate such performance in Table 4 by Precision, Recall, and F1 scores against manually annotated evidence sentences that are provided in the dataset. In this analysis, we do not perform relation prediction, but concern about entity pairs knowingly having certain relations. Specifically, for entity pair  $(e_{i_h}, e_{i_t})$  with relation  $j$ , we calculate the importance score  $g_{ij}^{(-n)}$  for each sentence and cut off evidence/non-evidence sentences with a threshold based on the development F1 score.
As seen, all base models achieve above  $60\%$  F1, suggesting that the proposed importance score is indeed indicative for predicting evidence and non-evidence sentences.
With the proposed SIEF framework, the performance improves for all metrics, with an average improvement of 2.95 F1 points across three base models. This further verifies that our SIEF framework not only improves relation extraction performance, but also is able to better detect evidence and non-evidence sentences, which is important for the interpretability of machine learning models.
Robustness of DocRE models. We further investigate the robustness of DocRE models by showing the difference between the predicted distributions with and without non-evidence sentences. We show in Figure 5 the scatter plots of the probability  $P$  based on the entire document and the probability  $\hat{P}_{ij}^{(-n)}$  with a random non-evidence sentence removed.
As shown in the figure, the points of the base
Probability Distribution with and without non-evidence
Figure 5: Robustness of DocRE models.
models (left magenta plots) scatters over a wider range, whereas our SIEF training (right cyan plots) makes them more concentrated on the diagonal, indicating that the prediction  $P_{ij}$  on the entire document is mostly the same as  $\hat{P}_{ij}^{(-n)}$  with a non-evidence removed. This shows the robustness of SIEF-trained models, as they are less sensitive to non-evidences sentences for DocRE.
Analysis on hyperparameter  $\beta$ . Our SIEF framework has one hyperparameter  $\beta$  that controls how strict we treat a sentence as evidence or non-evidence (Section 4.3). We analyze the effect of  $\beta$  in Figure 4.
Method	BiLSTM	HeterGSAN	GAIN
Ign F1	F1	Ign F1	F1	Ign F1	F1
Base	48.87	50.94	52.17	54.40	53.05	55.29
+SIEF	52.08	54.20	54.49	56.30	55.07	56.96
+Rand	50.63	52.63	52.75	54.70	53.41	55.63
+NoMention	51.56	53.79	54.07	55.95	54.66	56.52
Table 5: Results of our approach and other heuristics.
Method	BiLSTM	HeterGSAN	GAIN
Ign F1	F1	Ign F1	F1	Ign F1	F1
Base	48.87	50.94	52.17	54.40	53.05	55.29
+SIEF	52.08	54.20	54.49	56.30	55.07	56.96
+GTruth	50.36	52.56	52.65	54.69	53.75	55.87
Table 6: Comparing our sentence focusing loss with learning from groundtruth labels (denoted by GTruth).
As seen, our SIEF approach consistently benefits the base models with a large range of  $\beta$  values. Intuitively, if  $\beta$  is too small, very few sentences will be treated as non-evidence and our sentence focusing loss is less effective; if  $\beta$  is too large, it has a high false positive rate of non-evidence sentences. Empirically, a moderate  $\beta$  around (0.6-0.8) yields the highest performance. From the plots, we also see that our hyperparameter  $\beta$  is insensitive to the base models, justifying our design of Eqn. (3).
Sentence importance score VS other heuristics. To investigate the effectiveness of our sentence importance score in Eqn. (3), we compare it with several alternative heuristics: 1) We randomly select half of the sentences as the nonevidence set, denoted by Rand; and 2) We consider the non-evidence set as the sentences without entity mentions, denoted by NoMention.
The results of the performance in terms of F1 and Ign F1 on the development set are shown in Table 5. As seen, the simple heuristic Rand outperforms the base model, as Rand can be thought of as noisy data augmentation. The NoMention heuristic outperforms Rand, as sentences without entity mentions are more likely to be non-evidence. Moreover, SIEF is superior to both Rand and NoMention, showing that our sentence importance scores is a more effective indicator of evidence and non-evidence sentences.
Our sentence focusing loss VS learning from groundtruth. We encourage the DocRE models to generate consistent output probabilities with and without non-evidence (Section 4.2) by a cross-entropy loss between two soft distributions  $P_{ij}$  and  $\hat{P}_{ij}^{(-n)}$ . To investigate the effect of such a sentence focusing loss, we compare it with an
Figure 6: Case Study.
alternative choice: we learn  $\hat{P}_{ij}^{(-n)}$  directly from the groundtruth label  $r_{ij}$ .
Table 6 shows the results on the development set in terms of F1 and Ign F1. As seen, both methods can improve the performance of the base models. This confirms that removing non-evidence sentences can serve as a way of data augmentation, boosting the performance of DocRE models. Moreover, we observe that our sentence focusing loss is better than learning from the groundtruth labels, showing that the soft predictions provide more information than one-hot labels, consistent with knowledge distillation literature (Hinton et al., 2015).
Case Study. Figure 6 shows a case study of GAIN and GAIN+SIEF models. For the entity pair (Brad Wilk, Rage Against the Machine), both GAIN and GAIN+SIEF predicts the relation "MemberOf", which is consistent with the reference. We see that Sentence 3 is non-evidence, and in principle, it should not affect DocRE prediction in this case. However, the base GAIN model makes a wrong prediction "not MemberOf", as the predicted probability is below the threshold, which is determined by validation based on predicted binary probabilities of all relations. By contrast, our SIEF model is able to make correct predictions when different non-evidence sentences are removed, demonstrating its robustness.
In this paper, we propose a novel Sentence Information Estimation and Focusing (SIEF) approach to document relation extraction (DocRE).
We design a sentence importance score and a sentence focusing loss to encourage the model to focus on evidence sentences. The proposed SIEF is a general framework, and can be combined with various base DocRE models. Experimental results show that SIEF consistently improves the performance of base models in different domains, and that it improves the robustness of DocRE models.
We are grateful to the anonymous reviewers and meta reviewers for their insightful comments and suggestions. This work is funded by the National Key Research and Development Program of China (No. 2020AAA0108000). The corresponding authors are Kehai Chen and Tiejun Zhao. Lili Mou is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant No. RGPIN2020-04465, the Amii Fellow Program, the Canada CIFAR AI Chair Program, a UAHJIC project, a donation from DeepMind, and Compute Canada (www.computecanada.ca).
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2895-2905.
Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, pages 4925-4936.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186.
George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ACE) program - tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation, pages 837-840.
Pankaj Gupta, Subburam Rajaram, Hinrich Schütze, and Thomas A. Runkler. 2019. Neural relation
extraction within and across sentence boundaries. In Proceedings of the Association for the Advance of Artificial Intelligence, pages 6513-6520.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In Proceedings of the Annual Conference on Neural Information Processing Systems Deep Learning and Representation Learning Workshop.
Quzhe Huang, Shengqi Zhu, Yansong Feng, Yuan Ye, Yuxuan Lai, and Dongyan Zhao. 2021. Three sentences are all you need: Local path enhanced document relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, pages 998-1004.
Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: A resource for chemical disease relation extraction. Database: The Journal of Biological Databases and Curation.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, pages 1003-1011.
Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. Reasoning with latent structure refinement for document-level relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1546-1557.
Patrick Pantel and Marco Pennacchiotti. 2006. *Espresso: Leveraging generic patterns for automatically harvesting semantic relations*. In *Proceedings of the International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics*, pages 113-120.
Chris Quirk and Hoifung Poon. 2017. Distant supervision for relation extraction beyond the sentence boundary. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, pages 1171-1182.
Linfeng Song, Yue Zhang, Daniel Gildea, Mo Yu, Zhiguo Wang, and Jinsong Su. 2019. Leveraging dependency forest for neural medical relation extraction. Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing.
Daniil Sorokin and Iryna Gurevych. 2017. Context-aware representations for knowledge base relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1784-1789.
Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William W. J. Wang. 2019. Fine-tune BERT for DocRED with two-step process. arXiv preprint arXiv:1909.11898.
Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, and Yi Chang. 2020. A novel cascade binary tagging framework for relational triple extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1476-1488.
Minguang Xiao and Cong Liu. 2016. Semantic relation classification via hierarchical recurrent neural network with attention. In Proceedings of the International Conference on Computational Linguistics: Technical Papers, pages 1254-1263.
Wang Xu, Kehai Chen, and Tiejun Zhao. 2021a. Discriminative reasoning for document-level relation extraction. In *Findings of the Annual Meeting of the Association for Computational Linguistics*, pages 1653-1663.
Wang Xu, Kehai Chen, and Tiejun Zhao. 2021b. Document-level relation extraction with reconstruction. In Proceedings of the Association for the Advance of Artificial Intelligence, 16, pages 14167-14175.
Yan Xu, Ran Jia, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. 2016. Improved relation classification by deep recurrent neural networks with data augmentation. In Proceedings of the International Conference on Computational Linguistics: Technical Papers, pages 1461-1470.
Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1785-1794.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 764-777.
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential reasoning learning for language representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 7170-7186.
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. 2020. Dialogue-based relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 4927-4940.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of the International Conference on Computational Linguistics: Technical Papers, pages 2335-2344.
Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph based reasoning for document-level relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1630-1640.
Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extraction as semantic segmentation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 3999-4006.
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of the Association for the Advance of Artificial Intelligence, 16, pages 14612-14620.
Hao Zhu, Yankai Lin, Zhiyuan Liu, Jie Fu, Tat-Seng Chua, and Maosong Sun. 2019. Graph neural networks with generated parameters for relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1331-1339.