SlowGuess/NAACL_2022_OCR huggingface dataset:
Dat Quoc Nguyen and Karin Verspoor. 2018. Convolutional neural networks for chemical-disease relation extraction are improved with character-based word embeddings. BioNLP Workshop.
Sachin Pawar, Girish K Palshikar, and Pushpak Bhattacharyya. 2017. Relation extraction: A survey. arXiv preprint arXiv:1712.05191.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from context or names? an empirical study on neural relation extraction. In EMNLP.
Meng Qu, Xiang Ren, Yu Zhang, and Jiawei Han. 2018. Weakly-supervised relation extraction by pattern-enhanced embedding learning. In WWW.
Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Inter-sentence relation extraction with document-level graph convolutional neural network. In ACL.
Hengzhu Tang, Yanan Cao, Zhenyu Zhang, Jiangxia Cao, Fang Fang, Shi Wang, and Pengfei Yin. 2020. Hin: Hierarchical inference network for document-level relation extraction. In PAKDD.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In ICLR.
Patrick Verga, Emma Strubell, and Andrew McCallum. 2018. Simultaneously self-attending to all mentions for full-abstract biological relation extraction. In NAACL.
Difeng Wang, Wei Hu, Ermei Cao, and Weijian Sun. 2020. Global-to-local neural networks for document-level relation extraction. In EMNLP.
Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Wang. 2019. Fine-tune bert for docred with two-step process. arXiv preprint arXiv:1909.11898.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtopicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
David H Wolpert. 1992. Stacked generalization. Neural networks.
Ye Wu, Ruibang Luo, Henry CM Leung, Hing-Fung Ting, and Tak-Wah Lam. 2019. Renet: A deep learning approach for extracting gene-disease associations from literature. In RECOMB.
Yiqing Xie, Jiaming Shen, Sha Li, Yuning Mao, and Jiawei Han. 2022. Eider: Evidence-enhanced document-level relation extraction. In ACL (Findings).
Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021a. Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. In AAAI.
Han Xu, Zhang Zhengyan, Ding Ning, Gu Yuxian, Liu Xiao, Huo Yuqi, Qiu Jiezhong, Zhang Liang, Han Wentao, Huang Minlie, et al. 2021b. Pre-trained models: Past, present and future. arXiv preprint arXiv:2106.07139.
Wang Xu, Kehai Chen, and Tiejun Zhao. 2021c. Discriminative reasoning for document-level relation extraction. In ACL.
Wang Xu, Kehai Chen, and Tiejun Zhao. 2021d. Document-level relation extraction with reconstruction. In AAAI.
Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. 2020. Heterogeneous network representation learning: A unified framework with survey and benchmark. IEEE TKDE.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale document-level relation extraction dataset. In ACL.
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential reasoning learning for language representation. In EMNLP.
Shuang Zeng, Yuting Wu, and Baobao Chang. 2021. Sire: Separate intra-and inter-sentential reasoning for document-level relation extraction. In ACL.
Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph based reasoning for document-level relation extraction. In EMNLP.
Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extraction as semantic segmentation. In *IJCAI*.
Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang, Xi Chen, Wei Zhang, and Huajun Chen. 2019. Long-tail relation extraction via knowledge graph embeddings and graph convolution networks. In NAACL.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. 2017. Position-aware attention and supervised data improve slot filling. In EMNLP.
Zhenyu Zhang, Bowen Yu, Xiaobo Shu, Tingwen Liu, Hengzhu Tang, Wang Yubin, and Li Guo. 2020. Document-level relation extraction with dual-tier heterogeneous graph. In COLING.
Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. 2019. Learning deep bilinear transformation for fine-grained image representation. In NeurIPS.
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-level relation extraction with adaptive thresholding and localized context pooling. In AAAI.
Wenxuan Zhou, Hongtao Lin, Bill Yuchen Lin, Ziqi Wang, Junyi Du, Leonardo Neves, and Xiang Ren. 2020. Nero: A neural rule grounding framework for label-efficient relation extraction. In WWW.
A Multi-Task Learning Pipeline by Supervising Intermediate Steps (SAIS<sup>0</sup>
To explicitly teach the model to capture relevant contexts and entity type information for RE, we design four tasks to supervise the model's outputs in the corresponding intermediate steps. We illustrate the overall multi-task pipeline  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$  in Figure 4.
Figure 4: The overall multi-task learning pipeline of the proposed SAIS method  $(\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}})$ . By explicitly supervising the model's outputs in the intermediate steps via carefully designed tasks, we improve the RE performance.
B Ensemble Inference Algorithm with Evidence-based Data Augmentation (SAIS<sup>B</sup>
After training the multi-task pipeline  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$  proposed in Section 3, we further boost the model performance by evidence-based data augmentation and ensemble inference as discussed in Section 4. The detailed steps are explained in Algorithm 1 below.
We compare the proposed SAIS method against existing baselines based on three benchmarks: CDR (Li et al., 2016) and GDA (Wu et al., 2019) in Table 5, and DocRED (Yao et al., 2019) in Table 6. The details are explained in Section 5.
In particular, DocRED uses the MIT License, CDR is freely available for the research community, and GDA uses the GNU Affero General Public License. DocRED is constructed from Wikipedia and Wikidata and, therefore, contains information that names people. However, since our research focuses on identifying relations among real-world entities (including public figures) based on a given document, it is impossible to fully anonymize the dataset. We ensure that we only use publicly available information in our experiments. Our use of these datasets is consistent with their intended use. Although our method achieves state-of-the-art performance for RE and ER, using the predicted relations and evidence directly for downstream tasks without manual validation may increase the risk of errors carried forward due to the incorrect predictions. The experiments in this paper focus on English documents from biomedical and general domains, but our proposed framework can be easily extended to documents of other languages.
Model	CDR	GDA
BRAN (Verga et al., 2018)	62.1	-
CNN (Nguyen and Verspoor, 2018)	62.3	-
EoG (Christopoulou et al., 2019)	63.6	81.5
LSR (Nan et al., 2020)	64.8	82.2
SciBERT (Beltagy et al., 2019)	65.1	82.5
DHG (Zhang et al., 2020)	65.9	83.1
GLRE (Wang et al., 2020)	68.5	-
SSAN-SciBERT (Xu et al., 2021a)	68.7	83.7
ATLOP-SciBERT (Zhou et al., 2021)	69.4	83.9
SIRE-BioBERT (Zeng et al., 2021)	70.8	84.7
DocuNet-SciBERT (Zhang et al., 2021)	76.3	85.3
SAIS\(_{\text{RE+CR+ET}}^{\text{O}}\)-SciBERT (Ours)	79.0 ± 0.8	87.1 ± 0.3
SAIS\(_{\text{RE+ET}}^{\text{O}}\)-SciBERT	75.9 ± 0.9	86.1 ± 0.5
SAIS\(_{\text{RE+CR}}^{\text{O}}\)-SciBERT	74.5 ± 0.4	85.4 ± 0.2
SAIS\(_{\text{RE}}^{\text{O}}\)-SciBERT	72.8 ± 0.6	84.5 ± 0.3
Table 5: RE F1 results (\%) on the CDR and GDA test sets. We report the baseline performances from the corresponding papers.  $\mathrm{SAIS}_{\mathrm{RE} + \mathrm{CR} + \mathrm{ET}}^{\mathrm{O}}$  using three training tasks (i.e., RE, CR, and ET) scores the highest on both datasets and better than its variants with fewer training tasks.
Model	DocRED Dev	DocRED Test
Relation	Evidence	Relation	Evidence
Ign F1	F1	F1	Ign F1	F1	F1
CNN (Yao et al., 2019)	41.58	43.45	-	40.33	42.26	-
GAT (Veličković et al., 2018)	45.17	51.44	-	47.36	49.51	-
BiLSTM (Yao et al., 2019)	48.87	50.94	44.07	48.78	51.06	43.83
GCNN (Sahu et al., 2019)	46.22	51.52	-	49.59	51.62	-
EoG (Christopoulou et al., 2019)	45.94	52.15	-	49.48	51.82	-
AGGCN (Guo et al., 2019)	46.29	52.47	-	48.89	51.45	-
GEDA-BERTbase (Li et al., 2020)	54.52	56.16	-	53.71	55.74	-
GLRE-BERTbase (Wang et al., 2020)	-	-	-	55.40	57.40	-
LSR-BERTbase (Nan et al., 2020)	52.43	59.00	-	56.97	59.05	-
HeterGSAN-BERTbase (Xu et al., 2021d)	58.13	60.18	-	57.12	59.45	-
GAIN-BERTbase (Zeng et al., 2020)	59.14	61.22	-	59.00	61.24	-
DRN-BERTbase (Xu et al., 2021c)	59.33	61.39	-	59.15	61.37	-
SIRE-BERTbase (Zeng et al., 2021)	59.82	61.60	-	60.18	62.05	-
BERTbase (Wang et al., 2019)	-	54.16	-	-	53.20	-
BERT-TSbase (Wang et al., 2019)	-	54.42	-	-	53.92	-
HIN-BERTbase (Tang et al., 2020)	54.29	56.31	-	53.70	55.60	-
CorefBERTbase (Ye et al., 2020)	55.32	57.51	-	54.54	56.96	-
E2GRE-BERTbase (Huang et al., 2020)	55.22	58.72	47.14	-	-	-
SSAN-BERTbase (Xu et al., 2021a)	57.03	59.19	-	56.06	58.41	-
ATLOP-BERTbase (Zhou et al., 2021)	59.22	61.09	-	59.31	61.30	-
DocuNet-BERTbase (Zhang et al., 2021)	59.86	61.83	-	59.93	61.86	-
Eider-BERTbase (Xie et al., 2022)	60.51	62.48	50.71	60.42	62.47	51.27
SAISBAll-BERTbase (Ours)	59.98 ± 0.13	62.96 ± 0.11	53.70 ± 0.21	60.96	62.77	52.88
BERTlarge (Ye et al., 2020)	56.51	58.70	-	56.01	58.31	-
CorefBERTlarge (Ye et al., 2020)	56.82	59.01	-	56.40	58.83	-
RoBERTlarge (Ye et al., 2020)	57.19	59.40	-	57.74	60.06	-
CorefRoBERTlarge (Ye et al., 2020)	57.35	59.43	-	57.90	60.25	-
SSAN-RoBERTlarge (Xu et al., 2021a)	60.25	62.08	-	59.47	61.42	-
E2GRE-RoBERTlarge (Huang et al., 2020)	-	-	-	60.30	62.50	50.50
ATLOP-RoBERTlarge (Zhou et al., 2021)	61.32	63.18	-	61.39	63.40	-
DocuNet-RoBERTlarge (Zhang et al., 2021)	62.23	64.12	-	62.39	64.55	-
Eider-RoBERTlarge (Xie et al., 2022)	62.34	64.27	52.54	62.85	64.79	53.01
SAISBAll-RoBERTlarge (Ours)	62.23 ± 0.15	65.17 ± 0.08	55.84 ± 0.23	63.44	65.11	55.67
Table 6: RE and ER results (\%) on the develop and test sets of DocRED. Ign F1 refers to the F1 score excluding the relation instances mentioned in the train set. Baselines using  $\mathrm{BERT}_{\mathrm{base}}$  are separated into the graph-based (upper) and transformer-based (lower) groups. We report the test set scores from the official scoreboard and the baseline scores from the corresponding papers.  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{B}}$  achieves state-of-the-art performance on both RE and ER.