SlowGuess/NAACL_2022_OCR huggingface dataset:
Markus Eberts and Adrian Ulges. 2021. An end-to-end model for entity-level relation extraction using multi-instance learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3650-3660, Online. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803-5808, Hong Kong, China. Association for Computational Linguistics.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neural networks for one-shot image recognition. In ICML Deep Learning workshop.
Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687-692, New Orleans, Louisiana. Association for Computational Linguistics.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3036-3046, Minneapolis, Minnesota. Association for Computational Linguistics.
Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. Reasoning with latent structure refinement for document-level relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1546-1557, Online. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1-40, Jeju Island, Korea. Association for Computational Linguistics.
Dongyu Ru, Changzhi Sun, Jiangtao Feng, Lin Qiu, Hao Zhou, Weinan Zhang, Yong Yu, and Lei Li. 2021. Learning logic rules for document-level relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1239-1250, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Victor Sanh, Thomas Wolf, and Sebastian Ruder. 2019. A hierarchical multi-task approach for learning embeddings from semantic tasks. Proceedings of the AAAI Conference on Artificial Intelligence, 33:6949-6956.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Severine Verlinden, Klim Zaporojets, Johannes Deleu, Thomas Demeester, and Chris Develder. 2021. Injecting knowledge base information into end-to-end joint entity and relation extraction and coreference resolution. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1952-1957, Online. Association for Computational Linguistics.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5784-5789, Hong Kong, China. Association for Computational Linguistics.
Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021. Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14149-14157.
Liyan Xu and Jinho D. Choi. 2020. Revealing the myth of higher-order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527-8533, Online. Association for Computational Linguistics.
Liyan Xu and Jinho D. Choi. 2021. Adapted end-to-end coreference resolution system for anaphoric identities in dialogues. In Proceedings of the CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue, pages 55-62, Punta Cana,
Dominican Republic. Association for Computational Linguistics.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 764-777, Florence, Italy. Association for Computational Linguistics.
Klim Zaporojets, Johannes Deleu, Chris Develder, and Thomas Demeester. 2021. DWIE: An entity-centric dataset for multi-task document-level information extraction. Information Processing & Management, 58(4):102563.
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14612-14620.
A.1 Baseline: COREF
We use the Transformers-based end-to-end coreference model from Lee et al. (2018); Joshi et al. (2019) without higher-order inference (Xu and Choi, 2020) which still has near state-of-the-art performance on the standard COREF benchmark OntoNotes (Pradhan et al., 2012). We briefly introduce the model architecture as follows. The model first enumerates all possible spans over the document and performs topK pruning by mention scores, yielding a set of mention candidates. It then conducts a two-phase scoring to obtain the pairwise coreference scores: the first phase being a lightweight bilinear scoring, and the second phase being a slow but more accurate antecedent scoring.
In our setting, we remove the second phase and only use the bilinear scoring as mentioned in §2.1. We do not observe performance degradation on our experimented datasets, likely due to the fact that COREF in DocRED and DWIE is easier, e.g. pronouns are not annotated. In addition, we support predicting the singleton entity (entity with only one mention) in the same way as Xu and Choi (2021), by keeping all mention candidates whose mention scores  $>0$  regardless they co-refer with other mentions or not. Thereby a binary cross-entropy optimization on mention scores is added in the training loss.
A.2 +GC
For local graph pruning, we experiment the following two strategies. (1) randomly sample  $\gamma n$  nodes  $(\gamma \in (0,1]$  as a hyperparameter,  $n$  being the total number of nodes) as neighboring nodes; (2) keep top  $\gamma n$  neighboring nodes by highest sum of relation scores as a measurement of node saliency. We adopt the second strategy as it performs better in preliminary experiments.
We do not perform extra preprocessing for DocRED (Yao et al., 2019). However for DWIE (Zaporojets et al., 2021), there exist a tiny number of empty entities (clusters with zero mentions from the document for entity-linking purposes) in the annotations, which will raise errors in COREF evaluation. We perform the preprocessing step for DWIE that removes all empty entities and their involving relations.
TRN	DEV	TST	#T	#E	%S
DocRED	3053	998	1000	198.2	19.5	80.9%
DWIE	702	-	100	623.9	27.3	66.1%
Table 3: Statistics of the dataset DocRED and DWIE. TRN, DEV, TST are the numbers of documents in the training, development, and test set. #T and #E are the averaged numbers of tokens and entity clusters per document. %S is the averaged percentage of singleton entities out of all entities per document.
Table 3 lists important statistics of the two datasets. We only take the annotated training set for DocRED without using the distant supervised training set. As shown, both datasets have a large presence of singleton entities in their relation triples.
A.4 Experimental Settings
The Transformers encoder takes max input of two segments (up to 1024 subtokens per document) due to the GPU memory constraint. We employ the BERT learning rate as  $5 \times 10^{-5}$  and task learning rate as  $2 \times 10^{-4}$ .
For our proposed +GC setting, we set the margin  $m = 2$  in Eq (6) and  $\lambda$  for Eq (5) as  $10^{-3}$ . We set  $k = 24$  for local graph pruning that balances between performance and overhead.
For all our experiments, we use a batch size of 4 documents, and set 72/96 epochs for DocRED/DWIE respectively. All training is conducted on a Nvidia Titan RTX GPU.
A.5 Post-processing
The objective of the post-processing step is to map the entity ID of predicted entities according to gold entities. We substitute the entity ID of a predicted entity with its gold ID, if the predicted entity matches a gold entity; else, we assign a dummy ID to this predicted entity so that all its participating relation triples will be evaluated as incorrect by Codalab. After the entity ID mapping, we simply submit the predictions to Codalab without any further post-processing.