SlowGuess/NAACL_2021_OCR huggingface dataset:
C. Chen and Vincent Ng. 2014. Sinocoreferencer: An end-to-end chinese event coreference resolver. In LREC.
Chen Chen and Vincent Ng. 2016. Joint inference over a lightly supervised information extraction pipeline: Towards event coreference resolution for resource-scarce languages. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, page 2913-2920. AAAI Press.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multipooling convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 167-176, Beijing, China. Association for Computational Linguistics.
Zheng Chen and Heng Ji. 2009. Graph-based event coreference resolution. In Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4), pages 54-57, Suntec, Singapore. Association for Computational Linguistics.
Zheng Chen, Heng Ji, and Robert Haralick. 2009. A pairwise event coreference model, feature impact and evaluation for event coreference resolution. In Proceedings of the Workshop on Events in Emerging Text Types, pages 17-22, Borovets, Bulgaria. Association for Computational Linguistics.
Prafulla Kumar Choubey and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2124-2133, Copenhagen, Denmark. Association for Computational Linguistics.
Prafulla Kumar Choubey, Kaushik Raju, and Ruihong Huang. 2018. Identifying the most dominant event in a news article by mining event coreference relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 340-345, New Orleans, Louisiana. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Chase Duncan, Liang-Wei Chan, Haoruo Peng, Hao Wu, Shyam Upadhyay, Nitish Gupta, Chen-Tse Tsai, Mark Sammons, and Dan Roth. 2017. UI CCG TACKBP2017 submissions: Entity discovery and linking, and event nugget detection and co-reference. In Proceedings of the 2017 Text Analysis Conference, TAC 2017, Gaithersburg, Maryland, USA, November 13-14, 2017. NIST.
C. Harris, K. J. Millman, S. Walt, Ralf Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, Nathaniel J. Smith, R. Kern, Matti Picus, S. Hoyer, M. Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fern'andez del R'io, Mark Wiebe, P. Peterson, Pierre G'erard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, Christoph Gohlke, and T. E. Oliphant. 2020. Array programming with numpy. Nature, 585 7825:357-362.
Eduard Hovy, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013. Events are not simple: Identity, non-identity, and quasi-identity. In Workshop on Events: Definition, Detection, Coreference, and Representation, pages 21-28, Atlanta, Georgia. Association for Computational Linguistics.
Yin Jou Huang, Jing Lu, Sadao Kurohashi, and Vincent Ng. 2019. Improving event coreference resolution by learning argument compatibility from unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 785-795, Minneapolis, Minnesota. Association for Computational Linguistics.
Heng Ji and Ralph Grishman. 2011. Knowledge base population: Successful approaches and challenges. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1148-1158, Portland, Oregon, USA. Association for Computational Linguistics.
Heng Ji, Joel Nothman, H Trang Dang, and Sydney Informatics Hub. 2016. Overview of tac-kbp2016 trilingual edl and its impact on end-to-end cold-start kbp. Proceedings of TAC.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8(0):64-77.
Anders Krogh and John Hertz. 1992. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems, volume 4. Morgan-Kaufmann.
Tuan Lai, Quan Hung Tran, Trung Bui, and Daisuke Kihara. 2019. A gated self-attention memory network for answer selection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5953-5959, Hong Kong, China. Association for Computational Linguistics.
LDC. 2005. English annotation guidelines for events version 5.4.3. Linguistic Data Consortium, 1.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999-8009, Online. Association for Computational Linguistics.
Ying Lin, Liyuan Liu, Heng Ji, Dong Yu, and Jiawei Han. 2019. Reliability-aware dynamic feature composition for name tagging. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 165-174, Florence, Italy. Association for Computational Linguistics.
J. Lu and Vincent Ng. 2017a. Learning antecedent structures for event coreference resolution. 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 113-118.
Jing Lu and Vincent Ng. 2015. Utd's event nugget detection and coreference system at KBP 2015. In Proceedings of the 2015 Text Analysis Conference, TAC 2015, Gaithersburg, Maryland, USA, November 16-17, 2015, 2015. NIST.
Jing Lu and Vincent Ng. 2016. Event coreference resolution with multi-pass sieves. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3996-4003, Porto-rož, Slovenia. European Language Resources Association (ELRA).
Jing Lu and Vincent Ng. 2017b. Joint learning for event coreference resolution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 90-101, Vancouver, Canada. Association for Computational Linguistics.
Jing Lu and Vincent Ng. 2018. Event coreference resolution: A survey of two decades of research. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 5479-5486. International Joint Conferences on Artificial Intelligence Organization.
Yaojie Lu, Hongyu Lin, Jialong Tang, Xianpei Han, and Le Sun. 2020. End-to-end neural event coreference resolution. arXiv preprint arXiv:2009.08153.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS'13, page 3111-3119, Red Hook, NY, USA. Curran Associates Inc.
Teruko Mitamura, Zhengzhong Liu, and Eduard H. Hovy. 2016. Overview of TAC-KBP 2016 event nugget track. In Proceedings of the 2016 Text Analysis Conference, TAC 2016, Gaithersburg, Maryland, USA, November 14-15, 2016. NIST.
Thien Huu Nguyen, Adam Meyers, and Ralph Grishman. 2016. New york university 2016 system for KBP event nugget: A deep learning approach. In Proceedings of the 2016 Text Analysis Conference, TAC 2016, Gaithersburg, Maryland, USA, November 14-15, 2016. NIST.
Adam Paszke, S. Gross, Francisco Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Kopf, E. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, B. Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. ArXiv, abs/1912.01703.
Haoruo Peng, Yangqiu Song, and Dan Roth. 2016. Event detection and co-reference with minimal supervision. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 392-402, Austin, Texas. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
M. Recasens, E. Hovy, and M. Martí. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. *Lingua*, 121:1138–1152.
Mark Sammons, Haoruo Peng, Yangqiu Song, Shyam Upadhyay, Chen-Tse Tsai, Pavankumar Reddy, Subhro Roy, and Dan Roth. 2015. Illinois CCG TAC 2015 event nugget, entity discovery and linking, and slot filler validation systems. In Proceedings of the
2015 Text Analysis Conference, TAC 2015, Gaithersburg, Maryland, USA, November 16-17, 2015, 2015. NIST.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1958.
Lucy Vanderwende, Michele Banko, and Arul Menezes. 2004. Event-centric summary generation. In Working notes of the Document Understanding Conference 2004. ACL.
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57:45.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020a. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020b. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Xiaodong Yu, Wenpeng Yin, and Dan Roth. 2020. Paired representation learning for event and entity coreference. arXiv preprint arXiv:2010.12808.
Tongtao Zhang, Hongzhi Li, Heng Ji, and Shih-Fu Chang. 2015. Cross-document event coreference resolution based on cross-media features. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP2015).
Section A.1 describes our symbolic feature predictors. Section A.2 provides the details of the datasets we used. Section A.3 describes the hyperparameters and their value ranges that were explored. Section A.4 presents our reproducibility checklist.
Dataset	Train (# Docs)	Dev (# Docs)	Test (# Docs)
ACE-2005	529	30	40
KBP 2016	509	139	169
Table 9: Basic statistics of the datasets.
A.1 Symbolic Feature Predictors
In an end-to-end setting, we train and use OneIE (Lin et al., 2020) to identify event mentions along with their subtypes. For other symbolic features, we train a simple joint model. More specifically, given a document, our joint model first forms contextualized representations for the input tokens using SpanBERT (Joshi et al., 2020). Each event mention's representation is then defined as the average of the embeddings of the tokens in its trigger. After that, we feed the mentions' representations into classification heads for feature value prediction. Each classification head is a standard multi-layer feedforward network with softmax output units.
The event detection performance of OneIE on the test set of ACE 2005 is 74.7 (Type-F1 score). OneIE's performance on KBP 2016 is 55.20 (Type-F1 score). For reference, the performance of the event detection component of  $\mathrm{E}^3\mathrm{C}$  (Lu et al., 2020) on KBP 2016 is 55.38 (Type-F1 score).
A.2 Datasets Description
In this work, we use two English within-document coreference datasets: ACE 2005 and KBP 2016. The ACE 2005 English corpus contains fine-grained event annotations for 599 articles from a variety of sources. We use the same split as that stated in (Chen et al., 2015), where there are 529/30/40 documents in the train/dev/test split. In ACE, a strict notion of event coreference is adopted, which requires two event mentions to be coreferential if and only if they had the same agent(s), patient(s), time, and location. For KBP 2016, we follow the setup of (Lu and Ng, 2017a), where there are 648 documents that can be used for training and 169 documents for testing. We train our model on 509 documents randomly chosen from the training documents and tune parameters on the remaining 139 training documents. Different from ACE, KBP adopts a more relaxed definition of event coreference, where two event mentions can be coreferent as long as they intuitively refer to the same real-world event. Table 9 summarizes the basic statistics of the datasets.
We use SpanBERT (spanbert-base-cased) as the Transformer encoder (Wolf et al., 2020a; Joshi et al., 2020). We did hyperparameter tuning using the datasets' dev sets. For all the experiments, we pick the model which achieves the best AVG score on the dev set, and then evaluate it on the test set. For each of our models, two different learning rates are used, one for the lower pretrained Transformer encoder and one for the upper layer. The optimal hyperparameter values are variant-specific, and we experimented with the following range of possible values:  $\{8,16\}$  for batch size,  $\{3e - 5,4e - 5,$ $5e - 5\}$  for lower learning rate,  $\{1e - 4,2.5e - 4,5e - 4\}$  for upper learning rate, and  $\{50,100\}$  for number of training epochs. Table 10 shows the value of  $\epsilon$  we used for each symbolic feature type. In general, the larger the discrepancy between the train and test accuracies, the larger the value of  $\epsilon$ .
Dataset	Features	ε (predicted mentions)	ε (gold mentions)
ACE 2005	Type	0.00	0.10
Polarity	0.00	0.02
Modality	0.15	0.20
Genericity	0.15	0.20
Tense	0.25	0.30
KBP 2016	Type	0.05	-
Realis	0.10	-
Table 10: The value of  $\epsilon$  for each feature type.
A.4 Reproducibility Checklist
We present the reproducibility information of the paper. Due to license reason, we cannot provide downloadable links for ACE 2005 and KBP 2016.
Implementation Dependencies Libraries Pytorch 1.6.0 (Paszke et al., 2019), Transformers 3.0.2 (Wolf et al., 2020b), Numpy 1.19.1 (Harris et al., 2020), CUDA 10.2.
Computing Infrastructure The experiments were conducted on a server with Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz and NVIDIA Tesla V100 GPUs. The allocated RAM is 187G. GPU memory is 16G.
Average Runtime Table 11 shows the estimated average run time of our full model.
Number of Model Parameters The number of parameters in a baseline model is about 109.7M
Dataset	One Training Epoch	Evaluation (Dev Set)	Evaluation (Test Set)
ACE 2005	65.5 seconds	2.3 seconds	2.4 seconds
KBP 2016	103.6 seconds	10.7 seconds	11.8 seconds
Table 11: Estimated average runtime of our full model.
parameters. The number of parameters in a full model trained on the KBP 2016 dataset is about 111.4M parameters. The number of parameters in a full model trained on the ACE 2005 dataset is about 113.8M parameters.
Hyperparameters of Best-Performing Models Table 12 summarizes the hyperparameter configurations of best-performing models. Note that Table 10 already showed the hyperparameters used for the noisy training method.
Hyperparameters	ACE 2005 (end-to-end)	KBP 2016 (end-to-end)	ACE 2005 (gold mentions)
Symbolic Features Used	All (CDGM)	All (CDGM)	All (CDGM)
Noisy Training	Yes	Yes	Yes
Lower Learning Rate	4e-5	5e-5	5e-5
Upper Learning Rate	2.5e-4	5e-4	5e-4
Batch Size	16	8	8
Number Epochs	100	50	50
Table 12: Hyperparameters for best-performing models (refer to Table 10 for the hyperparameters used for the noisy training method).
Expected Validation Performance We repeat training five times for each best-performing model. We show the average validation performance in Table 13. Our validation scores on KBP 2016 are not comparable to that of (Lu et al., 2020), because we split the original 648 training documents into the final train set and dev set randomly. We still use the same test set. For each best-performing model, we report the test performance of the checkpoint with the best AVG score in the main paper.
Dataset	Avg. CoNLL score	Avg. AVG score
ACE 2005 (end-to-end)	60.20	58.80
KBP 2016 (end-to-end)	54.86	48.83
ACE 2005 (gold mentions)	81.9	83.02
Table 13: Average validation performance.