SlowGuess/NAACL_2022_OCR huggingface dataset:
Domain Event Detection Dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1652-1671, Online. Association for Computational Linguistics.
Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, and Jie Zhou. 2021b. CLEVE: Contrastive Pre-training for Event Extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6283-6297, Online. Association for Computational Linguistics.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip Yu. 2018. Zero-shot user intent detection via capsule neural networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3090-3099, Brussels, Belgium. Association for Computational Linguistics.
Wenhan Xiong, Jiawei Wu, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Imposing label-relational inductive bias for extremely fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 773-784, Minneapolis, Minnesota. Association for Computational Linguistics.
Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020. Improving AMR parsing with sequence-to-sequence pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2501-2511, Online. Association for Computational Linguistics.
Peng Xu and Denilson Barbosa. 2018. Neural fine-grained entity type classification with hierarchy-aware loss. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 16-25, New Orleans, Louisiana. Association for Computational Linguistics.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442-6454, Online. Association for Computational Linguistics.
Haoran Yan, Xiaolong Jin, Xiangbin Meng, Jiafeng Guo, and Xueqi Cheng. 2019. Event detection with multi-order graph convolution and aggregated attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5766-5770, Hong Kong, China. Association for Computational Linguistics.
Semih Yavuz, Izzeddin Gur, Yu Su, Mudhakar Srivatsa, and Xifeng Yan. 2016. Improving semantic parsing via answer type inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 149-159, Austin, Texas. Association for Computational Linguistics.
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 50-61, Online. Association for Computational Linguistics.
Wenxuan Zhou and Muhao Chen. 2021a. An improved baseline for sentence-level relation extraction. arXiv preprint arXiv:2102.01373.
Wenxuan Zhou and Muhao Chen. 2021b. Learning from noisy labels for entity-centric information extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5381-5392, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
We run all single-task UNIST<sub>BASE</sub> experiments on NVIDIA RTX 2080Ti GPUs, and all UNIST<sub>LARGE</sub> and multi-task experiments on NVIDIA RTX A5000 GPUs. UNIST<sub>BASE</sub> and UNIST<sub>LARGE</sub> use base and large versions of RoBERTa as encoders with 125M and 355M parameters respectively. We conduct hyperparameter search within the following range:
- learning rate:  $\{3\mathrm{e} - 6,5\mathrm{e} - 6,1\mathrm{e} - 5,2\mathrm{e} - 5\}$
- Batch size:  $\{32, 64, 128\}$
- Number of training epochs: {50, 100, 200, 500, 1000}
- Ranking loss margin  $\gamma: \{0.1, 0.2, 0.3\}$
Learning rate	5e-6
Dropout rate	0.1
Adam ε	1e-6
Adam β1	0.9
Adam β2	0.999
Gradient clipping	1.0
Warmup ratio	0.1
Ranking loss margin	0.1
We optimize our models using AdamW (Loshchilov and Hutter, 2019) with linear learning rate decay. The best model checkpoints are selected based on dev set performance. Tab. 8 lists common hyperparameters used across all experiments. All datasets used in our experiments are in English. More details of individual tasks and experiments are provided below.
A.1 UFET
The UFET dataset is publicly available on its official website. Tab. 9 shows the hyperparameters and dev F1 score for UFET experiments.
Table 8: Common hyperparameters used in all experiments.
Name	UNISTBASE	UNISTLARGE
Batch size	64	64
Number of training epochs	1000	1000
Dev F1	49.2	49.5
Table 9: Hyperparameters and dev F1 score for UFET experiments.
3https://www.cs.utexas.edu/\~eunsol/ html_pages/open-entity.html
A.2 TACRED
The TACRED dataset we use is licensed by LDC  ${}^{4}$  . Tab. 10 shows the hyperparameters and dev F1 score for TACRED experiments.
Name	UNISTBASE	UNISTLARGE
Batch size	64	64
Number of training epochs	100	100
Dev F1	73.9	75.3
A.3 MAVEN
The MAVEN dataset is publicly available via its official github repository  $^{5}$ . Tab. 11 shows the hyperparameters and dev F1 score for MAVEN experiments.
Table 10: Hyperparameters and dev F1 score for TA-CRED experiments.
Name	UNISTBASE	UNISTLARGE
Batch size	64	64
Number of training epochs	100	100
Dev F1	68.4	68.5
A.4 FewRel
The FewRel dataset is publicly available via its official github repository  $^{6}$ . We report the average accuracy of 10 runs on the dev set during evaluation. Tab. 12 shows the hyperparameters for FewRel experiments.
Table 11: Hyperparameters and dev F1 score for MAVEN experiments.
Name	UNISTBASE
Batch size	64
Number of training epochs	50
A.5 Multi-task Experiments
We conduct multi-task experiments on the combined UFET, TACRED, and MAVEN training sets. We up-sample UFET training set by a factor of 10 for more balanced training. Tab. 13 shows the hyperparameters and dev set F1 for multi-task experiments.
Table 12: Hyperparameters for FewRel experiments.
Name	UNISTBASE
Batch size	128
Number of training epochs	100
Dev F1	47.5 (UFET)
72.9 (TACRED)
68.3 (MAVEN)
Table 13: Hyperparameters and dev F1 score for multitask experiments.
B Ethics Considerations
Our experiments are all conducted on openly available and widely used datasets. We do not augment any information to those data in this research, hence this research is not expected to introduce any additional biased information to existing information in those data. However, the model may potentially capture biases reflective of the pre-trained language models and datasets we use for our experiments, in such biases have pre-existed in these pre-trained models or datasets. This is a common problem for models trained on large-scale data, and therefore we suggest conducting a thorough bias analysis before deploying our model in any real-world applications.