SlowGuess/NAACL_2022_OCR huggingface dataset:
Baselines. DMCNN (Chen et al., 2015) uses a CNN with dynamic multi-pooling to obtain trigger representations for classification. MOGANED (Yan et al., 2019) proposes a multi-order GCN to capture interrelation between event trigger and argument representations based on dependency trees. DMBERT (Wang et al., 2019) improves DMCNN by training a  $\mathrm{BERT}_{\mathrm{BASE}}$  model as sentence encoder with dynamic multi-pooling. BERT-CRF stacks a CRF layer on top of  $\mathrm{BERT}_{\mathrm{BASE}}$  to model multiple event correlations in a single sentence. CLEVE (Wang et al., 2021b) proposes a contrastive learning framework fine-tuned on large-scale corpus with AMR structures obtained from AMR parsers, and combines AMR graph representations from a GNN and text representations from RoBERTaLarge to classify event types.
Results. As shown in Tab. 4, UNIST is able to improve event typing over BERT-CRF, and outperform all baselines except CLEVE. Note that in addition to being initialized from the same RoBERTa model as UNIST, CLEVE is further fine-tuned on large-scale corpus with AMR structures obtained from a separate parsing model (Xu et al., 2020) that also requires large human-annotated data to train. This indicates much more expensive supervision signals used by CLEVE. In contrast, UNIST effectively captures the meaning of event types and learns to classify event triggers by only fine-tuning on MAVEN, while still achieving promising performance without the need of any additional annotated resources.
In this section, we provide a detailed analysis to better understand the generalizability of UNIST and the effects of incorporating task description. Specifically, we examine UNIST's performance on few-shot and zero-shot entity typing on UFET,
2We were unable to reproduce the results of RECENT (Lyu and Chen, 2021) due to an error in its evaluation process that wrongly corrected all false positive predictions during testing. After correcting that error, the performance of RECENT was observed to be below the other baselines, and hence has not been included in the result discussion.
Figure 2: Comparison between MLMET and UNIST on few-shot and zero-shot prediction on UFET.
zero-shot relation classification on FewRel (Han et al., 2018), and how UNIST performs without task descriptions.
Few-shot & Zero-shot Entity Typing. A large portion of UFET test set labels have very few or even no training instances. We focus on entity types with no more than 10 instances in the training set, and compare the performance of UNIST<sub>BASE</sub> with the previous SOTA model MLMET on these few-shot and zero-shot labels.
As shown in Fig. 2, the advantage of UNIST over MLMET becomes more evident for rarer labels. For the most challenging zero-shot labels, UNIST substantially outperforms MLMET by  $7.2\%$  in F1 score, suggesting that UNIST is better generalized to infer low-resource and unseen entity types.
Zero-shot Relation Classification. We conduct experiments on FewRel (Han et al., 2018), a widely used benchmark for low-resource relation classification. FewRel includes 64/16/20 non-overlapping relation types for train/dev/test with 700 sentences collected from Wikipedia for each relation type. We evaluate UNIST under the  $N$ -way-0-shot setting, where the goal is to predict the correct relation among  $N$  candidate relations without seen training examples. Following previous studies (Cetoli, 2020; Dong et al., 2021), we report 5-way-0-shot and 10-way-0-shot accuracy on the FewRel dev set.
We compare UNIST with following baselines: REGRAB (Qu et al., 2020) proposes a bayesian meta-learning method to infer the posterior distribution of relation prototypes initialized with knowledge graph embeddings. BERT-SQuAD (Cetoli, 2020) formulates zero-shot relation classification as a question answering problem, and fine-tunes a BERTLarge QA model trained on SQuAD 1.1 (Rajpurkar et al., 2016) to predict relation types. MapRE (Dong et al., 2021) proposes a contrastive
Model	5-way 0-shot	10-way 0-shot
\( \text{REGRAB}^{\dagger} \) (Qu et al., 2020)	52.5	37.5
BERT+SQuAD* (Cetoli, 2020)	86.0	76.2
MapRE (Dong et al., 2021)	90.7	81.5
\( UNIST_{BASE} \)	91.2	82.9
Table 5: Accuracy results of zero-shot relation classification on FewRel. * marks models based on large versions of PLMs. † Results for REGRAB are taken from Dong et al. (2021).
Model	UFET	TACRED	MAVEN
UNISTBASE	49.3	74.3	68.3
- without task description	49.2	72.9	68.2
Table 6: F1 results of ablation experiments without task description on UFET, TACRED and MAVEN.
pre-training framework that learns input and relation representations from large-scale relation-annotated data. All baselines, as well as UNIST, are fine-tuned on the FewRel training set, and then evaluated on the FewRel dev set with a new set of relation types completely disjoint from that of the training set.
As shown in Tab. 5, UNIST outperforms the best baseline MapRE by  $0.5\%$  and  $1.4\%$  in accuracy on 5-way-0-shot and 10-way-0-shot tasks without first pre-training on any relation-annotated data. This demonstrates that by effectively captures label semantics, UNIST allows better knowledge transfer to handle unseen relation types.
Effects of Task Description. We conduct an ablation experiment on task descriptions using UNIST to better understand their effects on downstream tasks. As shown in Tab. 6, the performance on TA-CRED degrades much more significantly compared to that on UFET and MAVEN after removing task description. In lexical typing, the token span to be classified tend to share similar semantics with its type, and in many cases can be easily matched to its type label without explicitly specifying the task. In contrast, relation types are usually not semantically similar to its subject and object, and task description helps bridge this gap.
3.5 Multi-task Learning
With a unified task formulation, UNIST facilities learning a single model to jointly train on and simultaneously solve different semantic typing tasks. For more balanced training, We train UNIST on
Model	U	T	M	FewRel
5-way 0-shot	10-way 0-shot
UNISTBASE,U	49.3	6.1	19.6	68.9	56.0
UNISTBASE,T	3.0	74.3	5.2	62.5	48.4
UNISTBASE,M	22.5	4.3	68.3	55.1	39.7
UNISTBASE,U+T+M	48.5	74.2	68.2	80.9	72.0
Table 7: F1 results by multi-task learning on UFET (U), TACRED (T), MAVEN (M), and zero-shot transfer to FewRel.
the combined training set of UFET, TACRED and MAVEN, and report F1 performance on their respective test sets by following their respective evaluation protocol. We also include performance of single-task UNIST for comparison.
As shown in Tab. 7, our multi-task model obtain generally comparable performance to dedicated UNIST models trained separately on each of the three semantic typing tasks. Despite a slight decrease in performance on some of the tasks,  $\mathrm{UNIST}_{\mathrm{U} + \mathrm{T} + \mathrm{M}}$  is still able to outperform several strong baselines discussed earlier. Hence, UNIST provides a possible solution for learning a compact, unified model with a joint semantic embedding space across different semantic typing tasks. Moreover, this leads to a well-structured embedding space that better allows zero-shot transfer to new semantic typing tasks. To provide a preliminary analysis on the potential of UNIST on cross-task transfer, we evaluate both single-task and multitask UNIST models on FewRel dev set without training on any FewRel data. While FewRel is also a relation classification dataset like TACRED,  $75\%$  of the relation types in FewRel dev set do not exist in TACRED. Results in Tab. 7 show that by jointly training on different semantic typing tasks within a unified framework, UNIST demonstrate significantly stronger transferability to the unseen FewRel task compared to single-task variants. It would be meaningful to see if incorporating more datasets and tasks into UNIST would further benefit cross-task transfer, especially to tasks with limited data available for training. We leave this as a direction for further investigation.
We present two lines of relevant research topics. Each has a large body of work which we can only provide as a highly selected summary.
Semantic Typing. Semantic typing tasks can be
generally categorized into lexical typing (e.g., entity typing, event typing) and relational typing (or classification). A large number of specialized approaches have been developed for individual semantic typing tasks. For example, prior studies on entity typing have exploited label dependencies and hierarchies (Xu and Barbosa, 2018; Xiong et al., 2019), capturing label relations with knowledge bases (Dai et al., 2019; Jin et al., 2019), as well as automatic data augmentation and denoising techniques (Onoe and Durrett, 2019; Dai et al., 2021) to deal with fine-grained type vocabularies. Relation classification has been tackled by modeling dependency structures (Zhang et al., 2018), learning span representations (Joshi et al., 2020), entity representations (Yamada et al., 2020), and injecting external knowledge into pre-trained language models (Peters et al., 2019; Zhang et al., 2019; Wang et al., 2021a). Nevertheless, most previous methods have formulated semantic typing as a multi-class classification problem without capturing label semantics.
Learning Label Semantics. Previous studies have attempted formulating typing tasks into other tasks that allow more effective learning of label semantics. Following this idea, semantic typing tasks have been reformulated as prompt-based learning (Ding et al., 2021; Han et al., 2021), natural language inference (Yin et al., 2019; Sainz et al., 2021), question answering (Levy et al., 2017; Li et al., 2019; Du and Cardie, 2020), and translation (Paolini et al., 2021). Another line of research that is more relevant to our approach focuses on learning semantic label embeddings such that candidate labels can be ranked based on their affinity with the input in the embedding space. Semantic label embeddings have been successfully applied to a variety of tasks such as hierarchical text classification (Chen et al., 2021; Shen et al., 2021) and intent detection (Xia et al., 2018). In the context of semantic typing tasks, Chen et al. (2020) propose a learning-to-rank framework for multi-axis event process typing with indirect supervision from label glosses. Chen and Li (2021) use a pre-trained sentence embedding model to learn relation label embeddings from label descriptions. Dong et al. (2021) propose a contrastive pre-training framework to learn input and relation representations from large-scale relation-annotated data. Unlike previous approaches, UNIST does not rely on external label knowledge, training data or task-specific model components. Instead, UNIST effectively
captures label semantics solely from label names, and unify different semantic typing tasks into a single framework by incorporating task descriptions to be jointly encoded with the input.
We propose UNIST, a unified framework for semantic typing that exploits label semantics to learn a joint semantic embedding space for both inputs and labels. By incorporating model-agnostic task descriptions, UNIST can be easily adapted to different semantic typing tasks without introducing task-specific model components. Experimental results show that UNIST offers both strong performance and generalizability on entity typing, relation classification, and event typing. Our unified framework also facilitates learning a single model to solve different semantic typing tasks simultaneously, with performance on par with dedicated models trained on individual tasks.
We appreciate the anonymous reviewers for their insightful comments and suggestions. This material is supported in part by the DARPA MCS program under Contract No. N660011924033 with the United States Office Of Naval Research, and by the National Science Foundation of United States Grant IIS 2105329.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895-2905, Florence, Italy. Association for Computational Linguistics.
Alberto Cetoli. 2020. Exploring the zero-shot limit of FewRel. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1447-1451, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Chih-Yao Chen and Cheng-Te Li. 2021. ZS-BERT: Towards zero-shot relation extraction with attribute representation learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3470-3479, Online. Association for Computational Linguistics.
Haibin Chen, Qianli Ma, Zhenxi Lin, and Jiangyue Yan. 2021. Hierarchy-aware label semantics matching net
work for hierarchical text classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4370-4379, Online. Association for Computational Linguistics.
Muhao Chen, Hongming Zhang, Haoyu Wang, and Dan Roth. 2020. What are you trying to do? semantic typing of event processes. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 531-542, Online. Association for Computational Linguistics.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multipooling convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 167-176, Beijing, China. Association for Computational Linguistics.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018. Ultra-fine entity typing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87-96, Melbourne, Australia. Association for Computational Linguistics.
Amir DN Cohen, Shachar Rosenman, and Yoav Goldberg. 2020. Relation classification as two-way span-prediction. arXiv preprint arXiv:2010.04829.
Hongliang Dai, Donghong Du, Xin Li, and Yangqiu Song. 2019. Improving fine-grained entity typing with entity linking. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6210-6215, Hong Kong, China. Association for Computational Linguistics.
Hongliang Dai, Yangqiu Song, and Haixun Wang. 2021. Ultra-fine entity typing with weak supervision from a masked language model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1790-1799, Online. Association for Computational Linguistics.
Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Gee Kim. 2021. Prompt-learning for fine-grained entity typing. arXiv preprint arXiv:2108.10604.
Manqing Dong, Chunguang Pan, and Zhipeng Luo. 2021. MapRE: An effective semantic mapping approach for low-resource relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2694-2704, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 671-683, Online. Association for Computational Linguistics.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. *Ptr: Prompt tuning with rules for text classification.* arXiv preprint arXiv:2105.11259.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809, Brussels, Belgium. Association for Computational Linguistics.
Hailong Jin, Lei Hou, Juanzi Li, and Tiansi Dong. 2019. Fine-grained entity typing via hierarchical multi graph convolutional networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4969-4978, Hong Kong, China. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333-342, Vancouver, Canada. Association for Computational Linguistics.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019. Entityrelation extraction as multi-turn question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1340-1350, Florence, Italy. Association for Computational Linguistics.
Qing Liu, Hongyu Lin, Xinyan Xiao, Xianpei Han, Le Sun, and Hua Wu. 2021. Fine-grained entity typing via label reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4611-4622, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Shengfei Lyu and Huanhuan Chen. 2021. Relation classification with entity type restriction. In Findings of the Association for Computational Linguistics: ACLIJCNLP 2021, pages 390-395, Online. Association for Computational Linguistics.
Yasumasa Onoe, Michael Boratko, Andrew McCallum, and Greg Durrett. 2021. Modeling fine-grained entity types with box embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2051-2064, Online. Association for Computational Linguistics.
Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise distantly-labeled data for entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2407-2417, Minneapolis, Minnesota. Association for Computational Linguistics.
Yasumasa Onoe and Greg Durrett. 2020. Interpretable entity representations through large-scale typing. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 612-624, Online. Association for Computational Linguistics.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. In International Conference on Learning Representations.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3661-3672, Online. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Meng Qu, Tianyu Gao, Louis-Pascal AC Xhonneux, and Jian Tang. 2020. Few-shot relation extraction via bayesian meta-learning on relation graphs. In International Conference on Machine Learning.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirre. 2021. Label verbalization and entailment for effective zero and few-shot relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1199-1212, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics.
Jiaming Shen, Wenda Qiu, Yu Meng, Jingbo Shang, Xiang Ren, and Jiawei Han. 2021. TaxoClass: Hierarchical multi-label text classification using only class names. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4239-4249, Online. Association for Computational Linguistics.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021a. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405-1418, Online. Association for Computational Linguistics.
Xiaozhi Wang, Xu Han, Zhiyuan Liu, Maosong Sun, and Peng Li. 2019. Adversarial training for weakly supervised event detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 998-1008, Minneapolis, Minnesota. Association for Computational Linguistics.
Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General