SlowGuess/NAACL_2022_OCR huggingface dataset:
Our model and variants Based on the Bayesian meta-learning approach (Qu et al., 2020), we present two kinds of implementations:
(1) Models denoted as  $\mathrm{REGRAB} + {}^{*}$  construct a global relation graph with relation embeddings. Detailed ablation analysis is conducted on those variations.  $\mathrm{REGRAB} + \mathrm{Proto}$  only replaces the original relation embeddings with definition representations by our ProtoEnc. Based on this, RE-GRAB+Rel+Proto further replaces original instance encoder with our pre-trained RelEnc.
(2) Other models that discard the global graph construction as introduced in section 3.2. Those models are denoted as "instance encoder" +Proto, e.g., Rel+Proto uses both of our pre-trained encoders, CP+Proto applies the public instance encoder CP (Peng et al., 2020) to our approach.
Results and analysis Table 2 shows the performance of different models. Our model Rel+Proto achieves state-of-the-art performance over strong baselines. This demonstrates the advantages of leveraging both extra knowledge from the definition text and pre-trained instance encoder.
Table 3 shows that our approach can further improve other strong pre-trained instance encoder CP by a large margin on both FewRel 1.0 and FewRel 2.0, verifying the wide applicability of our approach. We empirically found that REGRAB cannot be easily applied to FewRel 2.0 and MIML
Table 2: Classification accuracies  $(\%)$  on FewRel 1.0 test set. Results with  $\dagger$  are reported as published.
Method	5-way 1-shot	5-way 5-shot	10-way 1-shot	10-way 5-shot
FewRel 1.0
MTB	91.10	95.40	84.30	91.80
CP	95.10	97.10	91.20	94.70
CP+Proto	96.64	98.14	93.76	96.48
FewRel 2.0 Domain Adaptation
MTB	74.70	87.90	62.50	81.10
CP	79.70	84.90	68.10	79.80
CP+Proto	83.11	90.80	73.02	83.08
Table 3: Accuracy (%) on FewRel datasets. Our prototype learning method improves previous best pretrained RC model (CP) and the reported baseline results are from their paper (Peng et al., 2020).
often produced unstable results.
Ablation analysis of the pre-trained prototype encoder and instance encoder is conducted with REGRAB. Compared with REGRAB, REGRAB+Proto consistently improves the performance on four settings, indicating the semantic relationship by our prototype encoder is more effective than the structural relationship from KBs. Compared with REGRAB+Proto, REGRAB+Rel+Proto achieves obvious improvements over four settings, showing the importance of pre-trained instance encoder. Besides, removing the operation of explicitly constructing a global relation graph, Rel+Proto further improves REGRAB+Rel+Proto, verifying our assumption that relation definitions could imply semantic similarities of different relations, and they provide a better way to construct representations for relations than through the relationships between them.
From Table 2, we observe CP-RI, whose instance encoder is pre-trained with the open-sourced code on our pre-training dataset, performs worse than the published results of CP, indicating our pre-training dataset may be noisier. Compared with CP-RI, BERT-EM performs more poorly, implying that the model using the label of each instance may suffer from noisy labels. Thus our design to leverage MIL algorithm is necessary.
4.2 Generalizability in Zero-Shot Relation Learning
To test how generalizable the prototype encoder is, we apply it to unseen relations in zero-shot RC to construct the prototype representation from their definitions. We test how effective such a relation prototype representation is.
# relations	2	5	10	15	25
BERT-Def	48.98	19.83	9.34	7.98	3.98
ZS-BERT	87.60	57.84	38.54	30.73	24.05
BERT-Proto	84.01	65.60	52.46	44.80	34.72
Dataset and evaluation setting We use two datasets, NYT-25 and PubMed-10, whose relations are nonoverlapping with FewRel training data and are obtained from the FewRel website. NYT-25 contains 25 relations from Wikidata and its sentences are from New York Times; PubMed-10 contains 10 relations and both its relations and instances are from the biomedical domain. Each relation in the two datasets has 100 manually labeled instances. Relations from Wikidata have names and descriptions as the definition text, while the relations in PubMed-10 have only relation names as the definition text.
For N-way zero-shot RC setting, the classification difficulty is increased with the increase of candidate relation number N. We vary N from 2 to the max number. In each setting, a candidate set is made of N-1 negative relations and a positive relation.
Compared models We denote our combined RC model by the pre-trained prototype encoder and instance encoder as BERT-Proto, whose pre-training data exclude relations in NYT-25. To verify if the pre-trained language models understand the definition text for RC, we present a baseline BERT-Def that has the same model structure as BERT-Proto but does not perform the joint training with the instance encoder and prototype encoder.
We also present a competitive baseline ZSBERT (Chen and Li, 2021) that classifies sentences based on embedding similarity. Similar to BERTProto, ZS-BERT learns two functions to project instances and relation definitions into an embedding space. The difference is that it uses a fixed pre-trained model, sentenceBERT (Reimers and Gurevych, 2019), to encode the definition text into an attribute vector. The attribute vectors are used to regularize the instance encoder during training and to compare with instance embeddings for classification during testing. We train ZS-BERT with FewRel training data for adapting to RC tasks.
Table 4: Zero-shot classification accuracies (%) on NYT-25 with increasing candidate relations.
#relations	2	5	8	10
BERT-Def	45.68	21.92	11.80	12.10
ZS-BERT	50.00	23.32	13.60	13.00
BERT-Proto	63.78	32.82	21.40	18.00
Table 5: Zero-shot classification accuracies (%) on PubMed-10 with increasing candidate relations.
Figure 2: The t-SNE visualization of instance embeddings and definition representations (indicated by arrows) by our pre-trained BERT-Proto on selected relations. Each relation is represented by one color and the relations seen in pre-training are in circles and the unseen are crosses.
Zero-shot results and discussion From Table 4 and Table 5, we see BERT-Proto outperforms ZSBERT, showing that our model is competitive for zero-shot RC even applied to a new domain and under the most difficult setting, while ZS-BERT performs slightly better than random on PubMed10. We believe the advantages are mainly generated by the pre-training technique that helps learn a general mapping function, i.e. the prototype encoder. Note that BERT-Def presents almost random results, implying the ineffective encoding of relation definitions when no joint training is performed.
To figure out how relation representations are distributed in the semantic space, we visualize the representations by our pre-trained instance encoder and prototype encoder with TSNE (Van der Maaten and Hinton, 2008) method. We also present relations of seen and unseen in the same space. The seen relations are selected from the FewRel training data and the unseen are from NYT-25.
From Figure 2, we observe that: (1) The definition representations of seen relations are close to instance embeddings of the same relation, show
4https://github.com/thunlp/FewRel
ing that the prototype encoder could effectively map the definition text to its prototype vector. For some unseen relations, their own definition representations could also be the closest to their instance embeddings, providing effective prior knowledge for RC. (2) The distance between different relations reflects their similarities, which are mainly determined by contextual features, such as entity types, sentence structures and context semantics. For example, the relations of "performer", "screenwriter" and "present in work" all express the relationship between a person and an artwork. Their instance embeddings are much closer to each other than with the relations about locations such as "headquarters location", even though "present in work" is the unseen relation. This shows that pre-trained instance encoder could extract effective contextual features for relation representations and build a meaningful semantic space to guide our prototype encoder learning. (3) For new-emerging relations, their prototype vectors are determined by both the definition text and labeled instances, so as to benefit from the possible connections with seen relations through pre-trained prototype encoder and instance encoder.
Relation classification (RC) is pivotal for natural language understanding and has been studied for a long time (Chieu and Ng, 2002). Supervised machine learning approaches achieve remarkable progress on RC (Kambhatla, 2004; Hendrickx et al., 2009), but rely on high-quality labeled data. To relieve the heavy burden of manual annotation, researchers study RC under distant supervision (Mintz et al., 2009) or few-shot RC (Han et al., 2018; Gao et al., 2019b). The former focuses on robust classifier training with automatically labeled noisy data (Lin et al., 2016; Li et al., 2020). Our work belongs to the latter, which aims to learn general knowledge transferable to new relations.
Few-shot learning methods have been studied for image classification (Ravi and Larochelle, 2017) and some classical approaches such as prototype network (Snell et al., 2017), model-agnostic meta-learning (Finn et al., 2017) have been applied for RC (Han et al., 2018). Two types of efforts have been devoted to improving few-shot RC. Firstly, some approaches (Ye and Ling, 2019; Gao et al., 2019a; Wang et al., 2020; Han et al., 2021; Ren et al., 2020; Ohashi et al., 2021) design specific
model architectures such as using attention mechanism to model complex interactions between labeled instances. However, these approaches are still limited when the few labeled instances are atypical and does not reflect the general patterns of the relation. Secondly, researchers leverage extra information to complement the insufficient labeled data (Qu et al., 2020; Dong et al., 2020). Our method belongs to this line.
Some methods (Qu et al., 2020; Zhang et al., 2021) leverage extra knowledge from KBs but they cannot deal with relations not covered by the KBs, showing limited applicability. Similar to our work, some studies (Dong et al., 2020; Yang et al., 2020) use relation names or descriptions as extra information. They design specific modules to integrate the extra information, but the whole few-shot RC model suffers from over-fitting and unstable performance due to the limited number of training relations. In contrast, our method adopts pre-training techniques to learn a general mapping function, which is more applicable and proven effective for domain adaptation.
Some recent studies (Baldini Soares et al., 2019; Ding et al., 2021; Peng et al., 2020) conduct RC-oriented pre-training to learn a general-purpose instance encoder. Such instance encoders improve RC on both supervised and few-shot learning settings. We adopt the same idea of pre-training but focus on learning the general function of mapping relation definitions to the prototype space for relation classification.
This paper studies prototypical representation learning for few-shot relation classification, and the key idea is to encode the definition text as prior knowledge to help classify new relations. We proposed to train a general-purpose prototype encoder that could encode the definition text of any relation into the prototype space. An instance encoder and the prototype encoder are trained jointly with a multi-instance learning method on distantly labeled data. Applying our prior prototypes with a Bayesian meta-learning approach, our method outperforms previous state-of-the-art models by using pre-trained instance encoder on two datasets, verifying its wide applicability. Our prototype model also presents competitive performance on the zero-shot learning setting.
We would like to thank Meng Qu, Benyou Wang and other teammates of Yuyang Zhang for their insightful suggestions and valuable discussion. We also thank all the reviewers for the generous comments. This research work is partly supported by the National Natural Science Foundation of China under Grant No. 62025208.
Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1388-1398, Florence, Italy. Association for Computational Linguistics.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895-2905, Florence, Italy. Association for Computational Linguistics.
Chih-Yao Chen and Cheng-Te Li. 2021. ZS-BERT: Towards zero-shot relation extraction with attribute representation learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3470-3479, Online. Association for Computational Linguistics.
Hai Leong Chieu and Hwee Tou Ng. 2002. A maximum entropy approach to information extraction from semi-structured and free text. In Eighteenth National Conference on Artificial Intelligence, page 786-791, USA. American Association for Artificial Intelligence.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ning Ding, Xiaobin Wang, Yao Fu, Guangwei Xu, Rui Wang, Pengjun Xie, Ying Shen, Fei Huang, Hai-Tao Zheng, and Rui Zhang. 2021. Prototypical representation learning for relation extraction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Bowen Dong, Yuan Yao, Ruobing Xie, Tianyu Gao, Xu Han, Zhiyuan Liu, Fen Lin, Leyu Lin, and
Maosong Sun. 2020. Meta-information guided meta-learning for few-shot relation classification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1594-1605, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR.
Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. 2019a. Hybrid attention-based prototypical networks for noisy few-shot relation classification. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6407-6414. AAAI Press.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019b. FewRel 2.0: Towards more challenging few-shot relation classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6250-6255, Hong Kong, China. Association for Computational Linguistics.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809, Brussels, Belgium. Association for Computational Linguistics.
Yi Han, Linbo Qiao, Jianming Zheng, Zhigang Kan, Linhui Feng, Yifu Gao, Yu Tang, Qi Zhai, Dongsheng Li, and Xiangke Liao. 2021. Multi-view interaction learning for few-shot relation classification. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, page 649-658, New York, NY, USA. Association for Computing Machinery.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian
Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions (SEW-2009), pages 94–99, Boulder, Colorado. Association for Computational Linguistics.
Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2017. Distant supervision for relation extraction with sentence-level attention and entity descriptions. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 3060-3066. AAAI Press.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online. Association for Computational Linguistics.
Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. In ACL, pages 178-181.
Yang Li, Guodong Long, Tao Shen, Tianyi Zhou, Lina Yao, Huan Huo, and Jing Jiang. 2020. Self-attention enhanced selective gate with entity-aware embedding for distantly supervised relation extraction. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8269-8276. AAAI Press.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2124-2133, Berlin, Germany. Association for Computational Linguistics.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003-1011, Suntec, Singapore. Association for Computational Linguistics.
Sora Ohashi, Junya Takayama, Tomoyuki Kajiwara, and Yuki Arase. 2021. Distinct label representations for few-shot text classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 831-836, Online. Association for Computational Linguistics.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3661-3672, Online. Association for Computational Linguistics.
Meng Qu, Tianyu Gao, Louis-Pascal A. C. Xhonneux, and Jian Tang. 2020. Few-shot relation extraction via bayesian meta-learning on relation graphs. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 7867-7876. PMLR.
Sachin Ravi and Hugo Larochelle. 2017. Optimization as a model for few-shot learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Haopeng Ren, Yi Cai, Xiaofeng Chen, Guohua Wang, and Qing Li. 2020. A two-phase prototypical network model for incremental few-shot relation classification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1618-1629, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4077-4087.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Yingyao Wang, Junwei Bao, Guangyi Liu, Youzheng Wu, Xiaodong He, Bowen Zhou, and Tiejun Zhao. 2020. Learning to decouple relations: Few-shot relation classification with entity-guided attention and confusion-aware training. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5799-5809, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Kaijia Yang, Nantao Zheng, Xinyu Dai, Liang He, Shujian Huang, and Jiajun Chen. 2020. Enhance prototypical network with text descriptions for few-shot relation classification. In CIKM '20: The 29th
ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 2273-2276. ACM.
Zhi-Xiu Ye and Zhen-Hua Ling. 2019. Multi-level matching and aggregation network for few-shot relation classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2872-2881, Florence, Italy. Association for Computational Linguistics.
Jiawen Zhang, Jiaqi Zhu, Yi Yang, Wandong Shi, Congcong Zhang, and Hongan Wang. 2021. Knowledge-enhanced domain adaptation in few-shot relation classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '21, page 2183-2191, New York, NY, USA. Association for Computing Machinery.