SlowGuess/NAACL_2022_OCR huggingface dataset:
We compare our proposed method with eleven baselines in total. Based on the type of encoder, the comparable models are divided into two types, namely, two CNN-based models and nine BERT-based models. Specifically, CNN-based models include: 1) Proto-HATT (Gao et al., 2019a), prototypical networks modified with hybrid attention to focus on the crucial instances and features. 2) MLMAN (Ye and Ling, 2019), a multi-level matching and aggregation prototypical network. BERT-based models include: 3) Proto-BERT (Garcia and Brunna, 2018), a method that measures the similarity of prototypes and query instances for each relation. 4) BERT-PAIR (Gao et al., 2019b), a method that measures the similarity of sentence pairs. 5) REGRAB (Qu et al., 2020), a Bayesian meta learning method with an external global relation graph. 6) CTEG (Wang et al., 2020), a model that learns to decouple high co-occurrence relations, where two external information is added. 7) ConceptFERE (Yang et al., 2021), introducing the inherent concepts of entities to provide clues for relation prediction. 8) MTB (Baldini Soares et al., 2019), pre-train with their proposed matching the blank task on top of an existing BERT model. 9) CP (Peng et al., 2020), an entity masked con
Model	5-w	5-w	10-w	10-w
-1-s	-5-s	-1-s	-5-s
Proto-BERT	84.77	89.54	76.85	83.42
w/ relation info.
-Add	89.15	93.11	83.63	87.93
-Concat	80.34	85.11	73.78	80.85
w/ PRM	91.08	93.72	84.67	88.82
Table 3: Ablation Study in the validation set. w/ is the abbreviations of with. PRM is the abbreviation of Prototype Rectification Method/Module.
trastive pre-training framework for RE while utilizing prototype networks for fine-tuning on FSRE. 10) MapRE (Dong et al., 2021), a framework considering both label-agnostic and label-aware semantic mapping information in pre-training and fine-tuning. 11) HCRP (Han et al., 2021), introducing three modules containing Hybrid Prototype Learning, Relation-Prototype Contrastive Learning, and Task Adaptive Focal Loss for the model improvement.
To be noted, MTB, CP and MapRE all employ additional pre-training on BERT with Wikipedia data or contrastive learning to get better contextual representation. Moreover, we respectively use the original BERT and CP as our back-end language models. Therefore, among the 11 baselines mentioned above, Proto-BERT and CP are our most basic baseline.
5.3.2 NOTA Scenario
We compare our proposed method with three baselines in NOTA scenario: 1) BERT-PAIR (Gao et al., 2019b), a method that measures the similarity of sentence pairs. 2) MNAV, the Rank 1 method reported on CodaLab. 3) Ifc, the Rank 2 method reported on CodaLab.
6.1 General Scenario
All experimental results are shown in Table 1. CNN-based and BERT-based methods are both contained in the table. Proto-BERT represents the method on which our model is based, which means that this is the result of the model without introducing any improvement we propose. This result will also be analyzed and displayed in the ablation study. We apply our proposed method to BERT and CP. For obvious comparisons, the former is shown in the first part of BERT-based models, and the latter is shown in the second part of BERT-based models. The last two rows show the increase on
the test set compared to the basic models used by our method (i.e., Proto-BERT and CP).
There are several observations. We can observe that, regardless of using BERT or CP, our proposed model (i.e., PRM) outperforms all strong baselines. Particularly, when compared to the base model (i.e., Proto-BERT and CP), PRM achieves significant improvements, as shown in the last two rows of Table 1, further confirming the effectiveness of our innovation in explicitly utilizing relation information to guide prototype representation learning. Besides, the performances gains from the 5-w-1-s, 10-w-1-s setting over the current state-of-the-art model (i.e., HCRP) are larger than that of 5-w-5-s, 10-w-5-s, indicating that PRM is more suitable for the few-shot setting. The possible reason might be that when only one instance is given for each relation class (i.e., 1-shot condition), the Original Prototype is the statement representation of the one instance, which is more likely to deviate from the class center. Explicit constraints in PRM have a strong ability to pull this Original Prototype closer to the class center, while implicit constraints in HCRP are limited to dealing with such conditions.
6.2 NOTA Scenario
Results are shown in Table 2. MNAV and lfc are derived from CodaLab competition website that are 1st and 2nd methods respectively. The proposed PRM outperforms these methods with both BERT and CP baseline models, which demonstrates the effectiveness of PRM.
In order to analyze the effect of each component in our model (i.e., relations combination way and Prototype Rectification Method/Module), we conducted ablation study experiments on FewRel 1.0 and the results are reported in 3. Since labels of the test set of FewRel are not accessible, ablation experiments are performed on the validation set. There are two parts in Table 3. Proto-BERT means the baseline method without any relation information. The second part in the table represents the result obtained after adding a certain module, i.e., relations and PRM. In w/ relation info., instead of PRM, we perform two simple way: 1) Add: add relation representations to Original Prototypes directly. 2) Concat: concatenate relation representations and Original Prototypes, then through a linear layer for dimension reduction.
HCRP	PRM
Para.	110.66M	109.49M
Parameters to be adjusted
Training	learning rate
batch size
max iteration
Loss	λ	none
γ
Table 4: Comparison on the model complexity.
Model	5w1s	5w5s	10w1s	10w5s	Aver.
Proto-ADV	42.21	58.71	28.91	44.35	43.55
BERT-PAIR	67.41	78.57	54.89	66.85	66.93
HCRP (BERT)	76.34	83.03	63.77	72.94	74.02
PRM (BERT)	73.98	88.38	62.72	79.43	76.13
Table 5: Accuracy (%) of few-shot classification on the FewRel 2.0 domain adaptation test set, where Aver. denotes the average value of four settings.
We can obtain several observations. First, Add achieves better results than Proto-BERT, which proves that the idea of using relations to directly rectify the Original Prototypes is indeed effective. However, Concat obtains relatively poor results and is even inferior to the original Proto-BERT. A possible reason might be Concat needs to introduce an extra linear layer to reduce the dimension and thus bring some harmful parameters. Second, w/ PRM obtain further improvements on four settings, which demonstrates the effectiveness of PRM.
7.2 Model Complexity and Transferability
As we mentioned in Section 1, our method is simpler and has good transferability compared to the state-of-the-art, i.e., HCRP.
Complexity. The parameters of the two models are shown in Table 4. HCRP utilized three modules to jointly improve the model results, i.e., hybrid features generation, relation-prototype contrastive learning, and task adaptive loss function, whereas PRM only uses the GRM. More details and comparisons can be found in Appendix A, where PRM is compared with different modules of HCRP.
Transferability. We have already demonstrated that PRM still works in the NOTA scenario. In this section, we conduct experiments on domain adaptation settings with FewRel 2.0 and compare the results with HCRP, which is shown in Table 5. It can be seen that PRM is overall better than HCRP, which shows that PRM is also transferable to the domain adaptation setting. The possible reason
Figure 4: Prototypes Visualization. Left: Original Prototypes; Right: Rectified Prototypes
Figure 5: Instances Visualization. Left: Instances in Proto-BERT; Right: Instances in proposed PRM.
why PRM is worse than HCRP on 5w1s and 10w1s is that the FewRel 2.0 with domain adaptation setting only provides the name of relations without a specific description, which causes the model to fail to generate a strong relation representation for rectifying the prototypes.
7.3 Visualization
In order to further explore how PRM uses relations to rectify the original prototypes, we give the visualization results in Fig. 4, 5 with BERT on 5-way 1-shot of the validation set of FewRel 1.0. Fig. 4 and Fig. 5 show the visualization of prototypes and query instances respectively, where different colors represent different relation classes. From left to right in figures, Left means the original prototypes or statements of query instances without any relation information, Right means rectified prototypes or statements of query instances with the proposed PRM.
It can be seen that when the relations are not introduced into the model (Left), although the prototypes and instances can also be divided into different classes, the intra-class distances are not close enough, and there are multiple error points (i.e., black points). After introducing the relation information (Right), we can see that the error points are reduced while the representations of the same class are closer, especially for prototypes in Fig. 4. The observation shows that our proposed method of explicitly introducing relations has a part of the role of contrastive learning and is indeed beneficial to the improvement of the model.
In this paper, we proposed a prototype rectification method, PRM, with relations based on prototype framework, where a prototype rectification module is used for obtaining rectified prototypes. We further extended PRM to a none-of-the-above (NOTA) setting in few-shot learning. Extensive experiments demonstrate the effectiveness of the proposed method. We believe that the idea of finding global information to rectify prototypes explicitly with fewer parameters is general and can be extended to other few-shot tasks.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895-2905, Florence, Italy. Association for Computational Linguistics.
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
Yehuda Dar, Vidya Muthukumar, and Richard G Baraniuk. 2021. A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning. arXiv preprint arXiv:2109.02355.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bayu Distiawan, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. 2019. Neural relation extraction for knowledge base enrichment. In Proceedings of the
57th Annual Meeting of the Association for Computational Linguistics, pages 229-240.
Manqing Dong, Chunguang Pan, and Zhipeng Luo. 2021. Mapre: An effective semantic mapping approach for low-resource relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2694-2704.
Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. 2019a. Hybrid attention-based prototypical networks for noisy few-shot relation classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6407-6414.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019b. Fewrel 2.0: Towards more challenging few-shot relation classification. arXiv preprint arXiv:1910.07124.
Victor Garcia and Joan Bruna. 2018. Few-shot learning with graph neural networks. In 6th International Conference on Learning Representations, ICLR 2018.
Jiale Han, Bo Cheng, and Wei Lu. 2021. Exploring task difficulty for few-shot relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2605-2616.
Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. More data, more relations, more context and more openness: A review and outlook for relation extraction. arXiv preprint arXiv:2004.03186.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809.
Lifeng Jin, Linfeng Song, Yue Zhang, Kun Xu, Wei-yun Ma, and Dong Yu. 2020. Relation extraction exploiting full dependency forests. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8034-8041.
Shantanu Kumar. 2017. A survey of deep learning methods for relation extraction. arXiv preprint arXiv:1705.03645.
Ziran Li, Ning Ding, Zhiyuan Liu, Haitao Zheng, and Ying Shen. 2019. Chinese relation extraction with multi-grained information and external linguistic knowledge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4377-4386.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from context or names? an empirical study on neural relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3661-3672.
Meng Qu, Tianyu Gao, Louis-Pascal Xhonneux, and Jian Tang. 2020. Few-shot relation extraction via bayesian meta-learning on relation graphs. In International Conference on Machine Learning, pages 7867-7876. PMLR.
Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175.
Joaquin Vanschoren. 2018. Meta-learning: A survey. arXiv preprint arXiv:1810.03548.
Ricardo Vilalta and Youssef Drissi. 2002. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-95.
Yuxia Wang, Karin Verspoor, and Timothy Baldwin. 2020. Learning from unlabelled data for clinical semantic textual similarity. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 227-233.
Kaijia Yang, Nantao Zheng, Xinyu Dai, Liang He, Shujian Huang, and Jiajun Chen. 2020. Enhance prototypical network with text descriptions for few-shot relation classification. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2273-2276.
Shan Yang, Yongfei Zhang, Guanglin Niu, Qinghua Zhao, and Shiliang Pu. 2021. Entity concept-enhanced few-shot relation extraction. arXiv preprint arXiv:2106.02401.
Zhi-Xiu Ye and Zhen-Hua Ling. 2019. Multi-level matching and aggregation network for few-shot relation classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2872-2881.