SlowGuess/NAACL_2022_OCR huggingface dataset:
Evaluation Metrics. We follow the setting in the previous work (Simon et al., 2019) to convert pseudo labels predicted by clustering to relation labels. In each cluster, the relation label with the largest proportion among the cluster is assigned to all samples as the prediction label. For evaluation metrics, we adopt three commonly-used metrics (Wu et al., 2019; Hu et al., 2020; Zhang et al., 2021) to measure the effectiveness of clustering:  $\mathrm{B}^3$  (Bagga and Baldwin, 1998), Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI). For  $\mathrm{B}^3$ ,  $\mathrm{B}^3$  precision and recall correspondingly measure the correct rate of putting each sentence in its cluster or clustering all samples into a single class. Then  $\mathrm{B}^3\mathrm{F}_1$  is computed as the harmonic mean of the  $\mathrm{B}^3$  precision and recall:
$$
\begin{array}{l} \mathrm {B} ^ {3} \text {p r e c i s i o n} = \underset {X, Y} {\mathbb {E}} P (g (X) = g (Y) \mid c (X) = c (Y)) \\ \mathrm {B} ^ {3} \operatorname {r e c a l l} = \underset {X, Y} {\mathbb {E}} P (c (X) = c (Y) \mid g (X) = g (Y)) \\ \end{array}
$$
NMI measures the information shared between the predicted label and the ground truth. When data are partitioned perfectly, the NMI score is 1, while it becomes 0 when prediction and ground truth are independent. ARI is a metric to measure the degree of agreement between the cluster and golden distribution, which ranges in [-1,1]. The more consistent two distributions, the higher the score.
Baselines. We compare RCL to previous methods consisting of CNN (Zeng et al., 2014), Attention BiLSTM (Zhou et al., 2016), RSNs (Wu et al., 2019), MTB (Soares et al., 2019), ZS-BERT (Chen and Li, 2021). For CNN, Attention BiLSTM and MTB, these methods have great success in supervised relation extraction (SRE) but fail to perform zero-shot prediction. Specifically, we consider two variations of MTB which only differ in the backbone (MTB-BERT and MTB-RoBERTa). For fair comparison and zero-shot prediction, we make the relation representation from encoder become the output of the SRE model, instead of originally outputting a probability vector whose dimension is equal to the seen relations. The dimension of output vector is same as RCL. The K-Means is applied over output vector to generate zero-shot prediction. Although RSNs
SemEval2010 Task8
Model	P	R	F1	NMI	ARI
CNN	38.37	38.49	38.42	17.06	15.43
Att-BiLSTM	41.46	41.79	41.6	21.45	19.97
Supervised RSN	33.14	47.06	38.41	11.98	10.96
MTB-BERT	45.1	46.35	45.71	28.12	23.69
MTB-RoBERTa	42.71	44.84	43.71	24.52	21.01
ZS-BERT	33.86	36.33	35.03	12.47	9.53
RCL w/o RC	50.31	54.87	52.45	34.55	28.97
RCL	68.1	67.95	68.02	55.91	54.71
Table 1: Experimental results(%) on SemEval2010 Task8 in terms of  $\mathbf{B}^3$  precision,  $\mathbf{B}^3$  recall,  $\mathbf{B}^3$  F1, NMI, ARI. We also report the standard F1 score results in Table 5.
is a open relation extraction method, its Supervised RSN model also meets the setting of zero-shot. For ZS-BERT, the original relation descriptions are used for FewRel and we collect the descriptions of relations for SemEval2010 Task8 from open resources. Then we use the sentence embeddings for K-Means to predict unseen relations. Note that we set the dimension of sentence embeddings same as RCL for fair comparison.
Results on SemEval2010 Task8. Table 1 show the comparison results on SemEval2010 Task8. RCL achieves the best performance, significantly outperforming the previous state-of-the-art with  $22.31\%$  F1,  $27.79\%$  NMI and  $31.02\%$  ARI improvements. Due to the relations of SemEval2010 Task8 dataset with high similarity, baseline models severely suffer from similar errors and the performances of baselines are poor. Another reason why baselines perform poorly is that small number of seen relations and class imbalance are more challenging for model. Moreover, SemEval2010 Task8 is much less related to the general domains on which the transformers are pretrained. However, comparing with baselines, experimental results show RCL can effectively mitigates similar problems and better use the general knowledge of the pre-training language model.
Results on FewRel. For FewRel, the experimental results are shown in Table 2. From the results, we observe that our model RCL outperforms existing baselines on FewRel when targeting at different numbers of unseen relations  $m$ . Specifically, RCL achieves an average of  $2.87\%$  F1,  $1.98\%$  NMI and  $2.98\%$  ARI improvements compared to previous best results. Since relations on FewRel are class balance and sufficient, MTB-BERT and MTB-
Model	FewRel
F1	m=5 NMI	NMI	ARI	F1	m=10 NMI	NMI	ARI	F1	m=15 NMI	NMI	ARI	F1	m=30 NMI	NMI	ARI	F1	m=40 NMI	ARI	F1	Avg. NMI	ARI
CNN	74.47	68.51	66.31	60.87	64.59	53.79	55.3	62.35	49.87	39.15	54.61	35.49	34.09	53.46	30.37
Att-BiLSTM	82.75	79.36	76.63	75.89	79.1	71.46	69.84	75.94	66.03	50.76	66.99	47.64	45.01	64.66	42.23
Supervised RSN	73.33	67.89	64.49	59.11	64.96	48.66	50.99	59.98	39.74	26.01	44.31	18.71	23.55	48.26	18.08
MTB-BERT	88.06	85.32	84.03	81.08	83.95	76.22	78.62	83.57	74.83	63.51	76.61	59.98	60.35	75.9	54.54
MTB-RoBERTa	90.14	87.12	86.7	82.39	84.77	78.03	79.78	84.35	76.82	62.98	75.91	58.83	60.58	75.99	55.08
ZS-BERT	74.51	69.24	66.96	70.63	74.1	65.23	63.33	70.7	59.24	46.43	61.66	42.94	45.68	64.43	42.68
RCL w/o RC	73.58	68.23	64.5	70.52	74.28	59.53	58.02	64.67	51.74	39.89	54.62	33.7	33.09	50.15	28.57
RCL	90.73	87.41	86.72	84.52	86.73	80.23	81.48	85.64	78.18	67.75	79.21	64.43	65.74	79.09	61.1
RoBERTa perform well among competing models but their performance is still lower than RCL. The reason is that their approaches cannot well deal with similar problems. ZS-BERT performs worse than most competing models because ZS-BERT severely relies on the unseen relation descriptions for prediction, while our approach can perform well without external resources. In addition, we find that the improvement of RCL gets larger when  $m$  is larger, especially when  $m = 40$ . It is obvious that it becomes more difficult for prediction since the number of unseen relations increases leading to more seriously similar problems.
Ablation Study. To better validate our model, we conduct an ablation study on each module by correspondingly ablating one. Note that MTB-BERT is the version of RCL without contrastive learning module. From Table 1 and Table 2, we can see that combining these two modules can result in a noticeable performance gain over two datasets. Especially in SemEval2010 Task8, RCL w/o RC outperforms existing baselines by all evaluation metrics, which prove the effectiveness of contrastive learning. However, our proposed RCL significantly outperforms RCL w/o RC with  $15.57\%$  F1,  $21.36\%$  NMI and  $25.74\%$  ARI improvements. It demonstrates that these two modules are complementary on relation representation learning: contrastive learning focuses on learning the difference between instances and implicitly obtaining some knowledge about the difference between relations while relation classification can explicitly learn the difference between relations by identifying the relations but cannot learn the difference between similar instances and suffers from similar problems. When the number of unseen relations increases on FewRel, RCL w/o RC performs worse than competing methods due to without effectively learning relation difference, which also shows that both two modules are important to final model performance.
Table 2: Experimental results(%) produced by the baseline models and the proposed model RCL on FewRel dataset in terms of  $\mathrm{B}^3$  F-score, NMI, ARI.  $m$  is the number of unseen relations, and we vary  $m$  in [5, 10, 15, 30, 40] to examine how performance is affected. RCL w/o RC means RCL without relation classification module. In addition, we also report the standard F1 score results in Table 6.
SemEval2010 Task8	FewRel
Data augmentation	F1	F1
None	58.14	86.95
Random Mask	60.25	87.42
Feature Cutoff	59.92	88.46
Dropout	68.02	90.73
Dropout+Random Mask	67.53	89.84
Dropout+Feature Cutoff	65.51	89.23
Table 3: Experimental results(%) with different data augmentation strategies over two datasets in term of  $\mathbf{B}^3$  F1 score. For FewRel, we report the results on  $m = 5$ .
Effect of Data Augmentations. To study the effect of data augmentations, we consider six different data augmentation strategies for contrastive learning in our experiments, including None (i.e. doing nothing), Random Mask, Feature Cutoff, Dropout, Composition of Dropout and Feature Cutoff (Dropout+Feature Cutoff) and Composition of Dropout and Random Mask (Dropout+Random Mask).
The results are shown in Table 3. We can make the following observations: (a) Dropout is the most effective strategy, outperforming all competing strategies. It demonstrates that Dropout essentially acts as minimal data augmentation (Gao et al., 2021) and the noise produced by Dropout can make model learn the difference between similar instances better. (b) When compared with None, Random Mask and Feature Cutoff also improve performance across two datasets. Moreover, Dropout+Random Mask and Dropout+Feature Cutoff significantly outperform Random Mask and Feature Cutoff with roughly 6 and 2 points gain respectively while Dropout still outperforms these two composition of augmentations. It shows that different from the image domain (Chen et al., 2020), composition of augmentations is not always effective for the text domain. (c) We find that our model can improve performance on two datasets
Figure 3: Left: The results of RCL and MTB-BERT with different numbers of seen relations. Right: The performance of RCL with different fractions of unseen instances available for training. The number of unseen relations is set to 10 on FewRel.
Figure 4: t-SNE visualization of unseen relation representations learned by MTB-BERT and RCL on SemEval2010 Task8 dataset.
even without any data augmentation (None), especially for SemEval2010 Task8 (from 45.71 to 58.14). This is because None tunes the representation space by keeping each representation away from others, even if it has no effect on minimizing the distance between instance and its augmented view since the embeddings of augmented view are same with original instance. It also demonstrates that the effectiveness of the contrastive learning without external resources.
Effect of Number of Seen Relations. In this section, we study the effect of the number of seen relations on FewRel which contains sufficient relations. In our experiment, we vary the number of seen relations  $n$  from 10 (insufficient) to 70 (sufficient) and consistently set the number of unseen relations  $m$  to 10. Experimental results are presented in Figure 3. As the number of seen relations increases, RCL continuously outperforms MTB-BERT, which shows the effectiveness of our approach. More specifically, when  $n$  is set to 10, RCL can achieve  $90\%$  F1 score of the model trained on the full seen relations. In addition, the performance of RCL declines more slighter and smoother than MTB-BERT when seen relations gradually become insufficient (from 30 to 10), showing the robustness of our approach.
Capability under Few-shot Settings. In this section, we conduct the experiment of few-shot prediction by following the setting of Chen and Li (2021) to understand the capability of RCL. We move a small fraction of sentences of each unseen relation from test data to training data. Experimental results are shown in Figure 3. As expected on two datasets, RCL achieves more F1 score improvement with more unseen relation instances available at the training stage. When the fraction is set to  $4\%$ , RCL can achieve  $90\%$  F1 score on FewRel and  $80\%$  F1 score on SemEval2010 Task8. It shows the capability of few-shot learning for RCL.
(a) MTB-BERT
(b) RCL (ours)
Visualization of Relation Representations. To intuitively show how our approach helps to learn better relation representations on seen relations, we visualize the representations of unseen relations by using t-SNE (Van der Maaten and Hinton, 2008) to reduce the dimension to 2. We randomly choose 4 relations as unseen relation from SemEval2010 Task8 and the visualization results are shown in Figure 4. In each figure, relation instances are colored according to their ground-truth labels.
As we can see from Figure 4(a), the data points from MTB-BERT are mingled with different clusters, especially for red points. The reason is these instances possess similar relations or similar entities, and MTB-BERT has not learned the corresponding knowledge to deal with similar problems. However, as illustrated in Figure 4(b), RCL effectively mitigates these two types of similar problems since our approach can learn the difference between instances and the difference between seen relations. It again exhibits the effectiveness of the contrastive loss and multi-task learning structure. We also provide a case study in the Appendix F.
In this paper, we propose a jointly framework for zero-shot relation extraction to mitigate two types of similar errors: Similar Relations and Similar Entities. Different from conventional zero-shot relation extraction models which require external resources for training and test, our model does not require external resources. We demonstrate the effectiveness of our framework on two datasets, and our method achieves new state-of-the-art performance. Furthermore, we compare various data augmentation strategies for contrastive learning and provide fine-grained analysis for interpreting how our approach works.
Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 79-85.
Chih-Yao Chen and Cheng-Te Li. 2021. ZS-BERT: Towards zero-shot relation extraction with attribute representation learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3470–3479, Online. Association for Computational Linguistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR.
Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong. 2017. On sampling strategies for neural network-based collaborative filtering. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 767-776.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for un
supervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghda, Sebastian Padó, Marco Pennacchiotti, Lorenzo Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multiway classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33-38.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580.
Xuming Hu, Lijie Wen, Yusong Xu, Chenwei Zhang, and Philip Yu. 2020. SelfORE: Self-supervised relational feature learning for open relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3673-3682, Online. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333-342, Vancouver, Canada. Association for Computational Linguistics.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2124-2133, Berlin, Germany. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003-1011, Suntec, Singapore. Association for Computational Linguistics.
Abiola Obamuyide and Andreas Vlachos. 2018. Zero-shot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 72-78, Brussels, Belgium. Association for Computational Linguistics.
Longhua Qian, Guodong Zhou, Fang Kong, Qiaoming Zhu, and Peide Qian. 2008. Exploiting constituent dependencies for tree kernel-based semantic relation extraction. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 697-704, Manchester, UK. Coling 2008 Organizing Committee.
Swarnadeep Saha and Mausam. 2018. Open information extraction from conjunctive sentences. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2288-2299, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
David Sculley. 2010. Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web, pages 1177-1178.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough-to-beat data augmentation approach for natural language understanding and generation. arXiv preprint arXiv:2009.13818.
Étienne Simon, Vincent Guigue, and Benjamin Piwowarski. 2019. Unsupervised information extraction: Regularizing discriminative approaches with relation distribution losses. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1378-1387, Florence, Italy. Association for Computational Linguistics.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895-2905.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 885-895, New Orleans, Louisiana. Association for Computational Linguistics.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame,
Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Ruidong Wu, Yuan Yao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2019. Open relation extraction: Relational knowledge transfer from supervised data to unsupervised data. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 219-228, Hong Kong, China. Association for Computational Linguistics.
Shanchan Wu and Yifan He. 2019. Enriching pretrained language model with entity information for relation classification. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 2361-2364.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5065-5075, Online. Association for Computational Linguistics.
Dian Yu, Lifu Huang, and Heng Ji. 2017. Open relation extraction and grounding. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 854-864, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335-2344, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Dejiao Zhang, Feng Nan, Xiaokai Wei, Shang-Wen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021. Supporting clustering with contrastive learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5419-5430, Online. Association for Computational Linguistics.
Tian Zhang, Raghu Ramakrishnan, and Miron Livny. 1996. Birch: an efficient data clustering method for very large databases. ACM sigmoid record, 25(2):103-114.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based
bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 207-212, Berlin, Germany. Association for Computational Linguistics.
The statistics of SemEval2010 Task8 and FewRel are shown in Table 4. For SemEval2010 Task8, we use 9 relations except the Other relaiton. Because of small number of relations, class imbalance and relations with high similarity, the experiment on SemEval2010 Task8 is more challenging and close to real world setting. For FewRel, we use the train and valid split but not test split, because the test split is not publicly available.
#Instances	#Entities	#Relations	Avg.Len.
SemEval2010 Task8	10,717	7,984	10	18.84
FewRel	56,000	72,954	80	24.95