SlowGuess/NAACL_2021_OCR huggingface dataset:
ZSL Settings. We randomly select  $m$  relations as unseen ones ( $m = |Y_{u}|$ ), and randomly split the whole dataset into training and testing data, meanwhile ensuring that these  $m$  relations do not appear in training data so that  $Y_{s} \cap Y_{u} = \emptyset$ . We repeat the experiment 5 times for random selection of  $m$  relations and random training-testing splitting, and report the average results. We will also vary  $m$  to examine how performance is affected. We use Precision (P), Recall (R), and F1 as the evaluation metrics. As for the hyperparameters and configuration of ZS-BERT, we use Adam (Kingma and Ba, 2014) as the optimizer, in which the initial learning rate is  $5e - 6$ , the hidden layer size is 768, the dimension of input sentence embedding and attribute vector is 1024, the batch size is 4,  $\gamma = 7.5$ , and  $\alpha = 0.4$ .
Competing Methods. The compared methods consist of two categories, supervised relation extraction (SRE) models and text entailment models. The former includes CNN-based SRE (Zeng et al., 2014), Bi-LSTM SRE (Zhang et al., 2015), Attentional Bi-LSTM SRE (Zhou et al., 2016), and R-BERT (Wu and He, 2019). These SRE models use different ways to extract features from the input sentences and perform prediction. They have achieved great performance with fully supervision but fail to carry out zero-shot prediction. To make them capable of zero-shot prediction, also to have fair comparison, instead of originally using a softmax layer to output a probability vector whose dimension is equal to the seen relations, we change the last hidden layer of each SRE competing method to a fully-connected layer with a tanh activation function, and the embedding dimension  $d$  is the same as ZS-BERT. The nearest neighbor search is applied over input sentence embeddings and relation attribute vectors to generate zero-shot prediction.
Two text entailment models, ESIM (Chen et al., 2016) and CIM (Rocktäschel et al., 2015), are also used for comparison. These two models follow a
Table 2: Results with different  $m$  values in percentage.
Wiki-ZSL	FewRel
m=5	m=5
P	R	F1	P	R	F1
CNN	30.31	32.17	30.92	36.41	38.69	37.42
Bi-LSTM	36.73	40.44	38.62	41.99	50.25	45.66
Att Bi-LSTM	35.58	41.26	38.21	39.52	47.24	42.95
R-BERT	39.22	43.27	41.15	42.19	48.61	45.17
ESIM	48.58	47.74	48.16	56.27	58.44	57.33
CIM	49.63	48.81	49.22	58.05	61.92	59.92
ZS-BERT	71.54	72.39	71.96	76.96	78.86	77.90
m=10	m=10
P	R	F1	P	R	F1
CNN	20.86	23.61	22.08	22.37	28.15	24.85
Bi-LSTM	25.33	27.91	26.56	24.52	32.02	27.77
Att Bi-LSTM	24.98	29.13	26.90	24.24	31.32	27.28
R-BERT	26.18	29.69	27.82	25.52	33.02	28.20
ESIM	44.12	45.46	44.78	42.89	44.17	43.52
CIM	46.54	47.90	45.57	47.39	49.11	48.23
ZS-BERT	60.51	60.98	60.74	56.92	57.59	57.25
m=15	m=15
P	R	F1	P	R	F1
CNN	14.58	17.68	15.92	14.17	20.26	16.67
Bi-LSTM	16.25	18.94	17.49	16.83	27.62	20.92
Att Bi-LSTM	16.93	18.54	17.70	16.48	26.36	20.28
R-BERT	17.31	18.82	18.03	16.95	19.37	18.08
ESIM	27.31	29.62	28.42	29.15	31.59	30.32
CIM	29.17	30.58	29.86	31.83	33.06	32.43
ZS-BERT	34.12	34.38	34.25	35.54	38.19	36.82
well-known implementation (Obamuyide and Vlachos, 2018) that formulates zero-shot relation extraction as a text entailment task, which accepts sentence and relation description as input, and output a binary label indicating whether they are semantically matched. ESIM uses bi-LSTM (Hochreiter and Schmidhuber, 1997; Graves and Schmidhuber, 2005) to encode two input sequences, passes them through the local inference model, and produces the prediction via a softmax layer. CIM replaces the bi-LSTM block with a conditional version, i.e., the representation of sentence is conditioned on its relation description. Note that although there exist other zero-shot relation extraction approaches such as the approach proposed by Levy et al. (2017), their approach to formulate the ZSL task and their data requirement are quite different with our present work. To be specific, their method requires pre-defined question template, whereas our model does not. Hence it would be unfair to compare with those approaches.
Main Results. The experiment results by varying  $m$  unseen relations are shown in Table 2. First, it can be apparently found that the proposed ZS-BERT steadily outperforms the competing methods over two datasets when targeting at different numbers of unseen relations. The superiority of ZS-BERT gets more significant on  $m = 5$ . Such results
Figure 3: Effects on varying the margin parameter  $\gamma$  and balance coefficient  $\alpha$  with  $m = 10$  on both datasets.
Figure 4: Left: Results of ZS-BERT with different fractions of unseen instances available for training, in which 0.0 refers to the zero-shot result. Right: Results on different distance functions and varied  $m$ .
not only validate the effectiveness of leveraging relation descriptions, but also prove the usefulness of the proposed multi-task learning structure that better encodes the semantics of input sentences and have relation attribute vectors been differentiated from each other. Second, although the text entailment models ESIM and CIM perform well among competing methods, their performance is still obviously lower than ZS-BERT. The reason is that their approaches cannot precisely distinguish the semantics of input sentences and relation descriptions in the embedding space. Third, we also find that the improvement of ZS-BERT gets larger when  $m$  is smaller. Increasing  $m$  weakens the superiority of ZS-BERT. It is straightforward that as the number of unseen relations increases, it becomes more difficult to predict the right relation since the possible choices have increased. We also speculate another underlying reason is that although ZS-BERT can effectively capture the latent attributes for each relation, relations themselves could be to some extent semantically similar to one another, and more unseen relations will increase the possibility that obtains a predicted relation that is semantically close but actually wrong. To verify this conjecture, we will give an example in the case study.
Hyperparameter Sensitivity. We examine how
primary hyperparameters, including the value of margin parameter  $\gamma$  and the balance coefficient  $\alpha$  in Eq. 2, affect the performance of ZS-BERT. By fixing  $m = 10$  and varying  $\gamma$  and  $\alpha$ , the results in terms of F1 scores on two datasets are exhibited on Figure 3. It is noteworthy that  $\gamma$  does have an impact on performance, since it brings the condition on whether to increase the loss value, which is determined by the difference between the positive pair and negative pair. Nevertheless, not always the higher values of  $\gamma$  lead to better performance. This is reasonable that when  $\gamma$  is too low, the distance between the positive pair and negative pair would not be far enough. Thus, when performing nearest neighbor search, it is more likely to reach the wrong relations. In contrast, when  $\gamma$  gets too high, it is hard for the training process to converge at the point that the distance between relations is expected to be that high. We would suggest setting  $\gamma = 7.5$  to derive satisfying results across datasets. As for the balance coefficient  $\alpha$  in the loss function, we find that  $\alpha = 0.4$  can achieve the best performance, indicating that the margin loss plays a more significant role in training ZS-BERT. Also notice that when  $\alpha = 1.0$ , the performance drops dramatically, showing that the margin loss is essential to our model. This is also reasonable that since our model relies on the quality of embeddings, therefore totally relying on cross entropy loss leads to failure of zero-shot prediction. The better separation between embeddings of different relations, the more likely our model can generate the accurate zero-shot prediction. In addition, while the nearest neighbor search is performed to generate the zero-shot prediction, we think the choice of distance computing function  $dist()$  can also be an hyperparameter. By applying inner product, Euclidean distance, and the cosine similarity as  $dist()$  in ZS-BERT, we report their F1 scores with different  $m$  on two datasets in the right of Figure 4. The results inform us that inner product is a proper distance function for zero-shot relation extraction with ZS-BERT.
Few-shot Prediction. To understand the capability of ZS-BERT, we conduct the experiment of few-shot prediction. By following the setting of an existing work (Obamuyide and Vlachos, 2018), we make a small fraction of unseen data instances available at the training stage. That said, for each originally unseen relation, we move a small fraction of its sentences, along with the relation de
Figure 5: t-SNE visualization of the sentence embeddings for similar (a)(c) and dissimilar relations (b)(d).
Table 3: List of four cases, in which head and tail entities are highlighted by green and blue, respectively.
Input Sentence	True	Predicted
(1)	When promoting Anaconda, Minaj confirmed plans of a tour in support of The Pinkprint in an interview with Carson Daly on AMP Radio.	tracklist	publisher
(2)	Heaven and Hell as they are understood in Christian theology are roughly analogous to the Jewish Olam habah and Gehenna, with certain major differences.	opposite of	influenced by
(3)	(TOEO) was an MMORPG set in the world of the popular Nameco PlayStation title, Tales of Eternia.	publisher	manufacturer
(4)	The inverse of admittance is impedance, and the real part of admittance is conductance.	opposite of	influenced by
scription, from the testing to the training stage. By varying the fraction in x-axis, we report the results of few-shot prediction in Figure 4. We can find that that ZS-BERT can reach about  $80\%$  on F1 score with only  $2\%$  of unseen instances as supervision. Such results demonstrate the ability to recognize rare samples and the capability of few-shot learning for the proposed ZS-BERT. As expected, the more instances belonging to unseen relations available at the training stage, the higher the F1 score is. When the fraction equals to  $10\%$ , ZS-BERT can even achieve  $90\%$  F1 score on Wiki-ZSL dataset.
We categorize four types of incorrectly predicted unseen relations for the analysis: (1) The predicted relation is not precise for the targeted entity pair but may be suitable for other entities that also appear in the sentence. (2) The true relation is not appropriate because it comes from distant supervision. (3) The predicted relation is ambiguous or is a synonym of other relations. (4) The relation is wrongly predicted but should be able to be correctly classified. For each of these four types, we provide an example listed in Table 3. In case (1), the targeted entities are Anaconda and The Pinkprint, and ZS-BERT yields publisher as the prediction, which is actually correct if the targeted entities are Anaconda and Minaj. This shows ZS-BERT is able to infer the possible relation for entities in the
given sentence, but sometimes could be misled by non-targeted entities even though we have an entity mask to indicate the targeted entities. In case (2), it shows the noise originated from distant labeling. That is, even human being cannot identify the relation between Heaven and Hell is opposite of in this specific sentence. They just happened to appear together and their relation recorded in Wiki-data is opposite of. In case (3), the predicted unseen relation is manufacturer, while the ground truth is publisher. Both manufacturer and publisher describe someone make or produce something, although their domains are slightly different. This exhibits the capability of ZS-BERT to identify the input sentence with an abstract attribute because relations possessing similar semantics will have similar attribute vectors in the embedding space. Finally, in case (4), the model gives a wrong prediction that is not even close or related, which may due to the noise or information loss when transferring knowledge between relations.
Among these four groups, we are especially interested in case (3) since the semantic similarity between relations in the embedding space greatly impacts the performance. We select five semantically-distant relations, and the other five relations that possess similar semantics between two or three of them, to inspect their distributions in the embedding space. We feed sentences with these relations and generate their embeddings using ZS-BERT and R-BERT (Wu and He, 2019) for comparison. We choose R-BERT because it is the strongest embedding-based competing method for zero-shot prediction by nearest neighbor search. Note that since the predictions by text entailment-based models, ESIM and CIM, neither resort to similarity search nor directly predict unseen relation at one time, we cannot have them compared in this analysis. We visualize the embedding space by
t-SNE (Maaten and Hinton, 2008), as shown in Figure 5. We can find that when the relations are somewhat similar in their meanings (Figure 5(a),(c)), some of the data points are mingled with different clusters, as they indeed have close semantic relationships. Take subsidiary and owned by as examples, Company A is a subsidiary of company B and Company A is owned by company B refer to the same thing. This happens on both ZS-BERT and R-BERT but to a different extent. It is obvious that the embeddings produced by R-BERT are more tangled. We also plot the other five relations that there is no ambiguity among them (Figure 5(b),(d)). Apparently their embeddings are more separated between different relations. It is also obvious that the embeddings generated by ZS-BERT lead to larger inter-relation distance. This again exhibits the usefulness of the proposed ranking loss and multi-task learning structure.
In this work, we present a novel and effective model, ZS-BERT, to tackle the zero-shot relation extraction task. With the multi-task learning structure and the quality of contextual representation learning, ZS-BERT can not only well embed input sentences to the embedding space, but also substantially improve the performance. We have also conducted extensive experiments to study different aspects of ZS-BERT, from hyperparameter sensitivity to case study, and eventually show that ZS-BERT can steadily outperform existing relation extraction models under zero-shot settings. Furthermore, learning effective embeddings for relations might also be helpful to semi-supervised learning or few-shot learning by utilizing prototypes of relations as the auxiliary information.
This work is supported by Ministry of Science and Technology (MOST) of Taiwan under grants 109-2636-E-006-017 (MOST Young Scholar Fellowship) and 109-2221-E-006-173, and also by Academia Sinica under grant AS-TP-107-M05.
J. L. Ba, K. Swersky, S. Fidler, and R. Salakhutdinov. 2015. Predicting deep zero-shot convolutional neural networks using textual descriptions. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 4247-4255.
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced LSTM for natural language inference. arXiv preprint arXiv:1609.06038.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards more challenging few-shot relation classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6250-6255, Hong Kong, China. Association for Computational Linguistics.
Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural networks, 18(5-6):602-610.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809, Brussels, Belgium. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2017. Distant supervision for relation extraction with sentence-level attention and entity descriptions. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 3060-3066. AAAI Press.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
E. Kodirov, T. Xiang, and S. Gong. 2017. Semantic autoencoder for zero-shot learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4447-4456.
C. H. Lampert, H. Nickisch, and S. Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 951-958.
C. H. Lampert, H. Nickisch, and S. Harmeling. 2014. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3):453-465.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333-342, Vancouver, Canada. Association for Computational Linguistics.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference on artificial intelligence.
Jiang Lu, Jin Li, Ziang Yan, Fenghua Mei, and Changshui Zhang. 2018. Attribute-based synthetic network (abs-net): Learning more from pseudo feature representations. Pattern Recognition, 80:129-142.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003-1011, Suntec, Singapore. Association for Computational Linguistics.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S. Corrado, and Jeffrey Dean. 2013. Zero-shot learning by convex combination of semantic embeddings.
Abiola Obamuyide and Andreas Vlachos. 2018. Zero-shot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 72-78, Brussels, Belgium. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kocisky, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Daniil Sorokin and Iryna Gurevych. 2017. Context-aware representations for knowledge base relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1784-1789, Copenhagen, Denmark. Association for Computational Linguistics.
Shanchan Wu and Yifan He. 2019. Enriching pretrained language model with entity information for relation classification. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, page 2361-2364, New York, NY, USA. Association for Computing Machinery.
Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. Question answering on Freebase via relation extraction and textual evidence. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2326-2336, Berlin, Germany. Association for Computational Linguistics.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335-2344, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Shu Zhang, Dequan Zheng, Xinchen Hu, and Ming Yang. 2015. Bidirectional long short-term memory networks for relation classification. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation, pages 73-78, Shanghai, China.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 207-212, Berlin, Germany. Association for Computational Linguistics.