SlowGuess/ACL_2020_OCR huggingface dataset:
$D^{(+)}$  and  $D^{(-)}$  using  $\mathbf{r}_i$  as features. To this end, we summarize the high-level training process of BiRRE, as shown in Algorithm 3. There can be zero, one or multiple hidden layers in the neural network. The detailed study of network structures will be discussed in the experiments.
3.6 Discussion
Orthogonal projections have been applied to predict various types of word relations (Ethayarajh, 2019). However, the mechanisms behind orthogonal projections in the embedding space for predicting such relations can not be fully explained by NLP researchers. In BiRRE, we use different numbers of matrices in M1 and M2, in order to capture the mappings between hybernyms and hyponyms. Due to the complicated nature of linguistics, such projections are not  $100\%$  correct. Hence, we learn the residual vectors and train a classifier (in M3) to decide which dimensions learned by M1 and M2 are best predictors for hypernymy relations. Therefore, the performance of BiRRE can be improved.
In this section, we conduct extensive experiments to evaluate the BiRRE model over various benchmarks. We also compare it with state-of-the-art to show its effectiveness.
The default word embeddings used by our model are pre-trained by the fastText model (Bojanowski et al., 2017) over the English Wikipedia corpus of version December 2019. We train the model by ourselves using their original codes. The embedding size is set as  $d = 300$ , according to their paper. In the implementation, the parameters  $\eta$  and  $N$  are set to  $10^{-3}$  and  $\max \{1, \lfloor \lg |D^{(+)}| \rfloor\}$  (an empirical formula), respectively. We also tune the model parameters in subsequent experiments. The neural network in M3 is fully connected and trained via the Adam algorithm with the dropout rate to be 0.1.
4.2 Experiment 1: Effectiveness of BiRRE
We use the largest hypernymy relation dataset (to our knowledge) from Shwartz et al. (2016) to test the effectiveness of BiRRE. It is created from various resources: WordNet, DBPedia, Wikidata and YAGO, and divided into random split and lexical split. Especially, the lexical split forces training, testing and validation sets contain distinct vocabularies, disabling "lexical memorization" (Levy
Method	Precision	Recall	F1	Precision	Recall	F1
Random Split	Lexical Split
Roller and Erk (2016)	0.926	0.850	0.886	0.700	0.964	0.811
Shwartz et al. (2016)	0.913	0.890	0.901	0.809	0.617	0.700
Glavas and Ponzetto (2017)	0.933	0.826	0.876	0.705	0.785	0.743
Rei et al. (2018)	0.928	0.887	0.907	0.826	0.860	0.842
BiRRE	0.945	0.932	0.938	0.880	0.918	0.898
Table 1: Performance of different approaches over the dataset (Shwartz et al., 2016).
et al., 2015). We follow the same evaluation steps of Shwartz et al. (2016); Rei et al. (2018) and report the results in Table 1. The network structure and parameters are tuned over the validation set.
Based on the results, BiRRE consistently outperforms state-of-the-art by  $3.1\%$  and  $5.6\%$  in terms of F1. Additionally, the performance gap between lexical and random splits has been narrowed down from  $6.5\%$  (Rei et al., 2018) to  $4.0\%$  (BiRRE). It shows that BiRRE alleviates "lexical memorization", compared to other distributional models. We also conduct pairwise statistical tests between Rei et al. (2018) and our outputs. It shows that BiRRE outperforms the approach significantly.
We tune the value of  $\lambda$  from 0.0 to 1.0 using the development set. The results over the lexical spilt of the dataset (Shwartz et al., 2016) are shown in Figure 3(a). Bigger  $\lambda$  means a larger effect of negative regularization. As seen, the usage of negative regularization improves the prediction performance by a large margin. A suitable choice of  $\lambda$  is generally around 0.4 to 0.6. As for the neural network structures, the number of hidden nodes does not have a large impact on the model performance. Hence, we only report the results when we use the same number of nodes in hidden layers as the dimension of word embeddings  $d$ , shown in Figure 3(b). Our results are consistent with previous research, which show that adding more hidden layers can decrease the prediction accuracy, leading to model overfitting.
4.3 Experiment 2: Supervised Hypernymy Classification
We evaluate BiRRE over two benchmark datasets: BLESS (Baroni and Lenci, 2011) and ENTAILMENT (Baroni et al., 2012), consisting of 14,547 and 2,770 labeled term pairs, respectively. For evaluation, we follow the same "leave-one-out" evaluation protocols as used in previous research (Yu et al., 2015; Luu et al., 2016; Nguyen et al., 2017).
(a) Tuning  $\lambda$
(b) Varying network depth
Figure 3: Tuning parameter  $\lambda$  and the depth of the neural network classifier for BiRRE.
All the experimental results are reported in terms of averaged accuracy. Because the two datasets do not have separate validation sets, we take the dataset (Shwartz et al., 2016) to tune parameters of BiRRE. To prevent "data leakage", we exclude all the data of the validation set that also appear in the test set for parameter tuning. We compare BiRRE against several previous supervised models (Mikolov et al., 2013; Yu et al., 2015; Luu et al., 2016; Nguyen et al., 2017; Wang et al., 2019a).
The averaged accuracy scores of all these methods are shown in Table 2. From the results, we can see that our model outperforms all previous baseline approaches, having the averaged accuracy of  $98\%$  and  $93\%$ , respectively. We also conduct the paired t-test, which shows that BiRRE sig
<sup>3</sup>We have also considered SemEval 2018 Task 9 (Camacho-Collados et al., 2018) for evaluation. However, this task focuses on the complete process of retrieving (or discovering) hypernyms for input terms from specific corpora. Hence, it is not suitable to evaluate BiRRE directly.
Method	BLESS	ENT.
Mikolov et al. (2013)	0.84	0.83
Yu et al. (2015)	0.90	0.87
Luu et al. (2016)	0.93	0.91
Nguyen et al. (2017)	0.94	0.91
Wang et al. (2019a)	0.97	0.92
BiRRE	0.98	0.93
nificantly outperforms classical models (Mikolov et al., 2013). Compared to the strongest competitor (Wang et al., 2019a), the accuracy of our model is also higher by  $1\%$ .
4.4 Experiment 3: Ablation Study of BiRRE
We further study the effectiveness of individual residual vectors for hypernymy classification and conduct the following ablation study. Each time, we only use a unidirectional residual vector as features (i.e.,  $res^{hypo}(\mathbf{x}_i, \mathbf{y}_i)$  and  $res^{hyper}(\mathbf{x}_i, \mathbf{y}_i)$ ). Additionally, we follow several previous papers (Yu et al., 2015; Luu et al., 2016; Nguyen et al., 2017), using the addition, offset and concatenation of embedding vectors as features (i.e.,  $\mathbf{x}_i + \mathbf{y}_i$ ,  $\mathbf{x}_i - \mathbf{y}_i$  and  $\mathbf{x}_i \oplus \mathbf{y}_i$  to train the neural networks for hypernymy classification. These three models are treated as naive baselines. The experimental settings are the same as in Experiments 1 and 2.
The experimental results over BLESS (Baroni and Lenci, 2011), ENTAILMENT (Baroni et al., 2012) and the lexical split of the dataset (Shwartz et al., 2016) are illustrated in Table 3. We have the following three observations. i) Traditional models using  $\mathbf{x}_i + \mathbf{y}_i$ ,  $\mathbf{x}_i - \mathbf{y}_i$  and  $\mathbf{x}_i \oplus \mathbf{y}_i$  as features do not yield satisfactory results. The most likely cause is that they suffer from the "lexical memorization" problem. ii) The hyponym residual vector  $res^{hypo}(\mathbf{x}_i, \mathbf{y}_i)$  is slightly more effective than the hypernym residual vector  $res^{hyper}(\mathbf{x}_i, \mathbf{y}_i)$ . It means that the more complicated hyponym generation process is more precise and suitable for our task. iii) By combining  $res^{hypo}(\mathbf{x}_i, \mathbf{y}_i)$  and  $res^{hyper}(\mathbf{x}_i, \mathbf{y}_i)$ , the proposed approach is more effective and outperforms previous methods.
4.5 Experiment 4: Hypernym Discovery
Yet another widely used evaluation framework is hypernym discovery, including three subtasks: i) ranked hyernym detection, ii) hyernymy direction
Table 2: Performance comparison for supervised hypernymy classification in terms of averaged accuracy. ENT. stands for ENTAILMENT.
Feature Set	BLESS	ENT.	Shwartz
xi + yi	0.76	0.77	0.72
xi - yi	0.79	0.74	0.73
xi ⊕ yi	0.81	0.80	0.77
res hypo(xi, yi)	0.92	0.87	0.84
res hyper(xi, yi)	0.89	0.84	0.82
ri (i.e., BiRRE)	0.99	0.93	0.88
Table 3: Ablation study results of BiRRE in terms of averaged accuracy. ENT. stands for ENTAILMENT.
classification and iii) graded lexical entailment, as presented in Nguyen et al. (2017); Roller et al. (2018); Le et al. (2019) and many others. These subtasks require algorithms to output unsupervised scores (or measures), indicating the level of hypernymy within a term pair. Therefore, this framework is not directly applicable to evaluate BiRRE.
We evaluate BiRRE on hypernym discovery by external supervision. For ranked hyernym detection, following Roller et al. (2018); Le et al. (2019), we consider five test sets: BLESS (Baroni and Lenci, 2011), EVAL (Santus et al., 2015), LEDs (Baroni et al., 2012), SHWARTZ (Shwartz et al., 2016) and WBLESS (Weeds et al., 2014). For each test set, we use the remaining four datasets (excluding all term pairs in the current test set) to train and tune the BiRRE model. For each term in the test set, we create a ranked list of candidate hypernyms by placing positive predictions over negative. Next, for candidate hypernyms with the same relation label, we rank them by norms of BiRRE vectors to produce the final ranked list.
For the hypernymy direction classification subtask, we use three test sets: BLESS (Baroni and Lenci, 2011), WBLESS (Weeds et al., 2014) and BIBLESS (Kiela et al., 2015). Because this subtask is directly evaluated in terms of accuracy, we train the supervised BiRRE model using the external dataset (Shwartz et al., 2016) (also excluding term overlaps) and report the performance. Another subtask evaluated in Roller et al. (2018); Le et al. (2019) is graded lexical entailment (Vulic et al., 2017). Because BiRRE only produces discrete outputs, how BiRRE can be adapted for graded lexical entailment is left as future work.
The experimental results are summarized in Table 4. For comparison, we take three recent models (Nguyen et al., 2017; Roller et al., 2018; Le et al., 2019) as strong baselines. Due to space limitation, for Roller et al. (2018), we only list the
Method	BLESS	EVAL	LEDS	SHWARTZ	WBLESS
Task: Ranked Hyernym Detection (Average Precision)
Nguyen et al. (2017)	0.45	0.54	-	-	0.85
Roller et al. (2018)	0.76	0.48	0.84	0.44	0.96
Le et al. (2019)	0.81	0.50	0.89	0.50	0.98
BiRRE	0.87	0.56	0.88	0.56	0.98
Method	BLESS	WBLESS	BIBLESS
Task: Hyernymy Direction Classification (Accuracy)
Nguyen et al. (2017)	0.92	0.87	0.81
Roller et al. (2018)	0.96	0.87	0.85
Le et al. (2019)	0.94	0.90	0.87
BiRRE	0.98	0.95	0.92
scores generated by "spmi(x, y)" due to its superiority. We can see that BiRRE consistently outperforms baselines over most of the datasets. As for LEDs and WBLESS, the results of BiRRE and the state-of-the-art (Le et al., 2019) are comparable. Hence, our supervised distributional model BiRRE can also address hypernym discovery, previously addressed by unsupervised hypernymy scores.
We need to claim that models in Table 4 use different knowledge sources (either patterns or distributional vectors) for parameter learning. Strictly speaking, the gaps of scores in this set of tasks do not necessarily reflect which method is better in all situations. It still remains an open question that how to evaluate all types of methods related to hypernymy detection in a unified framework.
4.6 Experiment 5: Choice of Different Word Embeddings
We also test our model using other types of word embeddings. We consider two other types of traditional word embeddings: Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014), as well as BERT (Devlin et al., 2019) representations without contexts<sup>4</sup>. Experiments are conducted over the same datasets as used in Experiment 3. The results are shown in Table 5, in terms of accuracy. As shown, the effect of fastText (Bojanowski et al., 2017) is slightly better than Word2Vec and GloVe. The representations of BERT do not yield satisfactory performance, probably due to the fact that the dimensionality of BERT is higher than other models, making the number of parameters in BiRRE
Table 4: Experimental results of ranked hyernym detection and hyernymy direction classification.
Word Embed.	BLESS	ENT.	Shwartz
Word2Vec	0.94	0.90	0.82
GloVe	0.96	0.88	0.83
BERT	0.87	0.85	0.77
Table 5: The performance of BiRRE using other word embeddings. ENT. stands for ENTAILMENT.
too large to be learned. Note that the study of deep neural language models is beyond the scope of this paper, which can be explored in the future.
In this paper, we present the BiRRE model for supervised hypernymy detection. It employs two projection-based hypernym and hyponym generation modules based on word embeddings to learn BiRRE vectors for hypernymy classification. Experimental results show that BiRRE outperforms state-of-the-arts over various benchmark datasets.
Future work includes i) improving projection learning to model complicated linguistic properties of hypernymy; ii) extending our model to address other tasks, such as graded lexical entailment (Vulic et al., 2017) and cross-lingual graded lexical entailment (Vulic et al., 2019); and iii) exploring how deep neural language models (such as BERT (Devlin et al., 2019), Transformer-XL (Dai et al., 2019), XLNet (Yang et al., 2019)) can improve the performance of hypernymy detection.
We would like to thank anonymous reviewers for their valuable comments. This work is supported by the National Key Research and Development Program of China under Grant No. 2016YFB1000904.
4The dimensions of Word2Vec and GloVe are the same as fastText. The pre-trained BERT model we use is Google's base model, released at https://github.com/google-research/bert.
Daniele Alfarone and Jesse Davis. 2015. Unsupervised learning of an IS-A taxonomy from a limited domain-specific corpus. In *IJCAI*, pages 1434-1441.
Rami Aly, Shantanu Acharya, Alexander Ossa, Arne Kohn, Chris Biemann, and Alexander Panchenko. 2019. Every child should have parents: A taxonomy refinement algorithm based on hyperbolic term embeddings. In ACL, pages 4811-4817.
Luis Espinosa Anke, José Camacho-Collados, Claudio Delli Bovi, and Horacio Saggion. 2016. Supervised distributional hypernym discovery via domain adaptation. In EMNLP, pages 424-435.
Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In EACL, pages 23-32.
Marco Baroni and Alessandro Lenci. 2011. How we blessed distributional semantic evaluation. In GEMS, pages 1-10.
Chris Biemann, Dmitry Ustalov, Alexander Panchenko, and Nikolay Arefyev. 2017. Negative sampling improves hypernymy extraction based on projection learning. In EACL, pages 543-550.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135-146.
Jose Camacho-Collados, Claudio Delli Bovi, Luis Espinosa Anke, Sergio Oramas, Tommaso Pasini, Enrico Santus, Vered Shwartz, Roberto Navigli, and Horacio Saggion. 2018. Semeval-2018 task 9: Hypernym discovery. In SemEval@NAACL-HLT, pages 712-724.
Haw-Shiuan Chang, ZiYun Wang, Luke Vilnis, and Andrew McCallum. 2018. Distributional inclusion vector embedding for unsupervised hypernymy detection. In NAACL, pages 485-495.
Hong-You Chen, Cheng-Syuan Lee, Keng-Te Liao, and Shou-de Lin. 2018. Word relation autoencoder for unseen hypernym extraction using word embeddings. In EMNLP, pages 4834-4839.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL, pages 2978-2988.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In *NAACL*, pages 4171–4186.
Kawin Ethayarajh. 2019. Rotate king to get queen: Word relationships as orthogonal transformations in embedding space. In EMNLP-IJCNLP, pages 3501-3506.
Stefano Faralli, Irene Finocchi, Simone Paolo Ponzetto, and Paola Velardi. 2019. Webisagraph: A very large hypernymy graph from a web corpus. In CLIC-IT.
Ruiji Fu, Jiang Guo, Bing Qin, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Learning semantic hierarchies via word embeddings. In ACL, pages 1199-1209.
Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann. 2018. Hyperbolic entailment cones for learning hierarchical embeddings. In ICML, pages 1632-1641.
Goran Glavas and Simone Paolo Ponzetto. 2017. Dual tensor model for detecting asymmetric lexicosemantic relations. In EMNLP, pages 1757-1767.
Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In *COLING*, pages 539-545.
William Held and Nizar Habash. 2019. The effectiveness of simple hybrid systems for hypernym discovery. In ACL, pages 3362-3367.
Douwe Kiela, Laura Rimell, Ivan Vulic, and Stephen Clark. 2015. Exploiting image generality for lexical entailment detection. In ACL-IJCNLP, pages 119-124.
Zornitsa Kozareva and Eduard H. Hovy. 2010. A semi-supervised method to learn and construct taxonomies using the web. In EMNLP, pages 1110-1118.
Matt Le, Stephen Roller, Laetitia Papaxanthos, Douwe Kiela, and Maximilian Nickel. 2019. Inferring concept hierarchies from text corpora via hyperbolic embeddings. In ACL, pages 3231-3241.
Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In *NAACL*, pages 970-976.
Anh Tuan Luu, Yi Tay, Siu Cheung Hui, and See-Kiong Ng. 2016. Learning term embeddings for taxonomic relation identification using dynamic weighting neural network. In EMNLP, pages 403-413.
Yuning Mao, Xiang Ren, Jiaming Shen, Xiaotao Gu, and Jiawei Han. 2018. End-to-end reinforcement learning for automatic taxonomy induction. In ACL, pages 2462-2472.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In *ICLR*.
Kim Anh Nguyen, Maximilian Köper, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Hierarchical embeddings for hypernymy detection and directionality. In EMNLP, pages 233-243.
Maximilian Nickel and Douwe Kiela. 2018. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In ICML, pages 3776-3785.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Marek Rei, Daniela Gerz, and Ivan Vulic. 2018. Scoring lexical entailment with a supervised directional similarity network. In ACL, pages 638-643.
Stephen Roller and Katrin Erk. 2016. Relations such as hypernymy: Identifying and exploiting hearst patterns in distributional vectors for lexical entailment. In EMNLP, pages 2163-2172.
Stephen Roller, Douwe Kiela, and Maximilian Nickel. 2018. Hearst patterns revisited: Automatic hypernym detection from large text corpora. In ACL, pages 358-363.
Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vector spaces with entropy. In EACL, pages 38-42.
Enrico Santus, Vered Shwartz, and Dominik Schlechtweg. 2017. Hypernyms under siege: Linguistically-motivated artillery for hypernymy detection. In EACL, pages 65-75.
Enrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. Evaluation 1.0: an evolving semantic dataset for training and evaluation of distributional semantic models. In LDL@IJCNLP, pages 64-69.
Yu Shi, Jiaming Shen, Yuchen Li, Najing Zhang, Xinwei He, Zhengzhi Lou, Qi Zhu, Matthew Walker, Myunghwan Kim, and Jiawei Han. 2019. Discovering hypernymy in text-rich heterogeneous information network by exploiting context granularity. In CIKM, pages 599-608.
Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016. Improving hypernymy detection with an integrated path-based and distributional method. In ACL, pages 2389-2398.
Peter D. Turney and Saif M. Mohammad. 2015. Experiments with three approaches to recognizing lexical entailment. NLE, 21(3):437-476.
Ivan Vulic, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017. Hyperlex: A large-scale evaluation of graded lexical entailment. Computational Linguistics, 43(4).
Ivan Vulic, Simone Paolo Ponzetto, and Goran Glavas. 2019. Multilingual and cross-lingual graded lexical entailment. In ACL, pages 4963-4974.
Chengyu Wang, Yan Fan, Xiaofeng He, and Aoying Zhou. 2019a. A family of fuzzy orthogonal projection models for monolingual and cross-lingual hypernymy prediction. In WWW, pages 1965-1976.
Chengyu Wang, Xiaofeng He, and Aoying Zhou. 2017a. A short survey on taxonomy learning from text corpora: Issues, resources and recent advances. In EMNLP, pages 1190-1203.
Chengyu Wang, Xiaofeng He, and Aoying Zhou. 2019b. Improving hypernymy prediction via taxonomy enhanced adversarial learning. In AAAI, pages 7128-7135.
Chengyu Wang, Junchi Yan, Aoying Zhou, and Xiaofeng He. 2017b. Transductive non-linear learning for Chinese hypernym prediction. In ACL, pages 1394-1404.
Zhongyuan Wang, Kejun Zhao, Haixun Wang, Xiaofeng Meng, and Ji-Rong Wen. 2015. Query understanding through knowledge-based conceptualization. In *IJCAI*, pages 3264-3270.
Julie Weeds, Daoud Clarke, Jeremy Reffin, David J. Weir, and Bill Keller. 2014. Learning to distinguish hypernyms and co-hyponyms. In *COLING*, pages 2249–2259.
Julie Weeds, David J. Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In COLING, pages 1015-1021.
Wentao Wu, Hongsong Li, Haixun Wang, and Kenny Qili Zhu. 2012. Probase: a probabilistic taxonomy for text understanding. In SIGMOD, pages 481-492.
Josuke Yamane, Tomoya Takatani, Hitoshi Yamada, Makoto Miwa, and Yutaka Sasaki. 2016. Distributional hypernym generation by jointly learning clusters and projections. In COLING, pages 1871-1879.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, pages 5754-5764.
Zheng Yu, Haixun Wang, Xuemin Lin, and Min Wang. 2015. Learning term embeddings for hypernymy identification. In IJCAI, pages 1390-1397.