Venue: ACL 2020

# BiRRE: Learning Bidirectional Residual Relation Embeddings for Supervised Hypernymy Detection

 Chengyu Wang\({}^{1,2}\), Xiaofeng He\({}^{3*}\)

\({}^{1}\) School of Software Engineering, East China Normal University \({}^{2}\) Alibaba Group

\({}^{3}\) School of Computer Science and Technology, East China Normal University

chyywang2013@gmail.com, hexf@cs.ecnu.edu.cn

###### Abstract

The hypernymy detection task has been addressed under various frameworks. Previously, the design of unsupervised hypernymy scores has been extensively studied. In contrast, supervised classifiers, especially distributional models, leverage the global contexts of terms to make predictions, but are more likely to suffer from "lexical memorization". In this work, we revisit supervised distributional models for hypernymy detection. Rather than taking embeddings of two terms as classification inputs, we introduce a representation learning framework named Bidirectional Residual Relation Embeddings (BiRRE). In this model, a term pair is represented by a BiRRE vector as features for hypernymy classification, which models the possibility of a term being mapped to another in the embedding space by hypernymy relations. A Latent Projection Model with Negative Regularization (LPMNR) is proposed to simulate how hypernyms and hyponyms are generated by neural language models, and to generate BiRRE vectors based on bidirectional residuals of projections. Experiments verify BiRRE outperforms strong baselines over various evaluation frameworks.

+
Footnote †: _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3630–3640

## 1 Introduction

As a type of linguistic resources, hypernymy relations refer to "is-a" relations between terms. Such relations are frequently exploited in a wide range of NLP tasks, including taxonomy induction Mao et al. (2018), lexical entailment Vulic et al. (2017) and Web query understanding Wang et al. (2015).

In the NLP community, the task of hypernymy detection has been studied under various frameworks, e.g., unsupervised hypernym discovery Roller et al. (2018); Chen et al. (2018); Chang et al. (2018), supervised hypernymy classification Shwartz et al. (2016); Nguyen et al. (2017), graded lexical entailment Vulic et al. (2017). To address unsupervised hypernym discovery, pattern-based and distributional approaches are two mainstream types of methods. Pattern-based approaches use Hearst patterns Hearst (1992) and their variants to extract hypernymy relations from texts Kozareva and Hovy (2010); Roller and Erk (2016). Distributional methods employ hypernymy measures (or called scores) to predict hypernymy based on distributional vectors Santus et al. (2014, 2017), alleviating the pattern sparsity issue. Le et al. (2019) combine Hearst patterns and hyperbolic embeddings for unsupervised hypernym detection.

Compared to unsupervised tasks, the supervised hypernymy detection task is formulated more directly, classifying a term pair as hypernymy or non-hypernymy based on two terms' representations Yu et al. (2015); Anke et al. (2016); Nguyen et al. (2017). Although this task definition is more straightforward, the corresponding methods receive criticism because they may suffer from "lexical memorization" Levy et al. (2015), referring to the phenomenon that they only learn whether a term is a "prototypical hypernym", rather than the actual relations between two terms. To address the problem, several methods combine other signals as inputs for hypernymy classifiers, such as dependency paths Shwartz et al. (2016) and the WordNet concept hierarchy Nguyen et al. (2017). Nonetheless, it is worth studying whether supervised classifiers can learn hypernymy relations purely based on distributional representations.

In this paper, we revisit supervised distributional models for hypernymy detection, and propose a representation learning framework named Bidirectional Residual Relation Embeddings (BiRRE). To handle "lexical memorization" Levy et al. (2015), we learn a BiRRE vector for each term pair as features for the classifier, avoiding using the two terms' embeddings directly. The BiRRE vector

models the possibility of a term being mapped to another in the embedding space by hypernymy relations, learned via existing neural language models and supervised signals of the training set. Specifically, we introduce the Latent Projection Model with Negative Regularization (LPMNR) to simulate how hypernyms and hyponyms are generated in the the embedding space. The BiRRE vectors are generated based on bidirectional residuals of projection results of LPMNR. Experiments over multiple public datasets and various evaluation frameworks prove that BiRRE outperforms strong baselines.

The rest of this paper is organized as follows. Section 2 summarizes the related work. The BiRRE framework is elaborated in Section 3, with experiments shown in Section 4. Finally, we conclude our paper and discuss the future work in Section 5.

## 2 Related Work

In this section, we overview related work on various tasks related to hypernymy detection. Due to space limitation, we focus on recent advances and refer readers to Wang et al. (2017a) for earlier work.

**Pattern-based approaches** date back to Hearst (1992), utilizing handcrafted patterns in English for text matching. An example of Hearst patterns is "[...] such as [...]". They are employed to build large-scale taxonomies Wu et al. (2012); Faralli et al. (2019). Although Hearst patterns are fairly simple, recent studies show they are highly useful for designing hypernymy measures Roller et al. (2018); Le et al. (2019). Other approaches aim at improving the coverage of generalized Hearst patterns by automatic pattern expansion Kozareva and Hovy (2010); Roller and Erk (2016), or considering other context-rich representations (such as Heterogeneous Information Networks Shi et al. (2019)). A potential drawback of pattern-based methods is that the recall of extraction results over specific domains is limited Alfarone and Davis (2015), as textual patterns are naturally sparse in the corpus.

To overcome the sparsity issue, **distributional hypernymy measures** model the degree of hypernymy within a term pair. A majority of these hypernymy measures are based on Distributional Inclusion Hypothesis (DIH) Weeds et al. (2004), meaning that a hypernym covers a broader spectrum of contexts, compared to its hyponyms. The improvements and variants of DIH include Santus et al. (2014); Chen et al. (2018); Chang et al. (2018) and many others. A comprehensive overview of distributional hypernymy measures can be found in Santus et al. (2017). Recently, Le et al. (2019) combine Hearst patterns and distributional vectors for hypernym detection. Additionally, the work of graded lexical entailment Vulic et al. (2017) and cross-lingual graded lexical entailment Vulic et al. (2019) aims at computing numerical scores, indicating the degree of hypernymy of a term pair.

For **supervised hypernymy classification**, traditional approaches employ distributional vectors of two terms as features, such as the Concat model, the Diff model, the SimDiff model Turney and Mohammad (2015). Recently, several approaches are proposed to learn hypernymy embeddings, considering the semantic hierarchies of concepts Yu et al. (2015); Luu et al. (2016); Nguyen et al. (2017); Chang et al. (2018); Nickel and Kiela (2018); Ganea et al. (2018); Rei et al. (2018); Chen et al. (2018). For example, Yu et al. (2015) learn hypernym and hyponym embeddings for a term by max-margin neural network. Nguyen et al. (2017) propose hierarchical embeddings for hypernymy classification, jointly trained over texts and the WordNet concept hierarchy. Rei et al. (2018) propose a directional similarity neural network based on word embeddings to predict the degree of hypernymy between two terms. Yet a number of models encode terms in the hyperbolic space, such as the hyperbolic Lorentz Model Nickel and Kiela (2018), Hyperbolic Entailment Cones Ganea et al. (2018), and others Le et al. (2019); Aly et al. (2019). The hyperbolic geometry is more capable of modeling the transitivity property of hypernymy. Additionally, patterns and distributional vectors can also be combined for supervised hypernymy prediction, as in Shwartz et al. (2016); Held and Habash (2019) and several systems submitted to SemEval 2018 Task 9 Camacho-Collados et al. (2018).

Another type of supervised models can be categorized as **projection-based approaches**, which model how to map embeddings of a term to those of its hypernyms. Fu et al. (2014) is most influential, followed by a number of variants. Biemann et al. (2017); Wang et al. (2017b, 2019) improve projection learning by considering explicit negative samples. The usage of orthogonal matrices is exploited in Wang et al. (2019). One advantage is that they do not perform classification on two terms' embeddings directly, alleviating "lexical memoria 

tion" (Levy et al., 2015). Compared to previous work, BiRRE is supervised, but does not minimize the classification error firstly. It uses LPMNR to learn hypernym/hyponym generation process by projection learning. Hence, it takes advantages of both traditional classification and projection-based approaches.

## 3 The BiRRE Framework

In this section, we first introduce the task description and the BiRRE framework. The detailed steps and justifications are elaborated subsequently.

### Task Description

Given two sets of term pairs: the training sets of hypernymy \(D^{(+)}=\{(x_{i},y_{i})\}\) and non-hypernymy relations \(D^{(-)}=\{(x_{i},y_{i})\}\), the task is to learn a classifier \(f\) to distinguish hypernymy vs. non-hypernymy relations. Particularly, \(y_{i}\) is a hypernym of \(x_{i}\) if \((x_{i},y_{i})\in D^{(+)}\). For non-hypernym relations, the relation types between two terms \(x_{i}\) and \(y_{i}\) in \(D^{(-)}\) can be reverse-hypernymy, synonymy, antonymy, or unrelated, depending on the respective task and dataset settings.

### General Framework

The BiRRE framework is shown in Figure 1, consisting of pre-processing and three major modules.

**Pre-processing:** The pre-processing step of the BiRRE framework requires minimal computation. For each term pair \((x_{i},y_{i})\in D^{(+)}\cup D^{(-)}\), we retrieve the corresponding embedding vectors from any neural language models (e.g., Word2Vec, GloVe), without fine-tuning. Denote normalized embeddings of \(x_{i}\) and \(y_{i}\) as \(\mathbf{x}_{i}\) and \(\mathbf{y}_{i}\), respectively.

**M1:** The hyponym projection module learns how to map embeddings of a hypernym to those of its hyponyms. Consider the example in Figure 2. There are usually one-to-many mappings (in semantics) from hypernyms to hyponyms. Hence, we map a hypernym to its \(N\) semantically diverse hyponyms by LPMNR. We denote the \(N\) hyponym embeddings w.r.t. \(y_{i}\) as \(hypo^{(1)}(\mathbf{y}_{i})\), \(\cdots\), \(hypo^{(N)}(\mathbf{y}_{i})\)1. Based on the difference between the true hyponym embeddings \(\mathbf{x}_{i}\) and the \(N\) predicted hyponym embeddings , we compute the hyponym residual vector \(resh^{hypo}(\mathbf{x}_{i},\mathbf{y}_{i})\) to measure the "goodness" of mapping from \(\mathbf{y}_{i}\) to \(\mathbf{x}_{i}\). As shown in Biemann et al. (2017), the explicit usage of negative samples (i.e., non-hypernymy relations) improves the performance of projection learning. In this module, we take \(D^{(+)}\) as the training set and \(D^{(-)}\) for regularization purposes.

Footnote 1: Because the training process is completed in the embedding space, our model learns to associate low-density hypernym regions with multiple numbers of high-density hyponym regions. Here, \(\mathbf{M}^{(1)}\mathbf{y}_{i},\cdots,\mathbf{M}^{(N)}\mathbf{y}_{i}\) may refer to the distributions of word embeddings of hyponyms, with no guarantee that they refers to actual word embeddings.

**M2:** The hypernym projection module learns how to map embeddings of a hyponym to those of its hypernyms. Based on Figure 2, such mappings tend to be simpler. Hence, we only learn one mapping model from a hyponym to embeddings of its hypernym. We denote th

Figure 1: The BiRRE framework for supervised hypernymy detection.

 

as \(hyper({\bf x}_{i})\). This step is learned by a simplified version of LPMNR. Similarly, we denote the hypernym residual vector as \(res^{hyper}({\bf x}_{i},{\bf y}_{i})\), measuring the "goodness" of mapping from \({\bf x}_{i}\) to \({\bf y}_{i}\). In this module, we also take \(D^{(+)}\) as the training set and \(D^{(-)}\) for regularization.

**M3:** Finally, the BiRRE vector (denoted as \({\bf r}_{i}\)) w.r.t. \((x_{i},y_{i})\) is computed by concatenating \(res^{hypo}({\bf x}_{i},{\bf y}_{i})\) and \(res^{hyper}({\bf x}_{i},{\bf y}_{i})\). A feedforward neural network is trained over \(D^{(+)}\) and \(D^{(-)}\) for hypernymy relation classification. The parameters of **M3** are learned by back propagation, with parameters of **M1** and **M2** fixed in this step.

### Hyponym Projection (M1)

Previously, several approaches Fu et al. (2014); Yamane et al. (2016) assume there is a \(d\times d\) projection matrix \({\bf M}\) such that \({\bf Mx}_{i}\approx{\bf y}_{i}\) where \(d\) is the dimension of word embeddings for \((x_{i},y_{i})\in D^{(+)}\). According to Wang et al. (2019a), the usage of orthogonal matrices has better performance for hypernymy prediction, as the the cosine similarity of \({\bf Mx}_{i}\) and \({\bf y}_{i}\) can be maximized when \({\bf Mx}_{i}\) and \({\bf y}_{i}\) are normalized.

Let \(\mathcal{M}=\{{\bf M}^{(1)},\cdots,{\bf M}^{(N)}\}\) be the parameter collection of our hyponym projection model (i.e., \(N~{}d\times d\) orthogonal projection matrices). For each hypernym \(y_{i}\), these \(N\) projection matrices map \({\bf y}_{i}\) to the embeddings of \(N\) semantically diverse hyponyms \({\bf M}^{(1)}{\bf y}_{i},\cdots,{\bf M}^{(N)}{\bf y}_{i}\). The major challenge is that the explicit semantics of \(N\) projections are unknown, and may vary across different datasets. To derive a unified solution for all scenarios, we introduce a latent variable \(\theta_{i}^{(p)}\in(0,1)\) to represent the weight of \((x_{i},y_{i})\in D^{(+)}\) w.r.t. the projection matrix \({\bf M}^{(p)}\) (\(p\in\{1,\cdots,N\}\), \(\sum_{(x_{i},y_{i})\in D^{(+)}}\theta_{i}^{(p)}=1\)). The objective of hyponym projection is as follows:2

Footnote 2: For simplicity, we omit the constraints of latent variables in the objective functions in this paper.

\[\begin{split}&\min_{\mathcal{M}}\sum_{(x_{i},y_{i})\in D^{(+)}} \sum_{p=1}^{N}\theta_{i,j}^{(p)}\|{\bf M}^{(p)}{\bf y}_{i}-{\bf x}_{i}\|^{2}\\ &\text{s. t.}~{}~{}~{}{\bf M}^{(p)T}{\bf M}^{(p)}={\bf I}_{d},p \in\{1,\cdots,N\}\end{split} \tag{1}\]

where \({\bf I}_{d}\) is the \(d\times d\) identity matrix.

A potential drawback of Eq. (1) is that it only considers hypernymy relations \(D^{(+)}\). The relation classification objective is not optimized. As Biemann et al. (2017) suggest, negative samples can of help for learning projection regularizers. The regularizers push the projected hyponym embeddings of a term further away from its non-hyponyms, making hypernymy and non-hypernymy relations more separable. Hence, we reformulate Eq. (1) as:

\[\begin{split}&\min_{\mathcal{M}}\frac{1}{|D^{(+)}|}\sum_{(x_{i},y_{ i})\in D^{(+)}}\sum_{p=1}^{N}\theta_{i}^{(p)}\|{\bf M}^{(p)}{\bf y}_{i}-{ \bf x}_{i}\|^{2}\\ &+\frac{\lambda}{|D^{(-)}|}\sum_{(x_{i},y_{i})\in D^{(-)}}\sum_{ p=1}^{N}\phi_{i}^{(p)}({\bf M}^{(p)}{\bf y}_{i})^{T}\cdot{\bf x}_{i}\\ &\text{s. t.}~{}~{}~{}{\bf M}^{(p)T}{\bf M}^{(p)}={\bf I}_{d},p \in\{1,\cdots,N\}\end{split} \tag{2}\]

where \(\lambda>0\) is the regularization balancing factor. The latent variable \(\phi_{i}^{(p)}\in(0,1)\) is the weight of the negative sample \((x_{i},y_{i})\in D^{(-)}\) w.r.t. \({\bf M}^{(p)}\). The constraint \(\sum_{(x_{i},y_{i})\in D^{(-)}}\phi_{i}^{(p)}=1\) also holds.

To the best of our knowledge, there is no standard off-the-shelf solution to Eq. (2). We slightly change the regularization term of Eq. (2). The objective function is changed as follows, which we refer as the Latent Projection Model with Negative Regularization (LPMNR):

\[\begin{split}&\min_{\mathcal{M}}\frac{1}{|D^{(+)}|}\sum_{(x_{i},y_{ i})\in D^{(+)}}\sum_{p=1}^{N}\theta_{i}^{(p)}\|{\bf M}^{(p)}{\bf y}_{i}-{\bf x }_{i}\|^{2}\\ &-\frac{\lambda}{|D^{(-)}|}\sum_{(x_{i},y_{i})\in D^{(-)}}\sum_{p= 1}^{N}\phi_{i}^{(p)}\|{\bf M}^{(p)}{\bf y}_{i}-{\bf x}_{i}\|^{2}\\ &\text{s. t.}~{}~{}~{}{\bf M}^{(p)}{\bf M}^{(p)}={\bf I}_{d},p \in\{1,\cdots,N\}\end{split} \tag{3}\]

Figure 2: Two types of hypernymy relation mappings.

 

Optimizing Eq. (3) is non-trivial due to the existence of the unknown weights and in . In this paper, we present a dual-iterative algorithm to solve the problem. All values of and are randomly initialized (with , ),  and . In each iteration, we update the values of , and . When all the values of and are fixed, Eq. (3) can be regarded as a variant of the Multi-Wahba problem Wang et al. (2019a). For simplicity, let . We extend their work and give an SVD based closed-form solution to Eq. (3) in Algorithm 1.

```
1:for\(p=1\) to do
2:
3:
4:endfor
```

**Algorithm 1** Closed-form Solution to Eq. (3)

It is trivial to see that the optimal values of each matrix is independent from each other. Hence, we only need to optimize:

For simplicity, let , with the superscript omitted. The problem can be transformed as:

Define the matrix . We re-write the objective function as: . Hence, we have transformed the problem into the Multi-Wahba problem Wang et al. (2019a). is minimized when the optimal value of is:

with .

After optimal values of are computed, the values of all are known. In this condition, we fix the values of and update all and . We turn the problem of minimizing Eq. (3) into the following problems:

(4)

(5)

We update and by constrained gradient descent where the updating formulas are:

(6)

(7)

where is the learning rate (a small decimal). and are updated values of are updated values of for the new iteration, respectively. After the update of all weights, we normalize the weights to satisfy: and . The iterative procedure continues until convergence, with the algorithm summarized in Algorithm 2.

After training, given outputs \(N\) hyponym embeddings: Outputs , , , . We define the hyponym residual vector as follows:

where is the index of the selected projection matrix that best fits for . We set  empirically as: .

Based on the objective in Eq. (3), if tends to be small. Otherwise, would be large. Hence, it is discriminative for hypernymy classification.

 



### Hypernym Projection (M2)

The hypernym projection module can be regarded as a simplified version of the previous module. Denote \({\bf Q}\) as the \(d\times d\) projection matrix. The objective of hypernym projection is formulated as follows:

\[\underset{{\bf Q}}{\text{min}}\frac{1}{|D^{(-)}|}\sum_{(x_{i},y_{ i})\in D^{(-)}}\lVert{\bf Q}{\bf x}_{i}-{\bf y}_{i}\rVert^{2}-\frac{\lambda}{|D^{(+)}|}\] \[\sum_{(x_{i},y_{i})\in D^{(+)}}\lVert{\bf Q}{\bf x}_{i}-{\bf y}_{ i}\rVert^{2}~{}~{}\mbox{s. t.}~{}~{}~{}{\bf Q}^{T}{\bf Q}={\bf I}_{d}\]

It can be solved by Algorithm 1 with weights reduced and \(N=1\). Similar to hyponym projection, we compute the hypernym residual vector \(res^{hyper}({\bf x}_{i},{\bf y}_{i})\) as follows:

\[res^{hyper}({\bf x}_{i})={\bf Q}{\bf x}_{i}-{\bf y}_{i}\]

```
1:Randomly initialize all \(\theta_{i}^{(p)}\) and \(\phi_{i}^{(p)}\);
2:while Eq. (3) does not converge do
3: Compute \({\cal M}\) by Algorithm 1;
4:for\(p=1\) to \(N\)do
5:while Eq. (4) does not converge do
6: Update and normalize \(\theta_{i,j}^{(p)}\) by Eq. (6);
7:end while
8:while Eq. (5) does not converge do
9: Update and normalize \(\phi_{i,j}^{(p)}\) by Eq. (7);
10:endwhile
11:end for
12:endwhile
```

**Algorithm 2** Optimization Algorithm for Eq. (3)

### Hypernym Relation Classification (M3)

For each pair \((x_{i},y_{i})\in D^{(+)}\cup D^{(-)}\), we generate the BiRRE vector \({\bf r}_{i}\) via the concatenation of two residual vectors:

\[{\bf r}_{i}=res^{hypo}({\bf x}_{i},{\bf y}_{i})\oplus res^{hyper}({\bf x}_{i},{ \bf y}_{i}) \tag{8}\]

A feed forward neural network is trained for hypernymy vs. non-hypernymy classification over \(D^{(+)}\) and \(D^{(-)}\) using \({\bf r}_{i}\) as features. To this end, we summarize the high-level training process of BiRRE, as shown in Algorithm 3. There can be zero, one or multiple hidden layers in the neural network. The detailed study of network structures will be discussed in the experiments.

### Discussion

Orthogonal projections have been applied to predict various types of word relations (Ethayarajh, 2019). However, the mechanisms behind orthogonal projections in the embedding space for predicting such relations can not be fully explained by NLP researchers. In BiRRE, we use different numbers of matrices in **M1** and **M2**, in order to capture the mappings between hypernyms and hyponyms. Due to the complicated nature of linguistics, such projections are not 100% correct. Hence, we learn the residual vectors and train a classifier (in **M3**) to decide which dimensions learned by **M1** and **M2** are best predictors for hypernymy relations. Therefore, the performance of BiRRE can be improved.

## 4 Experiments

In this section, we conduct extensive experiments to evaluate the BiRRE model over various benchmarks. We also compare it with state-of-the-art to show its effectiveness.

### Experimental Settings

The default word embeddings used by our model are pre-trained by the fastText model (Bojanowski et al., 2017) over the English Wikipedia corpus of version December 2019. We train the model by ourselves using their original codes. The embedding size is set as \(d=300\), according to their paper. In the implementation, the parameters \(\eta\) and \(N\) are set to \(10^{-3}\) and \(\max\{1,\lfloor\lg|D^{(+)}|\rfloor\}\) (an empirical formula), respectively. We also tune the model parameters in subsequent experiments. The neural network in M3 is fully connected and trained via the Adam algorithm with the dropout rate to be 0.1.

### Experiment 1: Effectiveness of BiRRE

We use the largest hypernymy relation dataset (to our knowledge) from Shwartz et al. (2016) to test the effectiveness of BiRRE. It is created from various resources: WordNet, DBPedia, Wikidata and YAGO, and divided into random split and lexical split. Especially, the lexical split forces training, testing and validation sets contain distinct vocabularies, disabling "lexical memorization" (Levy

et al., 2015). We follow the same evaluation steps of Shwartz et al. (2016); Rei et al. (2018) and report the results in Table 1. The network structure and parameters are tuned over the validation set.

Based on the results, BiRRE consistently outperforms state-of-the-art by 3.1% and 5.6% in terms of F1. Additionally, the performance gap between lexical and random splits has been narrowed down from 6.5% (Rei et al., 2018) to 4.0% (BiRRE). It shows that BiRRE alleviates "lexical memorization", compared to other distributional models. We also conduct pairwise statistical tests between Rei et al. (2018) and our outputs. It shows that BiRRE outperforms the approach significantly.

We tune the value of \(\lambda\) from 0.0 to 1.0 using the development set. The results over the lexical spilt of the dataset (Shwartz et al., 2016) are shown in Figure 3(a). Bigger \(\lambda\) means a larger effect of negative regularization. As seen, the usage of negative regularization improves the prediction performance by a large margin. A suitable choice of \(\lambda\) is generally around 0.4 to 0.6. As for the neural network structures, the number of hidden nodes does not have a large impact on the model performance. Hence, we only report the results when we use the same number of nodes in hidden layers as the dimension of word embeddings \(d\), shown in Figure 3(b). Our results are consistent with previous research, which show that adding more hidden layers can decrease the prediction accuracy, leading to model overfitting.

### Experiment 2: Supervised Hypernymy Classification

We evaluate BiRRE over two benchmark datasets: BLESS (Baroni and Lenci, 2011) and ENTAILMENT (Baroni et al., 2012), consisting of 14,547 and 2,770 labeled term pairs, respectively. For evaluation, we follow the same "leave-one-out" evaluation protocols as used in previous research (Yu et al., 2015; Luu et al., 2016; Nguyen et al., 2017). All the experimental results are reported in terms of averaged accuracy. Because the two datasets do not have separate validation sets, we take the dataset (Shwartz et al., 2016) to tune parameters of BiRRE. To prevent "data leakage", we exclude all the data of the validation set that also appear in the test set for parameter tuning. We compare BiRRE against several previous supervised models (Mikolov et al., 2013; Yu et al., 2015; Luu et al., 2016; Nguyen et al., 2017; Wang et al., 2019). 3

Footnote 3: We have also considered SemEval 2018 Task 9 (Camacho-Collados et al., 2018) for evaluation. However, this task focuses on the complete process of retrieving (or discovering) hypernyms for input terms from specific corpora. Hence, it is not suitable to evaluate BiRRE directly.

The averaged accuracy scores of all these methods are shown in Table 2. From the results, we can see that our model outperforms all previous baseline approaches, having the averaged accuracy of 98% and 93%, respectiv

Figure 3: Tuning parameter \(\lambda\) and the depth of the neural network classifier for BiRRE.

 

nificantly outperforms classical models (Mikolov et al., 2013). Compared to the strongest competitor (Wang et al., 2019a), the accuracy of our model is also higher by \(1\%\).

### Experiment 3: Ablation Study of BiRRE

We further study the effectiveness of individual residual vectors for hypernymy classification and conduct the following ablation study. Each time, we only use a unidirectional residual vector as features (i.e., \(res^{hypo}(\mathbf{x}_{i},\mathbf{y}_{i})\) and \(res^{hyper}(\mathbf{x}_{i},\mathbf{y}_{i})\)). Additionally, we follow several previous papers (Yu et al., 2015; Luu et al., 2016; Nguyen et al., 2017), using the addition, offset and concatenation of embedding vectors as features (i.e., \(\mathbf{x}_{i}+\mathbf{y}_{i}\), \(\mathbf{x}_{i}-\mathbf{y}_{i}\) and \(\mathbf{x}_{i}\oplus\mathbf{y}_{i}\) to train the neural networks for hypernymy classification. These three models are treated as naive baselines. The experimental settings are the same as in Experiments 1 and 2.

The experimental results over BLESS (Baroni and Lenci, 2011), ENTAILMENT (Baroni et al., 2012) and the lexical split of the dataset (Shwartz et al., 2016) are illustrated in Table 3. We have the following three observations. i) Traditional models using \(\mathbf{x}_{i}+\mathbf{y}_{i}\), \(\mathbf{x}_{i}-\mathbf{y}_{i}\) and \(\mathbf{x}_{i}\oplus\mathbf{y}_{i}\) as features do not yield satisfactory results. The most likely cause is that they suffer from the "lexical memorization" problem. ii) The hyponym residual vector \(res^{hypo}(\mathbf{x}_{i},\mathbf{y}_{i})\) is slightly more effective than the hypernym residual vector \(res^{hyper}(\mathbf{x}_{i},\mathbf{y}_{i})\). It means that the more complicated hyponym generation process is more precise and suitable for our task. iii) By combining \(res^{hypo}(\mathbf{x}_{i},\mathbf{y}_{i})\) and \(res^{hyper}(\mathbf{x}_{i},\mathbf{y}_{i})\), the proposed approach is more effective and outperforms previous methods.

### Experiment 4: Hypernym Discovery

Yet another widely used evaluation framework is hypernym discovery, including three subtasks: i) ranked hyernym detection, ii) hyernym direction classification and iii) graded lexical entailment, as presented in Nguyen et al. (2017); Roller et al. (2018); Le et al. (2019) and many others. These subtasks require algorithms to output unsupervised scores (or measures), indicating the level of hypernymy within a term pair. Therefore, this framework is not directly applicable to evaluate BiRRE.

We evaluate BiRRE on hypernym discovery by external supervision. For ranked hyermym detection, following Roller et al. (2018); Le et al. (2019), we consider five test sets: BLESS (Baroni and Lenci, 2011), EVAL (Santus et al., 2015), LEDS (Baroni et al., 2012), SHWARTZ (Shwartz et al., 2016) and WBLESS (Weeds et al., 2014). For each test set, we use the remaining four datasets (excluding all term pairs in the current test set) to train and tune the BiRRE model. For each term in the test set, we create a ranked list of candidate hypernyms by placing positive predictions over negative. Next, for candidate hypernyms with the same relation label, we rank them by norms of BiRRE vectors to produce the final ranked list.

For the hypernymy direction classification subtask, we use three test sets: BLESS (Baroni and Lenci, 2011), WBLESS (Weeds et al., 2014) and BIBLESS (Kiela et al., 2015). Because this subtask is directly evaluated in terms of accuracy, we train the supervised BiRRE model using the external dataset (Shwartz et al., 2016) (also excluding term overlaps) and report the performance. Another subtask evaluated in Roller et al. (2018); Le et al. (2019) is graded lexical entailment (Vulic et al., 2017). Because BiRRE only produces discrete outputs, how BiRRE can be adapted for graded lexical entailment is left as future work.

The experimental results are summarized in Table 4. For comparison, we take three recent models (Nguyen et al., 2017; Roller et al., 2018; Le et al., 2019) as strong baselines. Due to space limitation, for Roller et al. (2018), we only list the

\begin{table}
\begin{tabular}{l l l} \hline
**Method** & **BLESS** & **ENT.** \\ \hline Mikolov et al. (2013) & 0.84 & 0.83 \\ Yu et al. (2015) & 0.90 & 0.87 \\ Luu et al. (2016) & 0.93 & 0.91 \\ Nguyen et al. (2017) & 0.94 & 0.91 \\ Wang et al. (2019a) & 0.97 & 0.92 \\ \hline
**BiRRE** & **0.98** & **0.93** \\ \hline \end{tabular}
\end{table}
Table 2: Performance comparison for supervised hypernymy classification in terms of averaged accuracy. ENT. stands for ENTAILMENT.

\begin{table}
\begin{tabular}{l c c c} \hline
**Feature Set** & **BLESS** & **ENT.** & **Shwartz** \\ \hline \(\mathbf{x}_{i}+\mathbf{y}_{i}\) & 0.76 & 0.77 & 0.72 \\ \(\mathbf{x}_{i}-\mathbf{y}_{i}\) & 0.79 & 0.74 & 0.73 \\ \(\mathbf{x}_{i}\oplus\mathbf{y}_{i}\) & 0.81 & 0.80 & 0.77 \\ \hline \(res^{hypo}(\mathbf{x}_{i},\mathbf{y}_{i})\) & 0.92 & 0.87 & 0.84 \\ \(res^{hyper}(\mathbf{x}_{i},\mathbf{y}_{i})\) & 0.89 & 0.84 & 0.82 \\ \hline \(\mathbf{r}_{i}\) (i.e., BiRRE) & **0.99** & **0.93** & **0.88** \\ \hline \end{tabular}
\end{table}
Table 3: Ablation study results of BiRRE in terms of averaged accuracy. ENT. stands for ENTAILMENT.

 

scores generated by "spmi(x, y)" due to its superiority. We can see that BiRRE consistently outperforms baselines over most of the datasets. As for LEDS and WBLESS, the results of BiRRE and the state-of-the-art Le et al. (2019) are comparable. Hence, our supervised distributional model BiRRE can also address hypernym discovery, previously addressed by unsupervised hypernymy scores.

We need to claim that models in Table 4 use different knowledge sources (either patterns or distributional vectors) for parameter learning. Strictly speaking, the gaps of scores in this set of tasks do not necessarily reflect which method is better in all situations. It still remains an open question that how to evaluate all types of methods related to hypernymy detection in a unified framework.

### Experiment 5: Choice of Different Word Embeddings

We also test our model using other types of word embeddings. We consider two other types of traditional word embeddings: Word2Vec Mikolov et al. (2013) and GloVe Pennington et al. (2014), as well as BERT Devlin et al. (2019) representations without contexts 4. Experiments are conducted over the same datasets as used in Experiment 3. The results are shown in Table 5, in terms of accuracy. As shown, the effect of fastText Bojanowski et al. (2017) is slightly better than Word2Vec and GloVe. The representations of BERT do not yield satisfactory performance, probably due to the fact that the dimensionality of BERT is higher than other models, making the number of parameters in BiRRE too large to be learned. Note that the study of deep neural language models is beyond the scope of this paper, which can be explored in the future.

Footnote 4: The dimensions of Word2Vec and GloVe are the same as fastText. The pre-trained BERT model we use is Google’s base model, released at [https://github.com/google-research/bert](https://github.com/google-research/bert).

## 5 Conclusion and Future Work

In this paper, we present the BiRRE model for supervised hypernymy detection. It employs two projection-based hypernym and hyponym generation modules based on word embeddings to learn BiRRE vectors for hypernymy classification. Experimental results show that BiRRE outperforms state-of-the-arts over various benchmark datasets.

Future work includes i) improving projection learning to model complicated linguistic properties of hypernymy; ii) extending our model to address other tasks, such as graded lexical entailment Vulic et al. (2017) and cross-lingual graded lexical entailment Vulic et al. (2019); and iii) exploring how deep neural language models (such as BERT Devlin et al. (2019), Transformer-XL Dai et al. (2019), XLNet Yang et al. (2019)) can improve the performance of hypernymy detection.

## Acknowledgements

We would like to thank anonymous reviewers for their valuable comments. This work is supported by the National Key Research and Development Program of China under Grant No. 2016YFB1000904.

\begin{table}
\begin{tabular}{l l l l l l} \hline

- & 0.85 \\ Roller et al. (2018) & 0.76 & 0.48 & 0.84 & 0.44 & 0.96 \\ Le et al. (2019) & 0.81 & 0.50 & **0.89** & 0.50 & **0.98** \\
**BiRRE** & **0.87** & **0.56** & 0.88 & **0.56** & **0.98** \\ \hline \hline \multicolumn{8}{l}{**Method**} & **BLESS** & **WBLESS** & **BIBLESS** & \\ \hline \multicolumn{8}{l}{**Task: Hyernmy Direction Classification (Accuracy)**} \\ \hline Nguyen et al. (2017) & 0.92 & 0.87 & 0.81 & & \\ Roller et al. (2018) & 0.96 & 0.87 & 0.85 & & \\ Le et al. (2019) & 0.94 & 0.90 & 0.87 & & \\
**BiRRE** & **0.98** & **0.95** & **0.92** & & \\ \hline \end{tabular}
\end{table}
Table 4: Experimental results of ranked hyernym detection and hyernymy direction classification.

\begin{table}
\begin{tabular}{l l l l} \hline
**Word Embed.** & **BLESS** & **ENT.** & **Shwartz** \\ \hline Word2Vec & 0.94 & 0.90 & 0.82 \\ GloVe & 0.96 & 0.88 & 0.83 \\ BERT & 0.87 & 0.85 & 0.77 \\ \hline \end{tabular}
\end{table}
Table 5: The performance of BiRRE using other word embeddings. ENT. stands for ENTAILMENT.

 



## References

* Alfarone and Davis (2015) Daniele Alfarone and Jesse Davis. 2015. Unsupervised learning of an IS-A taxonomy from a limited domain-specific corpus. In IJCAI, pp. 1434-1441. Cited by: SS6.
* R. Aly, S. Acharya, A. Ossa, A. Kohn, C. Biemann, and A. Panchenko (2019)Every child should have parents: a taxonomy refinement algorithm based on hyperbolic term embeddings. In ACL, pp. 4811-4817. Cited by: SS1.
* L. Espinosa Anke, J. Camacho-Collados, Claudio Delli Bovi, and H. Saggion (2016)Supervised distributional hypernym discovery via domain adaptation. In EMNLP, pp. 424-435. Cited by: SS1.
* M. Baroni, R. Bernardi, N. Do, and C. Shan (2012)Entailment above the word level in distributional semantics. In EACL, pp. 23-32. Cited by: SS1.
* M. Biorni and A. Lenci (2011)How we blessed distributional semantic evaluation. In GEMS, pp. 1-10. Cited by: SS1.
* C. Biemann, D. Ustalov, A. Panchenko, and N. Arefyev (2017)Negative sampling improves hypernymy extraction based on projection learning. In EACL, pp. 543-550. Cited by: SS1.
* P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov (2017)Enriching word vectors with subword information. TACL5, pp. 135-146. Cited by: SS1.
* J. Camacho-Collados, Claudio Delli Bovi, Luis Espinosa Anke, Sergio Oramas, Tommas Pasini, Enrico Santus, V. Shwartz, R. Navigli, and H. Saggion (2018)SemEval-2018 task 9: hypernym discovery. In SemEval@NAACL-HLT, pp. 712-724. Cited by: SS1.
* H. Chang, Z. Wang, L. Vilnis, and A. McCallum (2018)Distributional inclusion vector embedding for unsupervised hypernymy detection.. In NAACL, pp. 485-495. Cited by: SS1.
* H. Chen, C. Lee, K. Liao, and S. Lin (2018)Word relation autoencoder for unseen hypernym extraction using word embeddings. In EMNLP, pp. 4834-4839. Cited by: SS1.
* Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. Viet Le, and R. Salakhutdinov (2019)Transformer-xl: attentive language models beyond a fixed-length context. In ACL, pp. 2978-2988. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171-4186. Cited by: SS1.
* K. Ethayarajh (2019)Rotate king to get queen: word relationships as orthogonal transformations in embedding space. In EMNLP-IJCNLP, pp. 3501-3506. Cited by: SS1.
* S. Faralli, I. Finocchi, S. Paolo Ponzetto, and P. Velardi (2019)Webisagraph: a very large hypernymy graph from a web corpus. In CLIC-IT, Cited by: SS1.
* R. Fu, J. Guo, B. Qin, W. Che, H. Wang, and T. Liu (2014)Learning semantic hierarchies via word embeddings. In ACL, pp. 1199-1209. Cited by: SS1.
* O. Ganea, G. Becigneul, and T. Hofmann (2018)Hyperbolic entailment cones for learning hierarchical embeddings. In ICML, pp. 1632-1641. Cited by: SS1.
* G. Glavas and S. Paolo Ponzetto (2017)Dual tensor model for detecting asymmetric lexicosemantic relations. In EMNLP, pp. 1757-1767. Cited by: SS1.
* M. A. Hearst (1992)Automatic acquisition of hyponyms from large text corpora. In COLING, pp. 539-545. Cited by: SS1.
* William Held and Nizar Habash (2019)The effectiveness of simple hybrid systems for hypernym discovery. In ACL, pp. 3362-3367. Cited by: SS1.
* Douwe Kiela, Laura Rimell, Ivan Vulic, and Stephen Clark (2015)Exploiting image generality for lexical entailment detection. In ACL-IJCNLP, pp. 119-124. Cited by: SS1.
* Z. Kozareva and E. Hovy (2010)A semi-supervised method to learn and construct taxonomies using the web. In EMNLP, pp. 1110-1118. Cited by: SS1.
* M. Le, S. Roller, L. Papaxanthos, D. Kiela, and M. Nickel (2019)Inferring concept hierarchies from text corpora via hyperbolic embeddings. In ACL, pp. 3231-3241. Cited by: SS1.
* O. Levy, S. Remus, C. Biemann, and I. Dagan (2015)Do supervised distributional methods really learn lexical inference relations?. In NAACL, pp. 970-976. Cited by: SS1.
* A. T. Luu, Y. Tay, S. Cheung Hui, and See-Kiong Ng (2016)Learning term embeddings for taxonomic relation identification using dynamic weighting neural network. In EMNLP, pp. 403-413. Cited by: SS1.
* Y. Mao, X. Ren, J. Shen, X. Gu, and J. Han (2018)End-to-end reinforcement learning for automatic taxonomy induction. In ACL, pp. 2462-2472. Cited by: SS1.
* T. Mikolov, Kai Chen, G. Corrado, and J. Dean (2013)Efficient estimation of word representations in vector space. In ICLR, Cited by: SS1.
* K. A. Nguyen, M. Koper, S. Schulte im Walde, and N. Thang Vu (2017)Hierarchical embeddings for hypernymy detection and directionality. In EMNLP, pp. 233-243.



Maximilian Nickel and Douwe Kiela. 2018. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In _ICML_, pages 3776-3785.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, pages 1532-1543.
* Rei et al. (2018) Marek Rei, Daniela Gerz, and Ivan Vulic. 2018. Scoring lexical entailment with a supervised directional similarity network. In _ACL_, pages 638-643.
* Roller and Erk (2016) Stephen Roller and Katrin Erk. 2016. Relations such as hypernymy: Identifying and exploiting hearst patterns in distributional vectors for lexical entailment. In _EMNLP_, pages 2163-2172.
* Roller et al. (2018) Stephen Roller, Douwe Kiela, and Maximilian Nickel. 2018. Hearst patterns revisited: Automatic hypernym detection from large text corpora. In _ACL_, pages 358-363.
* Santus et al. (2014) Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vector spaces with entropy. In _EACL_, pages 38-42.
* Santos et al. (2017) Enrico Santus, Vered Shwartz, and Dominik Schlechtweg. 2017. Hypernyms under siege: Linguistically-motivated artillery for hypernymy detection. In _EACL_, pages 65-75.
* Santus et al. (2015) Enrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. Evalution 1.0: an evolving semantic dataset for training and evaluation of distributional semantic models. In _LDL@IJCNLP_, pages 64-69.
* Shi et al. (2019) Yu Shi, Jiaming Shen, Yuchen Li, Naijing Zhang, Xinwei He, ZhengZhi Lou, Qi Zhu, Matthew Walker, Myunghwan Kim, and Jiawei Han. 2019. Discovering hypernymy in text-rich heterogeneous information network by exploiting context granularity. In _CIKM_, pages 599-608.
* Shwartz et al. (2016) Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016. Improving hypernymy detection with an integrated path-based and distributional method. In _ACL_, pages 2389-2398.
* Turney and Mohammad (2015) Peter D. Turney and Saif M. Mohammad. 2015. Experiments with three approaches to recognizing lexical entailment. _NLE_, 21(3):437-476.
* Vulic et al. (2017) Ivan Vulic, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017. Hyperlex: A large-scale evaluation of graded lexical entailment. _Computational Linguistics_, 43(4).
* Vulic et al. (2019) Ivan Vulic, Simone Paolo Ponzetto, and Goran Glavas. 2019. Multilingual and cross-lingual graded lexical entailment. In _ACL_, pages 4963-4974.
* Wang et al. (2019a) Chengyu Wang, Yan Fan, Xiaofeng He, and Aoying Zhou. 2019a. A family of fuzzy orthogonal projection models for monolingual and cross-lingual hypernymy prediction. In _WWW_, pages 1965-1976.
* Wang et al. (2017a) Chengyu Wang, Xiaofeng He, and Aoying Zhou. 2017a. A short survey on taxonomy learning from text corpora: Issues, resources and recent advances. In _EMNLP_, pages 1190-1203.
* Wang et al. (2019b) Chengyu Wang, Xiaofeng He, and Aoying Zhou. 2019b. Improving hypernym prediction via taxonomy enhanced adversarial learning. In _AAAI_, pages 7128-7135.
* Wang et al. (2017b) Chengyu Wang, Junchi Yan, Aoying Zhou, and Xiaofeng He. 2017b. Transductive non-linear learning for chinese hypernym prediction. In _ACL_, pages 1394-1404.
* Wang et al. (2015) Zhongyuan Wang, Kejun Zhao, Haixun Wang, Xiaofeng Meng, and Ji-Rong Wen. 2015. Query understanding through knowledge-based conceptualization. In _IJCAI_, pages 3264-3270.
* Weeds et al. (2014) Julie Weeds, Daoud Clarke, Jeremy Reffin, David J. Weir, and Bill Keller. 2014. Learning to distinguish hypernyms and co-hyponyms. In _COLING_, pages 2249-2259.
* Woods et al. (2004) Julie Weeds, David J. Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In _COLING_, pages 1015-1021.
* Wu et al. (2012) Wentao Wu, Hongsong Li, Haixun Wang, and Kenny Qili Zhu. 2012. Probase: a probabilistic taxonomy for text understanding. In _SIGMOD_, pages 481-492.
* Yamane et al. (2016) Josuke Yamane, Tomoya Takatani, Hitoshi Yamada, Makoto Miwa, and Yutaka Sasaki. 2016. Distributional hypernym generation by jointly learning clusters and projections. In _COLING_, pages 1871-1879.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In _NeurIPS_, pages 5754-5764.
* Yu et al. (2015) Zheng Yu, Haixun Wang, Xuemin Lin, and Min Wang. 2015. Learning term embeddings for hypernymy identification. In _IJCAI_, pages 1390-1397.