Title: DISTRIBUTIONAL INCLUSION VECTOR EMBEDDING FOR UNSUPERVISED HYPERNYMY DETECTION Anonymous

Abstract: Modeling hypernymy, such as poodle is-a dog, is an important generalization aid to many NLP tasks, such as entailment, relation extraction, and question answering. Supervised learning from labeled hypernym sources, such as WordNet, limit the coverage of these models, which can be addressed by learning hypernyms from unlabeled text. Existing unsupervised methods either do not scale to large vocabularies or yield unacceptably poor accuracy. This paper introduces distributional inclusion vector embedding (DIVE), a simple-to-implement unsupervised method of hypernym discovery via per-word non-negative vector embeddings which preserve the inclusion property of word contexts. In experimental evaluations more comprehensive than any previous literature of which we are aware-evaluating on 11 datasets using multiple existing as well as newly proposed scoring functionswe find that our method provides up to double the precision of previous unsupervised methods, and the highest average performance, using a much more compact word representation, and yielding many new state-of-the-art results. In addition, the meaning of each dimension in DIVE is interpretable, which leads to a novel approach on word sense disambiguation as another promising application of DIVE.

1. INTRODUCTION:

Numerous applications benefit from compactly representing context distributions, which assign meaning to objects under the rubric of distributional semantics. In natural language processing, distributional semantics has long been used to assign meanings to words (that is, to lexemes in the dictionary, not individual instances of word tokens). The meaning of a word in the distributional sense is often taken to be the set of textual contexts (nearby tokens) in which that word appears, represented as a large sparse bag of words (SBOW). Without any supervision, word2vec (Mikolov et al., 2013) , among other approaches based on matrix factorization (Levy et al., 2015a) , successfully compress the SBOW into a much lower dimensional embedding space, increasing the scalability and applicability of the embeddings while preserving (or even improving) the correlation of geometric embedding similarities with human word similarity judgments.

While embedding models have achieved impressive results, context distributions capture more semantic features than just word similarity. The distributional inclusion hypothesis (DIH) (Weeds & Weir, 2003; Geffet & Dagan, 2005; Cimiano et al., 2005) posits that the context set of a word tends to be a subset of the contexts of its hypernyms. For a concrete example, most adjectives that can be applied to poodle can also be applied to dog, because dog is a hypernym of poodle. For instance, both can be obedient. However, the converse is not necessarily true -a dog can be straight-haired but a poodle cannot. Therefore, dog tends to have a broader context set than poodle. Many asymmetric scoring functions comparing SBOW based on DIH have been developed for automatic hypernymy detection (Weeds & Weir, 2003; Geffet & Dagan, 2005; Santus et al., 2017) .

Hypernymy detection plays a key role in many challenging NLP tasks, such as textual entailment (Sammons et al., 2011) , coreference (Ponzetto & Strube, 2006) , relation extraction (Demeester et al., 2016) and question answering (Huang et al., 2008) . Leveraging the variety of contexts and inclusion properties in context distributions can greatly increase the ability to discover taxonomic structure among words (Santus et al., 2017) . The inability to preserve these features limits the semantic representation power and downstream applicability of some popular existing unsupervised learning approaches such as word2vec.

Several recently proposed methods aim to encode hypernym relations between words in dense embeddings, such as Gaussian embedding (Vilnis & McCallum, 2015; Athiwaratkun & Wilson, 2017) , order embedding (Vendrov et al., 2016) , H-feature detector (Roller & Erk, 2016) , Hyper-Score (Nguyen et al., 2017) , dual tensor (Glavaš & Ponzetto, 2017) , Poincaré embedding (Nickel & Kiela, 2017) , and LEAR (Vulić & Mrkšić, 2017) . However, the methods focus on supervised or semi-supervised setting (Vendrov et al., 2016; Roller & Erk, 2016; Nguyen et al., 2017; Glavaš & Ponzetto, 2017; Vulić & Mrkšić, 2017) , do not learn from raw text (Nickel & Kiela, 2017) or lack comprehensive experiments on the hypernym detection task (Vilnis & McCallum, 2015; Athiwaratkun & Wilson, 2017) .

Recent studies (Levy et al., 2015b; Santus et al., 2017) have underscored the difficulty of generalizing supervised hypernymy annotations to unseen pairs -classifiers often effectively memorize prototypical hypernyms ('general' words) and ignore relations between words. These findings motivate us to develop more accurate and scalable unsupervised embeddings to detect hypernymy and propose several scoring functions to analyze the embeddings from different perspectives.

1.1. CONTRIBUTIONS:

• A novel unsupervised low-dimensional embedding method to model inclusion relations among word contexts via performing non-negative matrix factorization (NMF) on a weighted PMI matrix, which can be efficiently optimized using modified skip-grams.

• Several new asymmetric comparison functions to measure inclusion and generality properties and to evaluate different aspects of unsupervised embeddings.

• Extensive experiments on 11 datasets demonstrate the learned embeddings and comparison functions achieve state-of-the-art performances on unsupervised hypernym detection while requiring much less memory and compute than approaches based on the full SBOW.

• A qualitative experiment illustrates DIVE can be used to solve word sense disambiguation, especially when efficiently modeling word senses at multiple granularities is desirable.

2. METHOD:

The distributional inclusion hypothesis (DIH) suggests that the context set of a hypernym tends to contain the context set of its hyponyms. That is, when representing a word as the counts of contextual co-occurrences, the count in every dimension of hypernym y tends to be larger than or equal to the corresponding count of its hyponym x:

EQUATION

where x y means y is a hypernym of x, V is the set of vocabulary, and #(x, c) indicates the number of times that word x and its context word c co-occur in a small window with size |W | in corpus D.

Our goal is to produce lower-dimensional embeddings that preserve the inclusion property that the embedding of hypernym y is larger than or equal to the embedding of its hyponym x in every dimension. Formally, the desirable property can be written as

EQUATION

where d 0 is number of dimensions in the embedding space. We add additional non-negativity constraints, i.e. x[i] ≥ 0, y[i] ≥ 0, ∀i, in order to increase the interpretability of the embeddings (the reason will be explained later in this section). This is a challenging task. In reality, there are a lot of noise and systematic biases which cause the violation of DIH in Equation (1) (i.e. #(x, c) > #(y, c) for some neighboring word c), but the general trend can be discovered by processing several thousands of neighboring words in SBOW together. After the compression, the same trend has to be estimated in a much smaller embedding space which discards most of the information in SBOW, so it is not surprising to see most of the unsupervised hypernymy detection studies use SBOW (Santus et al., 2017) and the existing unsupervised embeddings like Gaussian embedding have degraded accuracy (Vulić et al., 2016) .

2.1. INCLUSION PRESERVING MATRIX FACTORIZATION:

Popular methods of unsupervised word embedding are usually based on matrix factorization (Levy et al., 2015a) . The approaches first compute a co-occurrence statistic between the wth word and the cth context word as the (w, c)th element of the matrix M To preserve DIH, we propose a novel word embedding method, distributional inclusion vector embedding (DIVE), which fixes the two flaws by performing non-negative factorization (NMF) on the matrix M , where

EQUATION

where k is a constant which shifts PMI value like SGNS, Z = |D| |V | is the average word frequency, and |V | is the vocabulary size.

The design encourages the inclusion property in DIVE (i.e. Equation ( 2)) to be satisfied because the property implies that Equation (1) (DIH) holds if the matrix is reconstructed perfectly. The derivation is simple: Since context vector c is non-negative, if the embedding of hypernym y is greater than or equal to the embedding of its hyponym x in every dimension,

x T c ≤ y T c. Then, M [x, c] ≤ M [y, c] tends to be true because w T c ≈ M [w, c]. This leads to #(x, c) ≤ #(y, c) because M [w, c] = log( #(w,c)|V | #(c)k

) and only #(w, c) change with w.

2.2. OPTIMIZATION:

Due to its appealing scalability properties during training time (Levy et al., 2015a) , we optimize our embedding based on the skip-gram with negative sampling (SGNS) (Mikolov et al., 2013) . The objective function of SGNS is

EQUATION

where w ∈ R, c ∈ R, c N ∈ R, k is a constant hyper-parameter indicating the ratio between positive and negative samples. Levy & Goldberg (2014) prove SGNS is equivalent to factorizing a shifted PMI matrix M , where M [w, c] = log( P (w,c)

P (w)•P (c) • 1 k ). By setting k = k•Z #(w)

and applying non-negativity constraints to the embeddings, DIVE can be optimized using the similar objective function:

l DIV E = w∈V c∈V #(w, c) log σ(w T c) + k w∈V Z #(w) c∈V #(w, c) E c N ∼P D [log σ(-w T c N )],

(5) where w ≥ 0, c ≥ 0, c N ≥ 0, σ is the logistic sigmoid function, and k is a constant hyper-parameter. P D is the distribution of negative samples, which we set to be the corpus word frequency distribution in this paper. Equation ( 5) is optimized by ADAM (Kingma & Ba, 2015) , a variant of stochastic gradient descent (SGD). The non-negativity constraint is implemented by projection (i.e., clipping any embedding which crosses the zero boundary after an update). The optimization process provides an alternative angle to explain how DIVE preserves DIH. The gradients for the word embedding w is

EQUATION

Assume hyponym x and hypernym y satisfy DIH in Equation (1) and the embeddings x and y are the same at some point during the gradient ascent. In the case, the gradients coming from negative sampling (the second term) decrease the same amount of embedding values for both x and y because k is a constant hyper-parameter. However, the embedding of hypernym y would get higher or equal positive gradients from the first term than x in every dimension because #(x, c) ≤ #(y, c). This means Equation (1) tends to imply Equation (2). Combining the analysis from the matrix factorization viewpoint, DIH in Equation ( 1) is approximately equivalent to the inclusion property in DIVE (i.e. Equation ( 2)).

2.3. PMI FILTERING:

For a frequent target word, there must be many neighboring words that incidentally appear near the target word without being semantically meaningful, especially when a large context window size is used. The unrelated context words cause noise in both the word vector and the context vector of DIVE. We address this issue by filtering out context words c for each target word w when the PMI of the co-occurring words is too small (i.e., log( P (w,c) P (w)•P (c) ) < log(k f )). That is, we set #(w, c) = 0 in the objective function. This preprocessing step is similar with computing PPMI in SBOW (Bullinaria & Levy, 2007) , where low PMI co-occurrences are removed from the count-based representation.

2.4. INTERPRETABILITY:

After applying the non-negativity constraint, we observe that each dimension roughly corresponds to a topic, as previous findings suggest (Pauca et al., 2004; Murphy et al., 2012) . This gives rise to a natural and intuitive interpretation of our word embeddings: the word embeddings can be seen as unnormalized probability distributions over topics. By removing the normalization of the target word frequency in the shifted PMI matrix, specific words have values in few dimensions (topics), while general words appear in more topics and correspondingly have high values in more dimensions, so the concreteness level of two words can be easily compared using the magnitude of their embeddings. In other words, general words have more diverse context distributions, so we need more dimensions to store the information in order to compress SBOW well (Nalisnick & Ravi, 2015) .

In Figure 1 , we present three mentions of the word core and its surrounding contexts. These various context words increase the embedding values in different dimensions. Each dimension of the learned embeddings roughly corresponds to a topic, and the more general or representative words for each topic tend to have the higher value in the corresponding dimension (e.g. words in the second column of the table). The embedding is able to capture the common contexts where the word core appears. For example, the context of the first mention is related to the atom topic (dimension id 1) and the electron topic (id 9), while the second and third mention occur in the computer architecture topic (id 2) and education topic (id 11), respectively.

3. EXPERIMENT SETUP:

We describe four experiments in Section 4-7. The first 3 experiments compare DIVE with other unsupervised embeddings and SBOW using different hypernymy scoring functions. In these experiments, unsupervised approaches refer to the methods that only train on plaintext corpus without using any hypernymy or lexicon annotation. The last experiment presents qualitative results on word sense disambiguation.

3.1. DATASETS AND TESTING SETUP:

The SBOW and embeddings are tested on 11 datasets. The first 4 datasets come from the recent review of Santus et al. (2017) : BLESS (Baroni & Lenci, 2011) , EVALution (Santus et al., 2015) , Lenci/Benotto (Benotto, 2015) , and Weeds (Weeds et al., 2014) . The next 4 datasets are downloaded from the code repository of the H-feature detector (Roller & Erk, 2016) : Medical (i.e., Levy 2014) (Levy et al., 2014) , LEDS (also referred to as ENTAILMENT or Baroni 2012) (Baroni et al., 2012) , TM14 (i.e., Turney 2014) (Turney & Mohammad, 2015) , and Kotlerman 2010 (Kotlerman et al., 2010) . In addition, the performance on the test set of HyperNet (Shwartz et al., 2016) (using the random train/test split), the test set of WordNet (Vendrov et al., 2016) , and all pairs in Hyper-Lex (Vulić et al., 2016) are also evaluated.

The F1 and accuracy measurements are sometimes very similar even though the quality of prediction varies, so average precision AP@all is adopted as the main evaluation metric. The HyperLex dataset has a continuous score on each candidate word pair, so we adopt Spearman rank coefficient ρ as suggested by the review study of Vulić et al. (2016) . Any OOV (out-of-vocabulary) word encountered in the testing data is pushed to the bottom of the prediction list (effectively assuming the word pair does not have a hypernym relation).

3.2. TRAINING SETUP:

We use WaCkypedia corpus (Baroni et al., 2009) , a 2009 Wikipedia dump, to compute SBOW and train the embedding. For the datasets without Part of Speech (POS) information (i.e. Medical, LEDS, TM14, Kotlerman 2010, and HyperNet) , the training data of SBOW and embeddings are raw text. For other datasets, we concatenate each token with the Part of Speech (POS) of the token before training the models except the case when we need to match the training setup of another paper.

All words are lower cased. Stop words and rare words (occurs less than 10 times) are removed during our preprocessing step. The number of embedding dimensions in DIVE d 0 is set to be 100. Other hyper-parameters used in the experiments are listed in the supplementary materials. The Table 1 : Comparison with previous unsupervised embeddings. All values are percentages. AP@all (%) for 10 datasets and Spearman ρ (%) for HyperLex. Word2Vec+C scores the word pairs using the cosine similarity on skip-grams (SGNS). GE+C and GE+KL computes cosine similarity and negative KL divergence on Gaussian embedding, respectively. hyper-parameters of DIVE were decided based on the performance of HyperNet training set. To train embeddings more efficiently, we chunk the corpus into subsets/lines of 100 tokens instead of using sentence segmentation. Preliminary experiments show that this implementation simplification does not hurt the performance.

In the following experiments, we train both SBOW and DIVE on only the first 512,000 lines (51.2 million tokens) because we find this way of training setting provides better performances (for both SBOW and DIVE) than training on the whole WaCkypedia or training on randomly sampled 512,000 lines. We suspect this is due to the corpus being sorted by the Wikipedia page titles, which makes some categorical words such as animal and mammal occur 3-4 times more frequently in the first 51.2 million tokens than the rest. The performances of training SBOW PPMI on the whole WaCkypedia is also provided for reference in Table 4 and Table 5 .

4. EXPERIMENT 1: COMPARISON WITH UNSUPERVISED EMBEDDINGS:

If a pair of words has the hypernym relation, the words tend to be similar and the hypernym should be more general than the hyponym. As in HyperScore (Nguyen et al., 2017) , we score the hypernym candidates by multiplying two factors corresponding to these properties. The C•∆S (i.e. the cosine similarity multiply the difference of summation) scoring function is defined as

EQUATION

where w p is the embedding of hypernym and w q is the embedding of hyponym.

As far as we know, Gaussian embedding (GE) is the only unsupervised embedding method which can capture the asymmetric relations between a hypernym and its hyponyms. Using the same training and testing setup, we use the code implemented by Athiwaratkun & Wilson (2017) 1 to train Gaussian embedding on the first 51.2 million tokens and test the embeddings on 11 datasets. Its hyper-parameters are determined using the same way as DIVE (i.e. maximizing the AP on Hyper-Net training set). We compare DIVE with GE2 in Table 1 , and the performances of random scores and only measuring word similarity using skip-grams are also presented for reference. As we can see, DIVE is usually significantly better than other baselines.

5. EXPERIMENT 2: HYPERNYMY SCORING FUNCTIONS ANALYSIS:

In Experiment 1, we show that there exists a scoring function (C•∆S) which detects hypernymy accurately using the embedding space of DIVE. Nevertheless, different scoring functions measure different signals in SBOW or embeddings. Since there are so many scoring functions and datasets available in the domain, we first introduce and test the performances of various scoring functions so as to select the representative ones for a more comprehensive evaluation of DIVE on the hypernymy detection tasks. We denote the embedding/context vector of the hypernym candidate and the hyponym candidate as w p and w q , respectively. The SBOW model which represents a word by the frequency of its neighboring words is denoted as SBOW Freq, while the SBOW which uses PPMI of its neighboring words as the features (Bullinaria & Levy, 2007) is denoted as SBOW PPMI.

5.1 UNSUPERVISED SCORING FUNCTIONS

5.1.1. SIMILARITY:

A hypernym tends to be similar to its hyponym, so we measure the cosine similarity between word vectors of the SBOW features (Levy et al., 2015b) or DIVE. We refer to the symmetric scoring function as Cosine or C for short in the following tables. We also train the original skip-grams with 100 dimensions and measure the cosine similarity between the resulting word2vec embeddings. This scoring function is referred to as Word2vec or W.

5.1.2. GENERALITY:

The distributional informativeness hypothesis (Santus et al., 2014) Weeds Precision (Weeds & Weir, 2003) and CDE (Clarke, 2009) both measure the magnitude of the intersection between feature vectors (|w p ∩ w q |). For example, w p ∩ w q is defined by the elementwise minimum in CDE. Then, both scoring functions divide the intersection by the magnitude of the potential hyponym vector (|w q |). invCL (Lenci & Benotto, 2012 ) (A variant of CDE) is also tested.

We choose these 3 functions because they have been shown to detect hypernymy well in a recent study (Santus et al., 2017) . However, it is hard to confirm that their good performances come from the inclusion property between context distributions -it is also possible that the context vectors of more general words have higher chance to overlap with all other words due to their high frequency.

For instance, considering a one dimension feature which stores only the frequency of words, the naive embedding could still have reasonable performance on the CDE function, but the embedding in fact only memorizes the general words without modeling relations between words (Levy et al., 2015b) and loses lots of inclusion signals in the word co-occurrence statistics.

In order to measure the inclusion property without the interference of the word frequency signal from the SBOW or embeddings, we propose a new measurement called asymmetric L 1 distance.

We first get context distributions d p and d q by normalizing w p and w q , respectively. Ideally, the context distribution of the hypernym d p will include d q . This suggests the hypernym distribution d p is larger than context distribution of the hyponym with a proper scaling factor ad q (i.e., max(ad qd p , 0) should be small). Furthermore, both distributions should be similar, so ad q should not be too different from d p (i.e., max(d p -ad q , 0) should also be small). Therefore, we define asymmetric L1 distance as

EQUATION

where w 0 is a constant which emphasizes the inclusion penalty. If w 0 = 1 and a = 1, AL 1 is equivalent to L1 distance. The lower AL 1 distance implies a higher chance of observing the hypernym relation. We tried w 0 = 5 and w 0 = 20. w 0 = 20 produces a worse micro-average AP@all on SBOW Freq, SBOW PPMI and DIVE, so we fix w 0 to be 5 in all experiments. An efficient way to solve the optimization in AL 1 is presented in the supplementary materials.

5.2. RESULTS AND DISCUSSIONS:

We show the micro average AP@all on 10 datasets using different hypernymy scoring functions in Table 2 . We can see the combination functions such as C•∆S and W•∆S perform the best overall. Among the unnormalized inclusion based scoring functions, CDE works the best. AL 1 performs well compared with other functions which remove the frequency signal such as Word2vec, Cosine, and SLQS Row. The summation is the most robust generality measurement. In the table, the scoring functions are applied to SBOW Freq, but the performances of hypernymy scoring functions on the other feature spaces (e.g. DIVE) have a similar trend.

6 EXPERIMENT 3: COMPARISON WITH SBOW

6.1. COMPARISON WITH PREVIOUSLY REPORTED RESULTS:

In Table 3 , DIVE with two of the best scoring functions (C•∆S and W•∆S) is compared with the previous unsupervised state-of-the-art approaches based on SBOW on different datasets.

There are several reasons which might cause the large performance gaps in some datasets. In addition to the effectiveness of DIVE, some improvements come from our proposed scoring functions. The fact that every paper uses a different training corpus also affects the performances. Furthermore, Santus et al. (2017) select the scoring functions and feature space for the first 4 datasets based on AP@100, which we believe is too sensitive to the hyper-parameter settings of different methods. To isolate the impact of each factor, we perform a more comprehensive comparison next.

6.2. PERFORMANCE ANALYSIS:

In this experiment, we examine whether DIVE successfully preserves the signals for hypernymy detection tasks, which are measured by the same scoring functions designed for SBOW. Summation difference (∆S) and CDE perform the best among generality and inclusion functions in Table 2 , (Lenci & Benotto, 2012) , APSyn (Santus et al., 2016) , and CDE (Clarke, 2009) are selected because they have the best AP@100 in the first 4 datasets (Santus et al., 2017) . Cosine similarity (Levy et al., 2015b) , balAPinc (Kotlerman et al., 2010) in 3 datasets (Turney & Mohammad, 2015) , SLQS (Santus et al., 2014) in HyperNet dataset (Shwartz et al., 2016) , and Freq ratio (FR) (Vulić et al., 2016) respectively. AL 1 could be used to examine the inclusion properties after removing the frequency signal. Therefore, we will present the results using these 3 scoring functions, along with W•∆S and C•∆S.

6.2.1. BASELINES:

In addition to classic representations such as SBOW Freq and SBOW PPMI, we compare distributional inclusion vector embedding (DIVE) with additional 4 baselines in Table 4 .

• SBOW PPMI with additional frequency weighting (PPMI w/ FW). Specifically, w[c] = max(log( P (w,c) P (w) * P (c) * Z #(w)

), 0). This forms the matrix reconstructed by DIVE when k = 1.

• DIVE without the PMI filter (DIVE w/o PMI) • NMF on shifted PMI: Non-negative matrix factorization (NMF) on the shifted PMI without frequency weighting for DIVE (DIVE w/o FW). This is the same as applying the nonnegative constraint on the skip-gram model. • K-means (Freq NMF): The method first uses Mini-batch k-means (Sculley, 2010) to cluster words in skip-gram embedding space into 100 topics, and hashes each frequency count in SBOW into the corresponding topic. If running k-means on skip-grams is viewed as an approximation of clustering the SBOW context vectors, the method can be viewed as a kind of NMF (Ding et al., 2005) . Let the N × N context matrix be denoted as M c , where the (i, j)th element stores the count of word j appearing beside word i. K-means hashing creates a N × 100 matrix G with orthonormal rows (G T G = I), where the (i, k)th element is 0 if the word i does not belong to cluster k. The orthonormal G is also an approximated solution of a type of NMF (M c ≈ F G T ) (Ding et al., 2005) . Hashing context vectors into topic vectors can be written as

M c G ≈ F G T G = F .

In the experiment, we also tried to apply a constant log(k) shifting to SBOW PPMI (i.e. max(P M I -log(k), 0)). We found that the performance degrades as k increases. Similarly, applying PMI filter to SBOW PPMI (set context feature to be 0 if the value is lower than log(k f )) usually makes the performances worse, especially when k f is large. Applying PMI filter to SBOW Freq only makes its performances closer to (but still much worse than) SBOW PPMI, so we omit this baseline as well.

In Table 4 , we first confirm the finding of the previous review study of Santus et al. (2017) : there is no single hypernymy scoring function which always outperforms others. One of the main reasons is that different datasets collect negative samples differently. This is also why we evaluate our method on many datasets to make sure our conclusions hold in general. For example, if negative samples come from random word pairs (e.g. WordNet dataset), a symmetric similarity measure is already a pretty good scoring function. On the other hand, negative samples come from related or similar words in HyperNet, EVALution, Lenci/Benotto, and Weeds, so only computing generality difference leads to the best (or close to the best) performance. The negative samples in many datasets are composed of both random samples and similar words (such as BLESS), so the combination of similarity and generality difference yields the most stable results.

DIVE performs similar or better on all the scoring functions compared with SBOW consistently across all datasets in Table 4 , while using many fewer dimensions (see Table 6 ). Its results on combination scoring functions outperform SBOW Freq. Meanwhile, its results on AL 1 outperform SBOW PPMI. The fact that combination scoring functions (i.e., W•∆S or C•∆S) usually outperform generality functions suggests that only memorizing general words is not sufficient. The best average performance on 4 and 10 datasets are both produced by W•∆S on DIVE.

SBOW PPMI improves the combination functions from SBOW Freq but sacrifices AP on the inclusion functions. It generally hurts performance to change the frequency sampling of PPMI (PPMI w/ FW) or compute SBOW PPMI on the whole WaCkypedia (all wiki) instead of the first 51.2 million tokens. The similar trend can also be seen in Table 5 . Note that AL 1 completely fails in HyperLex dataset using SBOW PPMI, which suggests that PPMI might not necessarily preserve the distributional inclusion property, even though it can have good performance on combination functions.

Removing the PMI filter from DIVE slightly drops the overall precision while removing frequency weights on shifted PMI (w/o FW) leads to poor performances. K-means (Freq NMF) produces similar AP compared with SBOW Freq, but has worse AL 1 scores. Its best AP scores on different datasets are also significantly worse than the best AP of DIVE. This means that only making word2vec (skip-grams with negative sampling) non-negative or naively accumulating topic distribution in contexts cannot lead to satisfactory embeddings. Table 7 : Spectral clustering on the non-zero DIVE dimensions of the query word. When the number of clusters is set to be 2, we present the top 4 dimensions in each cluster, which have the highest values on the query word embedding. Otherwise, the top 2 dimensions are presented. CID refers to cluster ID. The top 5 words with the highest values of each dimension are presented.

In addition to hypernymy detection, Athiwaratkun & Wilson (2017) show that the mixture of Gaussian distributions can also be used to discover multiple senses of each word. In our qualitative experiment, we show that DIVE can achieve the similar goal without fixing the number of senses before training the embedding.

Recall that each dimension roughly corresponds to one topic. Given a query word, the higher embedding value on a dimension implies higher likelihood to observe the word in the context of the topic. The embedding of a polysemy would have high values on different groups of topics/dimensions. This allows us to discover the senses by clustering the topics/dimensions of the polysemy. We use the embedding values as the feature each dimension, compute the pairwise similarity between dimensions, and apply spectral clustering (Stella & Shi, 2003) to group topics as shown in the Table 7 . See more implementation details in the supplementary materials.

In the word sense disambiguation tasks, it is usually challenging to determine how many senses/clusters each word should have. Many existing approaches fix the number of senses before training the embedding (Tian et al., 2014; Athiwaratkun & Wilson, 2017) . Neelakantan et al. (2014) make the number of clusters approximately proportional to the diversity of the context, but the assumption does not always hold. Furthermore, the training process cannot capture different granularity of senses. For instance, race in the car context could share the same sense with the race in the game topic because they all mean contest, but the race in the car context actually refers to the specific contest of speed. Therefore, they can also be viewed as separate senses (like the results in Table 7 ). This means the correct number of clusters is not unique, and the methods, which fixes the cluster numbers, need to re-train the embedding many times to capture such granularity.

In our approach, clustering dimensions is done after the training process of DIVE is completed, so it is fairly efficient to change the cluster numbers and hierarchical clustering is also an option. Similar to our method, Pelevina et al. ( 2016) also discover word senses by graph-based clustering. The main difference is that they cluster the top n words which are most related to the query word instead of topics. However, choosing the hyper-parameter n is difficult. Large n would make graph clustering algorithm inefficient, while small n would make less frequent senses difficult to discover.

8. RELATED WORK:

Most previous unsupervised approaches focus on designing better hypernymy scoring functions for sparse bag of word (SBOW) features. They are well summarized in the recent study (Santus et al., 2017) . Santus et al. (2017) also evaluate the influence of different contexts, such as changing the window size of contexts or incorporating dependency parsing information, but neglect scalability issues inherent to SBOW methods.

A notable exception is the Gaussian embedding model (Vilnis & McCallum, 2015) . The context distribution of each word is encoded as a multivariate Gaussian distribution, where the embeddings of hypernyms tend to have higher variance and overlap with the embedding of their hyponyms. However, since a Gaussian distribution is normalized, it is difficult to retain frequency information during the embedding process, and experiments on HyperLex (Vulić et al., 2016) demonstrate that a simple baseline only relying on word frequency can achieve good results. Follow-up work models contexts by a mixture of Gaussians (Athiwaratkun & Wilson, 2017) relaxing the unimodality assumption but achieves little improvement on hypernym detection tasks. Kiela et al. (2015) show that images retrieved by a search engine can be a useful source of information to determine the generality of lexicons, but the resources might not be available for some corpora such as scientific literature.

Order embedding (Vendrov et al., 2016) is a supervised approach to encode many annotated hypernym pairs (e.g. all of the whole WordNet (Miller, 1995) ) into a compact embedding space, where the embedding of a hypernym should be smaller than the embedding of its hyponym in every dimension.

Our method learns embedding from raw text, where a hypernym embedding should be larger than the embedding of its hyponym in every dimension. Thus, DIVE can be viewed as an unsupervised and reversed form of order embedding.

Other semi-supervised hypernym detection methods aim to generalize from sets of annotated word pairs using raw text corpora. The goal of HyperScore (Nguyen et al., 2017) is similar to our model: the embedding of a hypernym should be similar to its hyponym but with higher magnitude. However, their training process relies heavily on annotated hypernym pairs, and the performance drops significantly when reducing the amount of supervision. In addition to context distributions, previous work also leverages training data to discover useful text pattern indicating is-a relation (Shwartz et al., 2016; Roller & Erk, 2016) , but it remains challenging to increase recall of hypernym detection because commonsense facts like cat is-a animal might not appear in the corpus.

Non-negative matrix factorization (NMF) has a long history in NLP, for example in the construction of topic models (Pauca et al., 2004) . Non-negative sparse embedding (NNSE) (Murphy et al., 2012) and Faruqui et al. (2015) indicate that non-negativity can make embeddings more interpretable and improve word similarity evaluations. The sparse NMF is also shown to be effective in cross-lingual lexical entailment tasks but does not necessarily improve monolingual hypernymy detection (Vyas & Carpuat, 2016) . In our study, a new type of NMF is proposed, and the comprehensive experimental analysis demonstrates its state-of-the-art performances on unsupervised hypernymy detection.

9. CONCLUSIONS:

Compressing unsupervised SBOW models into a compact representation is challenging while preserving the inclusion, generality, and similarity signals which are important for hypernym detection. Our experiments suggest that simple baselines such as accumulating K-mean clusters and non-negative skip-grams do not lead to satisfactory performances in this task.

To achieve this goal, we proposed an interpretable and scalable embedding method called distributional inclusion vector embedding (DIVE) by performing non-negative matrix factorization (NMF) on a weighted PMI matrix. We demonstrate that scoring functions which measure inclusion and generality properties in SBOW can also be applied to DIVE to detect hypernymy, and DIVE performs the best on average, slightly better than SBOW while using many fewer dimensions.

Our experiments also indicate that unsupervised scoring functions, which combine similarity and generality measurements, work the best in general, but no one scoring function dominates across all datasets. A combination of unsupervised DIVE with the proposed scoring functions produces new state-of-the-art performances on many datasets under the unsupervised setup.

Finally, a qualitative experiment shows that clusters of the topics discovered by DIVE often correspond to the word senses, which allow us to do word sense disambiguation without the need to know the number of senses before training the embeddings. Score (Nguyen et al., 2017) and H-feature (Roller & Erk, 2016) . When we compare F1 with the results from other papers, we use 20 fold cross validation to determine prediction thresholds, as done by Roller & Erk (2016) . Note that HyperScore ignores POS in the testing data, so we follow the setup when comparing with it. In addition to the unsupervised approach, we also compare DIVE with semi-supervised approaches.

When there are sufficient training data, there is no doubt that the semi-supervised embedding approaches such as HyperNet (Shwartz et al., 2016) , H-feature detector (Roller & Erk, 2016) , and HyperScore (Nguyen et al., 2017) can achieve better performance than all unsupervised methods. However, in many domains such as scientific literature, there are often not many annotated hypernymy pairs (e.g. Medical dataset (Levy et al., 2014) ).

Since we are comparing an unsupervised method with semi-supervised methods, it is hard to fairly control the experimental setups and tune the hyper-parameters. In Table 8 , we only show several performances which are copied from the original paper when training data are limited3 . As we can see, the performance from DIVE is roughly comparable to the previous semi-supervised approaches trained on small amount of hypernym pairs. This demonstrates the robustness of our approach and the difficulty of generalizing hypernymy annotations with semi-supervised approaches.

A.2 GENERALITY ESTIMATION AND HYPERNYM DIRECTIONALITY DETECTION:

In Table 9 , we show the most general words in DIVE under different queries as constraints. We also present the accuracy of judging which word is a hypernym (more general) given word pairs with 

| 1 -|w q | 1 > 0 (|w p | 1 > |w q | 1 ).

From the table, we can see that the simple summation difference performs better than SQLS Sub, and DIVE predicts directionality as well as SBOW. Notice that whenever we encounter OOV, the directionality is predicted randomly. If OOV is excluded, the accuracy of predicting directionality using unsupervised methods can reach around 0.7-0.75.

A.3 EXPERIMENTAL DETAILS FOR HYPERNYMY DETECTION:

In HyperNet and WordNet, some hypernym relations are determined between phrases instead of words. Phrase embeddings are composed by averaging word embeddings or SBOW features. For WordNet, we assume the Part of Speech (POS) tags of the words are the same as the phrase. All part-of-speech (POS) tags in the experiments come from NLTK.

The window size |W | of SBOW, DIVE, and GE are set as 20 (left 10 words and right 10 words).

For DIVE, the number of epochs is 15, the learning rate is 0.001, the batch size is 128, the threshold in PMI filter k f is set to be 30, and the ratio between negative and positive samples (k) is 1.5. The hyper-parameters of DIVE were decided based on the performance of HyperNet training set. The window size of skip-grams (word2vec) is 10. The number of negative samples (k ) in skip-gram is set as 5. When composing skip gram into phrase embedding, average embedding is used.

For Gaussian embedding (GE), the number of mixture is 1, the number of dimension is 100, the learning rate is 0.01, the lowest variance is 0.1, the highest variance is 100, the highest Gaussian mean is 10, and other hyper-parameters are the default value in https://github.com/ benathi/word2gm. The hyper-parameters of GE were also decided based on the performance of HyperNet training set. When determining the score between two phrases, we use the average score of every pair of tokens in two phrases.

The number of testing pairs N and the number of OOV word pairs is presented in Table 11 .

A.4 CLUSTERING DIMENSIONS FOR WORD SENSE DISAMBIGUATION:

We use all the default hyper-parameters of the spectral clustering library in Scikit-learn 0.18.2 (Pedregosa et al., 2011) except the number of clusters is set manually. A simple way to prepare the feature of each dimension f (c i ) is to use the embedding values in that dimension w[c i ] of all the words in our vocabulary w ∈ V . That is,

EQUATION

However, clustering on the global features might group topics together based on the co-occurrence of words which are unrelated to the query words and we want to make the similarity dependent on the query word. For example, a country topic should be clustered together with a city topic if the query word is place, but it makes more sense to group the country topic with the money topic together if the query word is bank like we did in the word sense disambiguation experiment (Table 7 ). This means we want to focus on the geographical meaning of country when the query is related to geography, while focus on the economic meaning of country when the query is about economics.

To create query dependent similarity measurement, we only consider the embedding of words which are related to the query word when preparing the features of dimensions. Specifically, given a query word q, the feature vector of the ith dimension f (c i , q) is defined as:

EQUATION

where w q [c j ] is the value of jth dimension of query word embedding, C j (n) is the set of embeddings of top n words in the jth dimension, and the operator ⊕ means concatenation. This means instead of considering all the words in the vocabulary, we only take the top n words of every dimension j (n is fixed as 100 in the experiment), weight the feature based on how likely to observe query word in dimension j (w q [c j ]), and concatenate all features together. That is, when measuring the similarity of dimensions, we only consider the aspects related to query word (e.g. mostly considering words related to facility and money when the query word is bank).

After the features of all dimensions are collected, we normalize the feature of each dimension to have the norm 1, compute the pairwise similarity and run the spectral clustering to get the clustering results.

A.5 EFFICIENT WAY TO COMPUTE ASYMMETRIC L1 (AL 1 )

Recall that Equation (8) defines AL 1 as follows: 

AL 1 = L = min

:

https://github.com/benathi/word2gm

Notice that higher AP is reported in the previous literature: 80(Vilnis & McCallum, 2015) in LEDS, 74.2(Athiwaratkun & Wilson, 2017) in LEDS, and 20.6(Vulić et al., 2016) in HyperLex. The difference might be caused by different training and testing setup (e.g. their hyper-parameters might be determined by (parts of) testing dataset).

We neglect the performances from models trained on more than 10,000 hypernym pairs, models trained on the same datasets with more than 1000 hypernym pairs using cross-validation, and models using image classifier.

Figures and Tables:

[w, c]. Next, the matrix M is factorized such that M [w, c] ≈ w T c, where w is the low dimension embedding of wth word and c is the cth context embedding. The statistic in M [w, c] is usually related to pointwise mutual information: P M I(w, c) = log( P (w,c) P (w)•P (c) ), where P (w, c) = #(w,c) |D| , |D| = w∈V c∈V #(w, c) is number of co-occurrence word pairs in the corpus, P (w) = #(w) |D| , #(w) = c∈V #(w, c) is the frequency of the word w times the window size |W |, and similarly for P (c). For example, M [w, c] could be set as positive PMI (PPMI), max(P M I(w, c), 0), or shifted PMI, P M I(w, c) -log(k), like skip-grams with negative sampling (SGNS) (Levy et al., 2015a). Intuitively, since M [w, c] ≈ w T c, larger embedding values of w at every dimension seems to imply larger w T c, larger M [w, c], larger P M I(w, c), and thus larger co-occurrence count #(w, c). However, the derivation has two flaws: (1) c could be negative and (2) lower #(w, c) could still lead to larger P M I(w, c) as long as the #(w) is small enough.

Figure 1: The top 15 dimensions in the distributional inclusion vector embedding (DIVE) of the word core trained by the co-occurrence statistics of context words. The index of dimensions is sorted by the embedding values. The words in each row of the table are sorted by its embedding value in the dimension.

Figure 2: An example of AL 1 distance. If the word pair indeed has the hypernym relation, the context distribution of hyponym (d q ) tends to be included in the context distribution of hypernym (d p ) after proper scaling according to DIH. Thus, the context words only appear beside the hyponym candidate (ad q [c] -d p [c]) causes higher penalty.

Micro average AP@all (%) of 10 datasets using different scoring functions. The feature space is SBOW using word frequency.

Table content: <table><tr><td colspan="3">Word2vec (W) Cosine (C) SLQS Sub</td><td>SLQS Row (∆E)</td><td colspan="2">Summation (∆S) Two norm (∆Q)</td></tr><tr><td>24.8</td><td>26.7</td><td>27.4</td><td>27.6</td><td>31.5</td><td>31.2</td></tr><tr><td>W•∆E</td><td>C•∆E</td><td>W•∆S</td><td>C•∆S</td><td>W•∆Q</td><td>C•∆Q</td></tr><tr><td>28.8</td><td>29.5</td><td>31.6</td><td>31.2</td><td>31.4</td><td>31.1</td></tr><tr><td>Weeds</td><td>CDE</td><td>invCL</td><td>Asymmetric L1 (AL 1 )</td><td/><td/></tr><tr><td>19.0</td><td>31.1</td><td>30.7</td><td>28.2</td><td/><td/></tr></table>

Comparison with previous methods based on sparse bag of word (SBOW). All values are percentages. The results of invCL

Table content: <table/>

are compared.

Table content: <table><tr><td>Dataset</td><td colspan="2">BLESS EVALution</td><td colspan="2">LenciBenotto</td><td>Weeds</td><td>Medical</td></tr><tr><td>Metric</td><td/><td colspan="2">AP@all</td><td/><td/><td>F1</td></tr><tr><td>Baselines</td><td>5.1</td><td>invCL 35.3</td><td colspan="2">APSyn 38.2</td><td>CDE 44.1</td><td>Cosine 23.1</td></tr><tr><td>DIVE + C•∆S</td><td>16.3</td><td>33.0</td><td colspan="2">50.4</td><td>65.5</td><td>25.3</td></tr><tr><td>DIVE + W•∆S</td><td>18.6</td><td>32.3</td><td colspan="2">51.5</td><td>68.6</td><td>25.7</td></tr><tr><td>Dataset</td><td>LEDS</td><td>TM14</td><td colspan="3">Kotlerman 2010 HyperNet</td><td>HyperLex</td></tr><tr><td>Metric</td><td/><td>AP@all</td><td/><td/><td>F1</td><td>Spearman ρ</td></tr><tr><td>Baselines</td><td>73</td><td colspan="2">balAPinc 56</td><td>37</td><td>SLQS 22.8</td><td>Freq ratio 27.9</td></tr><tr><td>DIVE + C•∆S</td><td>83.5</td><td>57.2</td><td colspan="2">36.6</td><td>41.9</td><td>32.8</td></tr><tr><td>DIVE + W•∆S</td><td>86.4</td><td>57.3</td><td colspan="2">37.4</td><td>38.6</td><td>33.3</td></tr></table>

AP@all (%) of 10 datasets. The box at lower right corner compares the micro average AP across all 10 datasets. Numbers in different rows come from different feature or embedding spaces. Numbers in different columns come from different datasets and unsupervised scoring functions. We also present the micro average AP across the first 4 datasets (BLESS, EVALution, Lenci/Benotto and Weeds). All wiki means SBOW using PPMI features trained on the whole WaCkypedia. FW refers to frequency weighting on the shifted PMI matrix.

Table content: <table><tr><td colspan="2">AP@all (%)</td><td colspan="12">BLESS CDE AL1 ∆S W•∆S C•∆S CDE AL1 ∆S W•∆S C•∆S CDE AL1 ∆S W•∆S C•∆S EVALution Lenci/Benotto</td></tr><tr><td/><td>Freq</td><td>6.3</td><td>7.3</td><td>5.6</td><td>11.0</td><td>5.9</td><td>35.3 32.6 36.2</td><td>33.0</td><td>36.3</td><td colspan="3">51.8 47.6 51.0</td><td>51.8</td><td>51.1</td></tr><tr><td>SBOW</td><td>PPMI PPMI w/ FW</td><td>13.6 6.2</td><td>5.1 5.0</td><td>5.6 5.5</td><td>17.2 12.4</td><td>15.3 5.8</td><td>30.4 27.7 34.1 36.0 27.5 36.3</td><td>31.9 32.9</td><td>34.3 36.4</td><td colspan="3">47.2 39.7 50.8 52.0 43.1 50.9</td><td>51.1 51.9</td><td>52.0 50.7</td></tr><tr><td/><td>All wiki</td><td>12.1</td><td>5.2</td><td>6.9</td><td>12.5</td><td>13.4</td><td>28.5 27.1 30.3</td><td>29.9</td><td>31.0</td><td colspan="3">47.1 39.9 48.5</td><td>48.7</td><td>51.1</td></tr><tr><td/><td>Full</td><td>9.3</td><td>7.6</td><td>6.0</td><td>18.6</td><td>16.3</td><td>30.0 27.5 34.9</td><td>32.3</td><td>33.0</td><td colspan="3">46.7 43.2 51.3</td><td>51.5</td><td>50.4</td></tr><tr><td>DIVE</td><td>w/o PMI</td><td>7.8</td><td>6.9</td><td>5.6</td><td>16.7</td><td>7.1</td><td>32.8 32.2 35.7</td><td>32.5</td><td>35.4</td><td colspan="3">47.6 44.9 50.9</td><td>51.6</td><td>49.7</td></tr><tr><td/><td>w/o FW</td><td>9.0</td><td>6.2</td><td>7.3</td><td>6.2</td><td>7.3</td><td>24.3 25.0 22.9</td><td>23.5</td><td>23.9</td><td colspan="3">38.8 38.1 38.2</td><td>38.2</td><td>38.4</td></tr><tr><td colspan="2">Kmean (Freq NMF)</td><td>6.5</td><td>7.3</td><td>5.6</td><td>10.9</td><td>5.8</td><td>33.7 27.2 36.2</td><td>33.0</td><td>36.2</td><td colspan="3">49.6 42.5 51.0</td><td>51.8</td><td>51.2</td></tr><tr><td colspan="2">AP@all (%)</td><td colspan="12">Weeds CDE AL1 ∆S W•∆S C•∆S CDE AL1 ∆S W•∆S C•∆S CDE AL1 ∆S W•∆S C•∆S Micro Average (4 datasets) Medical</td></tr><tr><td/><td>Freq</td><td colspan="3">69.5 58.0 68.8</td><td>68.2</td><td>68.4</td><td>23.1 21.8 22.9</td><td>25.0</td><td>23.0</td><td colspan="3">19.4 19.2 14.1</td><td>18.4</td><td>15.3</td></tr><tr><td>SBOW</td><td colspan="4">PPMI PPMI w/ FW 67.6 52.2 69.4 61.0 50.3 70.3</td><td>69.2 68.7</td><td>69.3 67.7</td><td>24.7 17.9 22.3 23.2 18.2 22.9</td><td>28.1 25.8</td><td>27.8 22.9</td><td colspan="3">23.4 22.8 10.6 13.7 8.7 13.2</td><td>20.1 18.6</td><td>24.4 17.0</td></tr><tr><td/><td>All wiki</td><td colspan="3">61.3 48.6 70.0</td><td>68.5</td><td>70.4</td><td>23.4 17.7 21.7</td><td>24.6</td><td>25.8</td><td>22.3</td><td>8.9</td><td>12.2</td><td>17.6</td><td>21.1</td></tr><tr><td/><td>Full</td><td colspan="3">59.2 55.0 69.7</td><td>68.6</td><td>65.5</td><td>22.1 19.8 22.8</td><td>28.9</td><td>27.6</td><td>11.7</td><td>9.3</td><td>13.7</td><td>21.4</td><td>19.2</td></tr><tr><td>DIVE</td><td>w/o PMI</td><td colspan="3">60.4 56.4 69.3</td><td>68.6</td><td>64.8</td><td>22.2 21.0 22.7</td><td>28.0</td><td>23.1</td><td>10.7</td><td>8.4</td><td>13.3</td><td>19.8</td><td>16.2</td></tr><tr><td/><td>w/o FW</td><td colspan="3">49.2 47.3 45.1</td><td>45.1</td><td>44.9</td><td>18.9 17.3 17.2</td><td>16.8</td><td>17.5</td><td>10.9</td><td>9.8</td><td>7.4</td><td>7.6</td><td>7.7</td></tr><tr><td colspan="2">Kmean (Freq NMF)</td><td colspan="3">69.4 51.1 68.8</td><td>68.2</td><td>68.9</td><td>22.5 19.3 22.9</td><td>24.9</td><td>23.0</td><td colspan="3">12.6 10.9 14.0</td><td>18.1</td><td>14.6</td></tr><tr><td colspan="2">AP@all (%)</td><td colspan="12">LEDS CDE AL1 ∆S W•∆S C•∆S CDE AL1 ∆S W•∆S C•∆S CDE AL1 ∆S W•∆S C•∆S TM14 Kotlerman 2010</td></tr><tr><td colspan="2">SBOW Freq</td><td colspan="3">82.7 70.4 70.7</td><td>83.3</td><td>73.3</td><td>55.6 53.2 54.9</td><td>55.7</td><td>55.0</td><td colspan="3">35.9 40.5 34.5</td><td>37.0</td><td>35.4</td></tr><tr><td colspan="2">SBOW PPMI</td><td colspan="3">84.4 50.2 72.2</td><td>86.5</td><td>84.5</td><td>56.2 52.3 54.4</td><td>57.0</td><td>57.6</td><td colspan="3">39.1 30.9 33.0</td><td>37.0</td><td>36.3</td></tr><tr><td/><td>All wiki</td><td colspan="3">83.1 49.7 67.9</td><td>82.9</td><td>81.4</td><td>54.7 50.5 52.6</td><td>55.1</td><td>54.9</td><td colspan="3">38.5 31.2 32.2</td><td>35.4</td><td>35.3</td></tr><tr><td/><td>DIVE</td><td colspan="3">83.3 74.7 72.7</td><td>86.4</td><td>83.5</td><td>55.3 52.6 55.2</td><td>57.3</td><td>57.2</td><td colspan="3">35.3 31.6 33.6</td><td>37.4</td><td>36.6</td></tr><tr><td colspan="2">AP@all (%)</td><td colspan="12">HyperNet CDE AL1 ∆S W•∆S C•∆S CDE AL1 ∆S W•∆S C•∆S CDE AL1 ∆S W•∆S C•∆S WordNet Micro Average (10 datasets)</td></tr><tr><td colspan="2">SBOW Freq</td><td colspan="3">37.5 28.3 46.9</td><td>35.9</td><td>43.4</td><td>56.6 55.2 55.5</td><td>56.2</td><td>55.6</td><td colspan="3">31.1 28.2 31.5</td><td>31.6</td><td>31.2</td></tr><tr><td colspan="2">SBOW PPMI</td><td colspan="3">23.8 24.0 47.0</td><td>32.5</td><td>33.1</td><td>57.7 53.9 55.6</td><td>56.8</td><td>57.2</td><td colspan="3">30.1 23.0 31.1</td><td>32.9</td><td>33.5</td></tr><tr><td/><td>All wiki</td><td colspan="3">23.0 24.5 40.5</td><td>30.5</td><td>29.7</td><td>57.4 53.1 56.0</td><td>56.4</td><td>57.3</td><td colspan="3">29.0 23.1 29.2</td><td>30.2</td><td>31.1</td></tr><tr><td/><td>DIVE</td><td colspan="3">25.3 24.2 49.3</td><td>33.6</td><td>32.0</td><td>60.2 58.9 58.4</td><td>61.1</td><td>60.9</td><td colspan="3">27.6 25.3 32.1</td><td>34.1</td><td>32.7</td></tr></table>

Spearman ρ (%) in HyperLex.

Table content: <table><tr><td>Spearman ρ (%)</td><td colspan="3">HyperLex CDE AL 1 ∆S W•∆S C•∆S</td></tr><tr><td>SBOW Freq</td><td>31.7 19.6 27.6</td><td>29.6</td><td>27.3</td></tr><tr><td>SBOW PPMI</td><td>28.1 -2.3 31.8</td><td>34.3</td><td>34.5</td></tr><tr><td>All wiki</td><td>25.3 -2.2 28.0</td><td>30.5</td><td>31.0</td></tr><tr><td>DIVE</td><td>28.9 18.7 31.2</td><td>33.3</td><td>32.8</td></tr></table>

The average number of non-zero dimensions across all testing words in 10 datasets.

Table content: <table><tr><td colspan="3">SBOW Freq SBOW PPMI DIVE</td></tr><tr><td>5799</td><td>3808</td><td>20</td></tr></table>

Comparison with semi-supervised embeddings (with limited training data). All values are percentages. The number in parentheses beside each approach indicates the number of annotated hypernymy word pairs used to train the model. Semi-supervised embeddings include Hyper-

Table content: <table/>

We show the top 30 words with the highest embedding magnitude after dot product with the query embedding q (i.e. showing w such that |w T q| 1 is one of the top 30 highest values). The rows with the empty query word sort words based on |w| 1 .

Table content: <table><tr><td/><td>Dataset</td><td colspan="2">HyperLex</td><td colspan="3">EVALution LenciBenotto Weeds</td><td>Medical</td></tr><tr><td/><td>Metric</td><td colspan="2">Spearman ρ</td><td/><td>AP@all</td><td/><td>F1</td></tr><tr><td colspan="2">Baselines</td><td/><td/><td colspan="2">HyperScore (1337)</td><td/><td>H-feature (897)</td></tr><tr><td colspan="2">(#Training Hypernymy)</td><td>30</td><td/><td>39</td><td>44.8</td><td>58.5</td><td>26</td></tr><tr><td colspan="2">DIVE + C•∆S (0)</td><td>34.5</td><td/><td>33.8</td><td>52.9</td><td>70.0</td><td>25.3</td></tr><tr><td>Query</td><td/><td/><td/><td colspan="2">Top 30 general words</td><td/></tr><tr><td/><td>use</td><td>name</td><td/><td>system</td><td>include</td><td>base</td><td>city</td></tr><tr><td/><td>large</td><td>state</td><td/><td>group</td><td>power</td><td>death</td><td>form</td></tr><tr><td/><td>american</td><td>life</td><td/><td>may</td><td>small</td><td>find</td><td>body</td></tr><tr><td/><td>design</td><td>work</td><td colspan="2">produce</td><td>control</td><td>great</td><td>write</td></tr><tr><td/><td>study</td><td>lead</td><td/><td>type</td><td>people</td><td>high</td><td>create</td></tr><tr><td/><td>specie</td><td>species</td><td/><td>animal</td><td>find</td><td>plant</td><td>may</td></tr><tr><td/><td>human</td><td>bird</td><td/><td>genus</td><td>family</td><td>organism</td><td>suggest</td></tr><tr><td>species</td><td>gene</td><td>tree</td><td/><td>name</td><td>genetic</td><td>study</td><td>occur</td></tr><tr><td/><td>fish</td><td>disease</td><td/><td>live</td><td>food</td><td>cell</td><td>mammal</td></tr><tr><td/><td>evidence</td><td>breed</td><td colspan="2">protein</td><td>wild</td><td>similar</td><td>fossil</td></tr><tr><td/><td>system</td><td>use</td><td/><td>design</td><td>provide</td><td>operate</td><td>model</td></tr><tr><td/><td>standard</td><td>type</td><td colspan="2">computer</td><td colspan="2">application develop</td><td>method</td></tr><tr><td>system</td><td>allow</td><td>function</td><td/><td>datum</td><td>device</td><td>control</td><td>information</td></tr><tr><td/><td>process</td><td>code</td><td/><td>via</td><td>base</td><td>program</td><td>software</td></tr><tr><td/><td>network</td><td>file</td><td colspan="2">development</td><td>service</td><td>transport</td><td>law</td></tr><tr><td colspan="4">A SUPPLEMENTARY MATERIALS</td><td/><td/><td/></tr><tr><td colspan="6">A.1 COMPARISON WITH SEMI-SUPERVISED EMBEDDINGS</td><td/></tr></table>

Accuracy (%) of hypernym directionality prediction across 10 datasets.

Table content: <table><tr><td/><td>Micro Average (10 datasets)</td><td/><td/></tr><tr><td colspan="4">SBOW Freq + SLQS Sub SBOW Freq + S SBOW PPMI + S DIVE + S</td></tr><tr><td>64.4</td><td>66.8</td><td>66.8</td><td>67.0</td></tr></table>

Dataset sizes. N denotes the number of word pairs in the dataset, and OOV shows how many word pairs are not processed by all the methods in Table4 and Table 5.

Table content: <table><tr><td colspan="2">BLESS</td><td colspan="2">EVALution</td><td colspan="2">Lenci/Benotto</td><td colspan="2">Weeds</td><td colspan="2">Avg (4 datasets)</td></tr><tr><td>N</td><td>OOV</td><td>N</td><td>OOV</td><td>N</td><td>OOV</td><td>N</td><td>OOV</td><td>N</td><td>OOV</td></tr><tr><td colspan="3">26554 1507 13675</td><td>2475</td><td colspan="2">5010 1464</td><td>2928</td><td>643</td><td>48167</td><td>6089</td></tr><tr><td colspan="2">Medical</td><td colspan="2">LEDS</td><td colspan="2">TM14</td><td colspan="2">Kotlerman 2010</td><td colspan="2">HyperNet</td></tr><tr><td>N</td><td>OOV</td><td>N</td><td>OOV</td><td>N</td><td>OOV</td><td>N</td><td>OOV</td><td>N</td><td>OOV</td></tr><tr><td colspan="2">12602 3711</td><td>2770</td><td>28</td><td>2188</td><td>178</td><td>2940</td><td>89</td><td>17670</td><td>9424</td></tr><tr><td colspan="2">WordNet</td><td colspan="2">Avg (10 datasets)</td><td colspan="2">HyperLex</td><td/><td/><td/><td/></tr><tr><td>N</td><td>OOV</td><td>N</td><td>OOV</td><td>N</td><td>OOV</td><td/><td/><td/><td/></tr><tr><td>8000</td><td colspan="2">3596 94337</td><td>24110</td><td>2616</td><td>59</td><td/><td/><td/><td/></tr><tr><td colspan="2">hypernym relations in</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

The direction is classified correctly if the generality score is greater than 0 (hypernym is indeed predicted as the more general word). For instance, summation difference (∆S) classifies correctly if |w p

Table content: <table/>

Peer review (iclr2018):
 This work proposes DIVE (distributional inclusion vector embeddings), an unsupervised method for hypernymy discovery that preserves the inclusion property of old fashioned sparse BOW representations (i.e., that hypernyms tend to have higher counts in more varies contexts than hyponyms). The learned representations are evaluated on a large set of datasets and are shown to work well. The work is very thorough, and almost reads as a systematic study in places. Unfortunately, this makes it hard to follow sometimes, and although the work is generally interesting, I am left to wonder about its novelty and general impact on the field. The main proposition essentially comes down to a slight tweak in word2vec*s SGNS method: instead of shifting the PMI value of word co-occurrences by 1/k, we shift by #w/(k*|D|/|V|) and add a non-negativity constraint, i.e., we explicitly force Eq. 2 to be true in perfect reconstructions. This makes sense and I am surprised nobody has done it before. It feels like a minor tweak, however, and any major improvement seems to be largely dependent on the scoring function, rather than on this slightly different objective. Consequently, I feel like the paper is in two minds about what it wants to be: a systematic study of hypernymy detection methods, or an introduction of a novel algorithm for learning distributional inclusion embeddings. I like the main ideas of this work and I think it*s great that the study is so thorough, but I don*t think it should be accepted in its current form. Main concerns: - Presentation: the experiments show that (1) DIVE outperforms GE, HyperScore and H-Feature baselines; (2) which scoring function works best for DIVE; (3) DIVE outperforms SBOW in many cases, but not always, though better on average; and (4) we can use DIVE for WSD (only shown qualitatively). The paper spans 13 pages, excluding appendices, which is rather long. I feel that (2), (3) and (4) are part of a systematic study/review paper, while (1) could be interesting in and of itself, if it included a full comparison against alternative methods. The way results are presented in the tables is confusing, and it*s unclear why only one baseline is included in each case. - Comparison: it appears that there are methods missing from the results tables. On HyperLex, for example, results as high as 0.512 (Poincare embeddings), 0.540 (HyperVec) and 0.686 (LEAR) have been reported, while the highest result in the paper is 34.5. That*s a big difference. On Weeds* BLESS, it*s 68.6 versus e.g. 0.75 for Kiela et al. (unsupervised, using images) and 0.850 for HyperVec. If these results were omitted due to space, I think experiment (2) and (4) can safely be moved to the appendix. Especially for something that hinges on being a review paper, such as this work, it is important to be complete. In short, I think the presentation doesn*t help; I am left to wonder what the main contribution is of the work; and I think the comparison to previous work is incomplete. Questions: - Why use WaCkypedia? It*s old and, by now, small. It would be interesting to try all scoring functions and types of model (PPMI, PPMI with shift, PPMI with inclusion shift) trained on the exact same corpus and same negatives, and showing that it works best there. - General question: Is Vendrov*s test set only yes/no, e.g. if I set the threshold really low, I get 100% accuracy, or does it contain negatives? Same for BLESS. If so how valuable is this evaluation? - To what extent are the baselines tuned in the semi-supervised case? If the numbers are from papers, that should be mentioned. It would be better to use the same corpus, and the same amount of attention to tuning the results, for both cases. Minor/Typos: - Medicl - "From the recent review (Santus et al) ... suggested by the review study (Vulic et al)" this way of citing reviews isn*t very pretty