Venue: NAACL 2018

# Specialising Word Vectors for Lexical Entailment

Ivan Vulic\({}^{1}\)  and Nikola Mrksic\({}^{2}\)

\({}^{1}\) Language Technology Lab, University of Cambridge

\({}^{2}\) PolyAI

iv250@cam.ac.uk nikola@poly-ai.com

###### Abstract

We present lear (**L**exical **E**ntailment **A**ttract-**R**epel), a novel post-processing method that transforms any input word vector space to emphasise the asymmetric relation of _lexical entailment_ (LE), also known as the is-a or hyponymy-hypernymy relation. By injecting external linguistic constraints (e.g., WordNet links) into the initial vector space, the LE specialisation procedure brings true hyponymy-hypermy pairs closer together in the transformed Euclidean space. The proposed asymmetric distance measure adjusts the norms of word vectors to reflect the actual WordNet-style hierarchy of concepts. Simultaneously, a joint objective enforces semantic similarity using the symmetric cosine distance, yielding a vector space specialised for both lexical relations at once. lear specialisation achieves state-of-the-art performance in the tasks of hypernymy directionality, hypernymy detection, and graded lexical entailment, demonstrating the effectiveness and robustness of the proposed asymmetric specialisation model.

## 1 Introduction

Word representation learning has become a research area of central importance in NLP, with its usefulness demonstrated across application areas such as parsing Chen and Manning (2014), machine translation Zou et al. (2013), and many others Turian et al. (2010); Collobert et al. (2011). Standard techniques for inducing word embeddings rely on the _distributional hypothesis_Harris (1954), using co-occurrence information from large textual corpora to learn meaningful word representations Mikolov et al. (2013); Levy and Goldberg (2014); Pennington et al. (2014); Bojanowski et al. (2017).

A major drawback of the distributional hypothesis is that it coalesces different relationships between words, such as synonymy and topical relatedness, into a single vector space. A popular solution is to go beyond stand-alone unsupervised learning and fine-tune distributional vector spaces by using external knowledge from human- or automatically-constructed knowledge bases. This is often done as a _post-processing_ step, where distributional vectors are gradually refined to satisfy linguistic constraints extracted from lexical resources such as WordNet Faruqui et al. (2015); Mrksic et al. (2016), the Paraphrase Database (PPDB) Wieting et al. (2015), or BabelNet Mrksic et al. (2017); Vulic et al. (2017). One advantage of post-processing methods is that they treat the input vector space as a _black box_, making them applicable to any input space.

A key property of these methods is their ability to transform the vector space by _specialising_ it for a particular relationship between words.1 Prior work has predominantly focused on distinguishing between semantic similarity and conceptual relatedness Faruqui et al. (2015); Mrksic et al. (2017); Vulic et al. (2017). In this paper, we introduce a novel post-processing model which specialises vector spaces for the _lexical entailment_ (le) relation.

Footnote 1: Distinguishing between synonymy and antonymy has a positive impact on real-world language understanding tasks such as Dialogue State Tracking Mrksić et al. (2017).

Word-level lexical entailment is an _asymmetric_ semantic relation Collins and Quillian (1972); Beckwith et al. (1991). It is a key principle determining the organisation of semantic networks into hierarchical structures such as semantic ontologies Fellbaum (1998). Automatic reasoning about le supports tasks such as taxonomy creation Snow et al. (2006); Navigli et al. (2011), natural language inference Dagan et al. (2013); Bowman et al. (2015), text generation Biran and McKeown (2013), and metaphor detection Mohler et al. (2013).

Our novel le specialisation model, termed lear (**L**exical **E**ntailment **A**ttract-**R**epel), is inspired by Attract-Repel, a state-of-the-art general spe

cialisation framework (Mrksic et al., 2017).2 The key idea of lear, illustrated by Figure 1, is to pull desirable (attract) examples described by the constraints closer together, while at the same time pushing undesirable (repel) word pairs away from each other. Concurrently, lear (re-)arranges vector norms so that norm values in the Euclidean space reflect the hierarchical organisation of concepts according to the given le constraints: put simply, higher-level concepts are assigned larger norms. Therefore, lear simultaneously captures the hierarchy of concepts (through vector norms) and their similarity (through their cosine distance). The two pivotal pieces of information are combined into an _asymmetric distance measure_ which quantifies the LE strength in the specialised space.

Footnote 2: [https://github.com/nmrksic/attract-repel](https://github.com/nmrksic/attract-repel)

After specialising four well-known input vector spaces with lear, we test them in three standard word-level le tasks (Kiela et al., 2015b): **1)** hypernymy _directionality_; **2)** hypernymy _detection_; and **3)**_combined_ hypernymy detection/directionality. Our specialised vectors yield notable improvements over the strongest baselines for each task, with each input space, demonstrating the effectiveness and robustness of lear specialisation.

The employed asymmetric distance allows one to make graded assertions about hierarchical relationships between concepts in the specialised space. This property is evaluated using HyperLex, a recent _graded LE_ dataset (Vulic et al., 2017). The lear-specialised vectors push state-of-the-art Spearman's correlation from 0.540 to 0.686 on the full dataset (2,616 word pairs), and from 0.512 to 0.705 on its noun subset (2,163 word pairs).

The code for the lear model is available from: github.com/nmrksic/lear.

## 2 Methodology

### The Attract-Repel Framework

Let \(V\) be the vocabulary, \(A\) the set of Attract word pairs (e.g., _intelligent_ and _brilliant_), and \(R\) the set of Repel word pairs (e.g., _vacant_ and _occupied_). The Attract-Repel procedure operates over mini-batches of such pairs \({\cal B}_{A}\) and \({\cal B}_{R}\). For ease of notation, let each word pair \((x_{l},x_{r})\) in these two sets correspond to a vector pair \(({\bf x}_{l},{\bf x}_{r})\), so that a mini-batch of \(k_{1}\) word pairs is given by \({\cal B}_{A}=[({\bf x}^{1}_{l},{\bf x}^{1}_{r}),\ldots,({\bf x}^{k_{1}}_{l},{ \bf x}^{k_{1}}_{r})]\) (similarly for \({\cal B}_{R}\), which consists of \(k_{2}\) example pairs).

Next, the sets of pseudo-negative examples \(T_{A}=[({\bf t}^{1}_{l},{\bf t}^{1}_{r}),\ldots,({\bf t}^{k_{1}}_{l},{\bf t}^{ k_{1}}_{r})]\) and \(T_{R}=[({\bf t}^{1}_{l},{\bf t}^{1}_{r}),\ldots,({\bf t}^{k_{2}}_{l},{\bf t}^{ k_{2}}_{r})]\) are defined as pairs of _negative examples_ for each Attract and Repel example pair in mini-batches \({\cal B}_{A}\) and \({\cal B}_{R}\). These negative examples are chosen from the word vectors present in \({\cal B}_{A}\) or \({\cal B}_{R}\) so that, for each Attract pair \(({\bf x}_{l},{\bf x}_{r})\), the negative example pair \(({\bf t}_{l},{\bf t}_{r})\) is chosen so that \({\bf t}_{l}\) is the vector closest (in terms of cosine distance) to \({\bf x}_{l}\) and \({\bf t}_{r}\) is closest to \({\bf x}_{r}\). Similarly, for each Repeat pair \(({\bf x}_{l},{\bf x}_{r})\), the negative example pair \(({\bf t}_{l},{\bf t}_{r})\) is chosen from the remaining in-batch vectors so that \({\bf t}_{l}\) is the vector furthest away from \({\bf x}_{l}\) and \({\bf t}_{r}\) is furthest from \({\bf x}_{r}\).

The negative examples are used to: **a)** force Attract pairs to be closer to each other than to their respective negative examples; and **b)** to force Repel pairs to be further away from each other than from their negative examples. The first term of the cost function pulls Attract pairs together:

\[Att({\cal B}_{A},T_{A})~{}=\\ \sum_{i=1}^{k_{1}}\bigm{[}\tau\left(\delta_{att}+cos({\bf x}^{ i}_{l},{\bf t}^{i}_{l})-cos({\bf x}^{i}_{l},{\bf x}^{i}_{r})\right)\\ +\tau\left(\delta_{att}+cos({\bf x}^{i}_{r},{\bf t}^{i}_{r})- cos({\bf x}^{i}_{l},{\bf x}^{i}_{r})\right)\bigm{]} \tag{1}\]

Figure 1: An illustration of lear specialisation. leaf controls the arrangement of vectors in the transformed vector space by: **1)** emphasising symmetric similarity of le pairs through cosine distance (by enforcing small angles between \(\overline{terrier}^{t}\) and \(\overline{dlog}\) or \(\overline{dog}\) and \(\overline{animal}\)); and **2)** by imposing an le ordering using vector norms, adjusting them so that higher-level concepts have larger norms (e.g., \(|\overline{animal}>|\overline{dog}|>|\overline{terrier}|\)).

 

where \(cos\) denotes cosine similarity, \(\tau(x)=\max(0,x)\) is the hinge loss function and \(\delta_{att}\) is the attract margin which determines how much closer these vectors should be to each other than to their respective negative examples. The second part of the cost function pushes Repel word pairs away from each other:

\[Rep(\mathcal{B}_{R},T_{R})~{}=\\ \sum_{i=1}^{k_{2}}\big{[}\;\tau\left(\delta_{rep}+cos(\mathbf{x}_{ i}^{i},\mathbf{x}_{r}^{i})-cos(\mathbf{x}_{t}^{i},\mathbf{t}_{l}^{i})\right)\] \[\quad\quad+\!\tau\left(\delta_{rep}+cos(\mathbf{x}_{l}^{i}, \mathbf{x}_{r}^{i})-cos(\mathbf{x}_{r}^{i},\mathbf{t}_{r}^{i})\right)\big{]} \tag{2}\]

In addition to these two terms, an additional regularisation term is used to _preserve_ the abundance of high-quality semantic content present in the distributional vector space, as long as this information does not contradict the injected linguistic constraints. If \(V(\mathcal{B})\) is the set of all word vectors present in the given mini-batch, then:

\[Reg(\mathcal{B}_{A},\mathcal{B}_{R})=\sum_{\mathbf{x}_{i}\in V(\mathcal{B}_{ A}\cup\mathcal{B}_{R})}\lambda_{reg}\left\|\widehat{\mathbf{x}_{i}}-\mathbf{ x}_{i}\right\|_{2}\]

where \(\lambda_{reg}\) is the L2 regularization constant and \(\widehat{\mathbf{x}_{i}}\) denotes the original (distributional) word vector for word \(x_{i}\). The full Attract-Repel cost function is given by the sum of all three terms.

### LEAR: Encoding Lexical Entailment

In this section, the Attract-Repel framework is extended to model lexical entailment jointly with (symmetric) semantic similarity. To do this, the method uses an additional source of external lexical knowledge: let \(L\) be the set of _directed_ lexical entailment constraints such as _(corgi, dog)_, _(dog, animal)_, or _(corgi, animal)_, with lower-level concepts on the left and higher-level ones on the right (the source of these constraints will be discussed in Section 3). The optimisation proceeds in the same way as before, considering a mini-batch of le pairs \(\mathcal{B}_{L}\) consisting of \(k_{3}\) word pairs standing in the (directed) lexical entailment relation.

Unlike symmetric similarity, lexical entailment is an asymmetric relation which encodes a hierarchical ordering between concepts. Inferring the direction of the entailment relation between word vectors requires the use of an asymmetric distance function. We define three different ones, all of which use the word vector's norms to impose an ordering between high- and low-level concepts:

\[D_{1}(\mathbf{x},\mathbf{y}) = |\mathbf{x}|-|\mathbf{y}| \tag{3}\] \[D_{2}(\mathbf{x},\mathbf{y}) = \frac{|\mathbf{x}|-|\mathbf{y}|}{|\mathbf{x}|+|\mathbf{y}|}\] (4) \[D_{3}(\mathbf{x},\mathbf{y}) = \frac{|\mathbf{x}|-|\mathbf{y}|}{\max(|\mathbf{x}|,|\mathbf{y}|)} \tag{5}\]

The lexical entailment term (for the \(j\)-th asymmetric distance, \(j\in 1,2,3\)) is defined as:

\[LE_{j}(\mathcal{B}_{L})~{}=\sum_{i=1}^{k_{3}}D_{j}(\mathbf{x}_{i},\mathbf{y}_{ i}) \tag{6}\]

The first distance serves as the baseline: it uses the word vectors' norms to order the concepts, that is to decide which of the words is likely to be the higher-level concept. In this case, the magnitude of the difference between the two norms determines the 'intensity' of the le relation. This is potentially problematic, as this distance does not impose a limit on the vectors' norms. The second and third metric take a more sophisticated approach, using the ratios of the differences between the two norms and either: **a)** the sum of the two norms; or **b)** the larger of the two norms. In doing that, these metrics ensure that the cost function only considers the norms' ratios. This means that the cost function no longer has the incentive to increase word vectors' norms past a certain point, as the magnitudes of norm ratios grow in size much faster than the linear relation defined by the first distance function.

To model the semantic and the le relations jointly, the lear cost function jointly optimises the four terms of the expanded cost function:

\[C(\mathcal{B}_{A},T_{A},\mathcal{B}_{R},T_{R},\mathcal{B}_{L}, T_{L})~{}=~{}Att(\mathcal{B}_{S},T_{S})+\ldots\] \[~{}+~{}Rep(\mathcal{B}_{A},T_{A})~{}+~{}Reg(\mathcal{B}_{A}, \mathcal{B}_{R},\mathcal{B}_{L})~{}+~{}\ldots\] \[~{}+~{}Att(\mathcal{B}_{L},T_{L})~{}+~{}LE_{j}(B_{L})\]

LE Pairs as Attract ConstraintsThe combined cost function makes use of the batch of lexical constraints \(\mathcal{B}_{L}\) twice: once in the defined asymmetric cost function \(LE_{j}\), and once in the symmetric Attract term \(Att(\mathcal{B}_{L},T_{L})\). This means that words standing in the lexical entailment relation are forced to be similar both in terms of cosine distance (via the symmetric Attract term) and in terms of the asymmetric \(LE\) distance from Eq. (6).

Decoding Lexical EntailmentThe defined cost function serves to encode semantic similarity and 

le relations in the same vector space. Whereas the similarity can be inferred from the standard cosine distance, the lear optimisation embeds lexical entailment as a combination of the symmetric Attract term and the newly defined asymmetric \(LE_{j}\) cost function. Consequently, the metric used to determine whether two words stand in the le relation must combine the two cost terms as well. We define the le_decoding_ metric as:

Footnote 3: All vectors are 300-dimensional except for the 600-dimensional context2vec vectors; for further details regarding the architectures and training setup of the used vector collections, we refer the reader to the original papers. We also experimented with dependency-based SGNS vectors Levy and Goldberg (2014), observing similar patterns in the results.

\[I_{LE}(\mathbf{x},\mathbf{y})=dcos(\mathbf{x},\mathbf{y})+D_{j}(\mathbf{x}, \mathbf{y}) \tag{7}\]

where \(dcos(\mathbf{x},\mathbf{y})\) denotes the cosine distance. This decoding function combines the symmetric and the asymmetric cost term, in line with the combination of the two used to perform leaf specialisation. In the evaluation, we show that combining the two cost terms has a synergistic effect, with both terms contributing to stronger performance across all le tasks used for evaluation.

## 3 Experimental Setup

Starting Distributional VectorsTo test the robustness of lear specialisation, we experiment with a variety of well-known, publicly available English word vectors: **1)** Skip-Gram with Negative Sampling (SGNS) Mikolov et al. (2013) trained on the Polyglot Wikipedia Al-Rfou et al. (2013) by Levy and Goldberg (2014); **2)**Glove Common Crawl Pennington et al. (2014); **3)**context2vecMelamud et al. (2016), which replaces CBOW contexts with contexts based on bidirectional LSTMs Hochreiter and Schmidhuber (1997); and **4)**fastTextBojanowski et al. (2017), a SGNS variant which builds word vectors as the sum of their constituent character n-gram vectors.4

Footnote 4: footnotemark:

Linguistic ConstraintsWe use three groups of linguistic constraints in the learn specialisation model, covering three different relation types which are all beneficial to the specialisation process: directed **1)**_lexical entailment_ (le)_pairs_; **2)**_synonymy pairs_; and **3)**_antonymy pairs_. Synonyms are included as symmetric attract pairs (i.e., the \(\mathcal{B}_{A}\) pairs) since they can be seen as defining a trivial symmetric is-a relation Rei and Briscoe (2014); Vulic et al. (2017). For a similar reason, antonyms are clear repel constraints as they anticorrelate with the LE relation.5 Synonymy and antonymy constraints are taken from prior work Zhang et al. (2014); Ono et al. (2015): they are extracted from WordNet Fellbaum (1998) and Roget Kipfer (2009). In total, we work with 1,023,082 synonymy pairs (11.7 synonyms per word on average) and 380,873 autonomy pairs (6.5 per word).6

Footnote 5: [https://github.com/tricoin/AntonymDetection](https://github.com/tricoin/AntonymDetection)

As in prior work Nguyen et al. (2017); Nickel and Kiela (2017), le constraints are extracted from the WordNet hierarchy, relying on the transitivity of the le relation. This means that we include both direct and indirect le pairs in our set of constraints (e.g., (_pangasius, fish_), (_fish, animal_), and (_pangasius, animal_)). We retained only noun-noun and verb-verb pairs, while the rest were discarded: the final number of le constraints is 1,545,630.7

Footnote 6: We also experimented with additional 30,491 le constraints from the Paraphrase Database (PPDB) 2.0 Pavlick et al. (2015). Adding them to the WordNet-based le pairs makes no significant impact on the final performance. We also used synonymy and antonymy pairs from other sources, such as word pairs from PPDB used previously by Wieting et al. (2015), and BabelNet Navigli and Ponzetto (2012) used by Mrksic et al. (2017), reaching the same conclusions.

Training SetupWe adopt the original AttractRepel model setup without any fine-tuning. Hyperparameter values are set to: \(\delta_{att}=0.6\), \(\delta_{rep}=0.0\), \(\lambda_{reg}=10^{-9}\)Mrksic et al. (2017). The models are trained for 5 epochs with the AdaGrad algorithm Duchi et al. (2011), with batch sizes set to \(k_{1}=k_{2}=k_{3}=128\) for faster convergence.

## 4 Results and Discussion

We test and analyse lear-specialised vector spaces in two standard word-level le tasks used in prior work: hypernymy directionality and detection (Section 4.1) and graded le (Section 4.2).

### LE Directionality and Detection

The first evaluation uses three classification-style tasks with increased levels of difficulty. The tasks are evaluated on three datasets used extensively in the le literature Roller et al. (2014); Santus et al. (2014); Woods et al. (2014); Shwartz et al. (2017); Nguyen et al. (2017), compiled into an integrated evaluation set kiela et al. (2015b).7 

The first task, le directionality, is conducted on 1,337 le pairs originating from the bless evaluation set Baroni and Lenci (2011). Given a true le pair, the task is to predict the correct hypernym. With lear-specialised vectors this is achieved by simply comparing the vector norms of each concept in a pair: the one with the larger norm is the hypernym (see Figure 1).

The second task, le detection, involves a binary classification on the wbless dataset Weeds et al. (2014) which comprises 1,668 word pairs standing in a variety of relations (le, meronymy-holonymy, co-hyponymy, reversed le, no relation). The model has to detect a true le pair, that is, to distinguish between the pairs where the statement _\(X\) is a (type of)_\(Y\) is true from all other pairs. With lear vectors, this classification is based on the asymmetric distance score: if the score is above a certain threshold, we classify the pair as "true le", otherwise as "other". While Kiela et al. (2015) manually define the threshold value, we follow the approach of Nguyen et al. (2017) and cross-validate: in each of the 1,000 iterations, 2% of the pairs are sampled for threshold tuning, and the remaining 98% are used for testing. The reported numbers are therefore average accuracy scores.8

Footnote 8: We have conducted more le directionality and detection experiments on other datasets such as EVALution Santus et al. (2015), the \(N_{1}\vDash N_{2}\) dataset of Baroni et al. (2012), and the dataset of Lenci and Benotto (2012) with similar performances and findings. We do not report all these results for brevity and clarity of presentation.

The final task, le detection _and_ directionality, concerns a three-way classification on bibless, a relabeled version of whless. The task is now to distinguish both le pairs (\(\to 1\)) and reversed le pairs (\(\rightarrow-1\)) from other relations (\(\to 0\)), and then additionally select the correct hypernym in each detected le pair. We apply the same test protocol as in the le detection task.

Results and AnalysisThe original paper of Kiela et al. (2015) reports the following best scores on each task: 0.88 (bless), 0.75 (whless), 0.57 (bibless). These scores were recently surpassed by Nguyen et al. (2017), who, instead of post-processing, combine WordNet-based constraints with an SGNS-style objective into a joint model. They report the best scores to date: 0.92 (bless), 0.87 (wbless), and 0.81 (bibless).

The performance of the four leaf-specialised word vector collections is shown in Figure 2 (together with the strongest baseline scores for each of the three tasks). The comparative analysis confirms the increased complexity of subsequent tasks. lear specialisation of _each_ of the starting vector spaces consistently outperformed _all_ baseline scores across _all_ three tasks. The extent of the improvements is correlated with task difficulty: it is lowest for the easiest directionality task (\(0.92\to 0.96\)), and highest for the most difficult detection plus directionality task (\(0.81\to 0.88\)).

The results show that the two lear variants which do not rely on absolute norm values and

Figure 2: Summary of the results on three different word-level LE subtasks: (a) _directionality_; (b) _detection_; (c) _detection and directionality_. Vertical bars denote the results obtained by different input word vector spaces which are post-processed/specialised by our learn specialisation model using three variants of the asymmetric distance (\(D_{1}\), \(D_{2}\), \(D_{3}\)), see Section 2. Thick horizontal red lines refer to the best reported scores on each subtask for these datasets; the baseline scores are taken from Nguyen et al. (2017).

 

perform a normalisation step in the asymmetric distance (D2 and D3) have an edge over the D1 variant which operates with unbounded norms. The difference in performance between D2/D3 and D1 is even more pronounced in the graded LE task (see Section 4.2). This shows that the use of unbounded vector norms diminishes the importance of the symmetric cosine distance in the combined asymmetric distance. Conversely, the synergistic combination used in D2/D3 does not suffer from this issue.

The high scores achieved with each of the four word vector collections show that lear is not dependent on any particular word representation architecture. Moreover, the extent of the performance improvements in each task suggests that learn is able to reconstruct the concept hierarchy coded in the input linguistic constraints.

Moreover, we have conducted a small experiment to verify that the lear method can generalise beyond what is directly coded in pairwise external constraints. A simple WordNet lookup baseline yields accuracy scores of 0.82 and 0.80 on the directionality and detection tasks, respectively. This baseline is outperformed by Lear: its scores are 0.96 and 0.92 on the two tasks when relying on the same set of WordNet constraints.

Importance of Vector NormsTo verify that the knowledge concerning the position in the semantic hierarchy actually arises from vector norms, we also manually inspect the norms after leaf specialisation. A few examples are provided in Table 1. They indicate a desirable pattern in the norm values which imposes a hierarchical ordering on the concepts. Note that the original distributional SGNS model Mikolov et al. (2013) does not normalise vectors to unit length after training. However, these norms are not at all correlated with the desired hierarchical ordering, and are therefore useless for le-related applications: the non-specialised distributional SGNS model scores 0.44, 0.48, and 0.34 on the three tasks, respectively.

### Graded Lexical Entailment

Asymmetric distances in the lear-specialised space quantify the degree of lexical entailment between any two concepts. This means that they can be used to make fine-grained assertions regarding the hierarchical relationships between concepts. We test this property on HyperLex Vulic et al. (2017), a gold standard dataset for evaluating how well word representation models capture graded le, grounded in the notions of _concept (proto)typicality_Rosch (1973); Medin et al. (1984) and _category vagueness_Kamp and Partee (1995); Hampton (2007) from cognitive science. HyperLex contains 2,616 word pairs (2,163 noun pairs and 453 verb pairs) scored by human raters in the \([0,6]\) interval following the question _"To what degree is X a (type of) Y?"9_

Footnote 9: From another perspective, one might say that graded le provides finer-grained human judgements on a continuous scale rather than simplifying the judgements into binary discrete decisions. For instance, the HyperLex score for the pair _(girl, person)_ is 5.91/6, the score for _(guest, person)_ is 4.33, while the score for the reversed pair _(person, guest)_ is 1.73.

As shown by the high inter-annotator agreement on HyperLex (0.85), humans are able to consistently reason about graded le.10 However, current state-of-the-art representation architectures are far from this ceiling. For instance, Vulic et al. (2017) evaluate a plethora of architectures and report a high-score of only 0.320 (see the summary table in Figure 3). Two recent representation models Nickel and Kiela (2017); Nguyen et al. (2017) focused on the LE relation in particular (and employing the same set of WordNet-based constraints as lear) report the highest score of 0.540 (on the entire dataset) and 0.512 (on the noun subset).

Footnote 10: For further details concerning HyperLex, we refer the reader to the resource paper Vulic et al. (2017). The dataset is available at: [http://people.ds.cam.ac.uk/iv250/hyperlex.html](http://people.ds.cam.ac.uk/iv250/hyperlex.html)

Results and AnalysisWe scored all HyperLex pairs using the combined asymmetric distance described by Equation (7), and then computed Spearman's rank correlation with the ground-truth ranking. Our results, together with the strongest base-line scores, are summarised in Figure 3.

The summary table in Figure 3(c) shows the HyperLex performance of several prominent le models. We provide only a quick outline of these models here; further details can be found in the original papers. freq-ratio exploits the fact that more general concepts tend to occur more frequently in textual corpora. gns (cos) uses non-specialised

\begin{table}
\begin{tabular}{l c l r r r} \hline \hline  & **Norm** & \multicolumn{2}{r}{**Norm**} & \multicolumn{1}{c}{**Norm**} \\ \cline{2-5} terrier & 0.87 & laptop & 0.60 & cabriddet & 0.74 \\ dog & 2.64 & computer & 2.96 & car & 3.59 \\ mammal & 8.57 & machine & 6.15 & vehicle & 7.78 \\ vertebrate & 10.96 & device & 12.09 & transport & 8.01 \\ animal & 11.91 & artifact & 17.71 & instrumentality & 14.56 \\ organism & 20.08 & object & 23.55 & – & — \\ \hline \hline \end{tabular}
\end{table}
Table 1: L2 norms for selected concepts from the WordNet hierarchy. Input: fastText; Lear: D2.

 

SGNS vectors and quantifies the LE strength using the symmetric cosine distance between vectors. A comparison of these models to the best-performing lear vectors shows the extent of the improvements achieved using the specialisation approach.

lear-specialised vectors also outperform slQSsim(Santus et al., 2014) and visualKiela et al. (2015b), two le detection models similar in spirit to lear. These models combine symmetric semantic similarity (through cosine distance) with an asymmetric measure of lexical generality obtained either from text (slqs-sim) or visual data (visual). The results on HyperLex indicate that the two generality-based measures are too coarse-grained for graded le judgements. These models were originally constructed to tackle le directionality and detection tasks (see Section 4.1), but their performance is surpassed by leaf on those tasks as well. The visual model outperforms slqs-sim. However, its numbers on bless (0.88), wbless (0.75), and bibless (0.57) are far from the top-performing lear vectors (0.96, 0.92, 0.88).11

Footnote 11: We note that slqs and visual do not leverage any external knowledge from WordNet, but the visual model leverages external visual information about concepts.

wn-best denotes the best result with asymmetric similarity measures which use the WordNet structure as their starting point Wu and Palmer (1994); Pedersen et al. (2004). This model can be observed as a model that directly looks up the full WordNet structure to reason about graded lexical entailment. The reported results from Figure 3 suggest it is more effective to quantify the LE relation strength by using WordNet as the source of constraints for specialisation models such as HyperVec or Lear.

word2gaussVilnis and McCallum (2015) represents words as multivariate \(K\)-dimensional Gaussians rather than points in the embedding space: it is therefore naturally asymmetric and was used in le tasks before, but its performance on HyperLex indicates that it cannot effectively capture the subtleties required to model graded le. However, note that the comparison is not strictly fair as word2Gauss does not leverage any external knowledge. An interesting line for future research is to embed external knowledge within this representation framework.

Most importantly, lear outperforms three recent (and conceptually different) architectures: order-embVendrov et al. (2016), PoincareNickel and Kiela (2017), and HyperVec Nguyen et al. (2017). Like lear, all of these models complement distributional knowledge with external linguistic constraints extracted from WordNet. Each model uses a different strategy to exploit the hierarchical relationships encoded in these constraints (their approaches are discussed in Section 5).12However, lear, as the first le-oriented post-processor, is able to utilise the constraints more effectively than its competitors. Another advantage of lear is its applicability to any input

Figure 3: Results on the graded LE task defined by HyperLex. Following Nickel and Kiela (2017), we use Spearman’s rank correlation scores on: **a)** the entire dataset (2,616 noun and verb pairs); and **b)** its noun subset (2,163 pairs). The summary table shows the performance of other well-known architectures on the full HyperLex dataset, compared to the best results achieved using lear specialisation.

 

vector space.

Figures 3(a) and 3(b) indicate that the two lear variants which rely on norm ratios (D2 and D3), rather than on absolute (unbounded) norm differences (D1), achieve stronger performance on HyperLex. The highest correlation scores are again achieved by D2 with all input vector spaces.

### Further Discussion

Why Symmetric + Asymmetric?In another experiment, we analyse the contributions of both lerelated terms in the learn combined objective function (see Section 2.2). We compare three variants of lear: **1)** a symmetric variant which does not arrange vector norms using the \(LE_{j}(\mathcal{B}_{L})\) term (symonly); **2)** a variant which arranges norms, but does not use le constraints as additional symmetric attract constraints (asym-only); and **3)** the full lear model, which uses both cost terms (full). The results with one input space (similar results are achieved with others) are shown in Table 2. This table shows that, while the stand-alone asym-only term seems more beneficial than the sym-only one, using the two terms jointly yields the strongest performance across all LE tasks.

Le and Semantic SimilarityWe also test whether the asymmetric \(LE\) term harms the (norm-independent) cosine distances used to represent semantic similarity. The lear model is compared to the original attract-repel model making use of the same set of linguistic constraints. Two true semantic similarity datasets are used for evaluation: SimLex-999 Hill et al. (2015) and SimVerb-3500 Gerz et al. (2016). There is no significant difference in performance between the two models, both of which yield similar results on SimLex (Spearman's rank correlation of \(\approx\) 0.71) and SimVerb (\(\approx\) 0.70). This proves that cosine distances remain preserved during the optimization of the asymmetric objective performed by the joint lear model.

## 5 Related Work

Vector Space SpecialisationA standard approach to incorporating external information into vector spaces is to pull the representations of similar words closer together. Some models integrate such constraints into the training procedure: they modify the prior or the regularisation Yu and Dredze (2014); Xu et al. (2014); Bian et al. (2014); Kiela et al. (2015), or use a variant of the SGNS-style objective Liu et al. (2015); Osborne et al. (2016); Nguyen et al. (2017). Another class of models, popularly termed _retrofitting_, fine-tune distributional vector spaces by injecting lexical knowledge from semantic databases such as WordNet or the Paraphrase Database Faruqui et al. (2015); Jauhar et al. (2015); Wieting et al. (2015); Nguyen et al. (2016); Mrksic et al. (2016); Mrksic et al. (2017).

lear falls into the latter category. However, while previous post-processing methods have focused almost exclusively on specialising vector spaces to emphasise semantic similarity (i.e., to distinguish between similarity and relatedness by explicitly pulling synonyms closer and pushing antonyms further apart), this paper proposed a principled methodology for specialising vector spaces for asymmetric hierarchical relations (of which _lexical entailment_ is an instance). Its starting point is the state-of-the-art similarity specialisation framework of Mrksic et al. (2017), which we extend to support the inclusion of hierarchical asymmetric relationships between words.

Word Vectors and Lexical EntailmentSince the hierarchical le relation is one of the fundamental building blocks of semantic taxonomies and hierarchical concept categorisations Beckwith et al. (1991); Fellbaum (1998), a significant amount of research in semantics has been invested into its automatic detection and classification. Early work relied on asymmetric directional measures Weeds et al. (2004); Clarke (2009); Kotterman et al. (2010); Lenci and Benotto (2012), i.a.) which were based on the distributional inclusion hypothesis Geffet and Dagan (2005) or the distributional informativeness or generality hypothesis Herbelot and Ganesalingam (2013); Santus et al. (2014). However, these approaches have recently been superseded by methods based on word embeddings. These methods build dense real-valued vectors for capturing the LE relation, either directly in the LE-focused space Vilnis and McCallum (2015); Vendrov et al. (2016),

\begin{table}
\begin{tabular}{l l l l l} \hline \hline  & wless & bibless & hil-a & hil-n \\ \cline{2-5}
**lear variant** & & & & \\ sym-only & 0.687 & 0.679 & 0.469 & 0.429 \\ asym-only & 0.867 & 0.824 & 0.529 & 0.565 \\ full & **0.912** & **0.875** & **0.686** & **0.705** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Analysing the importance of the synergy in the full Lear model on the final performance on whpless, bless, HyperLex-All (hl-a) and HyperLex-Nouns (hl-n). Input: fastText. D2.

 

2016; Henderson and Popa, 2016; Nickel and Kiela, 2017; Nguyen et al., 2017) or by using the vectors as features for supervised LE detection models (Tuan et al., 2016; Shwartz et al., 2016; Nguyen et al., 2017; Glavas and Ponzetto, 2017).

Several le models embed useful hierarchical relations from external resources such as WordNet into le-focused vector spaces, with solutions coming in different flavours. The model of Yu et al. (2015) is a dynamic distance-margin model optimised for the le detection task using hierarchical WordNet constraints. This model was extended by Tuan et al. (2016) to make use of contextual sentential information. A major drawback of both models is their inability to make directionality judgements. Further, their performance has recently been surpassed by the HyperVec model of Nguyen et al. (2017). This model combines WordNet constraints with the SGNS distributional objective into a joint model. As such, the model is tied to the SGNS objective and any change of the distributional modelling paradigm implies a change of the entire HyperVec model. This makes their model less versatile than the proposed lear framework. Moreover, the results achieved using learn specialisation achieve substantially better performance across all le tasks used for evaluation.

Another model similar in spirit to lear is the order-emb model of Vendrov et al. (2016), which encodes hierarchical structure by imposing a partial order in the embedding space: higher-level concepts get assigned higher per-coordinate values in a \(d\)-dimensional vector space. The model minimises the violation of the per-coordinate orderings during training by relying on hierarchical WordNet constraints between word pairs. Finally, the Poincare model of Nickel and Kiela (2017) makes use of hyperbolic spaces to learn general-purpose LE embeddings based on \(n\)-dimensional Poincare balls which encode both hierarchy and semantic similarity, again using the WordNet constraints. A similar model in hyperbolic spaces was proposed by Chamberlain et al. (2017). In this paper, we demonstrate that le-specialised word embeddings with stronger performance can be induced using a simpler model operating in more intuitively interpretable Euclidean vector spaces.

## 6 Conclusion and Future Work

This paper proposed lear, a vector space specialisation procedure which simultaneously injects symmetric and asymmetric constraints into existing vector spaces, performing joint specialisation for two properties: _lexical entailment_ and _semantic similarity_. Since the former is not symmetric, Lear uses an asymmetric cost function which encodes the hierarchy between concepts by manipulating the norms of word vectors, assigning higher norms to higher-level concepts. Specialising the vector space for both relations has a synergistic effect: leaf-specialised vectors attain state-of-the-art performance in judging semantic similarity and set new high scores across four different lexical entailment tasks. The code for the lear model is available from: github.com/nmrksic/lear.

In future work, we plan to apply a similar methodology to other asymmetric relations (e.g., _meronymy_), as well as to investigate fine-grained models which can account for differing path lengths from the WordNet hierarchy. We will also extend the model to reason over words unseen in input lexical resources, similar to the recent post-specialisation model oriented towards specialisation of unseen words for similarity (Vulic et al., 2018). We also plan to test the usefulness of LEG specialised vectors in downstream natural language understanding tasks. Porting the model to other languages and enabling cross-lingual applications such as cross-lingual lexical entailment (Upadhyay et al., 2018) is another future research direction.

## Acknowledgments

We thank the three anonymous reviewers for their insightful comments and suggestions. We are also grateful to the TakeLab research group at the University of Zagreb for offering support to computationally intensive experiments in our hour of need. This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909).

## References

* Al-Rfou et al. (2013) Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In _Proceedings of CoNLL_, pages 183-192.
* Baroni et al. (2012) Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In _Proceedings of EACL_, pages 23-32.
* Baroni and Lenci (2011) Marco Baroni and Alessandro Lenci. 2011. How we 

BLESSed distributional semantic evaluation. In _Proceedings of the GEMS 2011 Workshop_, pages 1-10.
* Beckwith et al. (1991) Richard Beckwith, Christiane Fellbaum, Derek Gross, and George A. Miller. 1991. WordNet: A lexical database organized on psycholinguistic principles. _Lexical acquisition: Exploiting on-line resources to build a lexicon_, pages 211-231.
* Bian et al. (2014) Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Knowledge-powered deep learning for word embedding. In _Proceedings of ECML-PKDD_, pages 132-148.
* Biran and McKeown (2013) Or Biran and Kathleen McKeown. 2013. Classifying taxonomic relations between pairs of Wikipedia articles. In _Proceedings of IJCNLP_, pages 788-794.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the ACL_, 5:135-146.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of EMNLP_, pages 632-642.
* Chamberlain et al. (2017) Benjamin Paul Chamberlain, James Clough, and Marc Peter Deisenroth. 2017. Neural embeddings of graphs in hyperbolic space. _CoRR_, abs/1705.10359.
* Chen and Manning (2014) Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of EMNLP_, pages 740-750.
* Clarke (2009) Daoud Clarke. 2009. Context-theoretic semantics for natural language: An overview. In _Proceedings of the Workshop on Geometrical Models of Natural Language Semantics (GEMS)_, pages 112-119.
* Collins and Quillian (1972) Allan M. Collins and Ross M. Quillian. 1972. Experiments on semantic memory and language comprehension. _Cognition in Learning and Memory_.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. _Synthesis Lectures on Human Language Technologies_, 6(4):1-220.
* Duchi et al. (2011) John C. Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _Journal of Machine Learning Research_, 12:2121-2159.
* Faruqui et al. (2015) Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In _Proceedings of NAACL-HLT_, pages 1606-1615.
* Fellbaum (1998) Christiane Fellbaum. 1998. _WordNet_.
* Geffet and Dagan (2005) Maayan Geffet and Ido Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In _Proceedings of ACL_, pages 107-114.
* Gerz et al. (2016) Daniela Gerz, Ivan Vulic, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. In _Proceedings of EMNLP_, pages 2173-2182.
* Glavas and Ponzetto (2017) Goran Glavas and Simone Paolo Ponzetto. 2017. Dual tensor model for detecting asymmetric lexiccosemantic relations. In _Proceedings of EMNLP_, pages 1758-1768.
* Hampton (2007) James A. Hampton. 2007. Typicality, graded membership, and vagueness. _Cognitive Science_, 31(3):355-384.
* Harris (1954) Zellig S. Harris. 1954. Distributional structure. _Word_, 10(23):146-162.
* Henderson and Popa (2016) James Henderson and Diana Popa. 2016. A vector space for distributional semantics for entailment. In _Proceedings of ACL_, pages 2052-2062.
* Herbelot and Ganesalingam (2013) Aurelie Herbelot and Mohan Ganesalingam. 2013. Measuring semantic content in distributional vectors. In _Proceedings of ACL_, pages 440-445.
* Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. _Computational Linguistics_, 41(4):665-695.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Computation_, 9(8):1735-1780.
* Jauhar et al. (2015) Sujay Kumar Jauhar, Chris Dyer, and Eduard H. Hovy. 2015. Ontologically grounded multi-sense representation learning for semantic vector space models. In _Proceedings of NAACL_, pages 683-693.
* Kamp and Partee (1995) Hans Kamp and Barbara Partee. 1995. Prototype theory and compositionality. _Cognition_, 57(2):129-191.
* Kiela et al. (2015a) Douwe Kiela, Felix Hill, and Stephen Clark. 2015a. Specializing word embeddings for similarity or relatedness. In _Proceedings of EMNLP_, pages 2044-2048.
* Kiela et al. (2015b) Douwe Kiela, Laura Rimell, Ivan Vulic, and Stephen Clark. 2015b. Exploiting image generality for lexical entailment detection. In _Proceedings of ACL_, pages 119-124.
* Kipfer (2009) Barbara Ann Kipfer. 2009. _Roget's 21st Century Thesaurus (3rd Edition)_. Philip Lief Group.
* Kotlerman et al. (2010) Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. _Natural Language Engineering_, 16(4):359-389 

* Lenci and Benotto (2012) Alessandro Lenci and Giulia Benotto. 2012. Identifying hypernyms in distributional semantic spaces. In _Proceedings of *SEM_, pages 75-79.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In _Proceedings of ACL_, pages 302-308.
* Liu et al. (2015) Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. 2015. Learning semantic word embeddings based on ordinal knowledge constraints. In _Proceedings of ACL_, pages 1501-1511.
* Medin et al. (1984) Douglas L. Medin, Mark W. Altom, and Timothy D. Murphy. 1984. Given versus induced category representations: Use of prototype and exemplar information in classification. _Journal of Experimental Psychology_, 10(3):333-352.
* Melamud et al. (2016) Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. Context2vec: Learning generic context embedding with bidirectional LSTM. In _Proceedings of CoNLL_, pages 51-61.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Proceedings of NIPS_, pages 3111-3119.
* Mohler et al. (2013) Michael Mohler, David Bracewell, Marc Tomlinson, and David Hinote. 2013. Semantic signatures for example-based linguistic metaphor detection. In _Proceedings of the First Workshop on Metaphor in NLP_, pages 27-35.
* Mrksic et al. (2017) Nikola Mrksic, Diarmuid O Seaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In _Proceedings of ACL_, pages 1777-1788.
* Mrksic et al. (2016) Nikola Mrksic, Diarmuid O Seaghdha, Blaise Thomson, Milica Gasic, Lina Maria Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In _Proceedings of NAACL-HLT_.
* Mrksic et al. (2017) Nikola Mrksic, Ivan Vulic, Diarmuid O Seaghdha, Ira Leviant, Roi Reichart, Milica Gasic, Anna Korhonen, and Steve Young. 2017. Semantic specialisation of distributional word vector spaces using monolingual and cross-lingual constraints. _Transactions of the ACL_, 5:309-324.
* Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. _Artificial Intelligence_, 193:217-250.
* Navigli et al. (2011) Roberto Navigli, Paola Velardi, and Stefano Faralli. 2011. A graph-based algorithm for inducing lexical taxonomies from scratch. In _Proceedings of IJCAI_, pages 1872-1877.
* Nguyen et al. (2017) Kim Anh Nguyen, Maximilian Koper, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Hierarchical embeddings for hypernymy detection and directionality. In _Proceedings of EMNLP_, pages 233-243.
* Nguyen et al. (2016) Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016. Integrating distributional lexical contrast into word embeddings for anticonsynorm distinction. In _Proceedings of ACL_, pages 454-459.
* Nickel and Kiela (2017) Maximilian Nickel and Douwe Kiela. 2017. Poincare embeddings for learning hierarchical representations. In _Proceedings of NIPS_.
* Ono et al. (2015) Masataka Ono, Makoto Miwa, and Yutaka Sasaki. 2015. Word embedding-based antonym detection using thesauri and distributional information. In _Proceedings of NAACL-HLT_, pages 984-989.
* Osborne et al. (2016) Dominique Osborne, Shashi Narayan, and Shay Cohen. 2016. Encoding prior knowledge with eigenword embeddings. _Transactions of the ACL_, 4:417-430.
* Pavlick et al. (2015) Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In _Proceedings of ACL_, pages 425-430.
* Measuring the relatedness of concepts. In _Proceedings of AAAI_, pages 1024-1025.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of EMNLP_, pages 1532-1543.
* Rei and Briscoe (2014) Marek Rei and Ted Briscoe. 2014. Looking for hyponyms in vector space. In _Proceedings of CoNLL_, pages 68-77.
* Roller et al. (2014) Stephen Roller, Katrin Erk, and Gemma Boleda. 2014. Inclusive yet selective: Supervised distributional hypernymy detection. In _Proceedings of COLING_, pages 1025-1036.
* Rosch (1973) Eleanor H. Rosch. 1973. Natural categories. _Cognitive Psychology_, 4(3):328-350.
* Santus et al. (2014) Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vector spaces with entropy. In _Proceedings of EACL_, pages 38-42.
* Santos et al. (2015) Enrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. EVALution 1.0: an evolving semantic dataset for training and evaluation of distributional semantic models. In _Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applications_, pages 64-69 

Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016. Improving hypernymy detection with an integrated path-based and distributional method. In _Proceedings of ACL_, pages 2389-2398.
* Shwartz et al. (2017) Vered Shwartz, Enrico Santus, and Dominik Schlechtweg. 2017. Hypernyms under siege: Linguistically-motivated artillery for hypernymy detection. In _Proceedings of EACL_, pages 65-75.
* Snow et al. (2006) Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In _Proceedings of ACL_, pages 801-808.
* Tuan et al. (2016) Luu Anh Tuan, Yi Tay, Siu Cheung Hui, and See Kiong Ng. 2016. Learning term embeddings for taxonomic relation identification using dynamic weighting neural network. In _Proceedings of EMNLP_, pages 403-413.
* Turian et al. (2010) Joseph P. Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In _Proceedings of ACL_, pages 384-394.
* Upadhyay et al. (2018) Shyam Upadhyay, Yogarshi Vyas, Marine Carpuat, and Dan Roth. 2018. Robust cross-lingual hypernymy detection using dependency context. In _Proceedings of NAACL-HLT_.
* Vendrov et al. (2016) Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2016. Order-embeddings of images and language. In _Proceedings of ICLR (Conference Track)_.
* Vilnis and McCallum (2015) Luke Vilnis and Andrew McCallum. 2015. Word representations via Gaussian embedding. In _Proceedings of ICLR (Conference Track)_.
* Vulic et al. (2017) Ivan Vulic, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017. Hyperlex: A large-scale evaluation of graded lexical entailment. _Computational Linguistics_.
* Vulic et al. (2018) Ivan Vulic, Goran Glavas, Nikola Mrksic, and Anna Kornonen. 2018. Post-specialisation: Retrofitting vectors of words unseen in lexical resources. In _Proceedings of NAACL-HLT_.
* Vulic et al. (2017a) Ivan Vulic, Nikola Mrksic, and Anna Korhonen. 2017a. Cross-lingual induction and transfer of verb classes based on word vector space specialisation. In _Proceedings of EMNLP_, pages 2536-2548.
* Vulic et al. (2017b) Ivan Vulic, Nikola Mrksic, Roi Reichart, Diarmuid O Seaghdha, Steve Young, and Anna Korhonen. 2017b. Morph-fitting: Fine-tuning word vector spaces with simple language-specific rules. In _Proceedings of ACL_, pages 56-68.
* Weeds et al. (2014) Julie Weeds, Daoud Clarke, Jeremy Reffin, David Weir, and Bill Keller. 2014. Learning to distinguish hypernyms and co-hyponyms. In _Proceedings of COLING_, pages 2249-2259.
* Weeds et al. (2004) Julie Weeds, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In _Proceedings of COLING_, pages 1015-1021.
* Wieting et al. (2015) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. _Transactions of the ACL_, 3:345-358.
* Wu and Palmer (1994) Zhibiao Wu and Martha Palmer. 1994. Verb semantics and lexical selection. In _Proceedings of ACL_, pages 133-138.
* Xu et al. (2014) Chang Xu, Yalong Bai, Jiang Bian, Bin Gao, Gang Wang, Xiaoguang Liu, and Tie-Yan Liu. 2014. RCNET: A general framework for incorporating knowledge into word representations. In _Proceedings of CIKM_, pages 1219-1228.
* Yu and Dredze (2014) Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In _Proceedings of ACL_, pages 545-550.
* Yu et al. (2015) Zheng Yu, Haixun Wang, Xucmin Lin, and Min Wang. 2015. Learning term embeddings for hypernymy identification. In _Proceedings of IJCAI_, pages 1390-1397.
* Zhang et al. (2014) Jingwei Zhang, Jeremy Salwen, Michael Glass, and Alfio Glozzo. 2014. Word semantic representations using bayesian probabilistic tensor factorization. In _Proceedings of EMNLP_, pages 1522-1531.
* Zou et al. (2013) Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In _Proceedings of EMNLP_, pages 1393-1398.