LSTM Language Model for Hypernym Discovery
Hayato Hashimoto and Shinsuke Mori
Kyoto University, Yoshida Honmachi, Sakyo-ku, Kyoto 606-8501, Japan

Abstract. We introduce a new simple method in hypernym discovery task basedon the distributional inclusion hypothesis. Our method utilizes LSTM language
models to discover possible alternative words in the context of the given hyponym
word and find the hypernym words within it. We implemented our method andtested it on a well-known hypernym discovery task, SemEval 2018 task 9. The
experimental results showed that it achieved the state-of-the-art in comparison
with existing unsupervised methods.
Keywords: hypernym discovery · hypernymy detection · hypernym-hyponym
relationship · lexical entailment · is-a relationship · unsupervised learning · distributional inclusion hypothesis · distributional informativeness hypothesis · relative information content · recurrent neural network · LSTM · language model ·contextual BLSTM

1Introduction

Hypernym-hyponym relationship, also known as is-a relationship, is essential to model
world knowledge for artificial intelligence. Several benchmark tasks [2, 3, 21, 25, 26]
have been proposed to evaluate hypernymy detection methods. A hypernym detection
task is a binary classification distinguishing word pairs v and w of is-a relations, or v ≺w in short, from other word pairs. This task setting is, however, sometimes criticizedthat it tends to give high scores to models memorizing the words which frequently
appear as hypernyms [22]. Additionally, the best models tend to be inconsistent amongdatasets, for example, figures reported by Chang et al. [7], for they depend on negative
examples contained in the datasets. For example, some dataset contains random wordpairs, and other datasets contain word pairs of has-a relationships as a negative example.
Recently, a new framework for evaluating hypernymy detection models is proposed,
namely hypernym discovery [9]. Hypernym discovery is the task of finding the appropriate hypernyms for a given input word or phrase from the whole vocabulary of the
corpus. This formulation enables us to test a system in settings more alike to real-world
applications. Thus we focused on a domain-specific task of well-known SemEval 2018
shared task [6] in our experiments.
In the background explained in section 2, we came up with an idea that simple andstraightforward approach based on alternativity of a word in its contexts may performbetter than existing methods attempting to represent the sense of a word by a single
vector. We utilize LSTM [12] language models (LMs) to calculate the word alternativityof w to v as the measure of being v ≺ w, and find the hypernyms of v within words
which have high alternativity to v.

2

Hayato Hashimoto and Shinsuke Mori

We implemented our idea and tested it in SemEval 2018 shared task. In the subsequent sections, we first describe background and our method in detail and then evaluate
it with its experimental results.

2

BackgroundThe distributional inclusion hypothesis (DIH) [10] is a theoretical foundation of unsupervised learning of hypernymy relations, extending the distributional hypothesis [11].
Several measures [8, 14, 15, 27] for the word co-occurrence frequency or PPMI matrix [5] have been proposed in accord with this hypothesis.
The distributional informativeness hypothesis [20] is another theoretical foundationof unsupervised learning inspired by DIH. They hypothesize that a context of a word
is more informative than the context of its hypernym, in the sense of the information
theory.
Prediction-based approach for word semantics has been made in recent research:
Baroni et al. [1] showed that vectors trained with Skip-Gram prediction task performbetter in estimating word relatedness and resolving word analogies than count-based
methods. For hypernymy, Chang et al. [7] proposed distributional inclusion vector embedding (DIVE), a model specially designed to capture the hypernymy-hyponym relations through Skip-Gram predictions, providing a firm baseline for hypernym discovery
task.

3

Hypernymy Measure based on a Language Model

3.1

Definition of the Hypernymy Relationship

Adopting one of the definitions of is-a relationship proposed by Brachman [4], we define the hypernymy as the following.
Definition 1. Let vi be one of the senses of the term v and Set(vi ) denotes the set ofthe entities, the concepts, or the collections of things (for collective terms like “group”)
covered by the sense vi . If and only if Set(vi ) ⊂ Set(wi ), then “w is a hypernym of v”(v ≺ w).
Hereafter, we assume a word has only one sense and equate the word sense wi with its
term w for simplicity.
3.2

Hypernyms are More Likely to Appear in the Context where theirHyponyms Appear

DIH is a statement that if v ≺ w, then all syntactical properties of w also hold for v andvice versa. This hypothesis is based on the assumption that a hypernym can substitutefor its hyponyms by definition.
But why a hypernym can substitute for its hyponym? Let us consider the case that
an author wants to write a sentence about some real-world entities E and needs todetermine the wording for E. Then, it is reasonable to assume that (s)he chooses theLSTM Language Model for Hypernym Discovery

3term v so that the sense of v covers E, and v has an appropriate level of abstraction andmatches the writing style. By the definition 1, it automatically holds that w covers E
when v covers E if v ≺ w. Therefore, (s)he can choose w instead of v in context where
v is appropriate, provided that w has an appropriate generality and matches the style.
Now let us consider the probability of occurrence of a certain word v in a given
context c based on this argument. First we assume that the author mentions entities E
in a context c with probability P (c, E), which derives the conditional probability (intheory):P (c, E),P (E | c) = R
dP (c, E)
E∈ΩER
where E∈ΩE dP (c, E) denotes the marginal probability integrated by E. The conditional probability P (v | c) can be modeled by the wording process as follows:
P (v | c) = PA (v | c) · PS (v | c) · PE∼P (E|c) (E ⊂ Set(v)),
where PE∼P (E|c) (E ⊂ Set(v)) denotes the probability that E ⊂ Set(v) holds for theset of entities E drawn from the distribution P (E | c), PA (v | c) denotes the probabilityof the word in the context c being at the same level of abstraction as v, and PS (v | c)denotes the probability of the word in the context c matches the writing style to thesame extent as v.
By the definition 1, if v ≺ w then Set(v) ⊂ Set(w), and therefore, PE∼P (E|c) (E ⊂Set(v)) ≤ PE∼P (E|c) (E ⊂ Set(w)). Ignoring the writing style, if PA (w | c) is constant or at least
PE∼P (E|c) (E ⊂ Set(w))
PA (v | c)≤,(1)
PA (w | c)PE∼P (E|c) (E ⊂ Set(v))
then it is concluded that P (v | c) ≤ P (w | c) for any context c. This result implies
that a hypernym is more frequent than its hyponym. It also implies that P (v, vc ) ≤P (w, vc ) if v ≺ w where P (v, vc ) denotes the joint probability that the words v andvc co-occur in the same bag-of-words, the same document, or the same latent topic,
or there are instances of a certain type of syntactic dependency between v and vc . Asa corollary, it rationalize the estimation of hypernymy by calculating Clarke’s degreeof entailment (CDE) [8] or L1 norm of the vectors from the frequency matrix of word
co-occurrences [7] .
3.3

Filling Blanks by Language Models to Discover Hypernyms

We propose to use fitness to the concrete context which considers the whole sentence
instead of bag-of-words or syntactic dependency relations.
Definition 2. concrete context of v is the text blanked at the target word v.
For example, “Antiviral drugs are ____ in treating influenza.” is a concrete context ofv = “effective”. For all w in the vocabulary V , we proposed to calculate the averaged
fitness to the concrete contexts of v, which we call the word alternativity of w to v.
The alternativity cannot be measured by counting word occurrences since the concrete

4Hayato Hashimoto and Shinsuke Moricontext is almost unique. Instead, we propose to use an LM to calculate the probabilityof alternative words filling in the blank.
We propose to define a measure for word alternativity of w to v, namely LMMeasure, as the following:
Definition 3. X-LM-Measure (v ≺ w) is the difference of log-likelihood or relative
information content averaged over all concrete contexts C[v] of v:
LM-Measure (v ≺ w) =E[log PLM (w | c) − log PLM (v | c)] ,c∈C[v]

where PLM is a word occurring probability according to the X language model.
3.4

Contextual Language Models

The conventional LMs predict the word only from the preceding words. This model
often mistakes the part-of-speech (POS), for example in section 3.1, the LSTM LM
predicted “not,” “generally,” or “the” to fill in the blank in the above example. Thus
we adopted to use the contextual language model (cLM) where both the preceding andthe succeeding words are provided. A cLM trained by the method described in the next
section predicted “effective,” “useful,” or “recommended” belonging to the proper POSs
to fill in the blank in the same example.
3.5

Learning Contextual Bidirectional LSTM

Contextual BLSTM [18] or cBLSTM is a cLM which consists of two LSTMs which
consumes word sequences forward and backward ( Fig. 1 ).
We calculated the logit vector h = hf,k−1 + hb,k+1 using hf,k−1 and hb,k+1
obtained from the (k − 1)-th and the (k + 1)-th output of the forward and backward
LSTMs, respectively. We tied the parameters of the linear layer with the embedding
layer following [13]. We trained it by minimizing the categorical cross entropy loss ofthe predicted probability distribution.
3.6

Fast Calculation of LM-Measure

For LMs or cLMs outputting word probability using softmax function to normalize thelogit vector h, the relative information content log PLM (w|c) − log PLM (v|c) can becalculated byexp(hw )exp(hv )− log P
exp(h
)
i
i∈V
i∈V exp(hi )= hv − hw ,log[softmax(h)]v − log[softmax(h)]w = log P

where the subscript hv denotes the scalar element of the vector h at the index corresponding to the word v. In our experiments, we obtained LM-Measure for all pairs ofwords in one path using the following procedure. First, we calculated hv − hw for all w
in the most common |V 0 | words for all v in the corpus in order of appearance. The sum
0of the relative information content was stored in a matrix WMeasure ∈ R|V |×|V | . Then,
we divided every row by the corpus frequency of the corresponding word. We ran thispath as fast as running one epoch of the training of the cBLSTM.

LSTM Language Model for Hypernym Discovery

Antiviral

drugs

are

eﬀectivein++eﬀective

in

5

Embed

+

+

+
UnembedSoftmax
Cross Entropy Loss
Antiviral

drugs

are

Fig. 1. cBLSTM combines forward and backward LSTMs to predict a word in a given context. The square boxes in the middle stand for LSTM cell and circles stand for hidden vectors.

6

Hayato Hashimoto and Shinsuke Mori
Table 1. Hyperparameters used to train the baseline models
word2vec
Window Size8
# of negative samples
25
cbow
1
Downsampling threashold 10−4
Training epochs
15
DIVE# of negative samples
30
Inclusion Shift
Yes
Window size
5
PMI filter size
5
Embedding size
100
Training epochs
15

4

Evaluation

4.1

Training Corpus

We retrieved the pre-tokenized corpus from the competition website of SemEval 20181
. The task consists of five subtasks: three for general domains for different languages(1A, 1B, and 1C) and two for specific domains (2A–medical and 2B–music). We conducted experiments only on 2A and 2B, since it appears to be more practical settings
for unsupervised learning while dataset is relatively small. The corpus originates fromseveral in-domain sources. We split the text into the first 1,000 lines for validation andrest for training corpus. We limited the vocabulary to the most common 80,000 words
in the corpus, and all out-of-vocabulary words were rendered to <unk>.
Randomly taken 1%, 10%, and 100% sentences from the training corpus are used
to train the baseline and proposed models to compare the degree of dependence on thesize of the corpus.
4.2

Model Setups

Baselines In the results section, table 2 cites the results of unsupervised methods summarized by Camacho-Collados et al. [6]. ADAPT [16] calculates cosine similarity ofword embedding vectors learned by skip-gram negative sampling with the given corpus. APSyn [19] is a symmetric measure to calculate the similarity of two words using
the rank of shared context in the PPMI matrix. SLQS [20] is an asymmetric measure
to calculate the differences in generality between two words by exploiting the distributional informativeness hypothesis.
In addition to these methods, we set two unsupervised baselines, namely W∆S and
ClogP.
W∆S: We adopted the W∆S hypernymy measure from [7] as a baseline. W∆S is the
cosine similarity of the word2vec [17] vectors multiplied by the generality signal∆S(v ≺ w) = kwk1 − kvk1 of DIVE.
1

https://competitions.codalab.org/competitions/17119

LSTM Language Model for Hypernym Discovery

7

We trained the word2vec model by the Mikolov’s code2 and the DIVE model by
the code provided by the authors3 . We searched for the best number of dimensions d ofword2vec vectors, but other hyper-parameters are fixed ( Table 1 ).
ClogP: Since LM-Measure gives high score to frequent words, we set the following
baseline to see if LM-Measure do more than selecting similar and common words:
ClogP [α] (v ≺ w) = cos (v, w) + α · log P (w),where v and w are word2vec representations of v and w, respectively, and α is a
hyper-parameter and P (w) is the frequency of w in the corpus divided by the total
number of words.
Training Language Models for LM-Measure We trained LSTM and cBLSTM using
step annealing for 20 epochs. Firstly we split the corpus into n parts and the first mwords of each part are feed into the LSTM or cBLSTM in a batch. The hidden states of
the LSTM or cBLSTM are passed to the next batch, which processes the next m words
4. We calculated the hypernymy measure as described in section 3 using |V 0 | = 20000.
LM-Measure Variations We added the following variations to the proposed LMMeasure model.
Label smoothing: To ensure that cBLSTM predicts a large variety of words, we adopted
the label smoothing technique [23, 24]. We imposed 0.9 and 0.1/(|V | − 1) for correctand incorrect predictions instead of 1 and 0 as a supervision signal when calculating
cross entropy during training.
Discounting word frequency: In general, a cLM gives a high probability to frequentwords. In the presence of validation dataset, we can tune the system to cancel thiseffect by adequately discounting the measure by the log-frequency of w. We calculated
LM-Measure (v ≺ w) − α · log P (w) as a hypernymy measure, where α is a hyperparameter.
POS filtering: In this variation, we filtered out the hypernym candidates in case that
their POS does not match the given hyponym word. We used the WordNet databaseas a POS dictionary to find if any synset of the candidate hypernym be a noun, for all
hyponym words are nouns in the dataset used in the experiment.
4.3

Validation and Test Dataset of Hypernym-Hyponym Pairs

We obtained the validation and test dataset from competition website for SemEval 2018.
We used dataset originally provided for training supervised systems as the validation
dataset. Validation and test dataset contains 500 hyponyms Vdataset = {v1 , v2 , ..., v500 }and the corresponding lists of hypernyms Wv1 , Wv2 , ..., Wv500 such that2

https://github.com/tmikolov/word2vec
https://github.com/iesl/Distributional-Inclusion-Vector-Embedding
4This feeding process disturbs the word order for backward LSTM of cBLSTM once in mwords, but it enables efficient learning since it can make n × m predictions at once. The
dimensions of the embedding vector and the hidden state is set to 300.
3

8

Hayato Hashimoto and Shinsuke Mori∀w ∈ Wvi (vi ≺ w), and each word vi is labeled with either “Concept” or “Entity.”
We excluded entities and left concepts because unsupervised learning is not suitable forfinding hypernyms of entities since there is only a little number of possible hypernyms,
being more like a classification problem.
4.4

Evaluation Details

As described by Camacho-Collados et al. [6], we estimated fifteen hypernym candidates
wv,1 , wv,2 , ..., wv,15 for each given word v ∈ Vdataset and calculated mean average
precision (MAP), mean reciprocal rank (MRR), and precision at k = 1, 5, and 15(P@k) by comparing them with the corresponding hypernyms Wv in the dataset. We
used the implementation provided by Camacho-Collados et al. [6] to calculate these
metrics. We excluded 180 stop words and the given word itself from the candidates.
The stop word list is basically the same as the one used in the DIVE.
For input phrases more than one word, we calculated the score of the last word. We
also conducted an experiment calculating hypernym scores using the averaged vectors
or the averaged LM-Measure for all words appeared in a phrase, but the performance
was inferior.
We tuned α and the dimension d of word2vec vectors by choosing the ones maximizing MAP of the validation pairs. We searched over (100, 300, 1000) for d and (1/5,
1/10, ..., 1/30, 0) for α.
4.5

Results and Discussion

Table 2 shows the result of our experiments. Fistly, we observed that even the ClogP
baseline surpasses the result reported in literature [6]. In hypernym discovery tasks, it
seems that it is effective to narrow down the large vocabulary by selecting more generaland similar words, and the word frequency is a reliable index to measure the generalityof a word.
Secondly, we observed that LM-Measure worked without tuning discounting hyperparameter. We suspect that contributions to the hypernymy measure of the similarity andthe generality are already balanced by the information content approach.
Thirdly, we observed that LM-Measure surpasses ClogP baseline, especially at
P@5 and P@15. These figures imply that it picked up a wider variety of correct hypernyms than ClogP baseline. To substantiate this point, we listed the correct hypernyms
which appear in the top 5 hypernym candidates list obtained by one method but do notappear in the top 15 hypernym candidates obtained by the other method ( Table 3 ) andexamined the hypernym candidates of “influenza” ( Table 5 .). It seems that ClogP
preferred co-occurring related concepts and failed to predict “disease” since it did not
co-occur with disease names since the authors avoid redundancy, while LM-Measuresucceed to predict “disease.”
We tested ability of LM-Measure to retrieve hypernyms from frequent (and hence
presumably generic) words by calculating the average precision for each hypernyms
and showing its dependency to the word frequency. Fig. 2 shows the result of this investigation. The figure shows that LM-Measure estimates hypernym candidates better

LSTM Language Model for Hypernym Discovery

9

Table 2. Results of the subtasks 2A and 2B of the SemEval 2018 task 9 Bold figures indicate
the best score in the same corpus size. Note that Anu is the model trained with both the corpus
and the WordNet.
Subtask
2A - medical domain
2B - music domain
Performance metric
MAP MRR P@1 P@5 P@15 MAP MRR P@1 P@5 P@15
Training data used: 1%
ClogP
10.03 29.99 29.20 8.87 7.03 4.03 11.67 9.78 3.47 2.66
cBLSTM-LM-Measure 14.54 39.30 36.80 13.91 10.26 6.85 19.87 15.92 6.45 5.10
+ discount15.02 39.81 36.80 14.52 10.49 6.82 19.70 15.64 6.56 5.08
+ smoothing
14.13 38.58 36.00 13.53 9.91 6.68 19.12 15.36 6.49 4.86
+ smoothing + discount 14.30 38.58 35.60 13.85 9.98 6.68 19.16 15.36 6.49 4.84
Training data used: 10%
ClogP
14.66 40.31 38.60 13.27 10.47 6.99 20.26 16.76 6.54 5.03
cBLSTM-LM-Measure 16.77 43.81 39.20 16.52 12.03 8.40 22.28 16.48 8.04 6.55
+ discount
16.97 43.87 38.40 16.74 12.11 8.40 22.28 16.48 8.04 6.55
+ smoothing
16.54 42.84 38.40 16.06 11.87 8.36 23.11 17.60 8.16 6.25
+ both
16.62 43.23 38.60 16.18 11.84 8.36 23.11 17.60 8.16 6.25
Training data used: 100%
ClogP
13.96 39.14 36.60 12.89 10.31 7.19 20.31 16.20 7.37 5.45
W∆S
3.20 9.21 6.00 3.32 2.45 2.98 8.64 5.03 3.05 2.36
LSTM-LM-Measure
12.62 33.19 25.40 13.09 9.01 5.11 14.03 9.22 5.40 4.09
+ POS filtering
14.49 38.95 33.00 14.55 10.07 5.92 16.04 10.06 6.14 4.70
+ POS filtering + discount 14.44 38.86 32.80 14.59 9.95 5.92 16.04 10.06 6.14 4.70
cBLSTM-LM-Measure 16.81 42.64 37.40 16.78 12.22 8.84 24.17 18.44 8.81 6.58
+ discount
16.91 42.77 37.40 16.98 12.28 8.84 24.17 18.44 8.81 6.58
+ smoothing
16.75 42.82 37.60 16.68 12.14 8.17 21.83 16.76 7.80 6.61
+ smoothing + discount 16.75 42.82 37.60 16.68 12.14 8.17 21.83 16.76 7.80 6.61
+ POS filtering
16.93 43.04 37.60 16.91 12.36 8.90 24.24 18.44 8.81 6.64
+ POS filtering + discount 17.02 43.18 37.60 17.03 12.37 8.90 24.24 18.44 8.81 6.64
Results of unsupervised methods for Concepts summerized by Camacho-Collados et al. [6]ADAPT [16]8.13 20.56- 8.32- 1.88 5.34
- 1.89- 10.84Anu
7.05 17.51- 7.29- 10.68 27.13(Team 13)
2.55 7.19- 2.52- 4.83 14.33- 4.51balAPInc [14]0.91 2.10- 1.08- 1.44 3.65
- 1.58APSyn [19]0.65 1.43- 0.72- 1.13 2.55
- 1.30SLQS [20]0.29 0.66- 0.33- 0.64 1.25
- 0.65
Table 4. Hypernyms found unique to each lanTable 3. Hypernyms found unique to eachguage model Contextual language model canmethod LM-Measure can find generic words asfind more specific words as hypernyms comhypernyms compared to ClogP.
pared to a non-contextual language model.
LM-Measure
ClogP
hypernym
count hypernym
countdisease83 enzyme
2drug
8 anemia2
pain
4 pigment
1
disorder3 neoplasm
1(other 15 words) 1 or 2 (other 20 words)
1

cBLSTM
LSTM
hypernym
count hypernym countdrug
2 disease2
allergen
1 blood1
constipation
1 fluid1
corticosteroid
1 ion
1(other 22 words)
1

10

Hayato Hashimoto and Shinsuke Mori

Table 5. Hypernym candidates for “influenza” The proposed LM-Measure successfully estimates the word “disease” and “infection” as hypernyms while ClogP wrongly estimates cooccurring words (“outbreak”, “seasonal”) and synonyms (“flu”).
LM-Measure
ClogP
Influenza respiratory virus RSV HIV viral can- Influenza H1N1 pandemic RSV ILI pH1N1
cer pandemic <eos> hepatitis disease H5N1 viruses outbreak SARI pdm09 ARI outbreaks
H1N1 adenovirus lung H3N2 infection infec- virus H3N2 H5N1 viral seasonal respiratory flu
tious human

if they have large corpus frequency, and for frequent hypernyms ClogP does not matchthe LM-Measure results even if the large α is set to prioritize the frequent words. It
seems that if there is larger difference in generality between two words, word2vec
vectors no longer show high cosine similarity.
In this investigation, we used the following defition of average precision AP of thehypernym w:
r(v,w)X
1Wv (wv,k )1 X k=1,
AP (w) =|Vw |r(v, w)v∈Vw

where Vw ⊂ Vdataset is the set of hyponyms of w in the test dataset, wv,k is the k-th
hypernym candidate of v, r is the rank of w (i.e. wv,r(v,w) = w), and(1 (w ∈ Wv , i.e. v ≺ w)
1Wv (w) =.
0 (otherwise)
One limitation of LM-Measure is that it sometimes considers the hyponym as a
hypernym. In Table 5 , LM-Measure wrongly estimates “H5N1” being a hypernym of“influenza,” while it is, in fact, a hyponym of “influenza.” It may reflects the fact that
medical research papers preferably describe this disease at the subtype level to retain
academic detailedness, and the inequality assumption (1) does not hold in this case.
Lastly, POS filtering shows large improvement in LSTM-LM-Measure but small
improvement in cBLSTM-LM-Measure as expected. The results also shows that POS
filtering alone is not enough to fill the gap between LSTM-LM-Measure and cBLSTMLM-Measure. Table 4 shows the correct hypernyms appear in the top 5 candidates
obtained by LM-Measure + POS filtering using one language model but do not appear
in the top 15 hypernym candidates obtained by LM-Measure + POS filtering using theother language model.

5Conclusion and Future Work

By taking the straightforward approach utilizing the distributional inclusion hypothesis, our proposed method achieved state-of-the-art in unsupervised hypernym discovery
tasks.

LSTM Language Model for Hypernym Discovery

11

0.7

Hypernym precision

0.6
0.5

LM-Measure
ClogP = 1/25
ClogP = 1/10
ClogP = 1/8
ClogP = 1/5

0.4
0.3
0.2
0.1
0.0

10 6

10 5
10 4
Hypernym frequency

10 3

Fig. 2. LM-Measure estimates hypernym candidates better if they have large corpus frequency. Hypernym precision of the proposed cBLSTM-Measure and the ClogB baseline using
different hyperparameters α were compared by the subtask 2A of the SemEval 2018 task 9. The
hypernyms were divided into eight bins according to their log frequency (horizontal axis) andmean precision (vertical axis) were calculated for each bin.

12

Hayato Hashimoto and Shinsuke MoriOur method can be extended from words to phrases by replacing a blank with nconsecutive blanks when calculating LM-Measure. Another direction of extension is
to use attentional models as the cLM. The Transformer LM [24] can be trained as acLM by masking.

6

Acknowledgement

This research was partially supported by the Cooperative Intelligence Joint Research
Chair with Honda Research Institute Japan Co., Ltd.
We thank Keisuke Shirai for indicating label smoothing technique, and anonymous
reviewers of NAACL-HLT 2019 for comments that greatly improved the manuscript.

Bibliography[1] Baroni, M., Dinu, G., Kruszewski, G.: Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In: Proceedingsof the 52nd Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers). pp. 238–247. Association for Computational Linguistics (2014). https://doi.org/10.3115/v1/P14-1023[2] Baroni, M., Lenci, A.: How we blessed distributional semantic evaluation. In: Proceedings of the Workshop on Geometrical Models of Natural Language Semantics(GEMS ’11). pp. 1–10. Association for Computational Linguistics, Stroudsburg,
PA, USA (2011), http://aclweb.org/anthology/W11-2501[3] Benotto, G.: Distributional models for semantic relations: A study on hyponymy
and antonymy. PhD Thesis, University of Pisa (2015), https://etd.adm.unipi.it/
theses/available/etd-04302015-171419/[4] Brachman, R.J.: What is-a is and isn’t: An analysis of taxonomic
links in semantic networks. Computer 16(10), 30–36 (Oct 1983).
https://doi.org/10.1109/MC.1983.1654194[5] Bullinaria, J.A., Levy, J.P.: Extracting semantic representations from word cooccurrence statistics: A computational study. Behavior research methods 39(3),510–526 (2007). https://doi.org/10.3758/BF03193020[6] Camacho-Collados, J., Delli Bovi, C., Espinosa Anke, L., Oramas, S., Pasini, T.,
Santus, E., Shwartz, V., Navigli, R., Saggion, H.: Semeval-2018 task 9: Hypernym discovery. In: Proceedings of the 12th International Workshop on Semantic Evaluation. pp. 712–724. Association for Computational Linguistics (2018),
http://aclweb.org/anthology/S18-1115[7] Chang, H.S., Wang, Z., Vilnis, L., McCallum, A.: Distributional inclusion vector
embedding for unsupervised hypernymy detection. In: Proceedings of the 2018
Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 485–
495. Association for Computational Linguistics, New Orleans, Louisiana (June2018), http://aclweb.org/anthology/N18-1045[8] Clarke, D.: Context-theoretic semantics for natural language: An overview. In:
Proceedings of the Workshop on Geometrical Models of Natural Language Semantics (GEMS ’09). pp. 112–119. Association for Computational Linguistics,
Stroudsburg, PA, USA (2009), http://aclweb.org/anthology/W09-0215[9] Espinosa Anke, L., Camacho-Collados, J., Delli Bovi, C., Saggion, H.: Supervised distributional hypernym discovery via domain adaptation. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 424–435. Association for Computational Linguistics (2016), http://aclweb.org/anthology/D16-1041[10] Geffet, M., Dagan, I.: The distributional inclusion hypotheses and lexical entailment. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. pp. 107–114. ACL ’05, Association for Computational Linguistics, Stroudsburg, PA, USA (2005). https://doi.org/10.3115/1219840.1219854

14

Hayato Hashimoto and Shinsuke Mori[11] Harris, Z.S.: Distributional structure. Word 10(2-3), 146–162 (1954).
https://doi.org/10.1080/00437956.1954.11659520[12] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation
9(8), 1735–1780 (Nov 1997). https://doi.org/10.1162/neco.1997.9.8.1735[13] Inan, H., Khosravi, K., Socher, R.: Tying word vectors and word classifiers: A loss
framework for language modeling. In 7th International Conference on Learning
Representations (2017), https://arxiv.org/abs/1611.01462[14] Kolterman, L., Dagan, I., Szpektor, I., Geffet, M.: Directional distributional similarity for lexical inference. Natural Language Engineering 16(4), 359–389 (2010).
https://doi.org/10.1017/S1351324910000124[15] Lenci, A., Benotto, G.: Identifying hypernyms in distributional semantic spaces.
In: Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task,
and Volume 2: Proceedings of the Sixth International Workshop on Semantic
Evaluation. pp. 75–79. SemEval ’12, Association for Computational Linguistics,
Stroudsburg, PA, USA (2012), http://aclweb.org/anthology/S12-1012[16] Maldonado, A., Klubička, F.: Adapt at semeval-2018 task 9: Skip-gram wordembeddings for unsupervised hypernym discovery in specialised corpora. In:
Proceedings of The 12th International Workshop on Semantic Evaluation. pp.
924–927. Association for Computational Linguistics (2018), http://aclweb.org/
anthology/S18-1151[17] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Workshop Proceedings of the International Conference on Learning Representations (ICLR 2013) (2013), http://arxiv.org/abs/1301.
3781[18] Mousa, A., Schuller, B.: Contextual bidirectional long short-term memory recurrent neural network language models: A generative approach to sentiment analysis. In: Proceedings of the 15th Conference of the European Chapter of the
Association for Computational Linguistics: Volume 1, Long Papers. pp. 1023–
1032. Association for Computational Linguistics, Valencia, Spain (April 2017),
http://aclweb.org/anthology/E17-1096[19] Santus, E., Lenci, A., Chiu, T.S., Lu, Q., Huang, C.R.: What a nerd! beating students and vector cosine in the esl and toefl datasets. In: Chair), N.C.C.,
Choukri, K., Declerck, T., Goggi, S., Grobelnik, M., Maegaard, B., Mariani, J.,
Mazo, H., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the TenthInternational Conference on Language Resources and Evaluation (LREC 2016).
European Language Resources Association (ELRA), Paris, France (may 2016),
http://www.lrec-conf.org/proceedings/lrec2016/summaries/517.html[20] Santus, E., Lenci, A., Lu, Q., Schulte im Walde, S.: Chasing hypernyms in vector spaces with entropy. In: Proceedings of the 14th Conference of the European
Chapter of the Association for Computational Linguistics, volume 2: Short Papers. pp. 38–42. Association for Computational Linguistics, Gothenburg, Sweden(April 2014), http://aclweb.org/anthology/E14-4008[21] Santus, E., Yung, F., Lenci, A., Huang, C.R.: Evalution 1.0: an evolving semantic
dataset for training and evaluation of distributional semantic models. In: Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applica-LSTM Language Model for Hypernym Discovery

15

tions. pp. 64–69. Association for Computational Linguistics, Beijing, China (July
2015), http://aclweb.org/anthology/W15-4208[22] Shwartz, V., Santus, E., Schlechtweg, D.: Hypernyms under siege: Linguisticallymotivated artillery for hypernymy detection. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics:
Volume 1, Long Papers. pp. 65–75. Association for Computational Linguistics,
Valencia, Spain (April 2017), http://aclweb.org/anthology/E17-1007[23] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016).
https://doi.org/10.1109/CVPR.2016.308[24] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.,
Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V.,
Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advancesin Neural Information Processing Systems 30, pp. 5998–6008. Curran Associates,
Inc. (2017), http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf[25] Vendrov, I., Kiros, R., Fidler, S., Urtasun, R.: Order-embeddings of images andlanguage. In: Conference Proceedings of the International Conference on Learning Representations (ICLR 2016) (2016), https://arxiv.org/abs/1511.06361[26] Weeds, J., Clarke, D., Reffin, J., Weir, D., Keller, B.: Learning to distinguish hypernyms and co-hyponyms. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. pp. 2249–
2259. Dublin City University and Association for Computational Linguistics(2014), http://aclweb.org/anthology/C14-1212[27] Weeds, J., Weir, D.: A general framework for distributional similarity. In: Proceedings of the 2003 Conference on Empirical Methods in Natural Language
Processing (EMNLP ’03). pp. 81–88. Association for Computational Linguistics,
Stroudsburg, PA, USA (2003). https://doi.org/10.3115/1119355.1119366