Introduction
The prominent model for representing semantics of words is the distributional vector space model BIBREF2 and the prevalent approach for constructing these models is the distributional one which assumes that semantics of a word can be predicted from its context, hence placing words with similar contexts in close proximity to each other in an imaginary high-dimensional vector space. Distributional techniques, either in their conventional form which compute co-occurrence matrices BIBREF2 , BIBREF3 and learn high-dimensional vectors for words, or the recent neural-based paradigm which directly learns latent low-dimensional vectors, usually referred to as embeddings BIBREF4 , rely on a multitude of occurrences for each individual word to enable accurate representations. As a result of this statistical nature, words that are infrequent or unseen during training, such as domain-specific words, will not have reliable embeddings. This is the case even if massive corpora are used for training, such as the 100B-word Google News dataset BIBREF5 .
Recent work on embedding induction has mainly focused on morphologically complex rare words and has tried to address the problem by learning transformations that can transfer a word's semantic information to its morphological variations, hence inducing embeddings for complex forms by breaking them into their sub-word units BIBREF6 , BIBREF7 , BIBREF8 . However, these techniques are unable to effectively model single-morpheme words for which no sub-word information is available in the training data, essentially ignoring most of the rare domain-specific entities which are crucial in the performance of NLP systems when applied to those domains.
On the other hand, distributional techniques generally ignore all the lexical knowledge that is encoded in dictionaries, ontologies, or other lexical resources. There exist hundreds of high coverage or domain-specific lexical resources which contain valuable information for infrequent words, particularly in domains such as health. Here, we present a methodology that merges the two worlds by benefiting from both expert-based lexical knowledge encoded in external resources as well as statistical information derived from large corpora, enabling vocabulary expansion not only for morphological variations but also for infrequent single-morpheme words. The contributions of this work are twofold: (1) we propose a technique that induces embeddings for rare and unseen words by exploiting the information encoded for them in an external lexical resource, and (2) we apply, possibly for the first time, vector space mapping techniques, which are widely used in multilingual settings, to map two lexical semantic spaces with different properties in the same language. We show that a transfer methodology can lead to consistent improvements on a standard rare word similarity dataset.
Methodology
We take an existing semantic space INLINEFORM0 and enrich it with rare and unseen words on the basis of the knowledge encoded for them in an external knowledge base (KB) INLINEFORM1 . The procedure has two main steps: we first embed INLINEFORM2 to transform it from a graph representation into a vector space representation (§ SECREF2 ), and then map this space to INLINEFORM3 (§ SECREF7 ). Our methodology is illustrated in Figure 1.
In our experiments, we used WordNet 3.0 BIBREF9 as our external knowledge base INLINEFORM0 . For word embeddings, we experimented with two popular models: (1) GloVe embeddings trained by BIBREF10 on Wikipedia and Gigaword 5 (vocab: 400K, dim: 300), and (2) w2v-gn, Word2vec BIBREF5 trained on the Google News dataset (vocab: 3M, dim: 300).
Knowledge Base Embedding
Our coverage enhancement starts by transforming the knowledge base INLINEFORM0 into a vector space representation that is comparable to that of the corpus-based space INLINEFORM1 . To this end, we use two techniques for learning low-dimensional feature spaces from knowledge graphs: DeepWalk and node2vec. DeepWalk uses a stream of short random walks in order to extract local information for a node from the graph. By treating these walks as short sentences and phrases in a special language, the approach learns latent representations for each node. Similarly, node2vec learns a mapping of nodes to continuous vectors that maximizes the likelihood of preserving network neighborhoods of nodes. Thanks to a flexible objective that is not tied to a particular sampling strategy, node2vec reports improvements over DeepWalk on multiple classification and link prediction datasets. For both these systems we used the default parameters and set the dimensionality of output representation to 100. Also, note than nodes in the semantic graph of WordNet represent synsets. Hence, a polysemous word would correspond to multiple nodes. In our experiments, we use the MaxSim assumption of BIBREF11 in order to map words to synsets.
To verify the reliability of these vector representations, we carried out an experiment on three standard word similarity datasets: RG-65 BIBREF12 , WordSim-353 similarity subset BIBREF13 , and SimLex-999 BIBREF14 . Table TABREF5 reports Pearson and Spearman correlations for the two KB embedding techniques (on WordNet's graph) and, as baseline, for our two word embeddings, i.e. w2v-gn and GloVe. The results are very similar, with node2vec proving to be slightly superior. We note that the performances are close to those of state-of-the-art WordNet approaches BIBREF15 , which shows the efficacy of these embedding techniques in capturing the semantic properties of WordNet's graph.
Semantic Space Transformation
Once we have the lexical resource INLINEFORM0 represented as a vector space INLINEFORM1 , we proceed with projecting it to INLINEFORM2 in order to improve the word coverage of the latter with additional words from the former. In this procedure we make two assumptions. Firstly, the two spaces provide reliable models of word semantics; hence, the relative within-space distances between words in the two spaces are comparable. Secondly, there exists a set of shared words between the two spaces, which we refer to as semantic bridges, from which we can learn a projection that maps one space into another.
As for the mapping, we used two techniques which are widely used for the mapping of semantic spaces belonging to different languages, mainly with the purpose of learning multilingual semantic spaces: Least squares BIBREF16 , BIBREF17 and Canonical Correlation Analysis BIBREF18 , BIBREF19 . These models receive as their input two vector spaces of two different languages and a seed lexicon for that language pair and learn a linear mapping between the two spaces. Ideally, words that are semantically similar across the two languages will be placed in close proximity to each other in the projected space. We adapt these models to the monolingual setting and for mapping two semantic spaces with different properties. As for the seed lexicon (to which in our setting we refer to as semantic bridges) in this monolingual setting, we use the set of monosemous words in the vocabulary which are deemed to have the most reliable semantic representations.
Specifically, let INLINEFORM0 and INLINEFORM1 be the corpus and KB semantic spaces, respectively, and INLINEFORM2 and INLINEFORM3 be their corresponding subset of semantic bridges, i.e., words that are monosemous according to the WordNet sense inventory. Note that INLINEFORM4 and INLINEFORM5 are vector matrices that contain representations for the same set of corresponding words, i.e., INLINEFORM6 . LS views the problem as a multivariate regression and learns a linear function INLINEFORM7 (where INLINEFORM8 and INLINEFORM9 are the dimensionalities of the KB and corpus spaces, respectively) on the basis of the following INLINEFORM10 -regularized least squares error objective and typically using stochastic gradient descent: DISPLAYFORM0
The enriched space INLINEFORM0 is then obtained as a union of INLINEFORM1 and INLINEFORM2 . CCA, on the other hand, learns two distinct linear mappings INLINEFORM3 and INLINEFORM4 with the objective of maximizing the correlation between the dimensions of the projected vectors INLINEFORM5 and INLINEFORM6 : DISPLAYFORM0
In this case, INLINEFORM0 is the union of INLINEFORM1 and INLINEFORM2 . In the next section we first compare different KB embedding and transformation techniques introduced in this section and then apply our methodology to a rare word similarity task.
Evaluation benchmark
To verify the reliability of the transformed semantic space, we propose an evaluation benchmark on the basis of word similarity datasets. Given an enriched space INLINEFORM0 and a similarity dataset INLINEFORM1 , we compute the similarity of each word pair INLINEFORM2 as the cosine similarity of their corresponding transformed vectors INLINEFORM3 and INLINEFORM4 from the two spaces, where INLINEFORM5 and INLINEFORM6 for LS and INLINEFORM7 and INLINEFORM8 for CCA. A high performance on this benchmark shows that the mapping has been successful in placing semantically similar terms near to each other whereas dissimilar terms are relatively far apart in the space. We repeat the computation for each pair in the reverse direction.
Comparison Study
Figure FIGREF1 shows the performance of different configurations on our three similarity datasets and for increasing sizes of semantic bridge sets. We experimented with four different configurations: two KB embedding approaches, i.e. DeepWalk and node2vec, and two mapping techniques, i.e. LS and CCA (cf. § SECREF2 ). In general, the optimal performance is reached when around 3K semantic bridges are used for transformation. DeepWalk and node2vec prove to be very similar in their performance across the three datasets. Among the two transformation techniques, CCA consistently outperforms LS on all three datasets when provided with 1000 or more semantic bridges (with 500, however, LS always has an edge). In the remaining experiments we only report results for the best configuration: node2vec with CCA. We also set the size of semantic bridge set to 5K.
Rare Word Similarity
In order to verify the reliability of our technique in coverage expansion for infrequent words we did a set of experiments on the Rare Word similarity dataset BIBREF6 . The dataset comprises 2034 pairs of rare words, such as ulcerate-change and nurturance-care, judged by 10 raters on a [0,10] scale. Table TABREF15 shows the results on the dataset for three pre-trained word embeddings (cf. § SECREF2 ), in their initial form as well as when enriched with additional words from WordNet.
Among the three initial embeddings, w2v-gn-500K provides the lowest coverage, with over 20% out-of-vocabulary pairs, whereas GloVe has a similar coverage to that of w2v-gn despite its significantly smaller vocabulary (400K vs. 3M). Upon enrichment, all the embeddings attain near full coverage (over 99%), thanks to the vocabulary expansion by rare words in WordNet. The enhanced coverage leads to consistent performance improvements according to both Pearson and Spearman correlations. The best performance gain is achieved for w2v-gn-500K (around 10% absolute gain) which proves the efficacy of our approach in inducing embeddings for rare words. The improvements are also statistically significant (p < 0.05) according to conducted one tailed t-test BIBREF20 , showing that the coverage enhancement could lead to improved performance even if lower-performing KB embedding and transformation are used.
Related Work
The main focus of research in embedding coverage enhancement has been on the morphologically complex forms BIBREF21 . BIBREF6 used recursive neural networks (RNNs) and neural language models in order to induce embeddings for morphologically complex words from their morphemes whereas BIBREF22 adapted phrase composition models for this purpose. BIBREF7 proposed a different model based on log-bilinear language models, mainly to have a compositional vector-based morphological representation that can be easily integrated into a machine translation decoder. These models often utilize a morphological segmentation toolkit, such as Morfessor BIBREF23 , in order to break inflected words into their morphological structures and to obtain segmentations for words in the vocabulary. BIBREF8 put forward a technique that does not rely on any external morphological analyzer and instead, induces morphological rules and transformations, represented as vectors in the same embedding space. Based on these rules a morphological graph is constructed and representations are inferred by analyzing morphological transformations in the graph. However, all these techniques fall short in inducing representations for single-morpheme words that are not seen frequently during training as they base their modeling on information available on sub-word units. In contrast, our transformation-based model can also induce embeddings for single-morpheme words that are infrequent or unseen in the training data, such as domain-specific entities.
Conclusions and Future Work
We presented a methodology for merging distributional semantic spaces and lexical ontologies and applied it to the task of extending the vocabulary of the former with the help of information extracted from the latter. We carried out an analysis for different KB embedding and semantic space mapping techniques and also showed that our methodology can lead to considerable enrichment of two standard word embedding models, leading to consistent improvements on the rare word similarity dataset. One interesting property of our approach is that it can be used in the reverse direction and for the completion of knowledge bases using the distributional information derived from text corpora. In future work, we plan to investigate this direction. We also intend to experiment with domain-specific lexical resources and measure the impact of coverage enhancement on a downstream NLP application.

Question: What other embedding models are tested?
Answer: GloVe embeddings trained by BIBREF10 on Wikipedia and Gigaword 5 (vocab: 400K, dim: 300), w2v-gn, Word2vec BIBREF5 trained on the Google News dataset (vocab: 3M, dim: 300), DeepWalk , node2vec

Question: How is performance measured?
Answer: To verify the reliability of the transformed semantic space, we propose an evaluation benchmark on the basis of word similarity datasets. Given an enriched space INLINEFORM0 and a similarity dataset INLINEFORM1 , we compute the similarity of each word pair INLINEFORM2 as the cosine similarity of their corresponding transformed vectors INLINEFORM3 and INLINEFORM4 from the two spaces, where INLINEFORM5 and INLINEFORM6 for LS and INLINEFORM7 and INLINEFORM8 for CCA.

Question: How are rare words defined?
Answer: judged by 10 raters on a [0,10] scale