Abstract: We propose a methodology that adapts graph embedding techniques (DeepWalk (Perozzi et al., 2014) and node2vec (Grover and Leskovec, 2016)) as well as cross-lingual vector space mapping approaches (Least Squares and Canonical Correlation Analysis) in order to merge the corpus and ontological sources of lexical knowledge. We also perform comparative analysis of the used algorithms in order to identify the best combination for the proposed system. We then apply this to the task of enhancing the coverage of an existing word embedding's vocabulary with rare and unseen words. We show that our technique can provide considerable extra coverage (over 99%), leading to consistent performance gain (around 10% absolute gain is achieved with w2v-gn-500K cf.\S 3.3) on the Rare Word Similarity dataset.

Introduction: The prominent model for representing semantics of words is the distributional vector space model BIBREF2 and the prevalent approach for constructing these models is the distributional one which assumes that semantics of a word can be predicted from its context, hence placing words with similar contexts in close proximity to each other in an imaginary high-dimensional vector space. Distributional techniques, either in their conventional form which compute co-occurrence matrices BIBREF2 , BIBREF3 and learn high-dimensional vectors for words, or the recent neural-based paradigm which directly learns latent low-dimensional vectors, usually referred to as embeddings BIBREF4 , rely on a multitude of occurrences for each individual word to enable accurate representations. As a result of this statistical nature, words that are infrequent or unseen during training, such as domain-specific words, will not have reliable embeddings. This is the case even if massive corpora are used for training, such as the 100B-word Google News dataset BIBREF5 . Recent work on embedding induction has mainly focused on morphologically complex rare words and has tried to address the problem by learning transformations that can transfer a word's semantic information to its morphological variations, hence inducing embeddings for complex forms by breaking them into their sub-word units BIBREF6 , BIBREF7 , BIBREF8 . However, these techniques are unable to effectively model single-morpheme words for which no sub-word information is available in the training data, essentially ignoring most of the rare domain-specific entities which are crucial in the performance of NLP systems when applied to those domains. On the other hand, distributional techniques generally ignore all the lexical knowledge that is encoded in dictionaries, ontologies, or other lexical resources. There exist hundreds of high coverage or domain-specific lexical resources which contain valuable information for infrequent words, particularly in domains such as health. Here, we present a methodology that merges the two worlds by benefiting from both expert-based lexical knowledge encoded in external resources as well as statistical information derived from large corpora, enabling vocabulary expansion not only for morphological variations but also for infrequent single-morpheme words. The contributions of this work are twofold: (1) we propose a technique that induces embeddings for rare and unseen words by exploiting the information encoded for them in an external lexical resource, and (2) we apply, possibly for the first time, vector space mapping techniques, which are widely used in multilingual settings, to map two lexical semantic spaces with different properties in the same language. We show that a transfer methodology can lead to consistent improvements on a standard rare word similarity dataset.

Question: What datasets are used to evaluate the model?
Answer: WN18 and FB15k