CoKE : Extending Contextualized Knowledge Embeddings for Word Sense Induction

Word Embeddings are able to capture lexico-semantic information but remain 1 flawed in their inability to assign unique representations to different senses of 2 a polysemous words. They also fail to include information from well curated 3 semantic lexicons and dictionaries. Previous approaches that integrate polysemy 4 and knowledge bases fall distinctly under two categories a)retrofitting vectors to 5 ontologies or b)learning from sense tagged corpora. While embeddings learned 6 from these methods are superior in understanding contextual similarity, they are 7 outperformed by single prototype word vectors on several relatedness tasks. In this 8 work, we introduce a new approach that can induce polysemy to any pre-trained 9 embedding space by jointly grounding contextualized sense representations and 10 word embeddings to a knowledge base. Along with word sense induction, the 11 resulting representations reduces the effect of vocabulary bias that arises in natural 12 language corpora and in turn embedding spaces. By grounding them to knowledge 13 bases they are able to learn multi-word representations and are also interpretable. 14 We evaluate our vectors across 12 datasets on several similarity and relatedness 15 tasks along with two extrinsic tasks ,we also evaluate against other transfer learning 16 methods and find that our approach consistently outperforms current state of the 17 art. 18 1 Related Work and Introduction 19 Distributed representations of words (Mikolov et al., 2013b) have proven to be successful in addressing 20 multiple drawbacks of symbolic representations which treats words as atomic units of meaning. By 21 grouping similar words and capturing analogical and lexical relationships, they are a popular choice 22 in several downstream NLP applications. 23 While these embeddings capture meaningful relationships, they come with their own set of drawbacks. 24 For instance, complete reliance on natural language corpora amplifies existing societal and vocabulary 25 bias that are inherent in datasets. A study by Bolukbasi et al., 2016 discussed societal biases in the 26 form of gender stereotypes present in these VSMs. Vocabulary bias is caused by words not seen in 27 the training corpora and also extends to bias in word usage where some words, often morphologically 28 complex words, are used less frequently than other words or phrases with the same meaning. This 29 also becomes evident in the relatively lower performance of word embeddings on the rare word 30 similarity task (Luong et al., 2013b). A recent approach by (Bojanowski et al., 2016a) proposes 31 using character n-gram representations to address the problem of out-of-vocabulary and rare words. 32 (Faruqui et al., 2014) also proposed retrofitting vectors to an ontology. However, these methods don’t 33 account for polysemy. 34 Polysemy is an important feature of language which causes words to have different meanings based 35 on the context in which they occur. For instance, the word ‘bank’ can mean ‘financial institution’ or 36 Submitted to 32nd Conference on Neural Information Processing Systems (NIPS 2018). Do not distribute. ‘land on either side of a river’. A well known drawback with word embeddings is the assignment of a 37 single vector representation to a word type, irrespective of polysemy. A large body of work has gone 38 into developing word sense disambiguation systems to identify the correct sense of a word based on 39 it’s context. The availability of such disambiguation systems coupled with the growing reliance of 40 NLP systems on distributional semantics has led to an increasing interest in obtaining powerful sense 41 representations. 42 Some of the previous work that has gone into learning sense representations includes unsupervised 43 learning techniques to cluster contexts and learn multi prototype vectors(Reisinger and Mooney 44 (2010) , Huang et al. (2012) and Wu and Giles (2015)). However, a common drawback with the 45 cluster based models is the difficulty in deciding the number of clusters apriori. ( Neelakantan et al. 46 (2015) , Tian et al. (2014) ,Cheng and Kartsaklis (2015) also learn multiple word embeddings by 47 modifying the Skip-Gram model. These approaches yield to sense representations that are limited in 48 terms of interpretability which makes it challenging to include in downstream tasks. 49 As a remedy to limitation in sense tagged corpora, Jauhar et al. (2015) and Rothe and Schütze (2015) 50 explored grounding word embeddings to ontologies to obtain sense representations. As a result, these 51 techniques drastically improved performance on several similarity tasks but an observed pattern is 52 that this leads to compromised performance on word relatedness tasks(Faruqui et al. (2014), Jauhar 53 et al. (2015)). We suspect this is a result of directly modifying word embedding spaces based on 54 ontology structure. 55 In this work, we present a novel approach that uses knowledge bases and sense representations to 56 directly induce polysemy to any predefined word embedding space. We show how our approach 57 allows the integration of ontological information and leads to improvements in both word similarity 58 and relatedness tasks. The advantages of this are plenty, it allows the integration of knowledge into 59 embedding spaces and can readily induce polysemy in them. We thus rely on a) Sense tagged corpora 60 to obtain contextualized sense representations. The objective of which is to capture sense relations 61 and interactions in naturally occurring corpora. The sense representations are interpretable and have 62 lexical mappings to a knowledge base. We use them to induce polysemy in word embedding spaces. 63 b) Pretrained word embeddings to capture many useful lexical relationships that are inherent in them 64 on account of being trained on large amounts of data. These relationships are not effectively captured 65 by sense representations due to the limited size of sense tagged corpora they are trained on. c) Lastly, 66 in order to account for the vocabulary bias which causes similar meaning words to be farther apart in 67 embedding spaces as a result of bias in co-occurrence statistics found in corpora, we use a knowledge 68 base to jointly ground word and sense representations. 69 We thus obtain unique multiple word sense representations that show superior performance in 70 similarity, relatedness and extrinsic tasks. They also show performance benefits when used with 71 transfer learning methods like CoVE (McCann et al. (2017)) and ELMo (Peters et al. (2018)) 72