Learning Context-Specific Word/Character Embeddings

Unsupervised word representations have demonstrated improvements in predictive generalization on various NLP tasks. Most of the existing models are in fact good at capturing the relatedness among words rather than their “genuine” similarity because the context representations are often represented by a sum (or an average) of the neighbor’s embeddings, which simplifies the computation but ignores an important fact that the meaning of a word is determined by its context, reflecting not only the surrounding words but also the rules used to combine them (i.e. compositionality). On the other hand, much effort has been devoted to learning a singleprototype representation per word, which is problematic because many words are polysemous, and a single-prototype model is incapable of capturing phenomena of homonymy and polysemy. We present a neural network architecture to jointly learn word embeddings and context representations from large data sets. The explicitly produced context representations are further used to learn context-specific and multiprototype word embeddings. Our embeddings were evaluated on several NLP tasks, and the experimental results demonstrated the proposed model outperformed other competitors and is applicable to intrinsically “character-based” languages.  Much recent research has been devoted to deep learning algorithms which achieved impressive results on various natural language processing (NLP) tasks. The best results obtained on supervised learning tasks involve an unsupervised learning phase, usually in an unsupervised pre-training step to learn distributed word representations (also known as word embeddings). Such semi-supervised learning strategy has been empirically proven to be successful by using unlabeled data to supplement the supervised models for better generalization (Collobert et al. 2011; Socher et al. 2011; dos Santos and Zadrozny 2014; Zheng, Chen, and Xu 2013; Pei, Ge, and Chang 2014). Due to the importance of unsupervised pre-training in deep neural network methods, many models have been proposed to learn word embeddings. The two main model families are: (a) predicting or scoring the target word based on its local context, such as the neural probabilistic language ), and continuous bag-of-words (CBOW) (Mikolov et al. 2013a); (b) using a word to predict its surrounding words, such as the Skip-gram (Mikolov et al. 2013a) as well as its extensions, multi-sense Skip-gram (MSSG) (Neelakantan et al. 2014) and proximity-ambiguity sensitive (PAS) Skipgram (Qiu et al. 2014). However, they suffer one or both of the following significant drawbacks. First, the context is often represented by a sum (or an average) of the surrounding words’ feature vectors. Those context representations may not represent word meanings well because the meaning of a word is affected by its adjacent words and the rules to combine them. Each context word in general does not contribute equally to the meaning of the target word, and thus the simple sum or average operation can not capture this compositionality. Second, most of the existing methods create a singleprototype embedding for each word. This single-prototype representation is problematic because many words are intrinsically polysemous, and a single-prototype model is incapable of capturing phenomena of homonymy and polysemy. It is also important to be reminded that languages differ in the degree of polysemy they exhibit. For example, Chinese with its (almost) complete lack of morphological marking for parts of speech certainly exhibits a higher degree of polysemy than English (Packard 2004). Recent models on learning multiple representations per word (Reisinger and Mooney 2010; Huang et al. 2012) generally work as follows: for each word, first cluster its contexts into a set of clusters, and then derive multiple representations (each for a cluster) of the word from the clusters of similar contexts. In these models, each context is represented by a vector composed of the occurrences of its neighboring words or a weighted average of the surrounding words’ vector. Such context definitions neglect the relative order of words in the context window, which impairs the quality of the multi-prototype representations derived by the clustering based on such context representations. In fact, the order of words does matter to the meaning of those they form. Besides, those models set the equal number of prototypes for each word, although different numbers were tested to determine the optimal number of prototypes. We argue that the degree of polysemy in polysemous words depends on the number of distinct contexts in which they occur, especially for the “character-based” languages (e.g. Chinese and Japanese). When a word appears in more different linguistic contexts, it may carry more meanings, and greater number of prototypes should be created for that word. It is unnecessary for rare words to learn several prototypes, whereas for most common words, if the number of word senses is greater than that of prototypes, certain prototype could be affected by the contexts associated with different meanings, and the learned prototype might not represent any one of the meanings well as it is influenced by several (and different) meanings of that word. We present a novel neural network architecture to learn multi-prototype word/character embeddings from large data sets, in which the context representations are produced by a convolutional layer, designed to capture the compositionality of context words or characters. This architecture allows us to take advantage of the better trained context representations and to learn multi-prototype word/character embeddings accordingly. The number of prototypes for each word/character can be different and selected automatically. Experimental results show that the embeddings learned by our model outperform competitors on the several NLP tasks across different languages by transferring the unsupervised representations into the supervised models. Specific Vector Model Many methods were proposed to learn distributed word representations from large unlabeled texts, but the questions still remain as to how meaning is produced by such unsupervised methods, and how the resulting word vectors represent that meaning. We describe here a neural networkbased model, named CSV (Context-Specific Vector), which can generate the context representation of a word/character, and learn the word/character vector carrying the meaning inferred by that context. The proposed network architecture contains a convolutional layer that is designed to produce the refined context representations reflecting the order of their constituents and the rules to combine them. The better generated context representations are used to learn contextspecific multi-prototype word/character embeddings by the sense induction with a simple “winner-takes-all” strategy. The Neural Network Architecture The network architecture is shown in Figure 1. From now on, a term “type” is used to refer to word or character, depending on which language is considered (e.g. word in English or character in Chinese). Each type is associated with a global vector and multiple sense vectors. The input to the network is a type’s context window, and the convolutional layer produces the representation (or vector) for that context. One of the type’s sense vectors is chosen by how well the sense fits into the context, and the network is trained to differentiate the selected sense vector from others. We use a window approach that assumes the meaning of a type depends mainly on its surrounding types. The types (except the target) in the window of size w (a hyper parameter) are fed into the network as indices that are used by a lookup operation to transform types into their global vectors. We consider a fixed-sized type dictionary D. The type’s global vectors are stored in a matrixM ∈ Rd×|D|, where d is the dimensionality of the vector space (a hyper-parameter to be chosen) and |D| is the size of the dictionary. Convolution • • • g(t−w/2) • • • Word/Character Context