Distributional initialization of neural networks

In Natural Language Processing (NLP), together with speech, text is one of the main sources of information. Computational systems that process raw text need to perform a transformation of text input into machine-readable format. Final performance of the NLP systems depends on the quality of these input representations, that is why the main objective for representation learning is to keep and highlight important features of the input tokens (characters, words, phrases, etc.). Traditionally, for Neural Networks (NNs) such input representations are one-hot vectors, where each word is represented with a vector of all-but-one zeros, with value 1 on the position that corresponds to the index of the word in the vocabulary. Such a representation only helps to differentiate words, but does not contain any usable information about relations between them. Word representations that are learned by NNs - word embeddings - are then arranged in a matrix, where each row corresponds to a particular word in a vocabulary and is retrieved by multiplication of the corresponding one-hot vector and the embedding matrix. These word embeddings are initialized randomly, and during training adjust their values to capture the contextual semantic information with respect to the training objective. When a word is frequent, it is seen often during training and its representation is updated frequently; for the same reason embeddings for rare words experience much less updates. This makes it difficult for NNs to learn good word embeddings for words that occur just several times in a corpus. In this work, we propose a method to improve quality of word embeddings of rare words. The main idea is to initialize a NN that learns embeddings with sparse distributional vectors that are precomputed for rare words from a given corpus. We introduce and investigate several methods for building such distributional representations: with different ways to combine one-hot representations of frequent and distributional representations of rare words, different similarity functions between distributional vectors, different normalization approaches applied to the representations in order to control the input signals' amplitude. We evaluate the performance of our proposed models on two tasks. On a word similarity judgment task, the embeddings of words are used to compute similarity scores between two words in given pairs; then these similarity scores are compared with human ratings. With use of the same NN architecture, word embeddings that are trained using distributional initialization show significantly better performance than word embeddings trained with traditional one-hot initialization. On language modeling task, where models compete in predicting probability of a given sequence of words, models with distributional initialization show minor improvements over models with one-hot initialization. We also study a very popular word2vec tool (Mikolov et al., 2013a) that is used to obtain word embeddings without supervision. The main question we ask is how much the quality of learned word embeddings depends on the initial random seed. The obtained results suggest that training with word2vec is stable and reliable.