Incrementally Learning the Hierarchical Softmax Function for Neural Language Models

Neural network language models (NNLMs) have attracted a lot of attention recently. In this paper, we present a training method that can incrementally train the hierarchical softmax function for NNMLs. We split the cost function to model old and update corpora separately, and factorize the objective function for the hierarchical softmax. Then we provide a new stochastic gradient based method to update all the word vectors and parameters, by comparing the old tree generated based on the old corpus and the new tree generated based on the combined (old and update) corpus. Theoretical analysis shows that the mean square error of the parameter vectors can be bounded by a function of the number of changed words related to the parameter node. Experimental results show that incremental training can save a lot of time. The smaller the update corpus is, the faster the update training process is, where an up to 30 times speedup has been achieved. We also use both word similarity/relatedness tasks and dependency parsing task as our benchmarks to evaluate the correctness of the updated word vectors. Neural network language models (NNLMs) (Bengio et al. 2003; Morin and Bengio 2005; Mnih and Hinton 2008; Turian, Ratinov, and Bengio 2010; Collobert et al. 2011; Mikolov et al. 2013a; Mikolov et al. 2013b; Levy and Goldberg 2014; Levy, Goldberg, and Dagan 2015) have attracted a lot of attention recently given their compact representation form and generalization property compared to the traditional lexical representations. It has been applied to many natural language processing tasks such as word similarity/relatedness and word analogy (Mikolov et al. 2013a; Mikolov et al. 2013b), working as features for part-of-speech tagging, chunking, named entity recognition, etc. (Turian, Ratinov, and Bengio 2010; Collobert et al. 2011). However, the above NNLMs only consider the static training corpus. There are a lot of applications, such as news and tweets text processing, requiring incremental update of the word vectors given the fact that the working domains are fast evolving. When the new corpus is relatively smaller than the old corpus, it will be much less efficient to retrain the word vectors with the combined corpus. In this paper, we consider the problem of training the NNLMs given new corpora incrementally. In particular,  we adopt the popular word2vec tool due to its simplicity and time efficiency, and comparable performance to other NNLMs (Mikolov et al. 2013a; Mikolov et al. 2013b; Levy, Goldberg, and Dagan 2015). To speedup the process of indexing and querying, word2vec employed two techniques called hierarchical softmax and negative sampling (Mikolov et al. 2013a; Mikolov et al. 2013b). Hierarchical softmax was first proposed by Mnih and Hinton (Mnih and Hinton 2008) where a hierarchical tree is constructed to index all the words in a corpus as leaves, while negative sampling is developed based on noise contrastive estimation (Gutmann and Hyvärinen 2012), and randomly samples the words not in the context to distinguish the observed data from the artificially generated random noise. It is empirically shown that hierarchical softmax performs better for infrequent words while negative sampling performs better for frequent words (Mikolov et al. 2013b). The reason is that hierarchical softmax builds a tree over the whole vocabulary, and the leaf nodes representing rare words will inevitably inherit their ancestors’ vector representations in the tree, which can be affected by other frequent words in the corpus. Thus, we choose hierarchical softmax due to its good performance on rare words, which can benefit the further incremental training for new corpus. When applying hierarchical softmax to NNLMs, there is a preprocessing step to build a hierarchical tree of words. When we incrementally incorporate more data, the hierarchical tree should be changed to reflect the change of data. For example, in word2vec, it uses the Huffman coding to construct the tree over the vocabulary because Huffman tree uses shorter codes for more frequent words with less visits to the leaves (Gutmann and Hyvärinen 2012), and results in faster training process. When frequencies of words change, Huffman tree will be changed. To handle this problem, we retain the paths from root to leaves of the old tree, and perform prefix matching between the old tree and the new tree. When updating the vectors for the matched ancestors of a leaf, we follow the original CBOW (Continuous Bag-of-Words) or Skip-gram algorithms in word2vec based on stochastic gradient ascent of log-likelihood. When updating the vectors for the different ancestors, we modify the old tree with stochastic gradient descent, while update the new tree with stochastic gradient ascent. In this way, we only modify all the nodes that need to be updated while retaining all the other nodes as the same as the ones trained based on the old corpus. Since the update process is independent for all the internal nodes, we also develop a parallel algorithm similar to word2vec, which makes our algorithm efficient when having more CPUs. In the experiments, we show that using this updating procedure, we can get almost the same CBOW and Skip-gram models as fully re-trained ones. We check both individual vector’s mean square error (MSE) and down-stream applications, word similarity/relatedness and dependency parsing, to prove the correctness of our model. In addition, we also provide a bound that can characterize our algorithm’s difference from fully re-trained models.