Improved Learning through Augmenting the Loss

We present two improvements to the well-known Recurrent Neural Network Language Models(RNNLM). First, we use the word embedding matrix to project the RNN output onto the output space and already achieve a large reduction in the number of free parameters while still improving performance. Second, instead of merely minimizing the standard cross entropy loss between the prediction distribution and the ”one-hot” target distribution, we minimize an additional loss term which takes into account the inherent metric similarity between the target word and other words. We show with experiments on the Penn Treebank Dataset that our proposed model (1) achieves significantly lower average word perplexity than previous models with the same network size and (2) achieves the new state of the art by using much fewer parameters than used in the previous best work. 1 Background & Introduction Language Modeling, which is the task of determining the probability that a sequence of words occur in a sentence, is an important task in Natural Language Processing(NLP). Its scope includes a large variety of tasks such as speech recognition, machine translation, part of speech tagging, and spelling correction [1]. Old statistical approaches for addressing this problem were based on n-grams and their associated properties. Although these methods seemed to work reasonably well, they needed a larger value of n in order to capture a better model for the language, and they needed a number of parameters exponential in the size of the vocabulary [2]. Researchers have proposed methods to improve these models [1]; nonetheless, they are still difficult to implement in practice. Recently, with the advances in neural network research and the increased availability of faster computers, neural networks have started to replace the old methods for capturing the language structure. Recurrent Neural Networks(RNN) were first proposed in [3] for addressing the language modelling task and attained a reasonably satisfactory performance. The follow-up work [4] improved the result by applying back-propagation in addition to the existing forward-propagation. In [5], Zaremba et. al. used Long Short-Term Memory(LSTM) which were fist introduced in [6], as the building block of their recurrent neural network for language modelling. Furthermore, the authors showed that appropriat application of dropout may avoid overfitting and lead to a better and more generalizable model. The authors were further able to achieve the state of the art for language modeling on the Penn Dataset Treebank corpus[7]. In particular, by training a two-layer LSTM Recurrent Neural Network with 1500 hidden states and appropriate dropout, they could reduce the test perplexity to 78.4 for a single model. However, Recurrent Neural Network Language Models need big models for proper training (2 RNN layers with 1500 hidden units in each in [5]). In addition, their performance rely on training two very large matrices, one for word embeddings and one for output projection, both of which essentially capture the same thing: projection from token space to word vector space and vice versa. In this paper we explain how imposing this structure may help us to achieve a much better accuracy even on smaller networks. We also propose a novel method for augmenting the loss function with ad additional term which utilizes the similarity between words captured in word embeddings.