Representation Degeneration Problem in Training Natural Language Generation Models

1 INTRODUCTION . Neural Network ( NN ) -based algorithms have made significant progresses in natural language generation tasks , including language modeling ( Kim et al. , 2016 ; Jozefowicz et al. , 2016 ) , machine translation ( Wu et al. , 2016 ; Britz et al. , 2017 ; Vaswani et al. , 2017 ; Gehring et al. , 2017 ) and dialog systems ( Shang et al. , 2015 ) . Despite the huge variety of applications and model architectures , natural language generation mostly relies on predicting the next word given previous contexts and other conditional information . A standard approach is to use a deep neural network to encode the inputs into a fixed-size vector referred as hidden state1 , which is then multiplied by the word embedding matrix ( Vaswani et al. , 2017 ; Merity et al. , 2018 ; Yang et al. , 2018 ; Press & Wolf , 2017 ) . The output logits are further consumed by the softmax function to give a categorical distribution of the next word . Then the model is trained through likelihood maximization . While studying the learnt models for language generation tasks , we observe some interesting and surprising phenomena . As the word embedding matrix is tied with the softmax parameters ( Vaswani et al. , 2017 ; Merity et al. , 2018 ; Inan et al. , 2017 ; Press & Wolf , 2017 ) , it has a dual role in the model , serving as the input in the first layer and the weights in the last layer . Given its first role as input word embedding , it should contain rich semantic information that captures the given context which will be further used for different tasks . Given its role as output softmax matrix , it should have enough capacity to classify different hidden states into correct labels . We compare it with ∗This work was done when Jun Gao was an intern at Microsoft Research Asia . 1The concept of hidden states has multiple meanings in the literature of neural networks . In this paper , we use hidden state as the input to the last softmax layer . word embeddings trained from Word2Vec ( Mikolov et al. , 2013 ) and the parameters in the softmax layer of a classical classification task ( we refer it as categorical embedding ) . As shown in Figure 1 , the word embeddings learnt from Word2Vec ( Figure 1 ( b ) ) and the softmax parameters learnt from the classification task ( Figure 1 ( c ) ) are diversely distributed around the origin using SVD projection ; in contrast , the word embeddings in our studied model ( Figure 1 ( a ) ) degenerated into a narrow cone . Furthermore , we find the embeddings of any two words in our studied models are positively correlated . Such phenomena are very different from those in other tasks and deteriorate model ’ s capacity . As the role of the softmax layer , those parameters can not lead to a large margin prediction for good generalization . As the role of word embeddings , the parameters do not have enough capacity to model the diverse semantics in natural languages ( Yang et al. , 2018 ; McCann et al. , 2017 ) . We call the problem described above the representation degeneration problem . In this paper , we try to understand why the problem happens and propose a practical solution to address it . We provide some intuitive explanation and theoretical justification for the problem . Intuitively speaking , during the training process of a model with likelihood loss , for any given hidden state , the embedding of the corresponding ground-truth word will be pushed towards the direction of the hidden state in order to get a larger likelihood , while the embeddings of all other words will be pushed towards the negative direction of the hidden state to get a smaller likelihood . As in natural language , word frequency is usually very low comparing to the size of a large corpus , the embedding of the word will be pushed towards the negative directions of most hidden states which drastically vary . As a result , the embeddings of most words in the vocabulary will be pushed towards similar directions negatively correlated with most hidden states and thus are clustered together in a local region of the embedding space . From the theoretical perspective , we first analyze the extreme case of non-appeared words . We prove that the representation degeneration problem is related to the structure of hidden states : the degeneration appears when the convex hull of the hidden states does not contain the origin and such condition is likely to happen when training with layer normalization ( Ba et al. , 2016 ; Vaswani et al. , 2017 ; Merity et al. , 2018 ) . We further extend our study to the optimization of low-frequency words in a more realistic setting . We show that , under mild conditions , the low-frequency words are likely to be trained to be close to each other during optimization , and thus lie in a local region . Inspired by the empirical analysis and theoretical insights , we design a novel way to mitigate the degeneration problem by regularizing the word embedding matrix . As we observe that the word embeddings are restricted into a narrow cone , we try to directly increase the size of the aperture of the cone , which can be simply achieved by decreasing the similarity between individual word embeddings . We test our method on two tasks , language modeling and machine translation . Experimental results show that the representation degeneration problem is mitigated , and our algorithm achieves superior performance over the baseline algorithms , e.g. , with 2.0 point perplexity improvement on the WikiText-2 dataset for language modeling and 1.08/0.93 point BLEU improvement on WMT 2014 English-German/German-English tasks for machine translation . 2 RELATED WORK . Language modeling and machine translation are important language generation tasks . Language modeling aims at predicting the next token given an ( incomplete ) sequence of words as context . A popular approach based on neural networks is to map the given context to a real-valued vector as a hidden state , and then pass the hidden state through a softmax layer to generate a distribution over all the candidate words . There are different choices of neural networks used in language modeling . Recurrent neural network-based model and convolutional neural network-based model are widely used ( Mikolov et al. , 2010 ; Dauphin et al. , 2017 ) . Neural Machine Translation ( NMT ) is a challenging task that has attracted lots of attention in recent years . Based on the encoder-decoder framework , NMT starts to show promising results in many language pairs . The evolving structures of NMT models in recent years have pushed the accuracy of NMT into a new level . The attention mechanism ( Bahdanau et al. , 2015 ) added on top of the encoder-decoder framework is shown to be useful to automatically find alignment structure , and single-layer RNN-based structure has evolved towards deep models ( Wu et al. , 2016 ) , more efficient CNN models ( Gehring et al. , 2017 ) , and well-designed self-attention models ( Vaswani et al. , 2017 ) . In this paper , we mainly study the expressiveness of word embeddings in language generation tasks . A trend for language generation in recent years is to share the parameters between word embeddings and the softmax layer , which is named as the weight tying trick . Many state-of-the-art results in language modeling and machine translation are achieved with this trick ( Vaswani et al. , 2017 ; Merity et al. , 2018 ; Yang et al. , 2018 ; Press & Wolf , 2017 ) . Inan et al . ( 2017 ) shows that weight tying not only reduces the number of parameters but also has theoretical benefits . 3 REPRESENTATION DEGENERATION PROBLEM . In this section , we empirically study the word embeddings learnt from sequence generation tasks , i.e. , machine translation , and introduce the representation degeneration problem in neural sequence generation . 3.1 EXPERIMENTAL DESIGN . Our analysis reported in this section is mainly based on the state-of-the-art machine translation model Transformer ( Vaswani et al. , 2017 ) . We use the official code ( Vaswani et al. , 2018 ) and set all the hyperparameters ( the configurations of the base Transformer ) as default . We train the model on the WMT 2014 English-German Dataset and achieve 27.3 in terms of BLEU score . Additionally , we also analyze the LSTM-based model ( Wu et al. , 2016 ) and find the observations are similar . In neural sequence generation tasks , the weights in word embeddings and softmax layer are tied . Those parameters can be recognized not only as the inputs in the first layer but also as the weights in the last layer . Thus we compare this weight matrix with a word embedding matrix trained from conventional word representation learning task , and also compare it with the parameters in softmax layer of a conventional classification task . For simplicity and representative , we choose to use Word2Vec ( Mikolov et al. , 2013 ) to obtain word embeddings and use MNIST as the classification task trained with a two-layer convolutional neural network ( Yue et al. , 2018 ; Liu et al. , 2016 ) . For the classification task , We simply treat the row of the parameter matrix in the last softmax layer as the embedding for the category , like how the word embedding is used in the softmax layer for neural language generation model with weight tying trick . To get a comprehensive understanding , we use low-rank approximation ( rank = 2 ) for the learned matrices by SVD and plot them in a 2D plane , as shown in Figure 1 . We also check the singular values distribution and find that our low-rank approximation is reasonable : other singular values are much smaller than the chosen ones . 3.2 DISCUSSION . In the classification task , the category embeddings ( blue triangles in Figure 1 ( c ) ) in the softmax layer are diversely distributed around the origin . This shows the directions of category embeddings are different from each other and well separated , which consequently leads to large margin classification results with good generalization . For the word embeddings learnt from Word2Vec ( Figure 1 ( b ) ) , the phenomena are similar , the embeddings are also widely distributed around the origin in the projection space , which shows different words have different semantic meanings . Observations on GLOVE ( Pennington et al. , 2014 ) reported in Mu et al . ( 2018 ) are also consistent with ours . We observe very different phenomena for machine translation . We can see from Figure 1 ( a ) that the word embeddings are clustered together and only lying in a narrow cone . Furthermore , we find the cosine similarities between word embeddings are positive for almost all cases . That is , the words huddle together and are not well separated in the embedding space . Clearly , as the role of word embeddings , which are the inputs to the neural networks , the word representations should be widely distributed to represent different semantic meanings . As the role of softmax in the output layer , to achieve good prediction of next word in a target sentence , a more diverse distribution of word embeddings in the space is expected to obtain a large margin result . However , such representations of words limit the expressiveness of the learnt model . We call such a problem the representation degeneration problem .

Summary: This work proposes a simple regularization term which penalize cosine similarity of word embedding parameters in the loss function. The motivation comes from empirical studies of word embedding parameters in three tasks, translation, word2vec and classification, and showed that the parameters for the translation task are not distributed when compared with other tasks. The problem is hypothesized by the rare word problem especially when parameters are tied for softmax and input embedding, and proposes a cosine similarity regularization. Experiments on English/German show consistent gains over non-regularized loss.