Improving Neural Language Generation with Spectrum Control

1 INTRODUCTION . Neural language generation ( NLG ) is an important task with many practical applications , such as automatic speech recognition ( Graves et al. , 2013 ; Toshniwal et al. , 2018 ) , text generation ( Bowman et al. , 2016 ; Radford et al. , 2019 ; Keskar et al. , 2019 ) , machine translation ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) and dialog systems ( Gao et al. , 2019a ; Tang et al. , 2019 ) . Most NLG models utilize a complex encoding model to map a given context into a hidden state vector , and then predict the next word distribution by multiplying the encoded vector with the output embedding layer , followed by a softmax layer . In the past few years , it has witnessed a significant progress in NLG by improving the encoding model , from the recurrent neural network ( RNN ) ( Bahdanau et al. , 2015 ; Jozefowicz et al. , 2016 ; Merity et al. , 2018a ) based models to the current Transformer-based models ( Vaswani et al. , 2017 ; Devlin et al. , 2019 ; Dai et al. , 2019 ; Radford et al. , 2019 ) . However , embeddings in the softmax output layer have been shown not capable enough to model the conditional probability ( Yang et al. , 2018 ) . Recently , Gao et al . ( 2019b ) pointed out another limitation of the output embeddings : the representation degeneration problem . They showed that the singular value distribution of the output embedding matrix tends to decay very fast , and the embedding space is squeezed into a narrow cone ( as shown in Figure 1 ( a ) and 1 ( c ) in 2-D plots ) . Such anisotropic shape ( Ethayarajh , 2019 ) is very different from what one would expect from an expressive word embedding space ( Arora et al. , 2016a ; Mu & Viswanath , 2018 ) . Therefore , several efforts ( Gao et al. , 2019b ; Wang et al. , 2019a ) have been made to address the degeneration problem . Unlike previous approaches that applied implicit regularization to singular values of the output embedding matrix , we propose a spectrum control ( SC ) approach , which was inspired by the spectral control technique used for Generative Adversarial Network ( GAN ) training ( Jiang et al. , 2019 ) , to explicitly control the singular value distribution . We first reparameterize the output embedding matrix W by its singular value decomposition ( SVD ) : W = UΣV > , where U , V are column orthonormal matrices , and Σ is a diagonal matrix of singular values . Then we guide the training of Σ by a predefined slow-decaying prior distribution , such as a polynomial decay distribution , or an exponential decay distribution . At the end of training , the distribution of singular values of the embedding matrix gets close to the prior distribution . Our spectrum control approach alleviates the representation degeneration problem by encouraging the diversity of word representations and improving isotropic property of these representations ( see Figure 2 ) , even on top of the powerful Transformer-based models . We further present a theoretical analysis to justify our method . The results suggest that it is beneficial to directly guide the singular value distribution of the embedding matrix throughout the training process and control the decay rate of these singular values . We demonstrate the effectiveness of our training framework with extensive experimental results on two tasks : language modeling and machine translation . Our spectrum control method outperforms the latest state-of-the-art TransformerXL model on WikiText-103 dataset for language modeling ; and obtains close to 1.5 BLEU improvement on IWSLT 2014 German-English translation task compared to the Transformer baseline model . 2 RELATED WORK . In this paper , we mainly focus on the output embedding matrix that is used in the softmax output layer for language generation tasks . This embedding matrix is also used as input word embeddings , which is known as weight tying trick . The weight tying not only reduces the number of parameters but also enjoys theoretical benefits ( Inan et al. , 2017 ) . Thus the weight tying has been successfully applied to many state-of-the-art models for language modeling and machine translation ( Merity et al. , 2018a ; Vaswani et al. , 2017 ; Yang et al. , 2018 ) . However , there are certain issues with the softmax output layer . Yang et al . ( 2018 ) first identified a problem called “ softmax bottleneck ” that the softmax output layer does not have enough capacity to model natural language due to its connection with the rank bottleneck in matrix factorization . A simple and effective method called Mixture of Softmaxes ( MoS ) was proposed to deal with this issue . Follow-up work in Kanai et al . ( 2018 ) ; Ganea et al . ( 2019 ) tried to replace softmax with alternative activation functions . Kanai et al . ( 2018 ) proposed to use sigsoftmax , which is composed of a multiplication of an exponential function and sigmoid function . Ganea et al . ( 2019 ) proposed a Linear-Monotonic-Softmax ( LMS ) model that generalized the approach in Kanai et al . ( 2018 ) by learning parametric point-wise increasing functions to optimally distort the logits before feeding them into the softmax layer . Pappas & Henderson ( 2019 ) instead resorted to a powerful deep 1Note that we project the original word embeddings to a 2 dimensional vector space using principal component analysis ( PCA ) for the purpose of visualization . residual nonlinear output mapping while using a single softmax function without modifying its dimensionality or rank . Another line of work focuses on increasing the expressive power of the output embedding matrix by adding a cosine similarity regularization term ( Gao et al. , 2019b ) , or adversarial noise ( Wang et al. , 2019a ) . Gao et al . ( 2019b ) analyzed the representation degeneration problem that the output embeddings tend to degenerate and be distributed into a narrow cone . A regularization term based on the summation of pairwise cosine similarity among all words was proposed to increase the representation power of word embeddings . Wang et al . ( 2019a ) pointed out the computation of regularization term in ( Gao et al. , 2019b ) depends on the size of the vocabulary and hence is costly . Instead , they proposed a simple yet highly effective adversarial training method that adds adversarial noises to the output embedding layer when training the models . They proved in theory that this adversarial training increases the distances between two different words , and thus encourages the diversity of the embedding vectors . Our work follows this line of thought , but takes a different approach : motivated by the spectrum control method for GAN training ( Jiang et al. , 2019 ) , we propose to directly guide the singular values by a slow-decaying prior distribution during the model training process . We show that the anisotropic behavior of the contextualized word representations from powerful Transformer-based models ( Ethayarajh , 2019 ) are alleviated by our spectrum control approach . 3 PROBLEM SETUP . In this section , we briefly introduce the neural models for language generation , and illustrate the singular value decay phenomena of existing neural language models . We first introduce some notations used in the rest of the paper . Notation : For a d-dimensional vector x ∈ Rd , we use ‖x‖q = ( ∑d i=1 |xi|q ) 1/q , where 0 < q < ∞ to denote its ` q-norm , and ‖x‖∞ = maxi |xi| to be its infinity norm . For a matrix A ∈ Rd1×d2 , let Ai∗ be the i-th row of A , and we use ‖A‖2 , ‖A‖F , ‖A‖1 to denote its spectral norm , Frobenius norm , and matrix 1-norm . Given two sequences { an } and { bn } , if there exists a constant 0 < C < ∞ such that an ≤ Cbn , we write an = O ( bn ) , and we use Õ ( · ) to hide the logarithmic factors . 3.1 NEURAL LANGUAGE GENERATION . We first briefly review the softmax output layer typically used in neural language generation models . We define the joint probability of a given length-n sentence of words ( tokens ) sn = ( y1 , . . . , yn ) as the following product of conditional probabilities P ( sn ) = n∏ t=1 P ( yt|ct ) , ( 3.1 ) where yt ∈ V is the t-th word in sentence sn , and V represents the word vocabulary , ct = y1 : t−1 = ( y1 , . . . , yt−1 ) is referred to as the context of word yt . In addition , the context ct is usually modeled by a fixed size vector ht ∈ Rd , which is referred to as the hidden state , using some neural networks such as LSTM ( Hochreiter & Schmidhuber , 1997 ) and Transformer ( Vaswani et al. , 2017 ) . Then , the probability distribution of the output word yt given the context ct , i.e. , P ( yt|ct ) in ( 3.1 ) , is parameterized as the following softmax function : P ( Yt = yt|ct ) = P ( Yt = yt|ht ) = exp ( h > t WI ( yt ) ∗ ) ∑N i=1 exp ( h > t Wi∗ ) , ( 3.2 ) where W ∈ RN×d is the weight matrix and is usually tied with the input word embedding matrix ( Press & Wolf , 2017 ; Inan et al. , 2017 ) , N = |V| is the vocabulary size , d is the embedding dimension , I ( yt ) represents the index of word yt in vocabulary V . In the following discussion , we call output weight matrix W as the word embedding matrix of the neural language model since it is tied with the input word embedding matrix . In this paper we focus on the output layer of the neural model for language generation , i.e. , the softmax layer in ( 3.2 ) , as in ( Yang et al. , 2018 ; Kanai et al. , 2018 ; Ganea et al. , 2019 ; Gao et al. , 2019b ) . We will examine the singular value distribution of W and propose a different approach to increase the expressive power of the word embedding . 3.2 FAST SINGULAR VALUE DECAY . As we mentioned before , the singular values of W tend to drop very fast and it may interact with the cone-shaped embedding space . More specifically , Figure 1 ( b ) and 1 ( d ) illustrate the distributions of the normalized singular values of the Transformer-XL based language model2 ( Dai et al. , 2019 ) and the Transformer-based machine translation model ( Vaswani et al. , 2017 ) trained on WikiText103 ( Merity et al. , 2018a ) and IWSLT 2014 De-En ( Cettolo et al. , 2014 ) datasets , respectively . The plots show a fast singular value decay phenomenon , i.e. , there is a huge drop between the first and remaining singular values . Such a phenomenon has also been observed in some previous work ( Gao et al. , 2019b ; Wang et al. , 2019a ) . Figure 1 ( a ) and 1 ( c ) present the distributions of the projected word embeddings of the aforementioned two models . We can see from the plots that the projected word embeddings are distributed into some narrow cone shapes , which implies an anisotropic property of the learned word representations , i.e. , the embedding vectors are not uniformly distributed in the space . The detailed analysis in Ethayarajh ( 2019 ) also confirms that contextualized word embeddings learnt from ELMo ( Peters et al. , 2018 ) ( LSTM-based model ) , BERT and GPT-2 ( Devlin et al. , 2019 ; Radford et al. , 2019 ) ( Transformer-based models ) indeed tend to be anisotropic . These contextualized word embeddings have shown great success on many NLP tasks . However , static word embeddings , such as Word2Vec ( Mikolov et al. , 2013 ) and GloVe ( Pennington et al. , 2014 ) have been shown ( Arora et al. , 2016b ; Mu & Viswanath , 2018 ) to be isotropic with great expressive power . Hence it may be also beneficial to increase the expressive power of the contextualized word embedding by increasing its isotropy . This motivates us to alleviate the fast singular value decay phenomenon to increase the isotropy of the learned word representations .

Summary: This paper deals with the representation degeneration problem in neural language generation, as some prior works have found that the singular value distribution of the (input-output-tied) word embedding matrix decays quickly. The authors proposed an approach that directly penalizes deviations of the SV distribution from the two prior distributions, as well as a few other auxiliary losses on the orthogonality of U and V (which are now learnable). The experiments were conducted on small and large scale language modeling datasets as well as the relatively small IWSLT 2014 De-En MT dataset.