Title: IMPROVING NEURAL LANGUAGE GENERATION WITH SPECTRUM CONTROL

Abstract: Recent Transformer-based models such as Transformer-XL and BERT have achieved huge success on various natural language processing tasks. However, contextualized embeddings at the output layer of these powerful models tend to degenerate and occupy an anisotropic cone in the vector space, which is called the representation degeneration problem. In this paper, we propose a novel spectrum control approach to address this degeneration problem. The core idea of our method is to directly guide the spectra training of the output embedding matrix with a slow-decaying singular value prior distribution through a reparameterization framework. We show that our proposed method encourages isotropy of the learned word representations while maintains the modeling power of these contextual neural models. We further provide a theoretical analysis and insight on the benefit of modeling singular value distribution. We demonstrate that our spectrum control method outperforms the state-of-the-art Transformer-XL modeling for language model, and various Transformer-based models for machine translation, on common benchmark datasets for these tasks.

1. INTRODUCTION:

Neural language generation (NLG) is an important task with many practical applications, such as automatic speech recognition (Graves et al., 2013; Toshniwal et al., 2018 ), text generation (Bowman et al., 2016; Radford et al., 2019; Keskar et al., 2019) , machine translation (Bahdanau et al., 2015; Vaswani et al., 2017) and dialog systems (Gao et al., 2019a; Tang et al., 2019) . Most NLG models utilize a complex encoding model to map a given context into a hidden state vector, and then predict the next word distribution by multiplying the encoded vector with the output embedding layer, followed by a softmax layer. In the past few years, it has witnessed a significant progress in NLG by improving the encoding model, from the recurrent neural network (RNN) (Bahdanau et al., 2015; Jozefowicz et al., 2016; Merity et al., 2018a) based models to the current Transformer-based models (Vaswani et al., 2017; Devlin et al., 2019; Dai et al., 2019; Radford et al., 2019) . However, embeddings in the softmax output layer have been shown not capable enough to model the conditional probability (Yang et al., 2018) . Recently, Gao et al. (2019b) pointed out another limitation of the output embeddings: the representation degeneration problem. They showed that the singular value distribution of the output embedding matrix tends to decay very fast, and the embedding space is squeezed into a narrow cone (as shown in Figure 1 (a) and 1(c) in 2-D plots). Such anisotropic shape (Ethayarajh, 2019) is very different from what one would expect from an expressive word embedding space (Arora et al., 2016a; Mu & Viswanath, 2018) . Therefore, several efforts (Gao et al., 2019b; Wang et al., 2019a) have been made to address the degeneration problem. Unlike previous approaches that applied implicit regularization to singular values of the output embedding matrix, we propose a spectrum control (SC) approach, which was inspired by the spectral control technique used for Generative Adversarial Network (GAN) training (Jiang et al., 2019) , to explicitly control the singular value distribution. We first reparameterize the output embedding matrix W by its singular value decomposition (SVD): W = UΣV , where U, V are column orthonormal matrices, and Σ is a diagonal matrix of singular values. Then we guide the training of Σ by a predefined slow-decaying prior distribution, such as a polynomial decay distribution, or an exponential decay distribution. At the end of training, the distribution of singular values of the embedding matrix gets close to the prior distribution. Our spectrum control approach alleviates the representation degeneration problem by encouraging the diversity of word representations and improving isotropic property of these representations (see Figure 2 ), even on top of the powerful Transformer-based models. We further present a theoretical analysis to justify our method. The results suggest that it is beneficial to directly guide the singular value distribution of the embedding matrix throughout the training process and control the decay rate of these singular values. We demonstrate the effectiveness of our training framework with extensive experimental results on two tasks: language modeling and machine translation. Our spectrum control method outperforms the latest state-of-the-art Transformer-XL model on WikiText-103 dataset for language modeling; and obtains close to 1.5 BLEU improvement on IWSLT 2014 German-English translation task compared to the Transformer baseline model.

2. RELATED WORK:

In this paper, we mainly focus on the output embedding matrix that is used in the softmax output layer for language generation tasks. This embedding matrix is also used as input word embeddings, which is known as weight tying trick. The weight tying not only reduces the number of parameters but also enjoys theoretical benefits (Inan et al., 2017) . Thus the weight tying has been successfully applied to many state-of-the-art models for language modeling and machine translation (Merity et al., 2018a; Vaswani et al., 2017; Yang et al., 2018) . However, there are certain issues with the softmax output layer. Yang et al. (2018) first identified a problem called "softmax bottleneck" that the softmax output layer does not have enough capacity to model natural language due to its connection with the rank bottleneck in matrix factorization. A simple and effective method called Mixture of Softmaxes (MoS) was proposed to deal with this issue. Follow-up work in Kanai et al. (2018) ; Ganea et al. (2019) tried to replace softmax with alternative activation functions. Kanai et al. (2018) proposed to use sigsoftmax, which is composed of a multiplication of an exponential function and sigmoid function. Ganea et al. (2019) proposed a Linear-Monotonic-Softmax (LMS) model that generalized the approach in Kanai et al. (2018) by learning parametric point-wise increasing functions to optimally distort the logits before feeding them into the softmax layer. Pappas & Henderson (2019) instead resorted to a powerful deep residual nonlinear output mapping while using a single softmax function without modifying its dimensionality or rank. Another line of work focuses on increasing the expressive power of the output embedding matrix by adding a cosine similarity regularization term (Gao et al., 2019b) , or adversarial noise (Wang et al., 2019a) . Gao et al. (2019b) analyzed the representation degeneration problem that the output embeddings tend to degenerate and be distributed into a narrow cone. A regularization term based on the summation of pairwise cosine similarity among all words was proposed to increase the representation power of word embeddings. Wang et al. (2019a) pointed out the computation of regularization term in (Gao et al., 2019b) depends on the size of the vocabulary and hence is costly. Instead, they proposed a simple yet highly effective adversarial training method that adds adversarial noises to the output embedding layer when training the models. They proved in theory that this adversarial training increases the distances between two different words, and thus encourages the diversity of the embedding vectors. Our work follows this line of thought, but takes a different approach: motivated by the spectrum control method for GAN training (Jiang et al., 2019) , we propose to directly guide the singular values by a slow-decaying prior distribution during the model training process. We show that the anisotropic behavior of the contextualized word representations from powerful Transformer-based models (Ethayarajh, 2019) are alleviated by our spectrum control approach.

3. PROBLEM SETUP:

In this section, we briefly introduce the neural models for language generation, and illustrate the singular value decay phenomena of existing neural language models. We first introduce some notations used in the rest of the paper. Notation: For a d-dimensional vector x ∈ R d , we use x q = ( d i=1 |x i | q ) 1/q , where 0 < q < ∞ to denote its q -norm, and x ∞ = max i |x i | to be its infinity norm. For a matrix A ∈ R d1×d2 , let A i * be the i-th row of A, and we use A 2 , A F , A 1 to denote its spectral norm, Frobenius norm, and matrix 1-norm. Given two sequences {a n } and {b n }, if there exists a constant 0 < C < ∞ such that a n ≤ Cb n , we write a n = O(b n ), and we use O(•) to hide the logarithmic factors.

3.1. NEURAL LANGUAGE GENERATION:

We first briefly review the softmax output layer typically used in neural language generation models. We define the joint probability of a given length-n sentence of words (tokens) s n = (y 1 , . . . , y n ) as the following product of conditional probabilities

EQUATION

where y t ∈ V is the t-th word in sentence s n , and V represents the word vocabulary, c t = y 1:t-1 = (y 1 , . . . , y t-1 ) is referred to as the context of word y t . In addition, the context c t is usually modeled by a fixed size vector h t ∈ R d , which is referred to as the hidden state, using some neural networks such as LSTM (Hochreiter & Schmidhuber, 1997) and Transformer (Vaswani et al., 2017) . Then, the probability distribution of the output word y t given the context c t , i.e., P(y t |c t ) in (3.1), is parameterized as the following softmax function:

P(Y t = y t |c t ) = P(Y t = y t |h t ) = exp(h t W I(yt) * ) N i=1 exp(h t W i * ) , (3.2)

where W ∈ R N ×d is the weight matrix and is usually tied with the input word embedding matrix (Press & Wolf, 2017; Inan et al., 2017) , N = |V| is the vocabulary size, d is the embedding dimension, I(y t ) represents the index of word y t in vocabulary V. In the following discussion, we call output weight matrix W as the word embedding matrix of the neural language model since it is tied with the input word embedding matrix.

In this paper we focus on the output layer of the neural model for language generation, i.e., the softmax layer in (3.2), as in (Yang et al., 2018; Kanai et al., 2018; Ganea et al., 2019; Gao et al., 2019b) . We will examine the singular value distribution of W and propose a different approach to increase the expressive power of the word embedding.

3.2. FAST SINGULAR VALUE DECAY:

As we mentioned before, the singular values of W tend to drop very fast and it may interact with the cone-shaped embedding space. More specifically, Figure 1 (b) and 1(d) illustrate the distributions of the normalized singular values of the Transformer-XL based language model2 (Dai et al., 2019) and the Transformer-based machine translation model (Vaswani et al., 2017) trained on WikiText-103 (Merity et al., 2018a) and IWSLT 2014 De-En (Cettolo et al., 2014) datasets, respectively. The plots show a fast singular value decay phenomenon, i.e., there is a huge drop between the first and remaining singular values. Such a phenomenon has also been observed in some previous work (Gao et al., 2019b; Wang et al., 2019a) .

Figure 1 (a) and 1(c) present the distributions of the projected word embeddings of the aforementioned two models. We can see from the plots that the projected word embeddings are distributed into some narrow cone shapes, which implies an anisotropic property of the learned word representations, i.e., the embedding vectors are not uniformly distributed in the space. The detailed analysis in Ethayarajh ( 2019) also confirms that contextualized word embeddings learnt from ELMo (Peters et al., 2018) (LSTM-based model), BERT and GPT-2 (Devlin et al., 2019; Radford et al., 2019) (Transformer-based models) indeed tend to be anisotropic. These contextualized word embeddings have shown great success on many NLP tasks. However, static word embeddings, such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) have been shown (Arora et al., 2016b; Mu & Viswanath, 2018 ) to be isotropic with great expressive power. Hence it may be also beneficial to increase the expressive power of the contextualized word embedding by increasing its isotropy. This motivates us to alleviate the fast singular value decay phenomenon to increase the isotropy of the learned word representations.

4. PROPOSED METHOD:

To alleviate the fast singular value decay phenomenon, we propose to guide the singular value distribution of the contextualized word embedding throughout the training. As a result, we can achieve a trade-off between modeling contextual information, that tends to make word representations anisotropic, and the expressive power of word representations, that tends to be isotropic.

4.1. SVD REPARAMETERIZATION:

Following the previous work (Jiang et al., 2019) , we propose to apply singular value decomposition (SVD) based reparameterization to the embedding matrix W, i.e., W = UΣV , where U ∈ R N ×d , V ∈ R d×d are column orthonormal matrices, and Σ ∈ R d×d is a diagonal matrix with Σ kk = σ k being the k-th largest singular value of W. Note that SVD reparameterization is standard and has been widely used in the literature such as model compression (Chen et al., 2018) , training DNNs (Zhang et al., 2018) , and analyzing word embeddings (Arora et al., 2016c) . Given the SVD reparameterization, we can control the singular values of the embedding matrix W by constraining the matrix E, and the conditional distribution in (3.2) can be rewritten as follows:

P(Y t = y t |y <t ) = exp h t (UΣV ) I(yt) * N i=1 exp h t (UΣV ) i * subject to U U = I, V V = I, Σ ∈ P, (4.1)

where P is a feasible set to represent the singular value distribution of the embedding matrix W.

To ensure the orthogonal constraints in (4.1), we propose to use the orthogonal regularization for U, V during the training process, which has been previously used in (Jiang et al., 2019) . In particular, we use the following regularization, which is a linear combination of Frobenius norm and spectral norm errors:

λ 1 U U -I 2 F + λ 2 V V -I 2 F + λ 3 U U -I 2 2 + λ 4 V V -I 2 2 , where {λ i } 4

i=1 are positive regularization parameters.

4.2. SPECTRUM CONTROL:

Recall the SVD of W = UΣV , and we consider the following two types of the singular value distribution for Σ, which is inspired by the eigenvalue distribution of kernel methods (Wei et al., 2017; Pacchiano et al., 2019) :

• Exponential decay: we say the singular values {σ k } d k=1 of W satisfy the exponential decay if

W ∈ P e (γ) = {W ∈ R N ×d | σ k ≤ c 1 exp(-c 2 k γ ), k = 1, . . . , d, γ > 0, c 1 , c 2 > 0 are universal constants}.

• Polynomial decay: we say the singular values {σ k } d k=1 of W satisfy the polynomial decay if

W ∈ P p (γ) = {W ∈ R N ×d | σ k ≤ c 1 k -γ , k = 1, . . . , d, γ > 0, c 1 > 0 is a universal constant}.

For P e (γ) and P p (γ), the parameter γ controls the rate of singular value decay: the larger γ is, the faster singular value decay will be. To ensure the learned word embedding matrix W have the desired singular value distributions, we propose to add the following regularizations to our training objective: R e

(Σ) = λ e d k=1 σ k -c 1 exp(-c 2 k γ )

2 for exponential decay and

R p (Σ) = λ p d k=1 σ k -c 1 k -γ 2

for polynomial decay, where λ e , λ p are positive regularization parameters. Although the concept of "spectrum control" was previously used in Jiang et al. (2019) to improve the training of GANs, our spectrum control method is trying to solve a totally different problem, i.e., neural language generation, and its motivation is coming from a very different perspective, i.e., the representation degeneration of the word representations. In addition, our method of controlling the singular value with prior distributions is significantly different from the penalty function used in their method. Finally, our method is essential to improve the performance of the neural language generation, while the penalty function proposed in Jiang et al. (2019) can deteriorate the training of neural language models, as we illustrated in Appendix B.

4.3. THEORETICAL ANALYSIS:

In this subsection, we show some theoretical insights on why our proposed method can improve the performance of the NLG model. In particular, we follow the similar setup as considered in (Gao et al., 2019b) , i.e., focusing on the optimization of the embedding matrix W ∈ R N ×d and assume all the other parameters are fixed and well-optimized. In practice, we use our method to train the models from scratch. Therefore, according to the output layer in (3.2), we consider the following empirical risk minimization problem: given a training dataset S = {(h i , y i )} n i=1 with each example drawn i.i.d. from some unknown but fixed distribution D, and h i ∈ R d as a hidden state, y i ∈ {1, . . . , N } as its associated label, our goal is to minimize the training loss as follows:

min W∈R N ×d L S (W) = 1 n n i=1 h i W, y i , (4.2)

where h i W, y i is the cross-entropy loss with respect to h i W and y i . The cross-entropy loss defined above is widely used to train NLG models, and is also used to compute the perplexity of the trained model, which is the benchmark criterion to evaluate the performance of language models. In addition, we define the expected loss as follows

L D (W) = E[ h W, y ],

where the expectation is taken over the distribution D of the training data.

Let W = arg min W∈P(γ) L S (W), where P(γ) = P e (γ) for exponential decay and P(γ) = P p (γ) for polynomial decay. We assume the right singular vector matrix satisfies V 1 ≤ V for all W ∈ P(γ) . Now, we are ready to provide the main theory of our method (The proof can be found in Appendix A).

Theorem 4.1. Under previously stated conditions, suppose that | (•)| ≤ B, is G-Lipschitz continuous, and h i ∞ ≤ H for all i = 1, . . . , n. If we choose γ > 1/2, then with probability at least 1 -δ, we have

L D ( W) ≤ min W∈P(γ) L S (W) + C 1 AN m-1 j=1 σ 2 j + m 1-2γ /(2γ -1) + C 2 B log(1/δ) √ n , (4.3) where C 1 , C 2 are absolute constants, m ∈ [2, d], A = GV H √ log d.

Remark 4.2. According to Theorem 4.1, the expected loss of the learned embedding matrix W consists of two terms. The first term represents the training loss, the second term is the generalization error gap. Specifically, the smaller the γ, the larger the feasible set P(γ), thus the smaller the training loss min W∈P(γ) L S (W). On the other hand, the larger the γ, the faster the singular value decays, and the smaller the generalization error gap. Therefore, our generalization error bound demonstrates an appealing property of our proposed method: by directly controlling the singular value distribution of the learned word embedding, we are able to achieve a trade-off between the training loss and generalization error. Remark 4.3. For the generalization error bound in (4.3), penalizing the largest singular value could reduce this upper bound. It validates the method proposed in Gao et al. (2019b) , which implicitly penalizes the largest singular value (c.f. Section 5 in Gao et al. (2019b) ). Compared with their method, the error term O N m 1-2γ /(2γ -1)/ √ n suggests that by explicitly manipulating the singular value distribution, our method has a better control of the tail sum of the singular values.

5. EXPERIMENTS:

We demonstrate the effectiveness of our proposed spectrum control algorithm on two tasks: language modeling and machine translation. We compare our results with the state-of-the-art models. In our experiments, we try both exponential and polynomial singular value decays, and present the one with the better result. The performances of exponential decay and polynomial decay are very close (See Appendix D). In practice we found that for large scale dataset its better to use polynomial decay and for small scale dataset its better to use exponential decay. We also present the training time and memory cost of our method in Appendix C.

5.1. LANGUAGE MODELING:

Datasets We consider two benchmark datasets for language modeling: WikiText-2 and WikiText-103, which consist of pre-processed Wikipedia articles and were introduced by Merity et al. (2018a) . WikiText-2 is a small dataset with around 2 million words and 30K vocabulary size, while WikiText-103 is a significantly large dataset with around 103 million words and 260K vocabulary size.

Model Configuration On the small WikiText-2 dataset, we implement our method based on the state-of-the-art AWD-LSTM model (Merity et al., 2018a) . It is a 3-layer LSTM model with 1150 dimensional hidden states and 400 dimensional embeddings. We also follow the same regularization and optimization procedures introduced in (Merity et al., 2018a) . The implementation of our method is based on the open-source code3 for AWD-LSTM. On the large WikiText-103 dataset, we implement our method based on the state-of-the-art Transformer-XL based models (Dai et al., 2019) . We follow the same settings reported in (Dai et al., 2019) , and our implementation is based on the official code4 for Transformer-XL. To evaluate the performance of our method more thoroughly, we consider two Transformer-XL models with different number of layers. The first is the standard Transformer-XL model with 16 layers used in (Dai et al., 2019) . For the second, we consider a smaller Transformer-XL model with just 4 layers and other configurations unchanged.

Parameters For the parameters {λ i } 4 i=1 of the orthogonal regularizations, we tune them by grid search over {0.01, 0.1, 1, 10}. For the parameters λ e , λ p of the spectrum control, we tune them over the grid {0.1, 1, 10, 100}. We try different singular value distributions, and the best distributions for AWD-LSTM and Transformer-XL are exponential and polynomial, respectively.

Results of the LSTM Model on WikiText-2:

We first present the results of language modeling on the small dataset WikiText-2 using LSTM models. In Table 1 we compare the validation/test perplexity5 of the baseline AWD-LSTM model (Merity et al., 2018a) , the cosine similarity regularization (MLE-CosReg) model (Gao et al., 2019b) , and the models trained using our method under three different settings (Merity et al., 2018a) : without finetune, with finetune and with further continuous cache pointer. Compared with the baselines, our method achieves 2.3/2.9/2.3 test perplexity reduction under all three settings; compared with the MLE-CosReg method, our method achieves 1.5/1.2/0.3 test perplexity reduction under all three settings.

Results of the Transformer-XL Model on WikiText-103:

We next show the results of language modeling on the large dataset WikiText-103 using Transformer-XL models. Table 2 compares the validation/test perplexity of Transformer-XL based models (Dai et al., 2019) and the models trained by our method on WikiText-103 dataset. The results demonstrate that our method consistently improves upon the small Transformer-XL model (0.9 test perplexity reduction) and the standard Transformer-XL model (0.8 test perplexity reduction). Analysis We study the output embedding matrix of Transformer-XL trained on WikiText-103 using our method. In particular, we want to evaluate the isotropy of the learned word representations. We consider the partition function Z(a) = N i=1 exp( w i , a ) introduced in (Arora et al., 2016b) , where w i is the i-th row of the embedding matrix W ∈ R N ×d and a ∈ S d-1 is a unit vector. According to Lemma 2.1 in (Arora et al., 2016b) , if the word representation vectors are isotropic, Z(a) is close to some constant with high probability for all unit vectors. Thus to empirically measure the isotropy of the learned word representations, we consider two criteria based on Z(a):

I 1 (W) = min a∈E Z(a) max a∈E Z(a) and I 2 (W) = a∈E (Z(a) -Z(a)) 2 |E| Z(a) 2 ,

where E is the set of eigenvectors of W W, as suggested by Mu & Viswanath (2018) . We also propose to check sampled standard deviation measure I 2 (W) (normalized by its average, i.e., Z(a)).

We have I 1 (W) ∈ [0, 1] and I 2 (W) ≥ 0. Larger I 1 (W) and smaller I 2 (W) indicate more isotropic for word embeddings. We uniformly sample 40K words from the vocabulary (around 260K) of WikiText-103 to compute these two criteria. The left half of Table 3 summarizes the values of I 1 (W) and I 2 (W), which are averaged over 10 runs, for the baseline method and our method. We can see from the results that our method significantly improves the isotropy of the learned word representations in terms of both criteria.

5.2. MACHINE TRANSLATION:

We also apply our spectrum control method to machine translation tasks. Given a source sentence s, the decoder of an neural machine translation (NMT) model is to predict the next word in the target sentence t and the previous decoded words in t. In the following we use the state-of-the-art Transformer-based NMT model as our baseline. (Gehring et al., 2017) . More specifically, we have 160K sentence pairs as the training data, 7K sentence pairs as the validation data, and we combine tst2010, tst2011, tst2012, dev2010 and dev2012 datasets to form our test data. For the large scale WMT 14 En-De, we have 4.5 million sentence pairs as our training data, and we use newstest2014 as our test data dataset (Cettolo et al., 2014) .

Model Configuration:

We implement our method on top of Transformer model (Vaswani et al., 2017) . In particular, we consider the Transformer-Base architecture (Vaswani et al., 2017) for IWSLT 2014 De-En, which has a 6-layer encoder and 6-layer decoder with 512 dimensional hidden states and embeddings, except that we choose the dimension of the inner feed-forward layer as 1024 instead of 2048 and the number of attention heads is set to be 4 rather than 8. For WMT 14 En-De, we consider the original Transformer-Base and Transformer-Big architectures (Vaswani et al., 2017) , which have a 6-layer encoder and 6-layer decoder with 512 and 1024 dimensional embeddings, respectively. Our implementation is based on the open-sourced code6 provided by Ott et al. (2018) .

We follow the same procedures as in the language modeling task to choose the regularization parameters. The best results for this task on IWSLT 2014 De-En are from the models where the singular values are controlled by exponential distribution, and the best results for this task on WMT 14 En-De are from the models where the singular values are controlled by polynomial distribution.

Results:

Comparisons of different methods in terms of BLEU scores for IWSLT 2014 De-En are summarized in Table 4 . Compared with the baseline models, our method improves the BLEU score7 from 34.01 to 35.50 on the German→English task, close to 1.5 gain on BLEU score; our method is also better than 35.18 reported in (Wang et al., 2019a) , and 35.44 reported in (Wang et al., 2019b) . Table 5 summarizes different methods in terms of BLEU scores for WMT 14 En-De. The results show that our method achieves 1.15 and 0.92 BLEU score improvements on this task for base and big models, respectively. In addition, our method is also better than 28.38 and 28.94 reported in Gao et al. (2019b) for base and big models.

Analysis We also study the learned word embedding matrix of the Transformer trained on IWSLT 2014 De-En using our method in terms of isotropy. The values of I 1 (W) and I 2 (W) are reported in the right half of Table 3 , we compute these two values based on all the tokens. It shows that the isotropy of the learned word representations using our method increases significantly in terms of these criteria, from very anisotropic to nearly isotropic. We also demonstrate the projected word embedding matrices and the singular value distributions of different methods in Figure 3 . The plots show that the learned word representations from our method are distributed isotropically in the space, which is in contrast to the narrow cone distribution from the baseline method. 

Method BLEU:

Transformer-Base (Vaswani et al., 2017) 27.30 Transformer-Base (Gao et al., 2019b) 28.38 Transformer-Base + Ours 28.45

Transformer-Big (Vaswani et al., 2017) 28.40 Transformer-Big (Gao et al., 2019b) 28.94 Transformer-Big + Ours 29.32 

6. CONCLUSIONS AND FUTURE WORK:

In this paper, we tackle the degeneration problem that occurs at the output embeddings in the softmax layer used in neural language generation models. We develop a novel spectrum control method to explicitly guide the spectra training of the output embedding matrix with some slow-decaying singular value prior distributions. Our proposed method is shown to alleviate the degeneration problem and improve isotropy of the learned contextualized word representations. Thorough experimental results demonstrate the advantage of our method over the state-of-the-art neural models for language model and machine translation. Since our work is orthogonal to Wang et al. (2019a) , it would be interesting to combine the adversarial softmax training with our spectrum control method, and investigate its performance. For the future work, we would also like to investigate how the frequency-agnostic word representations in (Gong et al., 2018) relate to the single value distribution of the output embedding matrix. W = UΣV and u k is the k-th row of U. Therefore, we have

R S (F k ) = E ρ sup f ∈F k 1 n n i=1 ρ i w k , h i = E ρ sup f ∈F k 1 n VΣu k , h ρ (A.1)

where h ρ = n i=1 ρ i h i and the last inequality is due to the definition of w k = VΣu k . Therefore, we have

R S (F k ) ≤ E ρ sup f ∈F k 1 n VΣu k 1 h ρ ∞ ≤ V n E ρ sup f ∈F k Σu k 1 h ρ ∞ ≤ V d j=1 σ 2 j n E ρ sup f ∈F k h ρ ∞ , (A.2)

where the second inequality is due to V 1 ≤ V and the last one comes from

Σu k 1 = d j=1 σ j |u kj | ≤ d j=1 σ 2 j .

In addition, according to Theorem 12 in Liang (2014) , we have

E ρ h ρ ∞ ≤ H 2n log d, Therefore, we can get R S (F k ) ≤ V H m i=1 σ 2 i + i>m σ 2 i √ n 2 log d.

As a result, we can get 

D COMPARISON OF TWO PRIORS:

In this section we present the performance of our method on different tasks using different prior distributions. Table 8 suggests that the overall performance of these two prior distributions on different tasks are similar. In addition, the results show that it is better to use polynomial decay for large scale dataset and exponential decay for small scale dataset.

:

Note that we project the original word embeddings to a

dimensional vector space using principal component analysis (PCA) for the purpose of visualization.

The plots are based on the small Transformer-XL model and Transformer-Base model, the detailed model configuration can be found in experimental setups.

https://github.com/salesforce/awd-lstm-lm

https://github.com/kimiyoung/transformer-xl

Lower perplexity means better language models.

https://github.com/pytorch/fairseq

Higher BLEU score represents higher quality of the machine translation.

Figures and Tables:

Figure 1: Projected word embeddings 1 and singular value distributions. (a) and (c): 2-D visualization of word embedding matrices of Transformer-XL for language modeling and Transformer for machine translation; (b) and (d): Normalized singular value distributions of embedding matrices.

Figure 2: Projected word embeddings and singular value distributions using our spectrum control method. (a) and (c): 2-D visualization of word embedding matrices of Transformer-XL for language modeling and Transformer for machine translation; (b) and (d): Normalized singular value distributions of embedding matrices.

Figure 3: (a) Word embedding for the vanilla Transformer, which has a narrow cone distribution; (b) Word embedding for Transformer using spectrum control, which has a uniform distribution; (c) Normalized singular value for different methods, which shows a slow decay of our method.

for P(γ) = P e (γ), we have σ j ≤ c 1 exp(-c 2 j γ ) ≤ c 1 /(c 2 j γ ), and for P(γ) = P p (γ), we have σ j ≤ c 3 j -γ . Thus for γ > 1/2, we have 1)1-2γ ,where c 4 = (c 1 /c 2 ) 2 if P(γ) = P e (γ) and c 4 = c 2 3 if P(γ) = P p (γ). Therefore, plugging this upper bound into (A.3), we have Where c 5 = c1/c5 if P(γ) = P e (γ) and c 5 = c 3 if P(γ) = P p (γ). According to Lemma A.1, since is G-Lipschitz continuous and | (•)| ≤ B, we have sup W∈P(γ) L D (W) -L S (W) ≤ c 6 GN V H log d

Comparison of different methods in terms of perplexity on WikiText-2 dataset for the task of language modeling.

Table content: <table><tr><td>Method</td><td colspan="3">Parameters Validation Test</td></tr><tr><td>Existing results</td><td/><td/><td/></tr><tr><td>Variational LSTM (Inan et al., 2017)</td><td>51M</td><td>91.5</td><td>87.0</td></tr><tr><td>2-layer skip connection LSTM (Mandt et al., 2017)</td><td>24M</td><td>69.1</td><td>65.9</td></tr><tr><td>w/o finetune</td><td/><td/><td/></tr><tr><td>AWD-LSTM (Merity et al., 2018a)</td><td>33M</td><td>69.1</td><td>66.0</td></tr><tr><td>MLE-CosReg (Gao et al., 2019b)</td><td>33M</td><td>68.2</td><td>65.2</td></tr><tr><td>Ours</td><td>33M</td><td>66.3</td><td>63.7</td></tr><tr><td>+ finetune</td><td/><td/><td/></tr><tr><td>AWD-LSTM (Merity et al., 2018a)</td><td>33M</td><td>68.6</td><td>65.8</td></tr><tr><td>MLE-CosReg (Gao et al., 2019b)</td><td>33M</td><td>67.1</td><td>64.1</td></tr><tr><td>Ours</td><td>33M</td><td>65.3</td><td>62.9</td></tr><tr><td>+ continuous cache pointer</td><td/><td/><td/></tr><tr><td>AWD-LSTM (Merity et al., 2018a)</td><td>33M</td><td>53.8</td><td>52.0</td></tr><tr><td>MLE-CosReg (Gao et al., 2019b)</td><td>33M</td><td>51.7</td><td>50.0</td></tr><tr><td>Ours</td><td>33M</td><td>51.1</td><td>49.7</td></tr></table>

Comparison of different methods in terms of perplexity on WikiText-103 dataset for the task of language modeling.

Table content: <table><tr><td>Method</td><td colspan="3">Parameters Validation Test</td></tr><tr><td>Existing results</td><td/><td/><td/></tr><tr><td>4 layer QRNN (Merity et al., 2018b)</td><td>151M</td><td>32.0</td><td>33.0</td></tr><tr><td>Hebbian + Cache (Rae et al., 2018)</td><td>-</td><td>29.7</td><td>29.9</td></tr><tr><td>Small Transformer-XL (Dai et al., 2019)</td><td>120M</td><td>29.6</td><td>30.4</td></tr><tr><td>Ours</td><td>120M</td><td>29.0</td><td>29.5</td></tr><tr><td>Standard Transformer-XL (Dai et al., 2019)</td><td>151M</td><td>23.1</td><td>24.0</td></tr><tr><td>Ours</td><td>151M</td><td>22.9</td><td>23.2</td></tr></table>

Comparison of different methods in terms of isotropy for the tasks of language modeling and machine translation. (For perfect isotropy, I 1 (W) = 1, I 2 (W) = 0.)

Table content: <table><tr><td/><td>Language Modeling</td><td/><td/><td>Machine Translation</td><td/></tr><tr><td>Method</td><td colspan="3">I1(W) I2(W) Method</td><td colspan="2">I1(W) I2(W)</td></tr><tr><td colspan="2">Standard Transformer-XL 0.24</td><td>0.037</td><td colspan="2">Transformer-Base 0.31</td><td>0.031</td></tr><tr><td>Ours</td><td>0.63</td><td>0.022</td><td>Ours</td><td>0.88</td><td>0.005</td></tr></table>

Comparison of different methods in terms of BLEU scores on the task of De→En machine translation, trained on IWSLT 2014 dataset. Datasets We compare various NMT models on the IWSLT 2014 German→English (De-En) and WMT 14 English→German (En-De) datasets. For IWSLT 2014 De-En, we follow the same setup as in

Table content: <table><tr><td/><td/><td>Method</td><td/><td/></tr><tr><td>IWSLT 2014 De→En</td><td>Adversarial (Wang et al., 2019a)</td><td>Dual-learning (Wang et al., 2019b)</td><td>Transformer-Base (Wang et al., 2019b)</td><td>Ours</td></tr><tr><td>BLEU</td><td>35.18</td><td>35.44</td><td>34.01</td><td>35.50</td></tr></table>

Comparison of different methods in terms of BLEU scores on the task of En→De machine translation, trained on WMT 2014 dataset.

Table content: <table/>

Comparisons of different priors on different datasets.

Table content: <table><tr><td>Dataset</td><td colspan="2">Exponential Decay Polynomial Decay</td></tr><tr><td>Language Modeling</td><td>Test PPL</td><td>Test PPL</td></tr><tr><td>WikiText-2</td><td>63.7</td><td>64.2</td></tr><tr><td>WikiText-103</td><td>23.4</td><td>23.2</td></tr><tr><td>Machine Translation</td><td>BLEU</td><td>BLEU</td></tr><tr><td>IWSLT 2014 De→En</td><td>35.50</td><td>35.40</td></tr><tr><td>WMT 14 En→De</td><td>28.37</td><td>28.45</td></tr></table>

Peer review (iclr2020):
Summary: This paper deals with the representation degeneration problem in neural language generation, as some prior works have found that the singular value distribution of the (input-output-tied) word embedding matrix decays quickly. The authors proposed an approach that directly penalizes deviations of the SV distribution from the two prior distributions, as well as a few other auxiliary losses on the orthogonality of U and V (which are now learnable). The experiments were conducted on small and large scale language modeling datasets as well as the relatively small IWSLT 2014 De-En MT dataset. Pros: + The paper is well-written with great clarity. The dimensionality of the involved matrices (and their decompositions) are clearly provided, and the approach is clearly described. The authors also did a great job providing the details of their experimental setup. + The experiments seem to show consistent improvements over the baseline methods (at least the ones listed by the authors) on a relatively extensive set of tasks (e.g., of both small and large scales, of two different NLP tasks). Via WT2 and WT103, the authors also showed that their method worked on both LSTM and Transformers (which it should, as the SVD on word embedding should be independent of the underlying architecture). + I think studying the expressivity of the output embedding matrix layer is a very interesting (and important) topic for NLP. (e.g., While models like BERT are widely used, the actual most frequently re-used module of BERT is its pre-trained word embeddings.) --------------------------------- I have a few questions/comments on the work as well: 1) One of the things that is not clearly described in the paper is how the proposed spectrum control method was injected into training in practice. In Section 4.3 (when you do the theoretical analysis), you assumed *all the other parameters are fixed and well-optimized*. Is that also what you did in training on WT2/WT103 (e.g., first pre-train a model such as Transformer-XL, and then fine-tune its embedding layer using the proposed decomposed method)? 2) How does the runtime and memory cost of your approach compare to the baselines? (e.g., you now need to compute , which can also be prohibitively large when the vocabulary size is large; for instance, on the 1-Billion word dataset). 3) I remember the base Transformer-XL model did not use the adaptive embedding/softmax (although they set the `--adaptive` flag, they did use `--div_val 1`). How does your method work in the adaptive setting, where the embedding size of different words could be very different (e.g., depending on the word frequency)? 4) Regarding the theoretical analysis in Section 4.3. Why is the cross-entropy loss function (essentially NLL loss + softmax) bounded (as is required by Theorem 4.1)? Given a fixed ground-truth , if the corresponding predicted likelihood is small (e.g., ), won*t the CE loss be very large? In that case, the generalization bound in Eq. (4.3) would be vacuous, as B would be very large too. Moreover, I*m also not fully convinced by what the theoretical analysis is trying to convey--- that your method *achieve a trade-off between the training loss and generalization error*--- but isn*t that what (any) regularizations are designed for? The proved bound is no different in nature from any generalization bound with a regularizer (e.g., weight decay), and it does not necessarily reflect the usefulness of the proposed approach. 5) In experiments, you only showed the better performance of the exponential and the polynomial singular value decays (cf. beginning of Sec. 5). Could you show both? Which one is better (and on what task), and by how much? If one is to use your method, which decay scheme do you recommend? 6) Why is MLE-CosReg (Gao et al. 2019b) not compared to in the WikiText-103 and the machine translation task? As the MLE-CosReg approach only involves regularizing the cosine similarity via , it should computationally be even slightly cheaper than the proposed method (you need to compute , which has the same complexity). They also tested on the larger-scale WMT En-De and De-En dataset (which contains 4.5M sentence pairs). Is there any reason that you chose IWSLT 2014 instead? --------------------------------- Some issues that didn*t impact the score: 7) It*d be useful to add labels to the x- and y-axis of the plots in Figure 1. 8) When implementing the method described in Section 4.2, did you explicitly sort the singular values in each iteration, or just set them to learnable parameters (along with learnable ) without sorting? If you do sort, do you also *sort* the columns of and (as I would expect a one-to-one mapping from to , for instance)? If you don*t sort, how do you make sure that ? 9) Why did you regularize and by both its Frobenius norm and its spectral norm? Does using only one of them compromise the performance? --------------------------------- Overall, I find this work well-written and well-motivated. The experiments seem to show consistent improvement when using the approach. I vote for weak accept, but I also look forward to the author*s response to the questions I raised above.