IMPROVING NEURAL LANGUAGE GENERATION WITH SPECTRUM CONTROL
Year: 2020

Abstract:
Recent Transformer-based models such as Transformer-XL and BERT have achieved huge success on various natural language processing tasks. However, contextualized embeddings at the output layer of these powerful models tend to degenerate and occupy an anisotropic cone in the vector space, which is called the representation degeneration problem. In this paper, we propose a novel spectrum control approach to address this degeneration problem. The core idea of our method is to directly guide the spectra training of the output embedding matrix with a slow-decaying singular value prior distribution through a reparameterization framework. We show that our proposed method encourages isotropy of the learned word representations while maintains the modeling power of these contextual neural models. We further provide a theoretical analysis and insight on the benefit of modeling singular value distribution. We demonstrate that our spectrum control method outperforms the state-of-the-art Transformer-XL modeling for language model, and various Transformer-based models for machine translation, on common benchmark datasets for these tasks.

heading: 1 INTRODUCTION
text: Neural language generation (NLG) is an important task with many practical applications, such as automatic speech recognition (Graves et al., 2013; Toshniwal et al., 2018), text generation (Bowman et al., 2016; Radford et al., 2019; Keskar et al., 2019), machine translation (Bahdanau et al., 2015; Vaswani et al., 2017) and dialog systems (Gao et al., 2019a; Tang et al., 2019). Most NLG models utilize a complex encoding model to map a given context into a hidden state vector, and then predict the next word distribution by multiplying the encoded vector with the output embedding layer, followed by a softmax layer. In the past few years, it has witnessed a significant progress in NLG by improving the encoding model, from the recurrent neural network (RNN) (Bahdanau et al., 2015; Jozefowicz et al., 2016; Merity et al., 2018a) based models to the current Transformer-based models (Vaswani et al., 2017; Devlin et al., 2019; Dai et al., 2019; Radford et al., 2019). However, embeddings in the softmax output layer have been shown not capable enough to model the conditional probability (Yang et al., 2018). Recently, Gao et al. (2019b) pointed out another limitation of the output embeddings: the representation degeneration problem. They showed that the singular value distribution of the output embedding matrix tends to decay very fast, and the embedding space is squeezed into a narrow cone (as shown in Figure 1(a) and 1(c) in 2-D plots). Such anisotropic shape (Ethayarajh, 2019) is very different from what one would expect from an expressive word embedding space (Arora et al., 2016a; Mu & Viswanath, 2018). Therefore, several efforts (Gao et al., 2019b; Wang et al., 2019a) have been made to address the degeneration problem. Unlike previous approaches that applied implicit regularization to singular values of the output embedding matrix, we propose a spectrum control (SC) approach, which was inspired by the spectral control technique used for Generative Adversarial Network (GAN) training (Jiang et al., 2019), to explicitly control the singular value distribution. We first reparameterize the output embedding matrix W by its singular value decomposition (SVD): W = UΣV>, where U,V are column orthonormal matrices, and Σ is a diagonal matrix of singular values. Then we guide the training of Σ by a predefined slow-decaying prior distribution, such as a polynomial decay distribution, or
an exponential decay distribution. At the end of training, the distribution of singular values of the embedding matrix gets close to the prior distribution. Our spectrum control approach alleviates the representation degeneration problem by encouraging the diversity of word representations and improving isotropic property of these representations (see Figure 2), even on top of the powerful Transformer-based models. We further present a theoretical analysis to justify our method. The results suggest that it is beneficial to directly guide the singular value distribution of the embedding matrix throughout the training process and control the decay rate of these singular values. We demonstrate the effectiveness of our training framework with extensive experimental results on two tasks: language modeling and machine translation. Our spectrum control method outperforms the latest state-of-the-art TransformerXL model on WikiText-103 dataset for language modeling; and obtains close to 1.5 BLEU improvement on IWSLT 2014 German-English translation task compared to the Transformer baseline model.

heading: 2 RELATED WORK
text: In this paper, we mainly focus on the output embedding matrix that is used in the softmax output layer for language generation tasks. This embedding matrix is also used as input word embeddings, which is known as weight tying trick. The weight tying not only reduces the number of parameters but also enjoys theoretical benefits (Inan et al., 2017). Thus the weight tying has been successfully applied to many state-of-the-art models for language modeling and machine translation (Merity et al., 2018a; Vaswani et al., 2017; Yang et al., 2018). However, there are certain issues with the softmax output layer. Yang et al. (2018) first identified a problem called “softmax bottleneck” that the softmax output layer does not have enough capacity to model natural language due to its connection with the rank bottleneck in matrix factorization. A simple and effective method called Mixture of Softmaxes (MoS) was proposed to deal with this issue. Follow-up work in Kanai et al. (2018); Ganea et al. (2019) tried to replace softmax with alternative activation functions. Kanai et al. (2018) proposed to use sigsoftmax, which is composed of a multiplication of an exponential function and sigmoid function. Ganea et al. (2019) proposed a Linear-Monotonic-Softmax (LMS) model that generalized the approach in Kanai et al. (2018) by learning parametric point-wise increasing functions to optimally distort the logits before feeding them into the softmax layer. Pappas & Henderson (2019) instead resorted to a powerful deep
1Note that we project the original word embeddings to a 2 dimensional vector space using principal component analysis (PCA) for the purpose of visualization.
residual nonlinear output mapping while using a single softmax function without modifying its dimensionality or rank. Another line of work focuses on increasing the expressive power of the output embedding matrix by adding a cosine similarity regularization term (Gao et al., 2019b), or adversarial noise (Wang et al., 2019a). Gao et al. (2019b) analyzed the representation degeneration problem that the output embeddings tend to degenerate and be distributed into a narrow cone. A regularization term based on the summation of pairwise cosine similarity among all words was proposed to increase the representation power of word embeddings. Wang et al. (2019a) pointed out the computation of regularization term in (Gao et al., 2019b) depends on the size of the vocabulary and hence is costly. Instead, they proposed a simple yet highly effective adversarial training method that adds adversarial noises to the output embedding layer when training the models. They proved in theory that this adversarial training increases the distances between two different words, and thus encourages the diversity of the embedding vectors. Our work follows this line of thought, but takes a different approach: motivated by the spectrum control method for GAN training (Jiang et al., 2019), we propose to directly guide the singular values by a slow-decaying prior distribution during the model training process. We show that the anisotropic behavior of the contextualized word representations from powerful Transformer-based models (Ethayarajh, 2019) are alleviated by our spectrum control approach.

heading: 3 PROBLEM SETUP
text: In this section, we briefly introduce the neural models for language generation, and illustrate the singular value decay phenomena of existing neural language models. We first introduce some notations used in the rest of the paper. Notation: For a d-dimensional vector x ∈ Rd, we use ‖x‖q = ( ∑d i=1 |xi|q)1/q , where 0 < q <∞ to denote its `q-norm, and ‖x‖∞ = maxi |xi| to be its infinity norm. For a matrix A ∈ Rd1×d2 , let Ai∗ be the i-th row of A, and we use ‖A‖2, ‖A‖F , ‖A‖1 to denote its spectral norm, Frobenius norm, and matrix 1-norm. Given two sequences {an} and {bn}, if there exists a constant 0 < C < ∞ such that an ≤ Cbn, we write an = O(bn), and we use Õ(·) to hide the logarithmic factors.

heading: 3.1 NEURAL LANGUAGE GENERATION
text: We first briefly review the softmax output layer typically used in neural language generation models. We define the joint probability of a given length-n sentence of words (tokens) sn = (y1, . . . ,yn) as the following product of conditional probabilities
P(sn) = n∏ t=1 P(yt|ct), (3.1)
where yt ∈ V is the t-th word in sentence sn, and V represents the word vocabulary, ct = y1:t−1 = (y1, . . . ,yt−1) is referred to as the context of word yt. In addition, the context ct is usually modeled by a fixed size vector ht ∈ Rd, which is referred to as the hidden state, using some neural networks such as LSTM (Hochreiter & Schmidhuber, 1997) and Transformer (Vaswani et al., 2017). Then, the probability distribution of the output word yt given the context ct, i.e., P(yt|ct) in (3.1), is parameterized as the following softmax function:
P(Yt = yt|ct) = P(Yt = yt|ht) = exp(h>t WI(yt)∗)∑N i=1 exp(h > t Wi∗) , (3.2)
where W ∈ RN×d is the weight matrix and is usually tied with the input word embedding matrix (Press & Wolf, 2017; Inan et al., 2017), N = |V| is the vocabulary size, d is the embedding dimension, I(yt) represents the index of word yt in vocabulary V . In the following discussion, we call output weight matrix W as the word embedding matrix of the neural language model since it is tied with the input word embedding matrix. In this paper we focus on the output layer of the neural model for language generation, i.e., the softmax layer in (3.2), as in (Yang et al., 2018; Kanai et al., 2018; Ganea et al., 2019; Gao et al., 2019b). We will examine the singular value distribution of W and propose a different approach to increase the expressive power of the word embedding.

heading: 3.2 FAST SINGULAR VALUE DECAY
text: As we mentioned before, the singular values of W tend to drop very fast and it may interact with the cone-shaped embedding space. More specifically, Figure 1(b) and 1(d) illustrate the distributions of the normalized singular values of the Transformer-XL based language model2 (Dai et al., 2019) and the Transformer-based machine translation model (Vaswani et al., 2017) trained on WikiText103 (Merity et al., 2018a) and IWSLT 2014 De-En (Cettolo et al., 2014) datasets, respectively. The plots show a fast singular value decay phenomenon, i.e., there is a huge drop between the first and remaining singular values. Such a phenomenon has also been observed in some previous work (Gao et al., 2019b; Wang et al., 2019a). Figure 1(a) and 1(c) present the distributions of the projected word embeddings of the aforementioned two models. We can see from the plots that the projected word embeddings are distributed into some narrow cone shapes, which implies an anisotropic property of the learned word representations, i.e., the embedding vectors are not uniformly distributed in the space. The detailed analysis in Ethayarajh (2019) also confirms that contextualized word embeddings learnt from ELMo (Peters et al., 2018) (LSTM-based model), BERT and GPT-2 (Devlin et al., 2019; Radford et al., 2019) (Transformer-based models) indeed tend to be anisotropic. These contextualized word embeddings have shown great success on many NLP tasks. However, static word embeddings, such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) have been shown (Arora et al., 2016b; Mu & Viswanath, 2018) to be isotropic with great expressive power. Hence it may be also beneficial to increase the expressive power of the contextualized word embedding by increasing its isotropy. This motivates us to alleviate the fast singular value decay phenomenon to increase the isotropy of the learned word representations.

heading: 4 PROPOSED METHOD
text: To alleviate the fast singular value decay phenomenon, we propose to guide the singular value distribution of the contextualized word embedding throughout the training. As a result, we can achieve a trade-off between modeling contextual information, that tends to make word representations anisotropic, and the expressive power of word representations, that tends to be isotropic.

heading: 4.1 SVD REPARAMETERIZATION
text: Following the previous work (Jiang et al., 2019), we propose to apply singular value decomposition (SVD) based reparameterization to the embedding matrix W, i.e., W = UΣV>, where U ∈ RN×d,V ∈ Rd×d are column orthonormal matrices, and Σ ∈ Rd×d is a diagonal matrix with Σkk = σk being the k-th largest singular value of W. Note that SVD reparameterization is standard and has been widely used in the literature such as model compression (Chen et al., 2018), training DNNs (Zhang et al., 2018), and analyzing word embeddings (Arora et al., 2016c). Given the SVD reparameterization, we can control the singular values of the embedding matrix W by constraining the matrix E, and the conditional distribution in (3.2) can be rewritten as follows:
P(Yt = yt|y<t) = exp
( h>t (UΣV >)I(yt)∗ )∑N
i=1 exp ( h>t (UΣV >)i∗ ) subject to U>U = I,V>V = I,Σ ∈ P, (4.1)
where P is a feasible set to represent the singular value distribution of the embedding matrix W. To ensure the orthogonal constraints in (4.1), we propose to use the orthogonal regularization for U,V during the training process, which has been previously used in (Jiang et al., 2019). In particular, we use the following regularization, which is a linear combination of Frobenius norm and spectral norm errors: λ1‖U>U − I‖2F + λ2‖V>V − I‖2F + λ3‖U>U − I‖22 + λ4‖V>V − I‖22, where {λi}4i=1 are positive regularization parameters.

heading: 4.2 SPECTRUM CONTROL
text: Recall the SVD of W = UΣV>, and we consider the following two types of the singular value distribution for Σ, which is inspired by the eigenvalue distribution of kernel methods (Wei et al., 2017; Pacchiano et al., 2019):
2The plots are based on the small Transformer-XL model and Transformer-Base model, the detailed model configuration can be found in experimental setups.
• Exponential decay: we say the singular values {σk}dk=1 of W satisfy the exponential decay if W ∈ Pe(γ) = {W ∈ RN×d | σk ≤ c1 exp(−c2kγ), k = 1, . . . , d, γ > 0, c1, c2 > 0 are universal constants}.
• Polynomial decay: we say the singular values {σk}dk=1 of W satisfy the polynomial decay if W ∈ Pp(γ) = {W ∈ RN×d | σk ≤ c1k−γ , k = 1, . . . , d, γ > 0, c1 > 0 is a universal constant}.
For Pe(γ) and Pp(γ), the parameter γ controls the rate of singular value decay: the larger γ is, the faster singular value decay will be. To ensure the learned word embedding matrix W have the desired singular value distributions, we propose to add the following regularizations to our training objective: Re(Σ) = λe ∑d k=1 ( σk − c1 exp(−c2kγ) )2 for exponential decay and
Rp(Σ) = λp ∑d k=1 ( σk − c1k−γ )2 for polynomial decay, where λe, λp are positive regularization parameters. Although the concept of “spectrum control” was previously used in Jiang et al. (2019) to improve the training of GANs, our spectrum control method is trying to solve a totally different problem, i.e., neural language generation, and its motivation is coming from a very different perspective, i.e., the representation degeneration of the word representations. In addition, our method of controlling the singular value with prior distributions is significantly different from the penalty function used in their method. Finally, our method is essential to improve the performance of the neural language generation, while the penalty function proposed in Jiang et al. (2019) can deteriorate the training of neural language models, as we illustrated in Appendix B.

heading: 4.3 THEORETICAL ANALYSIS
text: In this subsection, we show some theoretical insights on why our proposed method can improve the performance of the NLG model. In particular, we follow the similar setup as considered in (Gao et al., 2019b), i.e., focusing on the optimization of the embedding matrix W ∈ RN×d and assume all the other parameters are fixed and well-optimized. In practice, we use our method to train the models from scratch. Therefore, according to the output layer in (3.2), we consider the following empirical risk minimization problem: given a training dataset S = {(hi, yi)}ni=1 with each example drawn i.i.d. from some unknown but fixed distribution D, and hi ∈ Rd as a hidden state, yi ∈ {1, . . . , N} as its associated label, our goal is to minimize the training loss as follows:
min W∈RN×d
LS(W) = 1
n n∑ i=1 ` ( h>i W, yi ) , (4.2)
where ` ( h>i W, yi ) is the cross-entropy loss with respect to h>i W and yi. The cross-entropy loss defined above is widely used to train NLG models, and is also used to compute the perplexity of the trained model, which is the benchmark criterion to evaluate the performance of language models. In addition, we define the expected loss as followsLD(W) = E[` ( h>W, y ) ], where the expectation is taken over the distribution D of the training data. Let Ŵ = arg minW∈P(γ) LS(W), where P(γ) = Pe(γ) for exponential decay and P(γ) = Pp(γ) for polynomial decay. We assume the right singular vector matrix satisfies ‖V‖1 ≤ V for all W ∈ P(γ) . Now, we are ready to provide the main theory of our method (The proof can be found in Appendix A).
Theorem 4.1. Under previously stated conditions, suppose that |`(·)| ≤ B, ` is G-Lipschitz continuous, and ‖hi‖∞ ≤ H for all i = 1, . . . , n. If we choose γ > 1/2, then with probability at least 1− δ, we have
LD(Ŵ) ≤ min W∈P(γ)
LS(W) + C1AN
(√∑m−1 j=1 σ 2 j + √ m1−2γ/(2γ − 1) ) + C2B √ log(1/δ)
√ n
,
(4.3) where C1, C2 are absolute constants, m ∈ [2, d], A = GVH √ log d.
Remark 4.2. According to Theorem 4.1, the expected loss of the learned embedding matrix Ŵ consists of two terms. The first term represents the training loss, the second term is the generalization error gap. Specifically, the smaller the γ, the larger the feasible setP(γ), thus the smaller the training loss minW∈P(γ) LS(W). On the other hand, the larger the γ, the faster the singular value decays, and the smaller the generalization error gap. Therefore, our generalization error bound demonstrates an appealing property of our proposed method: by directly controlling the singular value distribution
of the learned word embedding, we are able to achieve a trade-off between the training loss and generalization error. Remark 4.3. For the generalization error bound in (4.3), penalizing the largest singular value could reduce this upper bound. It validates the method proposed in Gao et al. (2019b), which implicitly penalizes the largest singular value (c.f. Section 5 in Gao et al. (2019b)). Compared with their method, the error term Õ ( N √ m1−2γ/(2γ − 1)/ √ n )
suggests that by explicitly manipulating the singular value distribution, our method has a better control of the tail sum of the singular values.

heading: 5 EXPERIMENTS
text: We demonstrate the effectiveness of our proposed spectrum control algorithm on two tasks: language modeling and machine translation. We compare our results with the state-of-the-art models. In our experiments, we try both exponential and polynomial singular value decays, and present the one with the better result. The performances of exponential decay and polynomial decay are very close (See Appendix D). In practice we found that for large scale dataset its better to use polynomial decay and for small scale dataset its better to use exponential decay. We also present the training time and memory cost of our method in Appendix C.

heading: 5.1 LANGUAGE MODELING
text: Datasets We consider two benchmark datasets for language modeling: WikiText-2 and WikiText103, which consist of pre-processed Wikipedia articles and were introduced by Merity et al. (2018a). WikiText-2 is a small dataset with around 2 million words and 30K vocabulary size, while WikiText103 is a significantly large dataset with around 103 million words and 260K vocabulary size.
Model Configuration On the small WikiText-2 dataset, we implement our method based on the state-of-the-art AWD-LSTM model (Merity et al., 2018a). It is a 3-layer LSTM model with 1150 dimensional hidden states and 400 dimensional embeddings. We also follow the same regularization and optimization procedures introduced in (Merity et al., 2018a). The implementation of our method is based on the open-source code3 for AWD-LSTM. On the large WikiText-103 dataset, we implement our method based on the state-of-the-art Transformer-XL based models (Dai et al., 2019). We follow the same settings reported in (Dai et al., 2019), and our implementation is based on the official code4 for Transformer-XL. To evaluate the performance of our method more thoroughly, we consider two Transformer-XL models with different number of layers. The first is the standard Transformer-XL model with 16 layers used in (Dai et al., 2019). For the second, we consider a smaller Transformer-XL model with just 4 layers and other configurations unchanged.
Parameters For the parameters {λi}4i=1 of the orthogonal regularizations, we tune them by grid search over {0.01, 0.1, 1, 10}. For the parameters λe, λp of the spectrum control, we tune them over the grid {0.1, 1, 10, 100}. We try different singular value distributions, and the best distributions for AWD-LSTM and Transformer-XL are exponential and polynomial, respectively.
Results of the LSTM Model on WikiText-2 We first present the results of language modeling on the small dataset WikiText-2 using LSTM models. In Table 1 we compare the validation/test perplexity5 of the baseline AWD-LSTM model (Merity et al., 2018a), the cosine similarity regularization (MLE-CosReg) model (Gao et al., 2019b), and the models trained using our method under three different settings (Merity et al., 2018a): without finetune, with finetune and with further continuous cache pointer. Compared with the baselines, our method achieves 2.3/2.9/2.3 test perplexity reduction under all three settings; compared with the MLE-CosReg method, our method achieves 1.5/1.2/0.3 test perplexity reduction under all three settings.
Results of the Transformer-XL Model on WikiText-103 We next show the results of language modeling on the large dataset WikiText-103 using Transformer-XL models. Table 2 compares the validation/test perplexity of Transformer-XL based models (Dai et al., 2019) and the models trained by our method on WikiText-103 dataset. The results demonstrate that our method consistently improves upon the small Transformer-XL model (0.9 test perplexity reduction) and the standard Transformer-XL model (0.8 test perplexity reduction).
3https://github.com/salesforce/awd-lstm-lm 4https://github.com/kimiyoung/transformer-xl 5Lower perplexity means better language models.
Analysis We study the output embedding matrix of Transformer-XL trained on WikiText-103 using our method. In particular, we want to evaluate the isotropy of the learned word representations. We consider the partition function Z(a) = ∑N i=1 exp(〈wi,a〉) introduced in (Arora et al., 2016b), where wi is the i-th row of the embedding matrix W ∈ RN×d and a ∈ Sd−1 is a unit vector. According to Lemma 2.1 in (Arora et al., 2016b), if the word representation vectors are isotropic, Z(a) is close to some constant with high probability for all unit vectors. Thus to empirically measure the isotropy of the learned word representations, we consider two criteria based on Z(a):
I1(W) = mina∈E Z(a)
maxa∈E Z(a) and I2(W) =
√∑ a∈E(Z(a)− Z̄(a))2
|E|Z̄(a)2 ,
where E is the set of eigenvectors of W>W, as suggested by Mu & Viswanath (2018). We also propose to check sampled standard deviation measure I2(W) (normalized by its average, i.e., Z̄(a)). We have I1(W) ∈ [0, 1] and I2(W) ≥ 0. Larger I1(W) and smaller I2(W) indicate more isotropic for word embeddings. We uniformly sample 40K words from the vocabulary (around 260K) of WikiText-103 to compute these two criteria. The left half of Table 3 summarizes the values of I1(W) and I2(W), which are averaged over 10 runs, for the baseline method and our method. We can see from the results that our method significantly improves the isotropy of the learned word representations in terms of both criteria.

heading: 5.2 MACHINE TRANSLATION
text: We also apply our spectrum control method to machine translation tasks. Given a source sentence s, the decoder of an neural machine translation (NMT) model is to predict the next word in the target sentence t and the previous decoded words in t. In the following we use the state-of-the-art Transformer-based NMT model as our baseline.
Datasets We compare various NMT models on the IWSLT 2014 German→English (De-En) and WMT 14 English→German (En-De) datasets. For IWSLT 2014 De-En, we follow the same setup as in (Gehring et al., 2017). More specifically, we have 160K sentence pairs as the training data, 7K sentence pairs as the validation data, and we combine tst2010, tst2011, tst2012, dev2010 and dev2012 datasets to form our test data. For the large scale WMT 14 En-De, we have 4.5 million sentence pairs as our training data, and we use newstest2014 as our test data dataset (Cettolo et al., 2014).
Model Configuration We implement our method on top of Transformer model (Vaswani et al., 2017). In particular, we consider the Transformer-Base architecture (Vaswani et al., 2017) for IWSLT 2014 De-En, which has a 6-layer encoder and 6-layer decoder with 512 dimensional hidden states and embeddings, except that we choose the dimension of the inner feed-forward layer as 1024 instead of 2048 and the number of attention heads is set to be 4 rather than 8. For WMT 14 En-De, we consider the original Transformer-Base and Transformer-Big architectures (Vaswani et al., 2017), which have a 6-layer encoder and 6-layer decoder with 512 and 1024 dimensional embeddings, respectively. Our implementation is based on the open-sourced code6 provided by Ott et al. (2018). We follow the same procedures as in the language modeling task to choose the regularization parameters. The best results for this task on IWSLT 2014 De-En are from the models where the singular values are controlled by exponential distribution, and the best results for this task on WMT 14 En-De are from the models where the singular values are controlled by polynomial distribution.
Results Comparisons of different methods in terms of BLEU scores for IWSLT 2014 De-En are summarized in Table 4. Compared with the baseline models, our method improves the BLEU score7 from 34.01 to 35.50 on the German→English task, close to 1.5 gain on BLEU score; our method is also better than 35.18 reported in (Wang et al., 2019a), and 35.44 reported in (Wang et al., 2019b). Table 5 summarizes different methods in terms of BLEU scores for WMT 14 En-De. The results show that our method achieves 1.15 and 0.92 BLEU score improvements on this task for base and big models, respectively. In addition, our method is also better than 28.38 and 28.94 reported in Gao et al. (2019b) for base and big models.
Analysis We also study the learned word embedding matrix of the Transformer trained on IWSLT 2014 De-En using our method in terms of isotropy. The values of I1(W) and I2(W) are reported in the right half of Table 3, we compute these two values based on all the tokens. It shows that the isotropy of the learned word representations using our method increases significantly in terms of these criteria, from very anisotropic to nearly isotropic. We also demonstrate the projected word embedding matrices and the singular value distributions of different methods in Figure 3. The plots show that the learned word representations from our method are distributed isotropically in the space, which is in contrast to the narrow cone distribution from the baseline method.
6https://github.com/pytorch/fairseq 7Higher BLEU score represents higher quality of the machine translation.

heading: 6 CONCLUSIONS AND FUTURE WORK
text: In this paper, we tackle the degeneration problem that occurs at the output embeddings in the softmax layer used in neural language generation models. We develop a novel spectrum control method to explicitly guide the spectra training of the output embedding matrix with some slow-decaying singular value prior distributions. Our proposed method is shown to alleviate the degeneration problem and improve isotropy of the learned contextualized word representations. Thorough experimental results demonstrate the advantage of our method over the state-of-the-art neural models for language model and machine translation. Since our work is orthogonal to Wang et al. (2019a), it would be interesting to combine the adversarial softmax training with our spectrum control method, and investigate its performance. For the future work, we would also like to investigate how the frequency-agnostic word representations in (Gong et al., 2018) relate to the single value distribution of the output embedding matrix.

heading: A PROOF OF THEOREM 4.1
text: Preliminaries We briefly review the general statistical learning framework. More specifically, let X and Y be the feature and label spaces, and suppose D is an unknown distribution over X × Y . Let F ⊆ VX be the hypothesis class that we use to make prediction, and ` : V × Y → R be the loss function. In addition, we define the function class `F = {(x, y) → `(f(x), y) : f ∈ F} as the composition of the functions in F and loss `. Therefore, the goal is to minimize the expected risk LD = E(x,y)∼D`(f(x), y) with some function f ∈ F . According to the problem setup in Section 4.3, we have n i.i.d. training examples S = {(hi, yi)}ni=1 drawn from D, where the hidden state hi ∈ Rd is the feature vector and yi is the corresponding label. Suppose there exist a learning algorithm that maps the training dataset S to a function f ∈ F , and we want to measure the gap between the empirical risk LS and and the population risk LD, i.e., |LD − LS|, where LS = ∑n i=1 `(f(hi), yi)/n. This gap is known as the generalization error. To bound the generalization error, we use the Rademacher complexity (Bartlett & Mendelson, 2002). Let F ⊆ RZ be a function class and S = {z1, . . . , zn} be a set of examples of size n, the empirical Rademacher complexity is defined as
R̂S(F) := 1 n Eρ [ sup f∈F n∑ i=1 ρif(xi) ] ,
where ρ1, . . . , ρn are i.i.d. Rademacher random variables with P(ρi = 1) = P(ρi = −1) = 1/2. According to the loss function LS defined in (4.2), we are considering N function classes {Fk}Nk=1 with Fk = {f : h→ 〈wk,h〉,wk = ek>W,W ∈ P(γ),h ∈ Rd}, where ek is a d-dimensional vector with k-th element to be one and others to be zero, and the loss ` : RN → R. We have the following Lemma to bound the generalization error based on the empirical Rademacher complexity, which was proved in Corollary A.11 in Allen-Zhu et al. (2019).
Lemma A.1. (Allen-Zhu et al., 2019) If F1 . . . ,FN are N classes of functions Rd → R, the loss function ` : RN → R is G-Lipschitz continuous and |`(·)| ≤ B for any z ∼ D, then with probability at least 1− δ, we have
sup f1∈F1,...,fn∈FN ∣∣∣∣E[`(f1(z), . . . , fN(z))]− 1n n∑ i=1 `(f(zi)) ∣∣∣∣ ≤ C1G N∑ k=1 R̂S(Fk)
+ C2 B √
log(1/δ)√ n ,
where the expectation is taken over the distribution D.
Equipped with this lemma, we now present the proof of our main result.
Proof of Theorem 4.1. According to the problem setup in Section 4, we have N classes of functions {Fk}Nk=1 and let the singular value decomposition of W as
W = UΣV> and uk is the k-th row of U. Therefore, we have R̂S(Fk) = Eρ [ sup f∈Fk 1 n n∑ i=1 ρi〈wk,hi〉 ] = Eρ [ sup f∈Fk 1 n 〈VΣu>k ,hρ〉 ] (A.1)
where hρ = ∑n
i=1 ρihi and the last inequality is due to the definition of wk = VΣu>k . Therefore, we have
R̂S(Fk) ≤ Eρ [ sup f∈Fk 1 n ‖VΣu>k ‖1‖hρ‖∞ ] ≤ V n Eρ [ sup f∈Fk ‖Σu>k ‖1‖hρ‖∞ ]
≤ V √∑d j=1 σ 2 j n Eρ [ sup f∈Fk ‖hρ‖∞ ] , (A.2)
where the second inequality is due to ‖V‖1 ≤ V and the last one comes from ‖Σu>k ‖1 = ∑d j=1 σj|ukj| ≤ √∑d j=1 σ 2 j . In addition, according to Theorem 12 in Liang (2014) , we have Eρ [ ‖hρ‖∞ ] ≤ H √ 2n log d,
Therefore, we can get R̂S(Fk) ≤ V H √∑m i=1 σ 2 i + √∑ i>m σ 2 i√
n
√ 2 log d.
As a result, we can get N∑ k=1 R̂S(Fk) ≤ NVH √∑m i=1 σ 2 i + √∑ i>m σ 2 i√ n √ 2 log d. (A.3) Since for P(γ) = Pe(γ), we have σj ≤ c1 exp(−c2jγ) ≤ c1/(c2jγ), and for P(γ) = Pp(γ), we have σj ≤ c3j−γ . Thus for γ > 1/2, we have∑
j>m σ2j ≤ ∑ j>m c4 j2γ ≤ c4 ∫ ∞ m+1 x−2γdx ≤ c4 (2γ − 1) (m+ 1)1−2γ,
where c4 = (c1/c2)2 if P(γ) = Pe(γ) and c4 = c23 if P(γ) = Pp(γ). Therefore, plugging this upper bound into (A.3), we have N∑ k=1 R̂S(Fk) ≤ NVH √ 2 log d √∑m j=1 σ 2 j √ n + c5 NVH √ 2 log d √ m1−2γ/(2γ − 1)√ n .
Where c5 = c1/c5 if P(γ) = Pe(γ) and c5 = c3 if P(γ) = Pp(γ). According to Lemma A.1, since ` is G-Lipschitz continuous and |`(·)| ≤ B, we have
sup W∈P(γ)
∣∣LD(W)− LS(W)∣∣ ≤ c6GNVH√log d √∑m j=1 σ 2 j + √ m1−2γ
(2γ−1) √ n
+ c7B
√ log(1/δ)
n .

heading: B EXPERIMENTAL RESULTS FOR OTHER PENALTY FUNCTIONS
text: In this section, we also implement our method with the penalty function proposed in Jiang et al. (2019) for training GANs. We consider WikiText-2 dataset, and all the experimental settings are same as before. Table 6 demonstrates the performance of our method using the Spectrum Normalization (SN) and SN+D-Optimal Regularizer, which can achieve the best performance of training GANs in their paper. The results show that their proposed penalty function can deteriorate the training of neural language models. This is because their method is motivated from training GANs, and will encourage all the singular values close to the largest one. If we use such penalty function to train neural language models, the learned word representations will lose the power of modeling contextual information. Therefore, our proposed spectrum control method is essential to improve the training of neural language generation.

heading: C TRAINING TIME AND MEMORY COST
text: In this section, we compare the training time and memory cost of our method with the baseline method, i.e., AWD-LSTM (Merity et al., 2018a) and standard Trandformer-XL (Dai et al., 2019). More specifically, we test our method and the baseline method on the same machine on WikiText-2 WikiText-103, and WMT 14 datasets. For WikiText-2 dataset, we use one NVIDIA Tesla V100 GPU and set the batch size to be 80. For WikiText-103 dataset, we use four NVIDIA Tesla V100 GPU and set the batch size to be 40. For WMT 14, we use four NVIDIA Tesla V100 GPU and set the max token as 3500. The training time and memory cost for a single GPU are summarized in Table 7. The results illustrate that our method is only 1.17×, 1.18×, and 1.24× slower than the baseline methods on WikiText-2, WikiText-103 and WMT 14 datasets, respectively. In addition, our method will cost 1.06×, 1.32× and 1.11×memory compared with the baseline methods on WikiText2, WikiText-103, and WMT 14 datasets, respectively.

heading: D COMPARISON OF TWO PRIORS
text: In this section we present the performance of our method on different tasks using different prior distributions. Table 8 suggests that the overall performance of these two prior distributions on different tasks are similar. In addition, the results show that it is better to use polynomial decay for large scale dataset and exponential decay for small scale dataset.

Review summaries:
This paper deals with the representation degeneration problem in neural language generation, as some prior works have found that the singular value distribution of the (input-output-tied) word embedding matrix decays quickly. The authors proposed an approach that directly penalizes deviations of the SV distribution from the two prior distributions, as well as a few other auxiliary losses on the orthogonality of U and V (which are now learnable). The experiments were conducted on small and large scale language modeling datasets as well as the relatively small IWSLT 2014 De-En MT dataset.