Improving Neural Language Generation with Spectrum Control
Authors: Lingxiao Wang
Jing Huang
Kevin Huang
Ziniu Hu
Guangtao Wang
Quanquan Gu
Venue:  (2020)

Abstract: Recent Transformer-based models such as Transformer-XL and BERT have achieved huge success on various natural language processing tasks. However, contextualized embeddings at the output layer of these powerful models tend to degenerate and occupy an anisotropic cone in the vector space, which is called the representation degeneration problem. In this paper, we propose a novel spectrum control approach to address this degeneration problem. The core idea of our method is to directly guide the spectra training of the output embedding matrix with a slow-decaying singular value prior distribution through a reparameterization framework. We show that our proposed method encourages isotropy of the learned word representations while maintains the modeling power of these contextual neural models. We further provide a theoretical analysis and insight on the benefit of modeling singular value distribution. We demonstrate that our spectrum control method outperforms the state-of-the-art Transformer-XL modeling for language model, and various Transformer-based models for machine translation, on common benchmark datasets for these tasks.

Published as a conference paper at ICLR 2020
IMPROVING NEURAL LANGUAGE GENERATION WITH SPECTRUM CONTROL
Lingxiao Baang'JJing Huang", Kevin Huang", Ziniu Hu', Guangtao Wang?, Quanquan Gu'
2 'Department of Computer Science, University of California, Los Angeles JD AI Research, Mountain View, CA 94034
{lingxw bull, qgu}ccs. wcla.eclae edu (jing. huang, kevin, kevin.huang3, huang3, guangtao... @ com
ABSTRACT
Recent Transformer-based models such as Transformer-XL and BERT have achieved huge success on various natural language processing tasks. However, contextualized embeddings the output layer of these powerful models tend degenerate and occupy an anisotropic cone in the vector space, which is called the representation degeneration problem. this paper, we propose novel spec- trum control approach to address this degeneration problem. The core idea of our method is to directly guide the spectra training of the output embedding matrix with slow-decaying singular value prior distribution through reparameteriza- tion framework. We show that our proposed method encourages isotropy of the learned word representations while maintains the modeling power of these con- textual neural models. We further provide a theoretical analysis and insight on the benefit of modeling singular value distribution. We demonstrate that our spectrum control method outperforms the tfate-of-the-e Transformer-X -LL modeling for lan- guage model, and various Transformer-basse models for machine translation, on common benchmark datasets for these tasks.
INTRODUCTION
Neural language generation (NLG) an important task with many practical applications, such as automatic speech recognition GGrves et a 2013 Toshniwal et al. (018). text generation (Bowman cet al. 2016 Radford et al. 2019 Keskar et al.] 2019), machine translation (Bahdanau et al., 2015 Vaswani et al.) (017) and dialog systems (Gao et al. 2019a Tang et al. 2019). Most NLG models utilize a complex encoding model to map a given context into a hidden state vector, and then predict the next word distribution by multiplying the encoded vector with the output embedding layer, fol- lowed by a softmax layer. In the past few years, it has witnessed a significant progress in NLG by improving the encoding model, from the recurrent neural network (RNN) (Bahdanau et al.) 2015
Jozefowicz et al. 2016 Merity et al. 2018a) based models to the current Transformer-basse models (Vaswani al.) 2017: Devlin al. 2019 Dai et al. 2019 Radford al.] 2019). However, em-
beddings in the softmax output layer have been shown not capable enough to model the conditional
probability (Yang et 2018).
Recently, Gao et al. (2019b) pointed out another limitation of the output embeddings: the repre- sentation degeneration problem. They showed that the singular value distribution of the output embedding matrix tends to decay very fast, and the embedding space is squeezed into a narrow (as shown in Figure I(a) and i[e] 2-D plots). Such anisotropic shape (Ethayarajh 2019) is very different from what one would expect from an expressive word embedding space (Arora et al. 2016a Mu & Viswanath, 2018). Therefore, several efforts (Gao et al. 2019b Wang al. 2019a) have been made to address the degeneration problem.
Unlike previous approaches that applied implicit regularization to singular values of the output em- bedding matrix, we propose spectrum control (SC) approach, which was inspired by the spectral control technique used for Generative Adversarial Network (GAN) training (Jiang al. 2019), to explicitly control the singular value distribution. We first reparameterize the output embedding matrix W by its singular value decomposition (SVD): W UEVT, where U, V column orthonormal matrices, and Σ is a diagonal matrix of singular values. Then we guide the training of Σ by predefined slow-decaying prior distribution, such polynomial decay distribution, or

---

Published as a conference paper at ICLR 2020
-32
-1
F5
300
500
(a) Transformer-Xl.
(b) Singular Values
(c) Transformer
(d) Singular Values
Figure/ 1: Projected word embedding singular value distributions. (a) and (c): 2-D visualiza- tion of word embedding matrices of Transformer-XL for language modeling and Transformer for machine translation; (b) and (d): Normalized singular value distributions of embedding matrices.
4
150
26 21 41
140
Transformer-Xl S (b) Singular Values
(c) Transforme++SC
(d) Singular Values
Figure 2: Projected word embeddings and singular value distributions using our spectrum control method. (a) and (c): 2-D visualization of word embedding matrices of Transformer-XL. for lan- guage modeling and Transformer for machine translation; (b) and (d): Normalized singular value
distributions of embedding matrices.
an exponential decay distribution. the end of training, the distribution of singular values of the embedding matrix gets close to the prior distribution. Our spectrum control approach alleviates the representation degeneration problem by encouraging the diversity word representations and improving isotropic property of these representations (see Figure 2). Transformer- based models.
top of the powerful
We further present a theoretical analysis to justify our method. The results suggest that it is benefi- cial to directly guide the singular value distribution of the embedding matrix throughout the training process and control the decay rate of these singular values. We demonstrate the effectiveness of our training framework with extensive experimental results on two tasks: language modeling and ma- chine translation. Our spectrum control method outperforms the latest state-of-thh-arr Transformer- XL model on WikiTexx- dataset for language modeling; and obtains close to 1.5 BLEU improve- ment on IWSLT 2014 German- German-English translation task compared to the Transformer baseline model.
2 RELATED WORK
In this paper, we mainly focus on the output embedding matrix that is used in the softmax output layer for language generation tasks. This embedding matrix is also used as input word embeddings, which is known as weight tying trick. The weight tying not only reduces the number of parameters but also enjoys theoretical benefits (Inan et al. 2017). Thus the weight tying has been successfully applied to many state-of-the -het-art models for language modeling and machine translation (Merity al. 2018a Vaswani et al., 2017:) Yang et al. 2018).
However, there are certain issues with the softmax output layer. Yang et al. (2018) first identified a problem called "softmax bottleneck" that the softmax output layer does not have enough capacity to model natural language due to its connection with the rank bottleneck in matrix factorization. A simple and effective method called Mixture of Softmaxes (MoS) was proposed deal with this issue. Follow-up work in Kanai et al. (2018); Ganea et al. (2019) tried to replace softmax with al- ternative activation functions. Kanai et al. (2018) proposed to use sigsofmmax, which is composed of a multiplication of an exponential function and sigmoid function. Ganea et al. (2019) proposed a Manaa-MMnntnnii--- by learning parametric point-wise increasing functions to optimally distort the logits before feed- ing them into the softmax layer. Pappas Henderson (2019) instead resorted to a powerful deep
(LMS) model that generalized the approach Kanai et al. (2018)
Note that we project the original word emheddings a dimensional vector space using principal com- ponent analysis (PCA) for the purpose of visualization.
2

---

Published as a conference paper at ICLR 2020
residual nonlinear output mapping while using single softmax function without modifying its di- mensionality or rank.
Another line of work focuses on increasing the expressive power of the output embedding matrix by adding a cosine similarity regularization term (Gao al. 2019b), or adversarial noise (Wang al. 2019a). Gao et al. (2019b) analyzed the representation degeneration problem that the output embed- dings tend to degenerate and be distributed into a narrow cone. regularization term based on the summation of pairwise cosine similarity among all words was proposed to increase the representa- tion power word embeddings. Wang et al. (2019a) pointed out the computation of regularization term in (Gao et al.] 2019b) depends on the size of the vocabulary and hence is costly. Instead, they proposed a simple yet highly effective adversarial training method that adds adversarial noises to the output embedding layer when training the models. They proved in theory that this adversarial training increases the distances between two different words, and thus encourages the diversity of the embedding vectors. Our work follows this line of thought, but takes different approach: moti- vated by the spectrum control method for GAN training (Jiang et al., 2019), we propose to directly guide the singular values by a slow-decaying prior distribution during the model training process. We show that the anisotropic behavior of the contextualized word representations from powerful Transformer- meased models (Ethayarajh 2019) are alleviated by our spectrum control approach.
PROBLEM SETUP
In this section, we briefly introduce the neural models for language generation, and illustrate the sin- gular value decay phenomena of existing neural language models. We first introduce some notations used in the rest of the paper.
Notation:
For d-dimensional vector x E Idd, we use ||*||, EEin lo,/9)///, where 0 < q < oo
to denote its by -ormm and ||x||>< = max, |zg| to be its infinity norm. For a matrix A E Id xd2 let Ai be the i-th row of A, and we use || | || to denote its spectral norm, Frobenius norm, and matrix 1-norm. Given two sequences { {an) } and (b,, }, if there exists a constant 0 < C <
oo such that On I Cb,, we write Ob,)), and we use (() to hide the logarithmic factors.
3.1 NEURAL LANGUAGE GENERATION
We first briefly review the softmax output layer typically used in neural language generation models. We define the joint probability of given length-n sentence of words (tokens) 8,, = (yI,..., Yn) as the following product of conditional probabilities
Pss).) II P(ys/c)), [=1
(3.1)
where y. E V is the t-th word in sentence Sn, and V represents the word vocabulary, C₁ Vi-t-1 (Y1....,y--1) is referred to as the context of word y.- In addition, the context is usually modeled by a fixed size vector h, E Rdd which is referred to as the hidden state, using some neural networks such as LSTM (Hochreiter & Schmidhuber] 1997) and Transformer (Vaswani et al., (017). Then, the probability distribution of the output word yt given the context Ct, i.e., P(yz(c+) (3.1. parameterized as the following softmax function:
P(Y: = yrlc,) = IY,,, = yelb,) cxp(h, W WI(y))))
(3.2)
EN exp(h, W
where W RR is the weight matrix and is usually tied with the input word embedding matrix (Press& & Wolf 2017 Inan et al. 2017). N = | is the vocabulary size, d is the embedding dimen- sion, T(y:) represents the index of word y, in vocabulary V. In the following discussion, we call output weight matrix W as the word embedding matrix of the neural language model since it is tied with the input word embedding matrix.
In this paper we focus on the output layer of the neural model for language generation, iee. the softmax layer in (.2).. as in (Yang et al. 2018 Kanai et al. 2018 Ganea et al.) 2019: Gao et al.) 2019b)). We will examine the singular value distribution of W and propose a different approach to increase the expressive power of the word embedding.
3

---

Published as a conference paper at ICLR 2020
3.2 FAST SINGULAR VALUE DECAY
As we mentioned before. the singular values of W tend to drop very fast and it may interact with the cone-shaped embedding space. More specifically, Figure [(b] [(d) iIlsssrstt the distributions of the normalized singular values of the Transformer-Xi XL based language mode (Dai et al. 2019) and the Transformer-basee
machine translation model (Vaswani al. 2017) trained on Wikilex-- 103 (Merity et al. 201 8a) and /WSLT' 2014 De-En (Cettolo et al.) 2014) datasets, respectively. The plots show a fast singular value decay phenomenon, i.e., there is a huge drop between the first and remaining singular values. Such phenomenon has also been observed in some previous work Gao
ct al.
2019b Wang et al., 2019a).
Figure ((a) and [[c) present the distributions of the projected word embeddings of the aforemen- tioned two models. We can see from the plots that the projected word embeddings are distributed into some narrow cone shapes, which implies an anisotropic property of the learned word represen- tations, i.e. the embedding vectors are not uniformly distributed the space. The detailed analysis in Ethayarajh (2019) also confirms that contextualized word embeddings learnt from EL PPtters
al., 2018) LLSTM-hased model), BERT and GPT-2 (Devlin et al. 2019 Radford al. 2019) (Transformer-base models) indeed tend to be anisotropic.
These contextualized word embeddings have shown great success many NLP tasks. However, static word embeddings, such as Word2Vec (Mikolov et al. 2013) and GloVe (Pennington et al.) 2014] have been shown (Arora et al., 2016b Mu & Viswanath (018)) to be isotropic with great expressive power. Hence it may be also beneficial to increase the expressive power of the contex- tualized word embedding by increasing its isotropy. This motivates us to alleviate the fast singular value decay phenomenon to increase the isotropy of the learned word representations.
4 PROPOSED METHOD
To alleviate the fast singular value decay phenomenon, we propose to guide the singular value distribution of the contextualized word embedding throughout the training. As a result, we can achieve a trade-off between modeling contextual information, that tends to make word representa- tions anisotropic, and the expressive power of word representations, that tends to be isotropic.
4.1 SVD REPARAMETERIZATIO
Following the previous work (Jiang et al. 2019). we propose to apply singular value decomposition (SVD) based reparameterization
the embedding matrix W, i.e., W = UEVT where U E R. Nxd V E Rdxd are column orthonormal matrices, and Σ E Rddd is a diagonal matrix with standard and has been widely used in the literature such as model compression (Chen et al., 2018), training DNNs (Zhang et al.) 2018), and analyzing word embeddings (Arora et | [il][206gg 6c). Given the SVD reparameterization,,
Ees = og being the k-th largest singular value of W. Note that SVD reparameterization
can control the singular values of the embedding matrix W by constraining the matrix E, and the conditional distribution in (3.2) can be rewritten as follows:
PY, velv exp (h, (UEVT) )TG))))
Σ exp (h, (UEVT)..) subject
(4.1)
where a feasible set to represent the singular value distribution of the embedding matrix W. To ensure the orthogonal constraints in (4.1). we propose to use the orthogonal regularization for U. V during the training process, which has been previously used in (Jiang al. 2019). In par- ticular, we use the following regularization, which is a linear combination of Frobenius norm and spectral norm errors: Aı || U I| + X2 X² VV - I + AuI I1 + V V I where { are positive regularization parameters.
4.2 SPECTRUM CONTROL
Recall the SVD of W UEVT and consider the following two types of the singular value distribution for E, which is inspired by the eigenvalue distribution of kernel methods (Wei et al. 2017 Pacchiano al. 2019):
TThe plots are hased on the small Transformer-Xi model and Transformer- Base model, the detailed model configuration be found in experimental setups.
4

---

Published as a conference paper at ICLR 2020
Exponential decay: say the singular values fou of w satisfy the exponential de- cay if W E Pe(r) {W RNx oy I c exp(-cckk).k are universal constants].
....,,,,7 > 0, c1, C2
>
Polynomial decay: we say the singular values fox (o=]k=1 of W satisfy the polynomial decay W E Pp()) (W E R oh S c .,....d!" > 0, c1 > 0 is a universal constant).
For Pa(r) and Pp(T), the parameter Y controls the rate of singular value decay: the larger ty is, the faster singular value decay will be. To ensure the learned word embedding matrix W have the desired singular value distributions, we propose to add the following regularizations to our training objective: (E) = Ae Ei= (ox сi exp(-c2k")) for exponential decay and Rx( = Ap Ei_ (ox ei ) 2 for polynomial decay, where Xe, Ap are positive regulariza- tion parameters. Although the concept of "spectrum control" was previously used in Jiang et al. (2019) to improve the training of GANs, our spectrum control method is trying to solve totally different problem, i.e., neural language generation, and its motivation is coming from a very differ- ent perspective, i.e., the representation degeneration of the word representations. In addition, our method of controlling the singular value with prior distributions is significantly different from the penalty function used in their method. Finally, our method ís essential to improve the performance of the neural language generation, while the penalty function proposed in Jiang et al. (2019) can deteriorate the training of neural language models, illustrated in Appendix B
4.3 THEORETICAL ANALYSIS
In this subsection, show some theoretical insights why our proposed method can improve the performance of the NLG model. In particular, we follow the similar setup as considered in (Gao al. 2019b), i.e., focusing the optimization of the embedding matrix W E R xd and assume all the other parameters are fixed and well-optimized. In practice, we use our method to train the models from scratch. Therefore, according the output layer in (3.2). we consider the following empirical risk minimization problem: given a training dataset S = {(,, 3)} with each example drawn i..d. from some unknown but fixed distribution D, and h, E IR as a hidden state, y/2 E [1,...,,N as its associated label, our goal is minimize the training loss as follows:
1 R
Ls (W) WeRNxd
min
Σ E(h N,TW,W w.),),
(4.2)
where (h, W, y4) is the cross-entropy loss with respect to h. W and y/- The cross-entropy loss defined above is widely used to train NLG models, and is also used to compute the perplexity of the trained model, which is the benchmark criterion to evaluate the performance of language models. In addition, we define the expected loss as follows LD (W) (l w.y)]. where the expectation is taken over the distribution 'D of the training data.
Let W arg minwe (W)) where P(7) = (7) for exponential decay and P(7) = PP)())
for polynomial decay. We assume the right singular vector matrix satisfies || < V for all W E P(y) Now, we are ready to provide the main theory of our method (The proof can be found in Appendix A)
Theorem 4.1. Under previously stated conditions, suppose that [((.)] < B, l is G-Lipschitz con- tinuous, and |ibl S H for all = I...., n. we choose > 1/2, then with probability at least 1 - 6, we have
(D(W) I min Ls(W)+ + GAN (v EV 03 + WEP())
1)) + B2 log(1/8)
(4.3)
where C1, C2 are absolute constants, m E [2,d]. A
GV Mogd.
Remark 4.2. According to Theorem 4.1 the expected loss the learned embedding matrix W consists of two terms. The first term represents the training loss, the second term is the generalization error gap. Specifically, the smaller the y, the larger the feasible set P(7), thus the smaller the training loss minwep(7)) Ls( (W). On the other hand, the larger the y, the faster the singular value decays, and the smaller the generalization error gap. Therefore, our generalization error bound demonstrates an appealing property of proposed method: by directly controlling the singular value distribution
5

---

Published as a conference paper at ICLR 2020
of the learned word embedding, we are able achieve trade-off between the training loss and generalization error.
Remark 4.3. For the generalization error bound in 4.3) penalizing the largest singular value could
reduce this upper bound. It validates the method proposed in Gao et al. (2019b), which implicitly penalizes the largest singular value (c.f. Section 5 in Gao et al. (b).. Compared with their method, the error term O /ml-2x/10 singular value distribution, our method has a better control of the tail sum of the singular values.
1)/vn) suggests that by explicitly manipulating the
EXPERIMENTS
We demonstrate the effectiveness of our proposed spectrum control algorithm on two tasks: language modeling and machine translation. We compare our results with the state-of-the-ar models. In our experiments, we try both exponential and polynomial singular value decays, and present the one with the better result. The performances of exponential decay and polynomial decay are very close (See Appendix()). In practice we found that for large scale dataset its better to use polynomial decay and for small scale dataset its better use exponential decay. We also present the training time and memory cost of our method in Appendix C
5.1 LANGUAGE MODELING
Datasets We consider two benchmark datasets for language modeling: WikiTexI-2 and WikiText- 103, which consist of pre- pre-processed Wikipedia articles and were introduced by Merity et al. (2018a). WikiText-2 is small dataset with around 2 million words and 30K vocabulary size, while Wikilex- 103 is a significantly large dataset with around 103 million words and 260K vocabulary size.
Model Configuration On the small WikiText-2 dataset, we implement our method based on the state-of-the-ar AWD-LSTM model (Merity et al., 2018 8a). It is 3-layer LSTM model with 1150 dimensional hidden states and 400 dimensional embeddings. We also follow the same regulariza- tion and optimization procedures introduced in (Merity et al. 2018a). The implementation of our method is based on the open-source code for AWD-LSTM. On the large WikiText- 103 dataset, we implement our method based on the of-te--o-theaarr Transformer-X XL based models (Dai al., 2019). We follow the same settings reported in (Dai et al., 2019)), and our implementation is based on the official code"] for Transformer-Xl. To evaluate the performance of our method more thoroughly, we consider two Transformer-XL models with different number of layers. The first is the standard Transformer-Xe XL model with 16 layers used in (Dai et al. 2019). For the second, we consider a smaller Transformer-Xi model with just layers and other configurations unchanged.
Parameters For the parameters {>, of the orthogonal regularizations, tune them by grid search over {0.01, 0.1, 1, 10}. For the parameters de, Ap of the spectrum control, we tune them over the grid {0.1, 10, 1003. We try different singular value distributions, and the best distributions for AWD-LSTM and Transformer-XL are exponential and polynomial, respectively.
Results of the LSTM Model on WikiText-2 We first present the results of language modeling on the small dataset WikiTexr-2 2 using LSTM models. In Table we compare the validation/test perplex- ity5 of the baseline AWD-LSTM model (Merity et al. 2018a), the cosine similarity regularization (MLE-CosReg) model (Gao et al. 2019b), and the models trained using our method under three different settings (Merity al.) 201 Say: without finetune. with finetune and with further continu- ous cache pointer. Compared with the baselines, our method achieves 2.3/2.9/2. 2.3 test perplexity reduction under all three settings; compared with the MLE-CosRae method, our method achieves 1.5/1.2/0.3 test perplexity reduction under all three settings.
Results of the Transformer-XI. Model on WikiText-103 We next show the results of language modeling on the large dataset WikiText- using Transformer-X XL models. Table 2 compares the validation/test perplexity of Transformer-XI. based models (Dai et al. 2019) and the models trained by our method on WikiText- 103 dataset. The results demonstrate that our method consistently improves upon the small Transformer-Xi model (0.9 test perplexity reduction) and the standard Transformer-XX XL model (0.8 test perplexity reduction).
hhtps://gttuub.com. wom///kiiyouggttraaa tranor/ranffoome-xll II Lower perplexity means better language models.

---

Published as a conference paper at ICLR 2020
Table 1: Comparison of different methods in terms of perplexity on WikiText-2 dataset for the task of language modeling.
Method
Parameters
Validation
Test
Existing results
Variational LSTM (Inan 2017)
SIM 24M
91.5 69.1
87.0
2-layer skip connection LS TM Mandt et al. 2017)
65.9
w/o finetune
AWD-LSTM (Merity et al.
80
33M
69.1
66.0
MLE-CosReg (Gao el al, 2019b
33M
68.2
65.2
Ours
33M
66.3
63.7
finetune
AWD-LSTM (Merity et al. 201 Ra
33M
68.6
65.8
MLE-CosReg Gao el al. 20196
33M
67.9
Ours
33M
65.3
62.9
continuous cache pointer
AWD-LSTM (Merity et al MI.E-CosReg (Gao el al. 20196 Ours
88
33M
53.8
52.0
33M
51.7
50.0
33M
51.1
49.7
Table Comparison of different methods in terms of perplexity WikiText-103 dataset for the task of language modeling.
Method
Parameters
Validation
Test
Existing results
layer QRNN (Merity al.
2018b)
151M
32.0
33.0
Hebbian + Caclle (Rae et al.
2018
29.7
29.9
Small Transformer-X XL. (Dai al. Ours
2019
120M
29.6
30.4
120M
29.0
29.5
Standard Transformer-XI. Dai
2019)
151M
23.1
24.0
Ours
151M
22.9
23.2
Analysis We study the output embedding matrix of Transformer-Xo trained WikiText-103 using our method. In particular, we want to evaluate the isotropy of the learned word representations. We consider the partition function ((a) E exp( (w,,, introduced in (Arora et al. 2016b), where is the i-th row of the embedding matrix W E Rxd and E S is unit vector. According to Lemma 2.1 (Arora al.) (0166)),
word representation vectors are isotropic, Z (a) is close to constant with high probability for all unit vectors. Thus empirically measure the isotropy of the learned word representations, we consider two criteria based on (a):
I (W) min ((a)
Encr (Z(a) 7(a))²
and (w(
max.es ((a)
(E)2(a)2
where E is the set of cigenvectors of W W, as suggested by Mu & Viswanath (2018). We also propose check sampled standard deviation measure (2) (normalized by its average, i.e., Z(a)) We have (W) E [0. 1] and I2 (2(W)= 2 0. Larger (W) and smaller 12(W) indicate more isotropic for word embeddings. We uniformly sample 40K words from the vocabulary (around 260K) of WikiText-103 to compute these two criteria. The left half of Table 3 summarizes the values of (W)) and Iz( (W), which are averaged over 10 runs, for the baseline method and our method. We can from the results that method significantly improves the isotropy of the learned word representations in terms of both criteria.
5.2 MACHINE TRANSLATION
We also apply our spectrum control method to machine translation tasks. Given a source sentence the decoder of an neural machine translation (NMT) model is to predict the next word in the target sentence and the previous decoded words in t. In the following we use the state-of of-the-art Transformer-hased NMT model our baseline.

---

Published as a conference paper at ICLR 2020
Table 3: Comparison of different methods in terms of isotropy for the tasks of language modeling and machine translation. (For perfect isotropy, (W) = 1, (w) = 0.)
Language Modeling
Machine Translation
Method
I (W)
(W)
Method
I (W)
(W)
Standard Transformer-Xi
0.24
0.037
Transformer-Bal
0.31
0.031
Ours
0.63
0.022
Ours
0.88
0.005
Table Comparison of different methods in terms of BLEU scores on the task of Dn machine translation, trained on IWSLT 2014 dataset.
Method
IWSLT 2014 De
Adversarial (Wang al. 2019a)
Dual-learning earning
Transtormer-Ban
Ours
Wang et al. 2019b)
Wang et al. 2019b)
BLEU
38.18
35.44
34.01
35.50
Datasets We compare various NMT models on the IWSLT 2014 German->Englizn (De-En) and WMT 14 English~Geenin (En-De) datasets. For IWSLT 2014 De-En, we follow the same setup as in (Gehring al., 2017). More specifically,, we have 160K sentence pairs as the training data, 7K sentence pairs as the validation data, and we combine tsr2010, tst20/ I, 1st2012, dev2010 and dev2012 datasets to form our test data. For the large scale WMT 14 En-De, we have 4.5 million sentence pairs as our training data, and we use newstest2014 as our test data dataset (Cettolo et al. 2014).
Model Configuration We implement our method on top of Transformer model (Vaswani et al., 2017). In particular, we consider the Transformer- architecture (Vaswani et al.) (017)) for /WSL7 2014 De-En, which has a 6-layer encoder and 6-layer decoder with 512 dimensional hidden states and embeddings, except that choose the dimension of the inner feed-forward layer as 1024 in- stead of 2048 and the number of attention heads is set to be 4 rather than 8. For WMT 14 En-De, we consider the original Transformer- Base and Transformer-Bis Big architectures (Vaswani et al. 2017), which have a 6-layer encoder and 6-layer decoder with 512 and 1024 dimensional embeddings re- spectively. Our implementation is based on the open- sourced code provided by Ott et al. 2018 We follow the same procedures as in the language modeling task to choose the regularization param- eters. The best results for this task on /WSLTT' 2014 De-En are from the models where the singular values are controlled by exponential distribution, and the best results for this task on WMT 14 En-De are from the models where the singular values are controlled by polynomial distribution.
Results Comparisons of different methods in terms of BLEU scores for /WSLT 2014 De-En are summarized in T Compared with the baseline models, our method improves the BLEU score? from 34.01 to 35.50 on the German--Englist task, close to 1.5 gain on BLEU score; our method is also better than 35.18 reported in (Wang et al., 2019a), and 35.44 reported in (Wang al. 2019b). Table 5 summarizes different methods in terms of BLEU scores for WMT 14 En-De. The results show that our method achieves 1.15 and 0.92 BLEU score improvements on this task for base and big models, respectively. In addition, our method is also better than 28.38 and 28.94 reported in Gao al. 2019b) for base and big models.
Analysis We also study the learned word embedding matrix of the Transformer trained on /WSLT 2014 De-En using our method in terms of isotropy. The values of I, (W) and (2) are reported in the right half of Table 3 we compute these two values based on the tokens. shows that the isotropy of the learned word representations using our method increases significantly in terms of these criteria, from very anisotropic to nearly isotropic. We also demonstrate the projected word embedding matrices and the singular value distributions of different methods in Figure The plots show that the learned word representations from our method are distributed isotropically in the space, which is in contrast the narrow cone distribution from the baseline method.
"https:/github.com http://ghhuuboooyttorccrccc www/pptorch/faiigeeee
"Higher BLEU score represents higher quality of the machine translation.
8

---

Published as a conference paper at ICLR 2020
Table 5: Comparison of different methods in terms of BLEU scores on the task of En-*De machine
translation, trained on WMT 2014 dataset.
Method
BLEU
Transformer- Base (Vaswani el al. 2017
27.30 28.38
Transformer-Bel Baaee (Gao ct al 20196)
Transformer- Basc Ours
28.45
Transformer-Bi -Bg (Vaswani ct al. 2017
28.40
Transformer- Big (Gao 201 9b)
28.94
Transformer-Big Ours
29.32
Trandfo WMT
Namnalize Sngukar valu
84-
02-
0.0
0.-
0.44
-02
-34
-02
0.0
0.8 1.0 -0.6
-02
5.0
Singuler Vakie
(a) Transformer for NMT
(b) Transformer NMT+SC
(c) Singular Value Decay
distribution; (b) Word embedding for Transformer using spectrum control, which has uniform distribution; (c)
Figure 3: (a) Word embedding for the vanilla Transformer, which has narrow
Normalized singular value for different methods, which shows slow decay of our method.
6 CONCLUSIONS AND FUTURE WORK
In this paper, we tackle the degeneration problem that occurs at the output embeddings the soft- max layer used in neural language generation models. We develop novel spectrum control method to explicitly guide the spectra training of the output embedding matrix with some slow-decaying singular value prior distributions. Our proposed method is shown to alleviate the degeneration problem and improve isotropy of the learned contextualized word representations. Thorough ex- perimental results demonstrate the advantage of our method over the state-of-thes for language model and machine translation. Since our work is orthogonal to Wang et al.) (2019a), it would be interesting combine the adversarial softmax training with our spectrum control method, and investigate its performance. For the future work, we would also like to investigate how the rrequency-agnostic word representations in (Gong et al.) (0188) relate to the single value distribution of the output embedding matrix.
neural models
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers. NeurlPS, 2019.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach pmi-based word embeddings. TACI., 4:385-399 2016a.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pms-based word embeddings. TACL, 4:385-399, 2016b.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. simple but tough-to-beat baseline for sentence embeddings. 2016c.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In /CML,
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR, 2002.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. In Proceedings of the Twentieth Conference Computational Natural Language Learning, 2016.

---

Published as a conference paper at ICLR 2020
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report the 11th iwslt evaluation campaign, iwslt 2014. In SLT, 2014.
Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. Groupreduce: Block-wise low- rank approximation for neural language model shrinking. In Advances Neural Information Processing Systems, pp. (0988-10998, 2018.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. ACL, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geom- etry of bert, elmo, and gpt-2 embeddings. EMNLP, 2019.
Octavian-Eugee Ganea, Sylvain Gelly, Gary Bécigneul, and Aliaksei Severyn. Breaking the softmax bottleneck via learnable monotonic pointwise non- con-lincasstees ICML, 2019.
Jianfeng Gao, Michel Galley, Lihong Li, et al. Neural approaches to conversational ai. Foundations and Trends(R) in Information Retrieval, ((2-3))127-2999 2019a.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In ICLR, 2019b.
Jonas Gehring, Michael Auli, David Grangier. Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning, In ICML, pp. 1243-1252, 2017.
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Ren-Yanle Liu. Frage: frequency-agnostic word representation. NIPS, 2018.
A. Graves, A. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In ICASSP, 2013.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780) 1997.
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In ICLR, 2017.
Haoming Jiang, Zhehui Chen, Minshuo Chen, Feng Liu, Dingding Wang, and Tuo Zhao. On com- putation and generalization of generative adversarial networks under spectrum control. In ICLR, 2019.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. 2016.
Sckitoshi Kanai, Yasuhiro Fujiwara, Yuki Yamanaka, and Shuichi Adachi. Sigsoftmax: Reanalysis of the softmax bottleneck. In NIPS. 2018.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. drXiv:190.008858 2019.
Percy Liang. Cs229t/star231: Statistical learning theory (winter 2016). 2014.
Stephan Mandt, Matthew D Hoffmman, and David M Blei. Stochastic gradient descent approximate bayesian inference. JMLR, 2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In ICLR, 2018a.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. arXiv preprint arXiv: /803. 002008240, 2018b.
10

---

Published as a conference paper at ICLR 2020
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.3781, 2013.
Jiaqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In ICLR, 2018.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation, 2018.
Aldo Pacchiano, Niladri S Chatterji, and Peter L Bartlett. Online learning with kernel losses. ICML, 2019.
Nikolaos Pappas and James Henderson. Deep residual output layers for neural language generation. In ICMI, 2019.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In FACI., 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog. 1(8), 2019.
Jack Rae, Chris Dyer, Peter Dayan, and Timothy Lillicrap. Fast parametric learning with activation memorization. In ICML, 2018.
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric P. Xing, and Zhiting Hu. Target-guided open-domain conversation. In ACL, 2019.
Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara Sainath, and Karen Livescu. A comparison of techniques for language model integration in encoder-decoder speech recognition. In IEEE Workshop on Spoken Language Technology, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Dilin Wang, Chengyue Gong, and Qiang Liu. Improving neural language modeling via adversarial training. In ICML, 2019a.
Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu, Multi- agent dual learning. In ICLR, 2019b.
Yuting Wei, Fanny Yang, and Martin J Wainwright. Early stopping for kernel boosting algorithms: A general analysis with localized complexities. In NIPS, 2017.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In ICLR, 2018.
Jiong Zhang, Qi Lei, and Inderjit Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization. In International Conference on Machine Learning, pp. 5801-5809, 2018
11

---

Published as a conference paper at ICLR 2020
APPENDIX
A PROOF OF THEOREM 4.
Preliminaries We briefly review the general statistical learning framework. More specifically, let X' and y be the feature and label spaces, and suppose D is an unknown distribution over X' x V. Let F C V be the hypothesis class that we use to make prediction, and l V y + R be the loss function. In addition, we define the function class by = { (x, y) + {(((x,, y) ff E F} as the composition of the functions in F and loss f. Therefore, the goal is to minimize the expected risk LD = (f y) with some function f F.
According to the problem setup in Section 4.3 we have n i..d. training examples S {(h,, )= drawn from D, where the hidden state h, E R is the feature vector and y/ is the corresponding label. Suppose there exist a learning algorithm that maps the training dataset S to a function f E F, and we want to measure the gap between the empirical risk Ls and and the population risk LD. i.e., Ln Lsl. where Ls Ei (f.)),, /n. This gap is known as the generalization error. To bound the generalization error, we use the Rademacher complexity (Bartlett & Mendelson 2002). Let F C 2 be function class and S {zı,... 2n be a set of examples of size n, the empirical Rademacher complexity defined as
1 sup Σ Cuf((z)) Ens(x))) Rs(F)== = n
fEF i=1
where P1...., Pn. are i.i.d. Rademacher random variables with P(p: 1) = P(pp -1) = 1/22.
According to the loss function Ls defined in (.2 we are considering N function classes {F*] with F& = {f h -> (Wk, h), W e TW, W E P(y). E Rd), where e is a d-dimensional vector with k-th element to be one and others to be zero, and the loss R.N. - R. We have the following Lemma bound the generalization егтоr based on the empirical Rademacher complexity, which was proved in Corollary A.11 in Allen-Zhu et al. (2019).
Lemma A.1. (Allen-Zhu al., 2019) If F,.... FN are N classes of functions Rd + R, the loss function é IR + R is G-Lipschitz continuous and |e(-) S B for any D, then with probability at least 5, we have
E (()(f((z),..... .... fN(z))]]]]] n Σ e(f(z)))
< CGG Σ Rs(FF)
sup F
k=1
B Booog 1/5)
+ C2
vn
where the expectation is taken over the distribution D.
Equipped with this lemma, now present the proof of our main result.
Proof of Theorem 4.1 According to the problem setup in Section we have N classes of functions [Fi) and let the singular value decomposition of W as
12

---

Published as a conference paper at ICLR 2020
W UEVT T and Ug is the k-th row of U. Therefore, we have
Rs(F = p sup JEF, n
Σ
Pi(wKK: h.)
= EEp sup - (VEu hp) fEFk n
1
(A.1)
where h, = Ei= Püh; and the last inequality due to the definition of We = VEu. Therefore, we have
As(FF) S E,
sup JEJ n
VEu
V
S
E p
sup JEFk
11
0p/loo
oj
j=1
E p
sup fEFi
(A.2)
n
where the second inequality is due to ||
S V and the last one comes from
Eug |1 = Ej= uy|unsl S v /E= og. In addition, according to Theorem 12 in
Liang (2014). we have
E p h 1o0) S V 2n log d.
Therefore, we can get
m n=
of + Zixm o²
Rs(Fi) <
VH
/ log
220og d.
Vn
As result, we can get
m of Σ Ris (is NVH -
iim of + V
12 22log d.
(A.3)
Vn
Since for P()) = Pe(r), we have Jj S C1 exp((cc22) S c and for
P(7) = Pp(7), we have 0j S czj 7 Thus for > 1/2, we have
Σ
Σ
C4
< C4 L x 2r <
C4
(m + 1)
j2n
m+1
(27 1)
j>m
j>m
where ca (2)
) if P()) Pe(>) and = 33 if (1>)))))))))>))) = -Pp(7).
Therefore, plugging this upper bound into (A.3) we have
N
NVH WWH d
m
NVH log d m2 27 27/(27 1)
(s)
<
C5
Vn n
n
1=1
Where G c5 cf = Pe(7) and cs C3 i Pp()))
=
According to Lemma A.I since is G-Lipschitz continuous and |e(-) < B, we
have
m o3
(27 1)
(D(W) Ls(W)) < c6GNVH v log d
sup WEP(1)
n
log(1//4
+ C,B\
13

---

Published as a conference paper at ICLR 2020
Table 6: Comparison of different penalty functions in terms of perplexity on WikiText-2 dataset.
Method
Validation
Test
AWD-LSTM (Merity et al. 2018a)
69.1
66.0
Spectrum Normalization
76.3
72.5
SN+D-Optimal Regularizer
98.3
94.8
Ours
66.3
63.7
B EXPERIMENTAL RESULTS FOR OTHER PENALTY FUNCTIONS
In this section, we also implement our method with the penalty function proposed in Jiang al. (2019) for training GANs. We consider WikiText-2 dataset, and all the experimental settings are same as before. Table demonstrates the performance of our method using the Spectrum Normalization (SN) and SN+D-Optimal Regu- larizer, which can achieve the best performance of training GANs in their paper. The results show that their proposed penalty function can deteriorate the training of neural language models. This is because their method motivated from train- ing GANs, and will encourage all the singular values close to the largest one. If we use such penalty function to train neural language models, the learned word representations will lose the power of modeling contextual information. Therefore, our proposed spectrum control method is essential to improve the training of neural language generation.
C TRAINING TIME AND MEMORY COST
In this section, compare the training time and memory cost of method with the baseline method, i.e., AWD-LSTM (Merity et al. 2018a) and standard Trandformer-Xi (Dai et al., 2019). More specifically, we test method and the baseline method on the same machine on WikiText-2 WikiText-103, and WMT 14 datasets. For WikiText-2 dataset, batch size to be 80. For WikiText-103 103 dataset, we use four NVIDIA Tesla V100 GPU and set the batch size to be 40. For WMT 14, we four NVIDIA Tesla V100 GPU and set the max token as 3500. The training time and memory cost for a single GPU are summarized in Table 7 The results illustrate that our method is only 1.17x, 1. 1.18x, and 1.24x slower than the baseline methods on WikiTexx-2, WikiText-103 and WMT 14 datasets, respectively. In addition, our method will cost 1.06x, 1.32x and 1. x memory compared with the baseline methods on WikiText- 2, WikiText-103, and WMT 14 datasets, respectively.
one NVIDIA Tesla V100 GPU and set the
Table 7: Comparisons of our method and baseline method in terms of the average training time per
epoch and the memory cost.
Method
WikiTexx-2
WikiText-103
WMT 14
Training time
Baseline
61 sec 70 sec
2681 sec 3152 see
5991 sec 7421 sec
Ours
Memory cost
Baseline
8.9 GB
11.4 GB
14.2 GB 15.7 GB
Ours
9.5 GB
GB.0.BB
14

---

Published as a conference paper at ICLR 2020
Table 8: Comparisons of different priors on different datasets.
Dataset
Exponential Decay
Polynomial Decay
Language Modeling
Test PPL.
Test PPI.
WikiText-2 WikiText- 103
63.7
64.2
23.4
23.2
Machine Translation
BLEU
BLEU
IWSLT 2014 De
35.50 28.37
35.40
WMT 14 En->De
28.45
D COMPARISON OF TWO PRIORS
In this section we present the performance of our method on different tasks using different prior distributions. Table 8 suggests that the overall performance of these two prior distributions on different tasks are similar. In addition, the results show that it is better to use polynomial decay for large scale dataset and exponential decay for small scale dataset.
15


Reviews:
[{"review_id": "Skl7NjSCFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a regularizer for the output representation of transformer NNs, based on the singular value distribution to encourage learning of richer representations and avoid fast decay of singular values previously reported for NNs with softmax outputs.\nIn particular, the embedding matrix is parametrized as the product of a matrix U, a diagonal matrix Sigma and a matrix V. U and V are encouraged towards orhogonality using additional penalties similar to Lagrangian augmentation. Finally, a desired singular value distribution (exponential or polynomial decay) is encouraged by adding an appropriate regularization penalty on the entries of Sigma.\n\nThe authors present a generalization error bound that relates expected loss, training loss and singular value distribution to motiveate the choice of the regularizer.\nExperiments are provided for a machine translation and languate modeling, showing mild improvements of the proposed regularziaer over the state-of-the-art baselines.\n\nThe paper is well written, notation is clearly introduced and used in consistent manner, mathematical derivations are clear and easy to follow.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeHZ7XCtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose to apply Spectrum control regularization to the embedding of weight matrices in NLP problems such as language modeling and neural machine translation. Spectrum Control Regularization was originally proposed and applied to GANs (Jiang et al 2019)\n\nThe author motivate the approach by showing that the singular values of embedding weight matrices, although I am not convinced that it is such a big issue. In terms of experimental results authors show a very slight improvement over strong baseline models, that further shows an evidence that regularization singular values of embedding matrices is not very important. \n\nOverall the paper is written well, however the contribution is very marginal.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJg6QHLiYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper deals with the representation degeneration problem in neural language generation, as some prior works have found that the singular value distribution of the (input-output-tied) word embedding matrix decays quickly. The authors proposed an approach that directly penalizes deviations of the SV distribution from the two prior distributions, as well as a few other auxiliary losses on the orthogonality of U and V (which are now learnable). The experiments were conducted on small and large scale language modeling datasets as well as the relatively small IWSLT 2014 De-En MT dataset.\n\nPros:\n+ The paper is well-written with great clarity. The dimensionality of the involved matrices (and their decompositions) are clearly provided, and the approach is clearly described. The authors also did a great job providing the details of their experimental setup.\n+ The experiments seem to show consistent improvements over the baseline methods (at least the ones listed by the authors) on a relatively extensive set of tasks (e.g., of both small and large scales, of two different NLP tasks). Via WT2 and WT103, the authors also showed that their method worked on both LSTM and Transformers (which it should, as the SVD on word embedding should be independent of the underlying architecture).\n+ I think studying the expressivity of the output embedding matrix layer is a very interesting (and important) topic for NLP. (e.g., While models like BERT are widely used, the actual most frequently re-used module of BERT is its pre-trained word embeddings.)\n\n---------------------------------\n\nI have a few questions/comments on the work as well:\n\n1) One of the things that is not clearly described in the paper is how the proposed spectrum control method was injected into training in practice. In Section 4.3 (when you do the theoretical analysis), you assumed \"all the other parameters are fixed and well-optimized\". Is that also what you did in training on WT2/WT103 (e.g., first pre-train a model such as Transformer-XL, and then fine-tune its embedding layer using the proposed decomposed method)?\n\n2) How does the runtime and memory cost of your approach compare to the baselines? (e.g., you now need to compute $U^\\top U$, which can also be prohibitively large when the vocabulary size is large; for instance, on the 1-Billion word dataset).\n\n3) I remember the base Transformer-XL model did not use the adaptive embedding/softmax (although they set the `--adaptive` flag, they did use `--div_val 1`). How does your method work in the adaptive setting, where the embedding size $d$ of different words could be very different (e.g., depending on the word frequency)?\n\n4) Regarding the theoretical analysis in Section 4.3. Why is the cross-entropy loss function (essentially NLL loss + softmax) bounded (as is required by Theorem 4.1)? Given a fixed ground-truth $y_i$, if the corresponding predicted likelihood is small (e.g., $\\rightarrow 0$), won't the CE loss be very large? In that case, the generalization bound in Eq. (4.3) would be vacuous, as B would be very large too. Moreover, I'm also not fully convinced by what the theoretical analysis is trying to convey--- that your method \"achieve a trade-off between the training loss and generalization error\"--- but isn't that what (any) regularizations are designed for? The proved bound is no different in nature from any generalization bound with a regularizer (e.g., weight decay), and it does not necessarily reflect the usefulness of the proposed approach. \n\n5) In experiments, you only showed the better performance of the exponential and the polynomial singular value decays (cf. beginning of Sec. 5). Could you show both? Which one is better (and on what task), and by how much? If one is to use your method, which decay scheme do you recommend?\n\n6) Why is MLE-CosReg (Gao et al. 2019b) not compared to in the WikiText-103 and the machine translation task? As the MLE-CosReg approach only involves regularizing the cosine similarity via $\\text{Sum}(\\hat{W}\\hat{W}^\\top)$, it should computationally be even slightly cheaper than the proposed method (you need to compute $\\mathbf{U}^\\top \\mathbf{U}$, which has the same complexity). They also tested on the larger-scale WMT En-De and De-En dataset (which contains 4.5M sentence pairs). Is there any reason that you chose IWSLT 2014 instead?\n\n---------------------------------\n\nSome issues that didn't impact the score:\n\n7) It'd be useful to add labels to the x- and y-axis of the plots in Figure 1.\n\n8) When implementing the method described in Section 4.2, did you explicitly sort the singular values in each iteration, or just set them to learnable parameters (along with learnable $\\mathbf{U}, \\mathbf{V}$) without sorting? If you do sort, do you also \"sort\" the columns of $\\mathbf{U}$ and $\\mathbf{V}$ (as I would expect a one-to-one mapping from $\\sigma_i$ to $\\mathbf{U}_i$, for instance)? If you don't sort, how do you make sure that $\\sigma_i \\geq \\sigma_{i+1}$?\n\n9) Why did you regularize $\\mathbf{U}$ and $\\mathbf{V}$ by both its Frobenius norm and its spectral norm? Does using only one of them compromise the performance?\n\n---------------------------------\n\nOverall, I find this work well-written and well-motivated. The experiments seem to show consistent improvement when using the approach. I vote for weak accept, but I also look forward to the author's response to the questions I raised above.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

Blind review #2 summarizes it well:

Summary: This paper deals with the representation degeneration problem in neural language generation, as some prior works have found that the singular value distribution of the (input-output-tied) word embedding matrix decays quickly. The authors proposed an approach that directly penalizes deviations of the SV distribution from the two prior distributions, as well as a few other auxiliary losses on the orthogonality of U and V (which are now learnable). The experiments were conducted on small and large scale language modeling datasets as well as the relatively small IWSLT 2014 De-En MT dataset.

Pros:
+ The paper is well-written with great clarity. The dimensionality of the involved matrices (and their decompositions) are clearly provided, and the approach is clearly described. The authors also did a great job providing the details of their experimental setup.
+ The experiments seem to show consistent improvements over the baseline methods (at least the ones listed by the authors) on a relatively extensive set of tasks (e.g., of both small and large scales, of two different NLP tasks). Via WT2 and WT103, the authors also showed that their method worked on both LSTM and Transformers (which it should, as the SVD on word embedding should be independent of the underlying architecture).
+ I think studying the expressivity of the output embedding matrix layer is a very interesting (and important) topic for NLP. (e.g., While models like BERT are widely used, the actual most frequently re-used module of BERT is its pre-trained word embeddings.)

--

Discussion:

The reviewers agree that it is a very well written paper, and this is important as a conference paper to illuminate readers.

The one main objection is that spectrum control regularization was previously proposed and applied to GANs (Jiang et al ICLR 2019). However the authors convincingly point out that the technique is widely used, not only for GANs, and that application to neural language generation has quite different characteristics requiring a different, new approach: "our proposed prior distributions as shown in Figure 2 in our paper are fundamentally different from the singular value distributions learned using their penalty functions (See Figure 1 and Table 7 in Jiang et al.’s paper). Figure 1 in their paper suggests that their penalty function, i.e., D-optimal Reg, will encourage all the singular values close to 1, which is well aligned with their motivation for training GAN. However, if we use such penalty function to train neural language models, the learned word representations will lose the power of modeling contextual information, and can result in much worse results than the baseline methods."

--

Recommendation and justification:

I concur with the majority of reviewers that this paper is a weak accept. Though not revolutionary, it is well written, has usefully broad application, and is supported well empirically.
Decision: Accept (Poster)