Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a regularizer for the output representation of transformer NNs, based on the singular value distribution to encourage learning of richer representations and avoid fast decay of singular values previously reported for NNs with softmax outputs.
In particular, the embedding matrix is parametrized as the product of a matrix U, a diagonal matrix Sigma and a matrix V. U and V are encouraged towards orhogonality using additional penalties similar to Lagrangian augmentation. Finally, a desired singular value distribution (exponential or polynomial decay) is encouraged by adding an appropriate regularization penalty on the entries of Sigma.

The authors present a generalization error bound that relates expected loss, training loss and singular value distribution to motiveate the choice of the regularizer.
Experiments are provided for a machine translation and languate modeling, showing mild improvements of the proposed regularziaer over the state-of-the-art baselines.

The paper is well written, notation is clearly introduced and used in consistent manner, mathematical derivations are clear and easy to follow.


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Authors propose to apply Spectrum control regularization to the embedding of weight matrices in NLP problems such as language modeling and neural machine translation. Spectrum Control Regularization was originally proposed and applied to GANs (Jiang et al 2019)

The author motivate the approach by showing that the singular values of embedding weight matrices, although I am not convinced that it is such a big issue. In terms of experimental results authors show a very slight improvement over strong baseline models, that further shows an evidence that regularization singular values of embedding matrices is not very important. 

Overall the paper is written well, however the contribution is very marginal.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary: This paper deals with the representation degeneration problem in neural language generation, as some prior works have found that the singular value distribution of the (input-output-tied) word embedding matrix decays quickly. The authors proposed an approach that directly penalizes deviations of the SV distribution from the two prior distributions, as well as a few other auxiliary losses on the orthogonality of U and V (which are now learnable). The experiments were conducted on small and large scale language modeling datasets as well as the relatively small IWSLT 2014 De-En MT dataset.

Pros:
+ The paper is well-written with great clarity. The dimensionality of the involved matrices (and their decompositions) are clearly provided, and the approach is clearly described. The authors also did a great job providing the details of their experimental setup.
+ The experiments seem to show consistent improvements over the baseline methods (at least the ones listed by the authors) on a relatively extensive set of tasks (e.g., of both small and large scales, of two different NLP tasks). Via WT2 and WT103, the authors also showed that their method worked on both LSTM and Transformers (which it should, as the SVD on word embedding should be independent of the underlying architecture).
+ I think studying the expressivity of the output embedding matrix layer is a very interesting (and important) topic for NLP. (e.g., While models like BERT are widely used, the actual most frequently re-used module of BERT is its pre-trained word embeddings.)

---------------------------------

I have a few questions/comments on the work as well:

1) One of the things that is not clearly described in the paper is how the proposed spectrum control method was injected into training in practice. In Section 4.3 (when you do the theoretical analysis), you assumed "all the other parameters are fixed and well-optimized". Is that also what you did in training on WT2/WT103 (e.g., first pre-train a model such as Transformer-XL, and then fine-tune its embedding layer using the proposed decomposed method)?

2) How does the runtime and memory cost of your approach compare to the baselines? (e.g., you now need to compute $U^\top U$, which can also be prohibitively large when the vocabulary size is large; for instance, on the 1-Billion word dataset).

3) I remember the base Transformer-XL model did not use the adaptive embedding/softmax (although they set the `--adaptive` flag, they did use `--div_val 1`). How does your method work in the adaptive setting, where the embedding size $d$ of different words could be very different (e.g., depending on the word frequency)?

4) Regarding the theoretical analysis in Section 4.3. Why is the cross-entropy loss function (essentially NLL loss + softmax) bounded (as is required by Theorem 4.1)? Given a fixed ground-truth $y_i$, if the corresponding predicted likelihood is small (e.g., $\rightarrow 0$), won't the CE loss be very large? In that case, the generalization bound in Eq. (4.3) would be vacuous, as B would be very large too. Moreover, I'm also not fully convinced by what the theoretical analysis is trying to convey--- that your method "achieve a trade-off between the training loss and generalization error"--- but isn't that what (any) regularizations are designed for? The proved bound is no different in nature from any generalization bound with a regularizer (e.g., weight decay), and it does not necessarily reflect the usefulness of the proposed approach. 

5) In experiments, you only showed the better performance of the exponential and the polynomial singular value decays (cf. beginning of Sec. 5). Could you show both? Which one is better (and on what task), and by how much? If one is to use your method, which decay scheme do you recommend?

6) Why is MLE-CosReg (Gao et al. 2019b) not compared to in the WikiText-103 and the machine translation task? As the MLE-CosReg approach only involves regularizing the cosine similarity via $\text{Sum}(\hat{W}\hat{W}^\top)$, it should computationally be even slightly cheaper than the proposed method (you need to compute $\mathbf{U}^\top \mathbf{U}$, which has the same complexity). They also tested on the larger-scale WMT En-De and De-En dataset (which contains 4.5M sentence pairs). Is there any reason that you chose IWSLT 2014 instead?

---------------------------------

Some issues that didn't impact the score:

7) It'd be useful to add labels to the x- and y-axis of the plots in Figure 1.

8) When implementing the method described in Section 4.2, did you explicitly sort the singular values in each iteration, or just set them to learnable parameters (along with learnable $\mathbf{U}, \mathbf{V}$) without sorting? If you do sort, do you also "sort" the columns of $\mathbf{U}$ and $\mathbf{V}$ (as I would expect a one-to-one mapping from $\sigma_i$ to $\mathbf{U}_i$, for instance)? If you don't sort, how do you make sure that $\sigma_i \geq \sigma_{i+1}$?

9) Why did you regularize $\mathbf{U}$ and $\mathbf{V}$ by both its Frobenius norm and its spectral norm? Does using only one of them compromise the performance?

---------------------------------

Overall, I find this work well-written and well-motivated. The experiments seem to show consistent improvement when using the approach. I vote for weak accept, but I also look forward to the author's response to the questions I raised above.
review_title: Official Blind Review #2

Meta-review: Main content:

Blind review #2 summarizes it well:

Summary: This paper deals with the representation degeneration problem in neural language generation, as some prior works have found that the singular value distribution of the (input-output-tied) word embedding matrix decays quickly. The authors proposed an approach that directly penalizes deviations of the SV distribution from the two prior distributions, as well as a few other auxiliary losses on the orthogonality of U and V (which are now learnable). The experiments were conducted on small and large scale language modeling datasets as well as the relatively small IWSLT 2014 De-En MT dataset.

Pros:
+ The paper is well-written with great clarity. The dimensionality of the involved matrices (and their decompositions) are clearly provided, and the approach is clearly described. The authors also did a great job providing the details of their experimental setup.
+ The experiments seem to show consistent improvements over the baseline methods (at least the ones listed by the authors) on a relatively extensive set of tasks (e.g., of both small and large scales, of two different NLP tasks). Via WT2 and WT103, the authors also showed that their method worked on both LSTM and Transformers (which it should, as the SVD on word embedding should be independent of the underlying architecture).
+ I think studying the expressivity of the output embedding matrix layer is a very interesting (and important) topic for NLP. (e.g., While models like BERT are widely used, the actual most frequently re-used module of BERT is its pre-trained word embeddings.)

--

Discussion:

The reviewers agree that it is a very well written paper, and this is important as a conference paper to illuminate readers.

The one main objection is that spectrum control regularization was previously proposed and applied to GANs (Jiang et al ICLR 2019). However the authors convincingly point out that the technique is widely used, not only for GANs, and that application to neural language generation has quite different characteristics requiring a different, new approach: "our proposed prior distributions as shown in Figure 2 in our paper are fundamentally different from the singular value distributions learned using their penalty functions (See Figure 1 and Table 7 in Jiang et al.’s paper). Figure 1 in their paper suggests that their penalty function, i.e., D-optimal Reg, will encourage all the singular values close to 1, which is well aligned with their motivation for training GAN. However, if we use such penalty function to train neural language models, the learned word representations will lose the power of modeling contextual information, and can result in much worse results than the baseline methods."

--

Recommendation and justification:

I concur with the majority of reviewers that this paper is a weak accept. Though not revolutionary, it is well written, has usefully broad application, and is supported well empirically.
Decision: Accept (Poster)