Variational Smoothing in Recurrent Neural Network Language Models

Abstract: We present a new theoretical perspective of data noising in recurrent neural network language models (Xie et al., 2017). We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution (i.e.,  a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution). We use this insight to propose a more principled  method to apply at prediction time and propose natural extensions to data noising under the variational framework. In particular, we propose variational smoothing  with tied input and output embedding matrices and an element-wise variational smoothing method. We empirically verify our analysis on two benchmark language modeling datasets and demonstrate performance improvements over existing data noising methods.

Reviews and author responses:

We thank the reviewer for the thoughtful comments. 

We totally agree with the reviewer that a comparison on larger dataset (billion word benchmark Chelba et al. 2013) will make the results stronger. It is an interesting question on itself to the data nosing method. If we have much larger data, will such smoothing method still be effective in the context of neural language models. We see that as an important future direction.

We thank the reviewer for the thoughtful comments. 

We agree that it is always nice (1) to show state of the art results, and (2) to build baselines upon the state of the art models. We also think the current set of results provides sufficient evidence to back up the main claims we made in the paper. We will edit the paper to reflect more contents on the state of the art models as shown in Yang et al, 2017.

The main point of the paper is to provide theoretical foundations of data noising by formulating it as a Bayesian RNN with a particular variational distribution (as well as natural extensions under this framework). Data noising as a regularization technique has been presented in Xie et al., 2017. We made no further claims about its effectiveness, although our results agree with Xie et al. that it is complementary to existing approaches (see below for more details). Our experiments were designed to verify our theoretical analysis. It also shows that natural extensions under this theoretical framework can further improves the data noising approach.

Regarding your specific comments:
- Comparisons with existing regularization approaches: we compared with the most standard regularization approaches for RNN LM, dropout and L2 regularization. All of our baseline methods and methods with data noising and variational smoothing were trained with dropout and L2 regularization. Our results (and Xie et al., 2017) show that both data noising and variational smoothing are complementary to existing methods.
- Applications to other tasks: Xie et al., 2017 showed that data noising also works for machine translation. We did not perform experiments on other real-world tasks in this paper since they are beyond the scope of our theoretical analysis.

We thank the reviewer for the thoughtful comments. 

- Discussion in Section 3: Yes, Section 3 and Bayesian NNs in general are from earlier work that go back to MacKay (1992) and Neal (1995). Gal and Ghahramani (2016) showed that dropout in RNNs can be viewed under this framework. We use all of these as foundations for our analysis in the paper.

- Determining \sigma: We assume in the paper that \sigma is a very small constant.

- Page 4, Training paragraph: We wanted to say that we go through the $t$-th sequence multiple times (once per epoch), and that every time we go through the training data we sample different configurations. We agree that the wording is confusing, so we have changed it a bit to make this clear.

- \ell_2 regularization for VS: Thank you for pointing this out. There is a minor typo in our original submission. In addition to \theta_i, each KL term includes a dataset specific regularization coefficient \lambda. What is tuned in Section 5.1 is the lambda coefficient. We have fixed this in the paper.

- Elementwise smoothing and sampling a mask: In dropout, since we only need to decide whether to drop (0) or keep (1), we could sample a binary mask to do this and multiply the parameters with the mask matrix. Here, since we need to decide which word in the vocab we would like to use, we need to sample from {1, 2, …, V}.
After sampling, instead of multiplying the parameters with a matrix, we need to perform a lookup depending on sampled indices. Therefore, this regularization is a lot more expensive than standard dropout.


The paper presents a Bayesian/ variational interpretation of data noising in recurrent networks (Xie et al. 2017). Overall I found the paper interesting and well presented.

The authors first review the work of Xie et al. 2017, that proposes data noisy for regularizing recurrent networks. This is done by randomly replacing certain words in the context according to some distribution. Xie et al. 2017 showed that this is highly related to smoothing in n-gram models.

The authors take a Bayesian approach where there is a prior over the parameters p(W) . Computing the posterior for RNNs is generally intractable so they suggest using a variational distribution q(W) instead. They show how certain choices of the variational distribution give a similar effect to different types of smoothing (e.g. linear interpolation and Kneser Ney), as well as show how for instance, combining smoothing with dropout fits into their theory. 

Experimenetal results show that their approach outperforms vanilla LSTMs and the approach of Xie et al. 2017 on PTB and Wikitext-2 which are two common (although small) benchmarks for language modeling.

The paper could be improved by running a comparison on larger dataset (e.g. billion word benchmark Chelba et al. 2013)

 



This submission closely builds upon an earlier work (Xie et al., 2017, Gal and Ghahramani, 2016) and proposes a new data noising technique motivated by Bayesian RNNs. Specifically, the key contribution is to extend Gal and Ghahramani (2016) to word embedding noising, while drawing inspiration from trational data smoothing techniques. Some variants are discussed, including those motivated by linear interpolation and Kneser-Ney smoothing, just as in Xie et al., 2017. Empirical evaluation is performed with language modeling experiments, and the proposed methods outperforms comparable baselines. One can imagine such a technique could be useful in many other sequence tasks. 

The paper is well-motivated and clearly written, and the experiments seem reasonable to me. Therefore I would vote for acceptance. My concern, which is not major, is that the proposed method might be a bit incremental based on Xie et al. (2017) and Gal and Ghahramani (2016).

Pros:
- Theoratical justification seems reasonably sound to me.
- Strong empirical performance.

Cons: 
- It would be interesting to see how the proposed technique works when applied to state-of-the-art models.

Details:

- I'm not entirely familiar with Gal and Ghahramani (2016), but I'm assuming the discussion in Section 3 and how it extends to word embeddings are from this earlier work. Please correct me if I'm wrong, so that I can adjust my recommendation accordingly.

- I can't find anything describing how \sigma is determined.

- Page 4, the paragraph of `Training.` I can't parse `we go though sequence t multiple times`.

- \ell_2 Regularization for `VS` models. I'm confused here, isn't the coefficient for VS determined by Eq. 3? Why is it still tuned in Section 5.1?

- Elementwise smoothing: I'm confused why one needs to sample \alpha for each dimension. Can't it be done by sampling a mask, just as in dropout?

In this submission, the authors present a variational smoothing interpretation of the data noising approach presented in (Xie et al., 2017). Although the theoretical coverage of the problem gives interesting insights. However, a comparison to related work w.r.t. alternative regularization approaches is missing. Similarly, the perplexity values reported in the experimental results on Penn Treebank are far away from state-of-the-art results published by many competitors on this task, e.g. see the current state-of-the-art results on Penn Treebank by (Yang et al., 2017, https://arxiv.org/pdf/1703.02573.pdf and references therein). It is bad practice to ignore existing work completely like this. The interesting question here would be, inhowfar the presented smoothing/regularization methods are complementary to existing approaches, and if the presented methods do provide improvements on top of these.

Finally, the mode of language modeling evaluation presented here, without considering an actual language or speech processing task, provides limited insight w.r.t. its utility in actual applications. Moreover, the very limited size of the language modeling tasks chosen here is highly advantageous for smoothing/regularization approaches. It remains totally unclear, how the presented approaches would perform on more realistically sized tasks and within actual applications.


Meta-review: as r1 and r2 have pointed out, this work presents an interesting and potentially more generalizable extension of the earlier work on introducing noise as regularization in autoregressive language modelling. although it would have been better with more extensive evaluation that goes beyond unsupervised language modelling and toward conditional language modelling, but i believe this is all fine for this further work to be left as follow-up.

r3's concern is definitely valid, but i believe the existing evaluation set as well as exposition merit presentation and discussion at the conference, which was shared by the other reviewers as well as a programme chair.

Decision: accepted-poster-papers