VARIATIONAL SMOOTHING IN RECURRENT NEURAL NETWORK LANGUAGE MODELS
Year: 2019

Abstract:
We present a new theoretical perspective of data noising in recurrent neural network language models (Xie et al., 2017). We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution (i.e., a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution). We use this insight to propose a more principled method to apply at prediction time and propose natural extensions to data noising under the variational framework. In particular, we propose variational smoothing with tied input and output embedding matrices and an element-wise variational smoothing method. We empirically verify our analysis on two benchmark language modeling datasets and demonstrate performance improvements over existing data noising methods.

text: We present a new theoretical perspective of data noising in recurrent neural network language models (Xie et al., 2017). We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution (i.e., a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution). We use this insight to propose a more principled method to apply at prediction time and propose natural extensions to data noising under the variational framework. In particular, we propose variational smoothing with tied input and output embedding matrices and an element-wise variational smoothing method. We empirically verify our analysis on two benchmark language modeling datasets and demonstrate performance improvements over existing data noising methods.

heading: 1 INTRODUCTION
text: Recurrent neural networks (RNNs) are state of the art models in various language processing tasks. However, their performance heavily depends on proper regularization at training time (Melis et al., 2018b; Merity et al., 2018). The two predominant approaches to regularize RNNs are dropout (randomly zeroing out neurons; Srivastava et al., 2014) and `2 regularization (applying `2 penalty to model parameters; Hoerl & Kennard, 1970). Recently, Xie et al. (2017) proposed data noising to regularize language models. Their method is formulated as a data augmentation method that randomly replaces words with other words drawn from a proposal distribution. For example, we can use the unigram distribution which models the number of word occurrences in the training corpus, or a more sophisticated proposal distribution that takes into account the number of bigram types in the corpus. Data noising has been shown to improve perplexity on small and large corpora, and that this improvement is complementary to other regularization techniques and translates to improvements on downstream models such as machine translation.
Xie et al. (2017) derived connections between data noising and smoothing in classical n-gram language models, which we review in §2. In smoothing (Chen & Goodman, 1996), since empirical counts for unseen sequences are zero, we smooth our estimates by a weighted average of higher order and lower order n-gram models. There are various ways to choose the weights and the lower order models leading to different smoothing techniques, with Kneser-Ney smoothing widely considered to be the most effective. Xie et al. (2017) showed that the pseudocounts of a noised data correspond to a mixture of different n-gram models.
In this paper, we provide a new theoretical foundation for data noising and show that it can be understood as a form of Bayesian recurrent neural network with a particular variational distribution (§4.1 and §4.2). Our derivation relates data noising to dropout and variational dropout (Gal & Ghahramani, 2016b), and naturally leads to a data dependent `2 regularization coefficient. We use this insight to arrive at a more principled way to do prediction with data noising—i.e., by taking the mean of the variational distribution, as opposed to the mode—and propose several extensions under the variational framework in §4.3. Specifically, we show how to use variational smoothing with tied input and output embeddings and propose element-wise smooothing. In §5, we validate our analysis in language modeling experiments on the Penn Treebank (Marcus et al., 1994) and Wikitext-2 (Merity et al., 2017) datasets.

heading: 2 RECURRENT NEURAL NETWORK LANGUAGE MODELS
text: We consider a language modeling problem where the goal is to predict the next word xt given previously seen context words x<t = {x0, x1, . . . , xt−1}. Let W be parameters of a recurrent neural network, and p(xt | x<t;W) = RNN(x<t;W). Following previous work in language modeling (Melis et al., 2018b; Merity et al., 2018), we use LSTM (Hochreiter & Schmidhuber, 1997) as our RNN function, although other variants such as GRU (Cho et al., 2014) can be used as well.
Given a training corpus D = {x0, . . . , xT }, the likelihood we would like to maximize is:
L(W;D) = T∏ t=1 p(xt | x<t;W).
Directly optimizing the (log) likelihood above often leads to overfitting. We typically augment the objective function with a regularizer (e.g., `2 regularizer where R(W) = λ‖W‖22) or use dropout by randomly zeroing out neurons (Srivastava et al., 2014).
Data noising as smoothing. Xie et al. (2017) proposed a method to regularize recurrent neural network language models by noising the data. For each input word i in x<t (sometimes also the corresponding output word), we replace it with another word sampled from a proposal distribution T with probability γi. They introduced various methods on how to choose γi and T.
For example, if γi = γ for all i ∈ V and T is the unigram distribution of words in the corpus, ET[p(xt | noiseγ,T(x<t);W)] corresponds to a mixture of n-gram models with fixed weights (i.e., linear interpolation smoothing).
Another option is to set γi = γ distinct(i,•)
count(i) , where distinct(i, •) denotes the number of distinct continuations preceded by word i (i.e., the number of bigram types that has i as the first word), and count(i) denotes the number of times i appears in the corpus. For this choice of γi, when T is the unigram distribution, the expectation corresponds to absolute discounting; whereas if Ti =
distinct(•,i)∑ v∈V distinct(•,v)
and we replace both the input and output words, it corresponds to bigram Kneser-Ney smoothing. We summarize their proposed methods in Table 1.
At prediction (test) time, Xie et al. (2017) do not apply any noising and directly predict p(xt | x<t;W). They showed that a combination of smoothing and dropout achieves the best result on their language modeling and machine translation experiments.

heading: 3 BAYESIAN RECURRENT NEURAL NETWORKS
text: In Bayesian RNNs, we define a prior over our parameters W and consider the following maximization problem:
L(D) = ∫ T∏ t=1 p(xt | x<t;W)p(W)dW.
A common prior is the standard normal distribution p(W) = N(0, I).
For recurrent neural networks, the posterior over W given the data D is intractable. We approximate the posterior with a variational distribution q(W), and minimize the KL divergence between the variational distribution and the true posterior by:
KL(q(W)‖p(W | D)) = KL(q(W)‖p(W))− ∫ log p(D |W)q(W)dW + constant
= KL(q(W)‖p(W))− T∑ t=1 ∫ log p(xt | x<t;W)q(W)dW + constant
The integral is often approximated with Monte Carlo integration with one sample W̃ ∼ q(W):1∫ log p(xt | x<t;W)q(W)dW ≈ log p(xt | x<t;W̃)q(W̃)dW̃. (1)
At test time, for a new sequence y0, . . . , yT , we can either set W to be the mean of q(W) or sample and average the results: p(yt | y<t) = 1S ∑S s=1 p(yt | y<t;W̃s), where S is the number of samples and W̃s ∼ q(W).

heading: 4 VARIATIONAL SMOOTHING
text: We now provide theoretical justifications for data smoothing under the variational framework. In a recurrent neural network language model, there are three types of parameters: an input (word) embedding matrix E, an LSTM parameter matrix R, and an output embedding matrix O that produces logits for the final softmax function. We have W = {E,R,O}.

heading: 4.1 LINEAR INTERPOLATION SMOOTHING.
text: We first focus on the simplest data noising method—linear interpolation smoothing—and show how to extend it to other data noising methods subsequently.
Word embedding matrix E. Denote the word vector corresponding to word i ∈ V in the input embedding matrix E by ei. We obtain a similar effect to linear interpolation smoothing by using the following mixture of Gaussians variational distribution for ei:
ei ∼ q(ei) = (1− γ)N(ei, σI) + γ ∑ v∈V UvN(ev, σI)
= (1− γ + γUi)N(ei, σI) + γ ∑ v 6=i∈V UvN(ev, σI), (2)
where Uv is the unigram probability of word v and σ is small. In other words, with probability γUv , we replace the embedding ei with another embedding sampled from a normal distribution centered at ev .
Note that noising the input word is equivalent to choosing a different word embedding vector to be used in a standard recurrent neural network. Under the variational framework, we sample a different word embedding matrix for every sequence t at training time, since the integral is approximated with Monte Carlo by sampling from q(ei) (Eq. 1).
The above formulation is related to word embedding dropout (Dai & Le, 2015; Iyyer et al., 2015; Kumar et al., 2016), although in word embedding dropout q(ei) = (1− γ)N(ei, σI) + γN(0, σI).
KL term. For p(ei) = N(0, I), we use Proposition 1 in Gal & Ghahramani (2016a) and approximate the KL divergence between a mixture of Gaussians q(ei) and p(ei) as:
KL(q(ei)‖p(ei)) ≈ λ ∑ v∈V θiv 2 ( e>v ev + tr(σI)− V (1 + log 2π)− log |σI| ) = λ
∑ v∈V θiv 2 ‖ev‖22 + constant,
1In the followings, we use x̃ to denote a sample from a distribution q(x).
where V is the vocabulary size and θiv is the mixture proportion for word i: (1 − γ + γUi) for v = i and γ otherwise. In practice, the KL term directly translates to an `2 regularizer on each word embedding vector, but the regularization coefficient is data dependent. More specifically, the regularization coefficient for word vector ei taking into account contributions from KL(q(ev)‖p(ev)) for v ∈ V is:
λ (V − 1)γ + (1− γ + γUi)
2 = λ 1− (V − 2 + Ui)γ 2
(3)
In other words, the variational formulation of data smoothing results in a regularization coefficient that is a function of corpus statistics (the unigram distribution).
Other parameters. For other parameters R and O, we can use either simple variational distributions such as q(R) ∼ N(R, σI) and q(O) ∼ N(O, σI), which become standard `2 regularizers on these parameters; or incorporate dropout by setting q(ri) ∼ (1− α)N(ri, σI) + αN(0, σI), where α is the dropout probability and ri is the i-th row of R (Gal & Ghahramani, 2016b).
Training. In the original noising formulation (Xie et al., 2017), given a sequence {a, b, a, c}, it is possible to get a noised sequence {d, b, a, c}, since the decision to noise at every timestep is independent of others. In the variational framework, we use Monte Carlo integration W̃ ∼ q(W) for each sequence to compute p(xt | x<t;W̃). We can either sample one embedding matrix E per sequence similar to Gal & Ghahramani (2016b), or sample one embedding matrix per timestep (Melis et al., 2018a). While the first method is computationally more efficient (we use it in our experiments), if we decide to noise the first a to d, we will have a noised sequence where every a is replaced by d: {d, b, d, c}. At training time, we go through sequence x<t multiple times (once per epoch) to get different noised sequences.
Predictions. For predictions, Xie et al. (2017) do not noise any input. In the variational framework, this corresponds to taking the mode of the variational distribution, since 1 − γ is almost always greater than γ. They also reported that they did not observe any improvements by sampling. Using the mode is rather uncommon in Bayesian RNNs. A standard approach is to take the mean, so we can set:
ēi = (1− γ + γUi)ei + ∑ v 6=i∈V γUvev.
Extension to absolute discounting. It is straightforward to extend linear interpolation smoothing to absolute discounting by setting γi = γ distinct(i,•) count(i) in Eq. 2 and Eq. 3 above.

heading: 4.2 KNESER-NEY SMOOTHING
text: We now consider the variational analog of Kneser-Ney data noising. Instead of smoothing towards lower order n-grams, Kneser-Ney uses models that take into account contextual diversity. For example, for bigram Kneser-Ney smoothing, we replace the unigram distribution with Ki =
distinct(•,i)∑ v∈V distinct(•,v)
, where distinct(•, i) denotes the number of distinct bigrams that end with word i. As a result, even if bigrams such as “San Francisco” and “Los Angeles” appear frequently in the corpus; “Francisco” and “Angeles” will not have high probabilities in K since they often follow “San” and “Los”.
Recall that similar to absolute discounting, Kneser-Ney also uses γi = γ distinct(i,•)
count(i) . However, if we replace an input word xt−1, Xie et al. (2017) proposed to also replace the corresponding output word xt. The intuition behind this is that since the probability of replacing an input word xt−1 is proportional to the number of distinct bigrams that start with xt−1, when we replace the input word, we also need to replace the output word (e.g., if we replace “San” by a word sampled from K, we should also replace “Francisco”).
Word embedding matrix E. To get (bigram) Kneser-Ney smoothing, we use the following mixture of Gaussians variational distribution for ei:
ei ∼ q(ei) = (1− γi)N(ei, σI) + γi ∑ v∈V KvN(ev, σI)
= (1− γi + γiKi)N(ei, σI) + γi ∑ v 6=i∈V KvN(ev, σI). (4)
Output embedding matrix O. For the output embedding matrix, we also use the same variational distribution as the word embedding matrix:
oi ∼ q(oi) = (1− γi)N(oi, σI) + γi ∑ v∈V KvN(ov, σI)
= (1− γi + γiKi)N(oi, σI) + γi ∑ v 6=i∈V KvN(ov, σI). (5)
KL term. Following similar derivations in the previous subsection, it is straightforward to show that the approximated KL term introduces the following regularization term to the overall objective:
λ ∑ v∈V 1− (V − 2 +Kv)γv 2 ‖ev‖22 + constant,
and similarly for the output embedding matrix O.
Training. Recall that we use Monte Carlo integration W̃ ∼ q(W) for each sequence to compute p(xt | x<t;W̃) at training time. Since we need to noise the output word when the input word is replaced, we sample γ̃i at t− 1 (for xt−1 = i) and decide whether we will keep the original (input and output) words or replace them. If we decide to replace, we need to sample two new words from KvN(ev, σI) and KvN(ov, σI) respectively. For other words k that are not the target at the current timestep (i.e., k 6= xt), we can either assume that õk = ok or alternatively sample (which can be expensive since we need to sample additional V − 1 times per timestep).
Predictions For predictions, Xie et al. (2017) also take the mode of the variational distribution (assuming 1− γi is almost always greater than γi). We use the mean instead, similar to what we do in variational linear interpolation smoothing.

heading: 4.3 EXTENSIONS
text: Our derivations above provide insights into new methods under the variational smoothing framework. We describe three variants in the followings:
Tying input and output embeddings. Inan et al. (2017) and Press & Wolf (2017) showed that tying the input and output embedding matrices (i.e., using the same embedding matrix for both) improves language modeling performance and reduces the number of parameters significantly. We take inspirations from this work and sample both O and E from the same base matrix. As a result, we have fewer parameters to train due to this sharing mechanism, but we still have different samples of input and output embedding matrices per sequence.2 Similar to previous results in language modeling (Inan et al., 2017; Melis et al., 2018b), our experiments demonstrate that tying improves the performance considerably.
Combining smoothing and dropout. We can combine variational smoothing and variational dropout by modifying the variational distribution to incorporate a standard Gaussian component:
ei ∼ q(ei) = (1− α) { (1− γi)N(ei, σI) + γi
∑ v∈V KvN(ev, σI)
} + αN(0, σI),
2Alternatively, we could sample one matrix for both the input and output embeddings. We found that this approach is slightly worse in our preliminary experiments.
where α is the dropout probability. Note that in this formulation, we either drop an entire word by setting its embedding vector to zero (i.e., similar to word embedding dropout and blank noising) or choose an embedding vector from the set of words in the vocabulary.
However, it is more common to apply dropout to each dimension of the input embedding and output embedding matrix. In this formulation, we have:
ei,j ∼ q(ei,j) = (1− αi,j) { (1− γi)N(ei,j , σ) + γi
∑ v∈V KvN(ev,j , σ)
} + αi,jN(0, σ). (6)
At training time, while we sample α̃i,j multiple times (once per word embedding dimension for each word), we only sample γ̃i once per word to ensure that when the element is not noised, we still use the same base embedding vector. We use this variant of smoothing and dropout throughout our experiments for our models.
Element-wise smoothing. The variational formulation above allows us to derive a variant of data smoothing that samples each element of the embedding vector ei independently (and similarly for oi). Consider the variational distribution in Eq. 6. At training time, if we sample both α̃i,j and γ̃i,j multiple times (once per word embedding dimension for each word) we arrive at a new element-wise smoothing method. The main difference between this model and the previous model is that each dimension in the input (and output) embedding vector is sampled independently. As a result, the vector that is used is a combination of elements from various word vectors. Notice that the mean under this new scheme is still the same as sampling per vector, so we do not need to change anything at test time. One major drawback about this model is that it is computationally expensive since we need to sample each element of each embedding vector.

heading: 5 EXPERIMENTS

heading: 5.1 SETUP
text: We evaluate our approaches on two standard language modeling datasets: Penn Treebank (PTB) and Wikitext-2. We use a two-layer LSTM as our base language model. We perform non-episodic training with batch size 64 using RMSprop (Hinton, 2012) as our optimization method. We tune the RMSprop learning rate and `2 regularization hyperparameter λ for all models on a development set by a grid search on {0.002, 0.003, 0.004} and {10−4, 10−3} respectively, and use perplexity on the development set to choose the best model. We also tune γ from {0.1, 0.2, 0.3, 0.4}. We use recurrent dropout (Semeniuta et al., 2016) for R and set it to 0.2, and apply (element-wise) input and output embedding dropouts for E and O and set it to 0.5 when E,O ∈ RV×512 and 0.7 when E,O ∈ RV×1024 based on preliminary experiments. We tie the input and output embedding matrices in all our experiments (i.e., E = O), except for the vanilla LSTM model, where we report results for both tied and untied.3

heading: 5.2 MODELS
text: We compare the following methods in our experiments:
• Baseline: a vanilla LSTM language model. We evaluate two variants: with tied input and output embeddings and without.
• Data noising (DN): an LSTM language model trained with data noising using linear interpolation smoothing or bigram Kneser-Ney smoothing (Xie et al., 2017).
• Variational smoothing (VS): an LSTM language model with variational smoothing using linear interpolation or Kneser-Ney. For both models, we use the mean of the variational distribution at test time.4
3Our preliminary experiments are consistent with previous work (Inan et al., 2017; Melis et al., 2018b; Merity et al., 2018) that show tying the input and output embedding matrices results in better models with fewer numbers of parameters.
4We discuss results using sampling at test time in §5.4.
• Variational element-wise smoothing: for the smaller PTB dataset, we evaluate an LSTM language model that uses elementwise Kneser-Ney variational smoothing and dropout. We also use the mean at test time.

heading: 5.3 RESULTS
text: Our results are summarized in Table 2. Consistent with previous work on tying the input and output embedding matrices in language models (Inan et al., 2017; Melis et al., 2018b; Merity et al., 2018), we see a large reduction in perplexity (lower is better) when doing so. While our numbers are generally better, the results are also consistent with Xie et al. (2017) that show linear interpolation data noising is slightly better than vanilla LSTM with dropout, and that Kneser-Ney data noising outperforms these methods for both the medium (512) and large (1024) models.
Variational smoothing improves over data noising in all cases, both for linear interpolation and Kneser-Ney. Recall that the main differences between variational smoothing and data noising are: (1) using the mean at test time, (2) having a data dependent `2 regularization coefficient that comes from the KL term,5 and (3) how each method interacts with the input and output embedding tying mechanism.6 Our results suggest that the choice of the proposal distribution to sample from is less important for variational smoothing. In our experiments, Kneser-Ney outperforms linear interpolation on PTB but linear interpolation is slightly better on Wikitext-2.
Element-wise variational smoothing performs the best for both small and large LSTM models on PTB. We note that this improvement comes at a cost, since this method is computationally expensive.7 It
5The data dependent `2 coefficient penalizes vectors that are sampled more often higher. 6 In the variational framework, if we sample one matrix for the input and output embeddings, it effectively noises the output words even for linear interpolation. If we sample two matrices from the same base matrix, these matrices can be different at training time even if the parameters are tied. As described in §4.3, we use the latter in our experiments.
7Element-wise dropout can be implemented efficiently by sampling a mask (zero or one with some probability) and multiply the entire embedding matrix with this mask. In element-wise smoothing, we need to sample an index for each dimension and reconstruct the embedding matrix for each timestep.
took about one day to train the smaller model as opposed to a couple hours without element-wise smoothing. As a result, we were unable to train this on a much bigger dataset such as Wikitext-2. Nonetheless, our results show that applying smoothing in the embedding (latent) space results in better models.

heading: 5.4 DISCUSSIONS
text: Sampling at test time. In order to better understand another prediction method for these models, we perform experiments where we sample at test time for both data noising and variational smoothing (instead of taking the mode or the mean). We use twenty samples and average the log likelihood. Our results agree with Xie et al. (2017) that mentioned sampling does not provide additional benefits. In our experiments, the perplexities of 512- and 1024-dimensional DN-Kneser-Ney models increase to 96.5 and 85.1 (from 75.0 and 71.4) on the PTB validation set. For VS-Kneser-Ney models, the perplexities increase to 89.8 and 78.7 (from 74.5 and 70.8). Both our results and Xie et al. (2017) suggest that introducing data dependent noise at test time is detrimental for recurrent neural network language models.
Sensitivity to γ. We evaluate the sensitivity of variational smoothing to hyperparameter γ. Figure 1 shows perplexities on the PTB validation set for a variant of our models. The model generally performs well within the range of 0.1 and 0.3, but it becomes progressively worse as we increase γ since there is too much noise.
Other applications. While we focus on language modeling in this paper, the proposed technique is applicable to other language processing tasks. For example, Xie et al. (2017) showed that data smoothing improves machine translation, so our techniques can be used in that setup as well. It is also interesting to consider how variational smoothing interacts with and/or can be applied to memory augmented language models (Tran et al., 2016; Grave et al., 2017; Yogatama et al., 2018) and state-of-the-art language models (Yang et al., 2017). We leave these for future work.

heading: 6 CONCLUSION
text: We showed that data noising in recurrent neural network language models can be understood as a Bayesian recurrent neural network with a variational distribution that consists of a mixture of Gaussians whose mixture weights are a function of the proposal distribution used in data noising (e.g., the unigram distribution, the Kneser-Ney continuation distribution). We proposed using the mean of the variational distribution at prediction time as a better alternative to using the mode. We combined it with variational dropout, presented two extensions (i.e., variational smoothing with tied input and output embedding matrices and element-wise smoothing), and demonstrated language modeling improvements on Penn Treebank and Wikitext-2.

Review summaries:
This submission closely builds upon an earlier work (Xie et al., 2017, Gal and Ghahramani, 2016) and proposes a new data noising technique motivated by Bayesian RNNs. Specifically, the key contribution is to extend Gal and Ghahramani (2016) to word embedding noising, while drawing inspiration from trational data smoothing techniques. Some variants are discussed, including those motivated by linear interpolation and Kneser-Ney smoothing, just as in Xie et al., 2017. Empirical evaluation is performed with language modeling experiments, and the proposed methods outperforms comparable baselines. One can imagine such a technique could be useful in many other sequence tasks. 

In this submission, the authors present a variational smoothing interpretation of the data noising approach presented in (Xie et al., 2017). Although the theoretical coverage of the problem gives interesting insights. However, a comparison to related work w.r.t. alternative regularization approaches is missing. Similarly, the perplexity values reported in the experimental results on Penn Treebank are far away from state-of-the-art results published by many competitors on this task, e.g. see the current state-of-the-art results on Penn Treebank by (Yang et al., 2017, https://arxiv.org/pdf/1703.02573.pdf and references therein). It is bad practice to ignore existing work completely like this. The interesting question here would be, inhowfar the presented smoothing/regularization methods are complementary to existing approaches, and if the presented methods do provide improvements on top of these.