Variational Smoothing in Recurrent Neural Network Language Models

We present a new theoretical perspective of data noising in recurrent neural network language models ( Xie et al. , 2017 ) . We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution ( i.e. , a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution ) . We use this insight to propose a more principled method to apply at prediction time and propose natural extensions to data noising under the variational framework . In particular , we propose variational smoothing with tied input and output embedding matrices and an element-wise variational smoothing method . We empirically verify our analysis on two benchmark language modeling datasets and demonstrate performance improvements over existing data noising methods . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) are state of the art models in various language processing tasks . However , their performance heavily depends on proper regularization at training time ( Melis et al. , 2018b ; Merity et al. , 2018 ) . The two predominant approaches to regularize RNNs are dropout ( randomly zeroing out neurons ; Srivastava et al. , 2014 ) and ` 2 regularization ( applying ` 2 penalty to model parameters ; Hoerl & Kennard , 1970 ) . Recently , Xie et al . ( 2017 ) proposed data noising to regularize language models . Their method is formulated as a data augmentation method that randomly replaces words with other words drawn from a proposal distribution . For example , we can use the unigram distribution which models the number of word occurrences in the training corpus , or a more sophisticated proposal distribution that takes into account the number of bigram types in the corpus . Data noising has been shown to improve perplexity on small and large corpora , and that this improvement is complementary to other regularization techniques and translates to improvements on downstream models such as machine translation . Xie et al . ( 2017 ) derived connections between data noising and smoothing in classical n-gram language models , which we review in §2 . In smoothing ( Chen & Goodman , 1996 ) , since empirical counts for unseen sequences are zero , we smooth our estimates by a weighted average of higher order and lower order n-gram models . There are various ways to choose the weights and the lower order models leading to different smoothing techniques , with Kneser-Ney smoothing widely considered to be the most effective . Xie et al . ( 2017 ) showed that the pseudocounts of a noised data correspond to a mixture of different n-gram models . In this paper , we provide a new theoretical foundation for data noising and show that it can be understood as a form of Bayesian recurrent neural network with a particular variational distribution ( §4.1 and §4.2 ) . Our derivation relates data noising to dropout and variational dropout ( Gal & Ghahramani , 2016b ) , and naturally leads to a data dependent ` 2 regularization coefficient . We use this insight to arrive at a more principled way to do prediction with data noising—i.e. , by taking the mean of the variational distribution , as opposed to the mode—and propose several extensions under the variational framework in §4.3 . Specifically , we show how to use variational smoothing with tied input and output embeddings and propose element-wise smooothing . In §5 , we validate our analysis in language modeling experiments on the Penn Treebank ( Marcus et al. , 1994 ) and Wikitext-2 ( Merity et al. , 2017 ) datasets . 2 RECURRENT NEURAL NETWORK LANGUAGE MODELS . We consider a language modeling problem where the goal is to predict the next word xt given previously seen context words x < t = { x0 , x1 , . . . , xt−1 } . Let W be parameters of a recurrent neural network , and p ( xt | x < t ; W ) = RNN ( x < t ; W ) . Following previous work in language modeling ( Melis et al. , 2018b ; Merity et al. , 2018 ) , we use LSTM ( Hochreiter & Schmidhuber , 1997 ) as our RNN function , although other variants such as GRU ( Cho et al. , 2014 ) can be used as well . Given a training corpus D = { x0 , . . . , xT } , the likelihood we would like to maximize is : L ( W ; D ) = T∏ t=1 p ( xt | x < t ; W ) . Directly optimizing the ( log ) likelihood above often leads to overfitting . We typically augment the objective function with a regularizer ( e.g. , ` 2 regularizer where R ( W ) = λ‖W‖22 ) or use dropout by randomly zeroing out neurons ( Srivastava et al. , 2014 ) . Data noising as smoothing . Xie et al . ( 2017 ) proposed a method to regularize recurrent neural network language models by noising the data . For each input word i in x < t ( sometimes also the corresponding output word ) , we replace it with another word sampled from a proposal distribution T with probability γi . They introduced various methods on how to choose γi and T. For example , if γi = γ for all i ∈ V and T is the unigram distribution of words in the corpus , ET [ p ( xt | noiseγ , T ( x < t ) ; W ) ] corresponds to a mixture of n-gram models with fixed weights ( i.e. , linear interpolation smoothing ) . Another option is to set γi = γ distinct ( i , • ) count ( i ) , where distinct ( i , • ) denotes the number of distinct continuations preceded by word i ( i.e. , the number of bigram types that has i as the first word ) , and count ( i ) denotes the number of times i appears in the corpus . For this choice of γi , when T is the unigram distribution , the expectation corresponds to absolute discounting ; whereas if Ti = distinct ( • , i ) ∑ v∈V distinct ( • , v ) and we replace both the input and output words , it corresponds to bigram Kneser-Ney smoothing . We summarize their proposed methods in Table 1 . At prediction ( test ) time , Xie et al . ( 2017 ) do not apply any noising and directly predict p ( xt | x < t ; W ) . They showed that a combination of smoothing and dropout achieves the best result on their language modeling and machine translation experiments . 3 BAYESIAN RECURRENT NEURAL NETWORKS . In Bayesian RNNs , we define a prior over our parameters W and consider the following maximization problem : L ( D ) = ∫ T∏ t=1 p ( xt | x < t ; W ) p ( W ) dW . A common prior is the standard normal distribution p ( W ) = N ( 0 , I ) . For recurrent neural networks , the posterior over W given the data D is intractable . We approximate the posterior with a variational distribution q ( W ) , and minimize the KL divergence between the variational distribution and the true posterior by : KL ( q ( W ) ‖p ( W | D ) ) = KL ( q ( W ) ‖p ( W ) ) − ∫ log p ( D |W ) q ( W ) dW + constant = KL ( q ( W ) ‖p ( W ) ) − T∑ t=1 ∫ log p ( xt | x < t ; W ) q ( W ) dW + constant The integral is often approximated with Monte Carlo integration with one sample W̃ ∼ q ( W ) :1∫ log p ( xt | x < t ; W ) q ( W ) dW ≈ log p ( xt | x < t ; W̃ ) q ( W̃ ) dW̃ . ( 1 ) At test time , for a new sequence y0 , . . . , yT , we can either set W to be the mean of q ( W ) or sample and average the results : p ( yt | y < t ) = 1S ∑S s=1 p ( yt | y < t ; W̃s ) , where S is the number of samples and W̃s ∼ q ( W ) . 4 VARIATIONAL SMOOTHING . We now provide theoretical justifications for data smoothing under the variational framework . In a recurrent neural network language model , there are three types of parameters : an input ( word ) embedding matrix E , an LSTM parameter matrix R , and an output embedding matrix O that produces logits for the final softmax function . We have W = { E , R , O } . 4.1 LINEAR INTERPOLATION SMOOTHING .. We first focus on the simplest data noising method—linear interpolation smoothing—and show how to extend it to other data noising methods subsequently . Word embedding matrix E. Denote the word vector corresponding to word i ∈ V in the input embedding matrix E by ei . We obtain a similar effect to linear interpolation smoothing by using the following mixture of Gaussians variational distribution for ei : ei ∼ q ( ei ) = ( 1− γ ) N ( ei , σI ) + γ ∑ v∈V UvN ( ev , σI ) = ( 1− γ + γUi ) N ( ei , σI ) + γ ∑ v 6=i∈V UvN ( ev , σI ) , ( 2 ) where Uv is the unigram probability of word v and σ is small . In other words , with probability γUv , we replace the embedding ei with another embedding sampled from a normal distribution centered at ev . Note that noising the input word is equivalent to choosing a different word embedding vector to be used in a standard recurrent neural network . Under the variational framework , we sample a different word embedding matrix for every sequence t at training time , since the integral is approximated with Monte Carlo by sampling from q ( ei ) ( Eq . 1 ) . The above formulation is related to word embedding dropout ( Dai & Le , 2015 ; Iyyer et al. , 2015 ; Kumar et al. , 2016 ) , although in word embedding dropout q ( ei ) = ( 1− γ ) N ( ei , σI ) + γN ( 0 , σI ) . KL term . For p ( ei ) = N ( 0 , I ) , we use Proposition 1 in Gal & Ghahramani ( 2016a ) and approximate the KL divergence between a mixture of Gaussians q ( ei ) and p ( ei ) as : KL ( q ( ei ) ‖p ( ei ) ) ≈ λ ∑ v∈V θiv 2 ( e > v ev + tr ( σI ) − V ( 1 + log 2π ) − log |σI| ) = λ ∑ v∈V θiv 2 ‖ev‖22 + constant , 1In the followings , we use x̃ to denote a sample from a distribution q ( x ) . where V is the vocabulary size and θiv is the mixture proportion for word i : ( 1 − γ + γUi ) for v = i and γ otherwise . In practice , the KL term directly translates to an ` 2 regularizer on each word embedding vector , but the regularization coefficient is data dependent . More specifically , the regularization coefficient for word vector ei taking into account contributions from KL ( q ( ev ) ‖p ( ev ) ) for v ∈ V is : λ ( V − 1 ) γ + ( 1− γ + γUi ) 2 = λ 1− ( V − 2 + Ui ) γ 2 ( 3 ) In other words , the variational formulation of data smoothing results in a regularization coefficient that is a function of corpus statistics ( the unigram distribution ) . Other parameters . For other parameters R and O , we can use either simple variational distributions such as q ( R ) ∼ N ( R , σI ) and q ( O ) ∼ N ( O , σI ) , which become standard ` 2 regularizers on these parameters ; or incorporate dropout by setting q ( ri ) ∼ ( 1− α ) N ( ri , σI ) + αN ( 0 , σI ) , where α is the dropout probability and ri is the i-th row of R ( Gal & Ghahramani , 2016b ) . Training . In the original noising formulation ( Xie et al. , 2017 ) , given a sequence { a , b , a , c } , it is possible to get a noised sequence { d , b , a , c } , since the decision to noise at every timestep is independent of others . In the variational framework , we use Monte Carlo integration W̃ ∼ q ( W ) for each sequence to compute p ( xt | x < t ; W̃ ) . We can either sample one embedding matrix E per sequence similar to Gal & Ghahramani ( 2016b ) , or sample one embedding matrix per timestep ( Melis et al. , 2018a ) . While the first method is computationally more efficient ( we use it in our experiments ) , if we decide to noise the first a to d , we will have a noised sequence where every a is replaced by d : { d , b , d , c } . At training time , we go through sequence x < t multiple times ( once per epoch ) to get different noised sequences . Predictions . For predictions , Xie et al . ( 2017 ) do not noise any input . In the variational framework , this corresponds to taking the mode of the variational distribution , since 1 − γ is almost always greater than γ . They also reported that they did not observe any improvements by sampling . Using the mode is rather uncommon in Bayesian RNNs . A standard approach is to take the mean , so we can set : ēi = ( 1− γ + γUi ) ei + ∑ v 6=i∈V γUvev . Extension to absolute discounting . It is straightforward to extend linear interpolation smoothing to absolute discounting by setting γi = γ distinct ( i , • ) count ( i ) in Eq . 2 and Eq . 3 above .

Summary: This submission closely builds upon an earlier work (Xie et al., 2017, Gal and Ghahramani, 2016) and proposes a new data noising technique motivated by Bayesian RNNs. Specifically, the key contribution is to extend Gal and Ghahramani (2016) to word embedding noising, while drawing inspiration from trational data smoothing techniques. Some variants are discussed, including those motivated by linear interpolation and Kneser-Ney smoothing, just as in Xie et al., 2017. Empirical evaluation is performed with language modeling experiments, and the proposed methods outperforms comparable baselines. One can imagine such a technique could be useful in many other sequence tasks.