Variational Smoothing in Recurrent Neural Network Language Models

We present a new theoretical perspective of data noising in recurrent neural network language models ( Xie et al. , 2017 ) . We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution ( i.e. , a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution ) . We use this insight to propose a more principled method to apply at prediction time and propose natural extensions to data noising under the variational framework . In particular , we propose variational smoothing with tied input and output embedding matrices and an element-wise variational smoothing method . We empirically verify our analysis on two benchmark language modeling datasets and demonstrate performance improvements over existing data noising methods . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) are state of the art models in various language processing tasks . However , their performance heavily depends on proper regularization at training time ( Melis et al. , 2018b ; Merity et al. , 2018 ) . The two predominant approaches to regularize RNNs are dropout ( randomly zeroing out neurons ; Srivastava et al. , 2014 ) and ` 2 regularization ( applying ` 2 penalty to model parameters ; Hoerl & Kennard , 1970 ) . Recently , Xie et al . ( 2017 ) proposed data noising to regularize language models . Their method is formulated as a data augmentation method that randomly replaces words with other words drawn from a proposal distribution . For example , we can use the unigram distribution which models the number of word occurrences in the training corpus , or a more sophisticated proposal distribution that takes into account the number of bigram types in the corpus . Data noising has been shown to improve perplexity on small and large corpora , and that this improvement is complementary to other regularization techniques and translates to improvements on downstream models such as machine translation . Xie et al . ( 2017 ) derived connections between data noising and smoothing in classical n-gram language models , which we review in §2 . In smoothing ( Chen & Goodman , 1996 ) , since empirical counts for unseen sequences are zero , we smooth our estimates by a weighted average of higher order and lower order n-gram models . There are various ways to choose the weights and the lower order models leading to different smoothing techniques , with Kneser-Ney smoothing widely considered to be the most effective . Xie et al . ( 2017 ) showed that the pseudocounts of a noised data correspond to a mixture of different n-gram models . In this paper , we provide a new theoretical foundation for data noising and show that it can be understood as a form of Bayesian recurrent neural network with a particular variational distribution ( §4.1 and §4.2 ) . Our derivation relates data noising to dropout and variational dropout ( Gal & Ghahramani , 2016b ) , and naturally leads to a data dependent ` 2 regularization coefficient . We use this insight to arrive at a more principled way to do prediction with data noising—i.e. , by taking the mean of the variational distribution , as opposed to the mode—and propose several extensions under the variational framework in §4.3 . Specifically , we show how to use variational smoothing with tied input and output embeddings and propose element-wise smooothing . In §5 , we validate our analysis in language modeling experiments on the Penn Treebank ( Marcus et al. , 1994 ) and Wikitext-2 ( Merity et al. , 2017 ) datasets . 2 RECURRENT NEURAL NETWORK LANGUAGE MODELS . We consider a language modeling problem where the goal is to predict the next word xt given previously seen context words x < t = { x0 , x1 , . . . , xt−1 } . Let W be parameters of a recurrent neural network , and p ( xt | x < t ; W ) = RNN ( x < t ; W ) . Following previous work in language modeling ( Melis et al. , 2018b ; Merity et al. , 2018 ) , we use LSTM ( Hochreiter & Schmidhuber , 1997 ) as our RNN function , although other variants such as GRU ( Cho et al. , 2014 ) can be used as well . Given a training corpus D = { x0 , . . . , xT } , the likelihood we would like to maximize is : L ( W ; D ) = T∏ t=1 p ( xt | x < t ; W ) . Directly optimizing the ( log ) likelihood above often leads to overfitting . We typically augment the objective function with a regularizer ( e.g. , ` 2 regularizer where R ( W ) = λ‖W‖22 ) or use dropout by randomly zeroing out neurons ( Srivastava et al. , 2014 ) . Data noising as smoothing . Xie et al . ( 2017 ) proposed a method to regularize recurrent neural network language models by noising the data . For each input word i in x < t ( sometimes also the corresponding output word ) , we replace it with another word sampled from a proposal distribution T with probability γi . They introduced various methods on how to choose γi and T. For example , if γi = γ for all i ∈ V and T is the unigram distribution of words in the corpus , ET [ p ( xt | noiseγ , T ( x < t ) ; W ) ] corresponds to a mixture of n-gram models with fixed weights ( i.e. , linear interpolation smoothing ) . Another option is to set γi = γ distinct ( i , • ) count ( i ) , where distinct ( i , • ) denotes the number of distinct continuations preceded by word i ( i.e. , the number of bigram types that has i as the first word ) , and count ( i ) denotes the number of times i appears in the corpus . For this choice of γi , when T is the unigram distribution , the expectation corresponds to absolute discounting ; whereas if Ti = distinct ( • , i ) ∑ v∈V distinct ( • , v ) and we replace both the input and output words , it corresponds to bigram Kneser-Ney smoothing . We summarize their proposed methods in Table 1 . At prediction ( test ) time , Xie et al . ( 2017 ) do not apply any noising and directly predict p ( xt | x < t ; W ) . They showed that a combination of smoothing and dropout achieves the best result on their language modeling and machine translation experiments . 3 BAYESIAN RECURRENT NEURAL NETWORKS . In Bayesian RNNs , we define a prior over our parameters W and consider the following maximization problem : L ( D ) = ∫ T∏ t=1 p ( xt | x < t ; W ) p ( W ) dW . A common prior is the standard normal distribution p ( W ) = N ( 0 , I ) . For recurrent neural networks , the posterior over W given the data D is intractable . We approximate the posterior with a variational distribution q ( W ) , and minimize the KL divergence between the variational distribution and the true posterior by : KL ( q ( W ) ‖p ( W | D ) ) = KL ( q ( W ) ‖p ( W ) ) − ∫ log p ( D |W ) q ( W ) dW + constant = KL ( q ( W ) ‖p ( W ) ) − T∑ t=1 ∫ log p ( xt | x < t ; W ) q ( W ) dW + constant The integral is often approximated with Monte Carlo integration with one sample W̃ ∼ q ( W ) :1∫ log p ( xt | x < t ; W ) q ( W ) dW ≈ log p ( xt | x < t ; W̃ ) q ( W̃ ) dW̃ . ( 1 ) At test time , for a new sequence y0 , . . . , yT , we can either set W to be the mean of q ( W ) or sample and average the results : p ( yt | y < t ) = 1S ∑S s=1 p ( yt | y < t ; W̃s ) , where S is the number of samples and W̃s ∼ q ( W ) . 4 VARIATIONAL SMOOTHING . We now provide theoretical justifications for data smoothing under the variational framework . In a recurrent neural network language model , there are three types of parameters : an input ( word ) embedding matrix E , an LSTM parameter matrix R , and an output embedding matrix O that produces logits for the final softmax function . We have W = { E , R , O } . 4.1 LINEAR INTERPOLATION SMOOTHING .. We first focus on the simplest data noising method—linear interpolation smoothing—and show how to extend it to other data noising methods subsequently . Word embedding matrix E. Denote the word vector corresponding to word i ∈ V in the input embedding matrix E by ei . We obtain a similar effect to linear interpolation smoothing by using the following mixture of Gaussians variational distribution for ei : ei ∼ q ( ei ) = ( 1− γ ) N ( ei , σI ) + γ ∑ v∈V UvN ( ev , σI ) = ( 1− γ + γUi ) N ( ei , σI ) + γ ∑ v 6=i∈V UvN ( ev , σI ) , ( 2 ) where Uv is the unigram probability of word v and σ is small . In other words , with probability γUv , we replace the embedding ei with another embedding sampled from a normal distribution centered at ev . Note that noising the input word is equivalent to choosing a different word embedding vector to be used in a standard recurrent neural network . Under the variational framework , we sample a different word embedding matrix for every sequence t at training time , since the integral is approximated with Monte Carlo by sampling from q ( ei ) ( Eq . 1 ) . The above formulation is related to word embedding dropout ( Dai & Le , 2015 ; Iyyer et al. , 2015 ; Kumar et al. , 2016 ) , although in word embedding dropout q ( ei ) = ( 1− γ ) N ( ei , σI ) + γN ( 0 , σI ) . KL term . For p ( ei ) = N ( 0 , I ) , we use Proposition 1 in Gal & Ghahramani ( 2016a ) and approximate the KL divergence between a mixture of Gaussians q ( ei ) and p ( ei ) as : KL ( q ( ei ) ‖p ( ei ) ) ≈ λ ∑ v∈V θiv 2 ( e > v ev + tr ( σI ) − V ( 1 + log 2π ) − log |σI| ) = λ ∑ v∈V θiv 2 ‖ev‖22 + constant , 1In the followings , we use x̃ to denote a sample from a distribution q ( x ) . where V is the vocabulary size and θiv is the mixture proportion for word i : ( 1 − γ + γUi ) for v = i and γ otherwise . In practice , the KL term directly translates to an ` 2 regularizer on each word embedding vector , but the regularization coefficient is data dependent . More specifically , the regularization coefficient for word vector ei taking into account contributions from KL ( q ( ev ) ‖p ( ev ) ) for v ∈ V is : λ ( V − 1 ) γ + ( 1− γ + γUi ) 2 = λ 1− ( V − 2 + Ui ) γ 2 ( 3 ) In other words , the variational formulation of data smoothing results in a regularization coefficient that is a function of corpus statistics ( the unigram distribution ) . Other parameters . For other parameters R and O , we can use either simple variational distributions such as q ( R ) ∼ N ( R , σI ) and q ( O ) ∼ N ( O , σI ) , which become standard ` 2 regularizers on these parameters ; or incorporate dropout by setting q ( ri ) ∼ ( 1− α ) N ( ri , σI ) + αN ( 0 , σI ) , where α is the dropout probability and ri is the i-th row of R ( Gal & Ghahramani , 2016b ) . Training . In the original noising formulation ( Xie et al. , 2017 ) , given a sequence { a , b , a , c } , it is possible to get a noised sequence { d , b , a , c } , since the decision to noise at every timestep is independent of others . In the variational framework , we use Monte Carlo integration W̃ ∼ q ( W ) for each sequence to compute p ( xt | x < t ; W̃ ) . We can either sample one embedding matrix E per sequence similar to Gal & Ghahramani ( 2016b ) , or sample one embedding matrix per timestep ( Melis et al. , 2018a ) . While the first method is computationally more efficient ( we use it in our experiments ) , if we decide to noise the first a to d , we will have a noised sequence where every a is replaced by d : { d , b , d , c } . At training time , we go through sequence x < t multiple times ( once per epoch ) to get different noised sequences . Predictions . For predictions , Xie et al . ( 2017 ) do not noise any input . In the variational framework , this corresponds to taking the mode of the variational distribution , since 1 − γ is almost always greater than γ . They also reported that they did not observe any improvements by sampling . Using the mode is rather uncommon in Bayesian RNNs . A standard approach is to take the mean , so we can set : ēi = ( 1− γ + γUi ) ei + ∑ v 6=i∈V γUvev . Extension to absolute discounting . It is straightforward to extend linear interpolation smoothing to absolute discounting by setting γi = γ distinct ( i , • ) count ( i ) in Eq . 2 and Eq . 3 above .

Summary: In this submission, the authors present a variational smoothing interpretation of the data noising approach presented in (Xie et al., 2017). Although the theoretical coverage of the problem gives interesting insights. However, a comparison to related work w.r.t. alternative regularization approaches is missing. Similarly, the perplexity values reported in the experimental results on Penn Treebank are far away from state-of-the-art results published by many competitors on this task, e.g. see the current state-of-the-art results on Penn Treebank by (Yang et al., 2017, https://arxiv.org/pdf/1703.02573.pdf and references therein). It is bad practice to ignore existing work completely like this. The interesting question here would be, inhowfar the presented smoothing/regularization methods are complementary to existing approaches, and if the presented methods do provide improvements on top of these.