Title: VARIATIONAL SMOOTHING IN RECURRENT NEURAL NETWORK LANGUAGE MODELS

Abstract: We present a new theoretical perspective of data noising in recurrent neural network language models (Xie et al., 2017) . We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution (i.e., a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution). We use this insight to propose a more principled method to apply at prediction time and propose natural extensions to data noising under the variational framework. In particular, we propose variational smoothing with tied input and output embedding matrices and an element-wise variational smoothing method. We empirically verify our analysis on two benchmark language modeling datasets and demonstrate performance improvements over existing data noising methods.

1. INTRODUCTION:

Recurrent neural networks (RNNs) are state of the art models in various language processing tasks. However, their performance heavily depends on proper regularization at training time (Melis et al., 2018b; Merity et al., 2018) . The two predominant approaches to regularize RNNs are dropout (randomly zeroing out neurons; Srivastava et al., 2014) and 2 regularization (applying 2 penalty to model parameters; Hoerl & Kennard, 1970) . Recently, Xie et al. (2017) proposed data noising to regularize language models. Their method is formulated as a data augmentation method that randomly replaces words with other words drawn from a proposal distribution. For example, we can use the unigram distribution which models the number of word occurrences in the training corpus, or a more sophisticated proposal distribution that takes into account the number of bigram types in the corpus. Data noising has been shown to improve perplexity on small and large corpora, and that this improvement is complementary to other regularization techniques and translates to improvements on downstream models such as machine translation. Xie et al. (2017) derived connections between data noising and smoothing in classical n-gram language models, which we review in §2. In smoothing (Chen & Goodman, 1996) , since empirical counts for unseen sequences are zero, we smooth our estimates by a weighted average of higher order and lower order n-gram models. There are various ways to choose the weights and the lower order models leading to different smoothing techniques, with Kneser-Ney smoothing widely considered to be the most effective. Xie et al. (2017) showed that the pseudocounts of a noised data correspond to a mixture of different n-gram models.

In this paper, we provide a new theoretical foundation for data noising and show that it can be understood as a form of Bayesian recurrent neural network with a particular variational distribution ( §4.1 and §4.2). Our derivation relates data noising to dropout and variational dropout (Gal & Ghahramani, 2016b) , and naturally leads to a data dependent 2 regularization coefficient. We use this insight to arrive at a more principled way to do prediction with data noising-i.e., by taking the mean of the variational distribution, as opposed to the mode-and propose several extensions under the variational framework in §4.3. Specifically, we show how to use variational smoothing with tied input and output embeddings and propose element-wise smooothing. In §5, we validate our analysis in language modeling experiments on the Penn Treebank (Marcus et al., 1994) and Wikitext-2 (Merity et al., 2017) datasets.

2. RECURRENT NEURAL NETWORK LANGUAGE MODELS:

We consider a language modeling problem where the goal is to predict the next word x t given previously seen context words x <t = {x 0 , x 1 , . . . , x t-1 }. Let W be parameters of a recurrent neural network, and p(x t | x <t ; W) = RNN(x <t ; W). Following previous work in language modeling (Melis et al., 2018b; Merity et al., 2018) , we use LSTM (Hochreiter & Schmidhuber, 1997) as our RNN function, although other variants such as GRU (Cho et al., 2014) can be used as well.

Given a training corpus D = {x 0 , . . . , x T }, the likelihood we would like to maximize is:

L(W; D) = T t=1 p(x t | x <t ; W).

Directly optimizing the (log) likelihood above often leads to overfitting. We typically augment the objective function with a regularizer (e.g., 2 regularizer where R(W) = λ W 2

2 ) or use dropout by randomly zeroing out neurons (Srivastava et al., 2014) .

Data noising as smoothing. Xie et al. (2017) proposed a method to regularize recurrent neural network language models by noising the data. For each input word i in x <t (sometimes also the corresponding output word), we replace it with another word sampled from a proposal distribution T with probability γ i . They introduced various methods on how to choose γ i and T.

For example, if γ i = γ for all i ∈ V and T is the unigram distribution of words in the corpus, E T [p(x t | noise γ,T (x <t ); W)] corresponds to a mixture of n-gram models with fixed weights (i.e., linear interpolation smoothing).

Another option is to set γ:

i = γ distinct(i,•) count(i)

, where distinct(i, •) denotes the number of distinct continuations preceded by word i (i.e., the number of bigram types that has i as the first word), and count(i) denotes the number of times i appears in the corpus. For this choice of γ i , when T is the unigram distribution, the expectation corresponds to absolute discounting; whereas if

T i = distinct(•,i) v∈V distinct(•,v)

and we replace both the input and output words, it corresponds to bigram Kneser-Ney smoothing. We summarize their proposed methods in Table 1 . At prediction (test) time, Xie et al. (2017) do not apply any noising and directly predict p(x t | x <t ; W). They showed that a combination of smoothing and dropout achieves the best result on their language modeling and machine translation experiments.

Name:

Noised

γ i T x Blank noising x 0 γ T "_" = 1 Linear interpolation x 0 γ unigram Absolute discounting x 0 γ distinct(i,•) count(i) unigram Kneser-Ney x 0 , x 1 γ distinct(i,•) count(i) distinct(•,i) v∈V distinct(•,v)

Table 1 : Variants of data noising techniques proposed in Xie et al. (2017) for context word x 0 and target word x 1 . Blank noising replaces an input word with a blank word, denoted by "_".

3. BAYESIAN RECURRENT NEURAL NETWORKS:

In Bayesian RNNs, we define a prior over our parameters W and consider the following maximization problem:

L(D) = T t=1 p(x t | x <t ; W)p(W)d W .

A common prior is the standard normal distribution p(W) = N(0, I).

For recurrent neural networks, the posterior over W given the data D is intractable. We approximate the posterior with a variational distribution q(W), and minimize the KL divergence between the variational distribution and the true posterior by:

KL(q(W) p(W | D)) = KL(q(W) p(W)) -log p(D | W)q(W)d W + constant = KL(q(W) p(W)) - T t=1 log p(x t | x <t ; W)q(W)d W + constant

The integral is often approximated with Monte Carlo integration with one sample W ∼ q(W):

EQUATION

At test time, for a new sequence y 0 , . . . , y T , we can either set W to be the mean of q(W) or sample and average the results:

p(y t | y <t ) = 1 S S s=1 p(y t | y <t ; Ws )

, where S is the number of samples and Ws ∼ q(W).

4. VARIATIONAL SMOOTHING:

We now provide theoretical justifications for data smoothing under the variational framework. In a recurrent neural network language model, there are three types of parameters: an input (word) embedding matrix E, an LSTM parameter matrix R, and an output embedding matrix O that produces logits for the final softmax function. We have W = {E, R, O}.

4.1. LINEAR INTERPOLATION SMOOTHING.:

We first focus on the simplest data noising method-linear interpolation smoothing-and show how to extend it to other data noising methods subsequently.

Word embedding matrix E. Denote the word vector corresponding to word i ∈ V in the input embedding matrix E by e i . We obtain a similar effect to linear interpolation smoothing by using the following mixture of Gaussians variational distribution for e i :

EQUATION

where U v is the unigram probability of word v and σ is small. In other words, with probability γU v , we replace the embedding e i with another embedding sampled from a normal distribution centered at e v .

Note that noising the input word is equivalent to choosing a different word embedding vector to be used in a standard recurrent neural network. Under the variational framework, we sample a different word embedding matrix for every sequence t at training time, since the integral is approximated with Monte Carlo by sampling from q(e i ) (Eq. 1).

The above formulation is related to word embedding dropout (Dai & Le, 2015; Iyyer et al., 2015; Kumar et al., 2016) , although in word embedding dropout q(e i ) = (1 -γ)N(e i , σI) + γN(0, σI).

KL term. For p(e i ) = N(0, I), we use Proposition 1 in Gal & Ghahramani (2016a) and approximate the KL divergence between a mixture of Gaussians q(e i ) and p(e i ) as:

KL(q(e i ) p(e i )) ≈ λ v∈V θ i v 2 e v e v + tr(σI) -V (1 + log 2π) -log |σI| = λ v∈V θ i v 2 e v 2 2 + constant,

where V is the vocabulary size and θ i v is the mixture proportion for word i: (1 -γ + γU i ) for v = i and γ otherwise. In practice, the KL term directly translates to an 2 regularizer on each word embedding vector, but the regularization coefficient is data dependent. More specifically, the regularization coefficient for word vector e i taking into account contributions from KL(q(e v ) p(e v )) for v ∈ V is:

EQUATION

In other words, the variational formulation of data smoothing results in a regularization coefficient that is a function of corpus statistics (the unigram distribution).

Other parameters. For other parameters R and O, we can use either simple variational distributions such as q(R) ∼ N(R, σI) and q(O) ∼ N(O, σI), which become standard 2 regularizers on these parameters; or incorporate dropout by setting q(r i ) ∼ (1 -α)N(r i , σI) + αN(0, σI), where α is the dropout probability and r i is the i-th row of R (Gal & Ghahramani, 2016b) .

Training. In the original noising formulation (Xie et al., 2017) , given a sequence {a, b, a, c}, it is possible to get a noised sequence {d, b, a, c}, since the decision to noise at every timestep is independent of others. In the variational framework, we use Monte Carlo integration W ∼ q(W) for each sequence to compute p(x t | x <t ; W). We can either sample one embedding matrix E per sequence similar to Gal & Ghahramani (2016b) , or sample one embedding matrix per timestep (Melis et al., 2018a) . While the first method is computationally more efficient (we use it in our experiments), if we decide to noise the first a to d, we will have a noised sequence where every a is replaced by d: {d, b, d, c}. At training time, we go through sequence x <t multiple times (once per epoch) to get different noised sequences.

Predictions. For predictions, Xie et al. (2017) do not noise any input. In the variational framework, this corresponds to taking the mode of the variational distribution, since 1 -γ is almost always greater than γ. They also reported that they did not observe any improvements by sampling. Using the mode is rather uncommon in Bayesian RNNs. A standard approach is to take the mean, so we can set:

ēi = (1 -γ + γU i )e i + v =i∈V γU v e v .

Extension to absolute discounting. It is straightforward to extend linear interpolation smoothing to absolute discounting by setting γ i = γ distinct(i,•) count(i) in Eq. 2 and Eq. 3 above.

4.2. KNESER-NEY SMOOTHING:

We now consider the variational analog of Kneser-Ney data noising. Instead of smoothing towards lower order n-grams, Kneser-Ney uses models that take into account contextual diversity. For example, for bigram Kneser-Ney smoothing, we replace the unigram distribution with ,v) , where distinct(•, i) denotes the number of distinct bigrams that end with word i. As a result, even if bigrams such as "San Francisco" and "Los Angeles" appear frequently in the corpus; "Francisco" and "Angeles" will not have high probabilities in K since they often follow "San" and "Los".

K i = distinct(•,i) v∈V distinct(•

Recall that similar to absolute discounting, Kneser-Ney also uses γ i = γ distinct(i,•) count(i) . However, if we replace an input word x t-1 , Xie et al. (2017) proposed to also replace the corresponding output word x t . The intuition behind this is that since the probability of replacing an input word x t-1 is proportional to the number of distinct bigrams that start with x t-1 , when we replace the input word, we also need to replace the output word (e.g., if we replace "San" by a word sampled from K, we should also replace "Francisco").

Word embedding matrix E. To get (bigram) Kneser-Ney smoothing, we use the following mixture of Gaussians variational distribution for e i :

e i ∼ q(e i ) = (1 -γ i )N(e i , σI) + γ i v∈V K v N(e v , σI) = (1 -γ i + γ i K i )N(e i , σI) + γ i v =i∈V K v N(e v , σI).

(4)

Output embedding matrix O. For the output embedding matrix, we also use the same variational distribution as the word embedding matrix:

EQUATION

KL term. Following similar derivations in the previous subsection, it is straightforward to show that the approximated KL term introduces the following regularization term to the overall objective:

λ v∈V 1 -(V -2 + K v )γ v 2 e v 2 2 + constant,

and similarly for the output embedding matrix O.

Training. Recall that we use Monte Carlo integration W ∼ q(W) for each sequence to compute p(x t | x <t ; W) at training time. Since we need to noise the output word when the input word is replaced, we sample γi at t -1 (for x t-1 = i) and decide whether we will keep the original (input and output) words or replace them. If we decide to replace, we need to sample two new words from K v N(e v , σI) and K v N(o v , σI) respectively. For other words k that are not the target at the current timestep (i.e., k = x t ), we can either assume that õk = o k or alternatively sample (which can be expensive since we need to sample additional V -1 times per timestep).

Predictions For predictions, Xie et al. (2017) also take the mode of the variational distribution (assuming 1 -γ i is almost always greater than γ i ). We use the mean instead, similar to what we do in variational linear interpolation smoothing.

4.3. EXTENSIONS:

Our derivations above provide insights into new methods under the variational smoothing framework. We describe three variants in the followings:

Tying input and output embeddings. Inan et al. (2017) and Press & Wolf (2017) showed that tying the input and output embedding matrices (i.e., using the same embedding matrix for both) improves language modeling performance and reduces the number of parameters significantly. We take inspirations from this work and sample both O and E from the same base matrix. As a result, we have fewer parameters to train due to this sharing mechanism, but we still have different samples of input and output embedding matrices per sequence.2 Similar to previous results in language modeling (Inan et al., 2017; Melis et al., 2018b) , our experiments demonstrate that tying improves the performance considerably.

Combining smoothing and dropout. We can combine variational smoothing and variational dropout by modifying the variational distribution to incorporate a standard Gaussian component:

e i ∼ q(e i ) = (1 -α) (1 -γ i )N(e i , σI) + γ i v∈V K v N(e v , σI) + αN(0, σI),

where α is the dropout probability. Note that in this formulation, we either drop an entire word by setting its embedding vector to zero (i.e., similar to word embedding dropout and blank noising) or choose an embedding vector from the set of words in the vocabulary.

However, it is more common to apply dropout to each dimension of the input embedding and output embedding matrix. In this formulation, we have:

EQUATION

)

At training time, while we sample αi,j multiple times (once per word embedding dimension for each word), we only sample γi once per word to ensure that when the element is not noised, we still use the same base embedding vector. We use this variant of smoothing and dropout throughout our experiments for our models.

Element-wise smoothing. The variational formulation above allows us to derive a variant of data smoothing that samples each element of the embedding vector e i independently (and similarly for o i ). Consider the variational distribution in Eq. 6. At training time, if we sample both αi,j and γi,j multiple times (once per word embedding dimension for each word) we arrive at a new element-wise smoothing method. The main difference between this model and the previous model is that each dimension in the input (and output) embedding vector is sampled independently. As a result, the vector that is used is a combination of elements from various word vectors. Notice that the mean under this new scheme is still the same as sampling per vector, so we do not need to change anything at test time. One major drawback about this model is that it is computationally expensive since we need to sample each element of each embedding vector.

5. EXPERIMENTS:



5.1. SETUP:

We evaluate our approaches on two standard language modeling datasets: Penn Treebank (PTB) and Wikitext-2. We use a two-layer LSTM as our base language model. We perform non-episodic training with batch size 64 using RMSprop (Hinton, 2012) as our optimization method. We tune the RMSprop learning rate and 2 regularization hyperparameter λ for all models on a development set by a grid search on {0.002, 0.003, 0.004} and {10 -4 , 10 -3 } respectively, and use perplexity on the development set to choose the best model. We also tune γ from {0.1, 0.2, 0.3, 0.4}. We use recurrent dropout (Semeniuta et al., 2016) for R and set it to 0.2, and apply (element-wise) input and output embedding dropouts for E and O and set it to 0.5 when E, O ∈ R V ×512 and 0.7 when E, O ∈ R V ×1024 based on preliminary experiments. We tie the input and output embedding matrices in all our experiments (i.e., E = O), except for the vanilla LSTM model, where we report results for both tied and untied. 3

5.2. MODELS:

We compare the following methods in our experiments:

• Baseline: a vanilla LSTM language model. We evaluate two variants: with tied input and output embeddings and without.

• Data noising (DN): an LSTM language model trained with data noising using linear interpolation smoothing or bigram Kneser-Ney smoothing (Xie et al., 2017) .

• Variational smoothing (VS): an LSTM language model with variational smoothing using linear interpolation or Kneser-Ney. For both models, we use the mean of the variational distribution at test time. • Variational element-wise smoothing: for the smaller PTB dataset, we evaluate an LSTM language model that uses elementwise Kneser-Ney variational smoothing and dropout. We also use the mean at test time.

5.3. RESULTS:

Our results are summarized in Table 2 . Consistent with previous work on tying the input and output embedding matrices in language models (Inan et al., 2017; Melis et al., 2018b; Merity et al., 2018) , we see a large reduction in perplexity (lower is better) when doing so. While our numbers are generally better, the results are also consistent with Xie et al. (2017) that show linear interpolation data noising is slightly better than vanilla LSTM with dropout, and that Kneser-Ney data noising outperforms these methods for both the medium (512) and large (1024) models.

Variational smoothing improves over data noising in all cases, both for linear interpolation and Kneser-Ney. Recall that the main differences between variational smoothing and data noising are:

(1) using the mean at test time, (2) having a data dependent 2 regularization coefficient that comes from the KL term,5 and (3) how each method interacts with the input and output embedding tying mechanism. 6 Our results suggest that the choice of the proposal distribution to sample from is less important for variational smoothing. In our experiments, Kneser-Ney outperforms linear interpolation on PTB but linear interpolation is slightly better on Wikitext-2.

Element-wise variational smoothing performs the best for both small and large LSTM models on PTB. We note that this improvement comes at a cost, since this method is computationally expensive. took about one day to train the smaller model as opposed to a couple hours without element-wise smoothing. As a result, we were unable to train this on a much bigger dataset such as Wikitext-2. Nonetheless, our results show that applying smoothing in the embedding (latent) space results in better models.

5.4. DISCUSSIONS:

Sampling at test time. In order to better understand another prediction method for these models, we perform experiments where we sample at test time for both data noising and variational smoothing (instead of taking the mode or the mean). We use twenty samples and average the log likelihood. Our results agree with Xie et al. ( 2017) that mentioned sampling does not provide additional benefits. In our experiments, the perplexities of 512-and 1024-dimensional DN-Kneser-Ney models increase to 96.5 and 85.1 (from 75.0 and 71.4) on the PTB validation set. For VS-Kneser-Ney models, the perplexities increase to 89.8 and 78.7 (from 74.5 and 70.8). Both our results and Xie et al. (2017) suggest that introducing data dependent noise at test time is detrimental for recurrent neural network language models.

Sensitivity to γ. We evaluate the sensitivity of variational smoothing to hyperparameter γ. Figure 1 shows perplexities on the PTB validation set for a variant of our models. The model generally performs well within the range of 0.1 and 0.3, but it becomes progressively worse as we increase γ since there is too much noise.

Other applications. While we focus on language modeling in this paper, the proposed technique is applicable to other language processing tasks. For example, Xie et al. (2017) showed that data smoothing improves machine translation, so our techniques can be used in that setup as well. It is also interesting to consider how variational smoothing interacts with and/or can be applied to memory augmented language models (Tran et al., 2016; Grave et al., 2017; Yogatama et al., 2018) and state-of-the-art language models (Yang et al., 2017) . We leave these for future work.

6. CONCLUSION:

We showed that data noising in recurrent neural network language models can be understood as a Bayesian recurrent neural network with a variational distribution that consists of a mixture of Gaussians whose mixture weights are a function of the proposal distribution used in data noising (e.g., the unigram distribution, the Kneser-Ney continuation distribution). We proposed using the mean of the variational distribution at prediction time as a better alternative to using the mode. We combined it with variational dropout, presented two extensions (i.e., variational smoothing with tied input and output embedding matrices and element-wise smoothing), and demonstrated language modeling improvements on Penn Treebank and Wikitext-2.

:

In the followings, we use x to denote a sample from a distribution q(x).

Alternatively, we could sample one matrix for both the input and output embeddings. We found that this approach is slightly worse in our preliminary experiments.

Our preliminary experiments are consistent with previous work(Inan et al., 2017; Melis et al., 2018b;Merity et al., 2018) that show tying the input and output embedding matrices results in better models with fewer numbers of parameters.

We discuss results using sampling at test time in §5.4.

The data dependent 2 coefficient penalizes vectors that are sampled more often higher.

In the variational framework, if we sample one matrix for the input and output embeddings, it effectively noises the output words even for linear interpolation. If we sample two matrices from the same base matrix, these matrices can be different at training time even if the parameters are tied. As described in §4.3, we use the latter in our experiments.

Element-wise dropout can be implemented efficiently by sampling a mask (zero or one with some probability) and multiply the entire embedding matrix with this mask. In element-wise smoothing, we need to sample an index for each dimension and reconstruct the embedding matrix for each timestep.

Figures and Tables:

Figure 1: Validation set perplexities on PTB for VS: Kneser-Ney (512 dimensions).

4  Perplexity on PTB and Wikitext-2 datasets. DN and VS denote data noising and variational smoothing. The two numbers (*M/*M) in the # of params. column denote the number of parameters for PTB and Wikitext-2 respectively.

Table content: <table><tr><td>Model</td><td>LSTM hidden size</td><td># of params.</td><td>PTB Dev Test</td><td colspan="2">Wikitext-2 Dev Test</td></tr><tr><td>Vanilla LSTM (Xie et al., 2017)</td><td>512</td><td>-</td><td>84.3 80.4</td><td>-</td><td>-</td></tr><tr><td>Vanilla LSTM (Xie et al., 2017)</td><td>1500</td><td>-</td><td>81.6 77.5</td><td>-</td><td>-</td></tr><tr><td>DN: Kneser-Ney (Xie et al., 2017)</td><td>512</td><td>-</td><td>79.9 76.9</td><td>-</td><td>-</td></tr><tr><td>DN: Kneser-Ney (Xie et al., 2017)</td><td>1500</td><td>-</td><td>76.2 73.4</td><td>-</td><td>-</td></tr><tr><td>Var. dropout (Gal &amp; Ghahramani, 2016b)</td><td>1500</td><td>-</td><td>-73.4</td><td>-</td><td>-</td></tr><tr><td>Vanilla LSTM: untied</td><td>512</td><td colspan="4">14M/38M 89.6 84.5 106.3 100.8</td></tr><tr><td>Vanilla LSTM: tied</td><td/><td/><td>80.0 74.0</td><td>90.6</td><td>86.6</td></tr><tr><td>DN: linear interpolation</td><td>512</td><td>9M/21M</td><td>79.4 73.3</td><td>88.9</td><td>84.6</td></tr><tr><td>DN: Kneser-Ney</td><td/><td/><td>75.0 70.7</td><td>86.1</td><td>82.1</td></tr><tr><td>VS: linear interpolation</td><td/><td/><td>76.3 71.2</td><td>84.0</td><td>79.6</td></tr><tr><td>VS: Kneser-Ney VS: element-wise</td><td>512</td><td>9M/21M</td><td>74.5 70.6 70.5 66.8</td><td>84.9 -</td><td>80.9 -</td></tr><tr><td>Vanilla LSTM: untied</td><td>1024</td><td colspan="2">37M/85M 90.3 85.5</td><td>97.6</td><td>91.9</td></tr><tr><td>Vanilla LSTM: tied</td><td/><td/><td>75.9 70.2</td><td>85.2</td><td>81.0</td></tr><tr><td>DN: linear interpolation</td><td>1024</td><td>27M/50M</td><td>75.5 70.2</td><td>84.3</td><td>80.1</td></tr><tr><td>DN: Kneser-Ney</td><td/><td/><td>71.4 67.3</td><td>81.9</td><td>78.3</td></tr><tr><td>VS: linear interpolation</td><td/><td/><td>71.7 67.8</td><td>80.5</td><td>76.6</td></tr><tr><td>VS: Kneser-Ney VS: element-wise</td><td>1024</td><td>27M/50M</td><td>70.8 66.9 68.6 64.5</td><td>80.9 -</td><td>76.7 -</td></tr></table>

Peer review (iclr2019):
 The paper presents a Bayesian/ variational interpretation of data noising in recurrent networks (Xie et al. 2017). Overall I found the paper interesting and well presented. The authors first review the work of Xie et al. 2017, that proposes data noisy for regularizing recurrent networks. This is done by randomly replacing certain words in the context according to some distribution. Xie et al. 2017 showed that this is highly related to smoothing in n-gram models. The authors take a Bayesian approach where there is a prior over the parameters p(W) . Computing the posterior for RNNs is generally intractable so they suggest using a variational distribution q(W) instead. They show how certain choices of the variational distribution give a similar effect to different types of smoothing (e.g. linear interpolation and Kneser Ney), as well as show how for instance, combining smoothing with dropout fits into their theory. Experimenetal results show that their approach outperforms vanilla LSTMs and the approach of Xie et al. 2017 on PTB and Wikitext-2 which are two common (although small) benchmarks for language modeling. The paper could be improved by running a comparison on larger dataset (e.g. billion word benchmark Chelba et al. 2013)