Venue: NAACL 2019

# Show Some Love to Your \(n\)-grams:

A Bit of Progress and Stronger \(n\)-gram Language Modeling Baselines

Ehsan Shareghi, Daniela Gerz, Ivan Vulic, Anna Korhonen

Language Technology Lab, DTAL, University of Cambridge

{es776, dsg40, iv250, alk23}@cam.ac.uk

###### Abstract

In recent years neural language models (LMs) have set state-of-the-art performance for several benchmarking datasets. While the reasons for their success and their computational demand are well-documented, a comparison between neural models and more recent developments in \(n\)-gram models is neglected. In this paper, we examine the recent progress in \(n\)-gram literature, running experiments on 50 languages covering all morphological language families. Experimental results illustrate that a simple extension of Modified Kneser-Ney outperforms an lstm language model on 42 languages while a word-level Bayesian \(n\)-gram LM [18] outperforms the character-aware neural model [9] on average across all languages, and its extension which explicitly invects linguistic knowledge [10] on 8 languages. Further experiments on larger Europarl datasets for 3 languages indicate that neural architectures are able to outperform computationally much cheaper \(n\)-gram models: \(n\)-gram training is up to \(15,000\times\) quicker. Our experiments illustrate that standalone \(n\)-gram models lend themselves as natural choices for resource-lean or morphologically rich languages, while the recent progress has significantly improved their accuracy.

## 1 Introduction

Statistical language models (LMs) are the pivot for several natural language processing tasks where a model trained on a text corpus is required to assign a probability to a given sequence \(w_{1}w_{2}...w_{N}\) (denoted by \(w^{N}_{1}\)). This probability indicates how likely is for \(w^{N}_{1}\) to belong to the corpus and is decomposed into conditional probabilities of words given their preceding contexts as \(P(w^{N}_{1})=\prod_{i=1}^{N}P(w_{i}|w^{i-1}_{1})\).

In \(n\)-gram LMs the unbounded conditional probabilities \(P(w_{i}|w^{i-1}_{1})\) are approximated by imposing a finite-order Markov assumption, \(P(w_{i}|w^{i-1}_{1})\approx P(w_{i}|w^{i-1}_{i-n+1})\). Several smoothing techniques address the statistical sparsity issue for computing the conditional probabilities [13, 14, 15, 16], while others avoided the above approximation with unbounded hierarchical nonparametric Bayesian frameworks [12, 13].

Alternatively, neural LMs compute \(P(w_{i}|w^{i-1}_{1})\) via recurrent neural units which, in theory, are capable of encoding an unbounded context \(w^{i-1}_{1}\). In recent years, neural LMs have become the prominent class of language modeling and have established state-of-the-art results on almost all sufficiently large benchmarks [11, 18]. While outperforming \(n\)-grams in terms of predictive accuracy, the computational shortcomings of neural LMs are well-documented: Training neural LMs is computationally expensive to the point that running experiments on large data (\(\geq\) a few GiBs) is beyond the reach of academic research to this date [14, 15, 16]. 1 Similarly, querying is slower for neural LMs due to the required matrix-based operations, whereas most of the widely used \(n\)-gram LM toolkits rely on a few hash lookups and much cheaper scalar-based operations [17, 18].

Footnote 1: For instance, \(n\)-gram LMs could be trained on 32GiB of data on a single CPU with \(\sim\)32GiB of RAM in half a day [16]. A ballpark estimate for neural LMs, based on Puri et al. (2018), requires 26 Tesla V100 16GB GPUs to finish within the same amount of time while its financial cost is at least 100x higher.

Nonetheless, it has been shown that the best predictive performance is still achieved by combining the two models via a basic interpolation or a mixture model [15, 16]: this indicates that the progress in \(n\)-gram

state-of-the-art performance. Inspired by this, in this paper we shed new light on the most notable recent progress in \(n\)-gram statistical LMs which improves their predictive accuracy.

We demonstrate that under a recent massively multilingual experimental setup of Gerz et al. (2018a), more recent extensions of Kneser-Ney family of \(n\)-gram LMs Shareghi et al. (2016, 2017) are highly competitive with neural LMs. More specifically, we experiment on 50 languages from different morphological families, and illustrate that a word-level Bayesian \(n\)-gram LM Shareghi et al. (2017) outperforms the character-level informed neural counterpart Kim et al. (2016) on average, and its linguistically informed variant Gerz et al. (2018a) on 8 languages. On larger Europarl datasets we find that \(n\)-gram models cannot reach the performance peaks of computationally much more expensive neural models, but a \(2\times\) decrease in perplexity comes at the cost of \(15,000\times\) longer training time. Our work reveals that recent \(n\)-gram LMs should be used as strong baselines, especially in resource-lean LM data and for morphologically rich languages.

Additionally, \(n\)-gram LMs offer a stringent way of dealing with Out-of-Vocabulary (OOVs) and rare words in a _full vocabulary_ setting without relying on any pruning Heafield (2013). However, in neural LMs this remains an open question Kawakami et al. (2017); Kim et al. (2016); Cotterell et al. (2018), while a common practice is pruning the training corpus and imposing _closed vocabulary_ assumption Mikolov et al. (2010) where rare words at training and unseen words at test are treated as an UNK token. We provide the mathematical underpinnings of \(n\)-gram models and highlight how this popular treatment works in favor of neural LMs (in comparative studies), and enforces \(n\)-gram LMs to perform much worse than their full potential.

## 2 \(n\)-gram Language Models: Smoothing

We now provide an overview of established smoothing techniques for \(n\)-gram LMs and their recent extensions. Smoothing is typically achieved by interpolation, where the probability of seeing a word \(w_{i}\) after a context \(w^{j-1}_{i-n+1}~{}P(w_{i}|w^{j-1}_{i-n+1},\Theta)\) is smoothed by its probability after a shorter context, \(w^{j-1}_{i-n+2}\), and follows the following general form:

\[\small\beta(w_{i}|w^{j-1}_{i-n+1},\Theta)+\gamma(w^{j-1}_{i-n+1},\Theta)P(w_{ i}|w^{j-1}_{i-n+2},\Theta). \tag{1}\]

The term \(\beta(\cdot)\) represents the existing mass for the \(n\)-gram (e.g. via maximum likelihood estimation), \(\gamma\) is the weight used for redistributing the preserved mass (e.g. via discounting), and \(\Theta\) are the parameters of the smoothing technique. The recursion stops at the unigram level where the conditioning context is empty. The recursion at lower levels relies on different quantities (e.g. pseudo-counts) but for brevity we focus on the top level of recursion and only the first term, \(\beta(w_{i}|w^{j-1}_{i-n+1},\Theta)\), which suffices to highlight the key differences between smoothing techniques (see Table 1).

Kneser-Ney (kn).The key parameters of knKneser and Ney (1995) are the \(k\)-gram specific discounts \(D_{k}\) which control the amount of preserved and redistributed mass at \(k\)th level of recursion. While learning the discounts (on held-out data) is a possibility, the following estimation is shown to work well in practice:

\[D_{k}=1-2\frac{n_{2}(k)}{n_{1}(k)}.\frac{n_{1}(k)}{n_{1}(k)+2n_{2}(k)}. \tag{2}\]

It captures the characteristics of different \(k\)-gram orders by looking at the number of unique \(k\)-grams which occurred once, \(n_{1}(k)\), or twice, \(n_{2}(k)\) in the training data, defined as follows:

\[n_{j}(k)=\begin{cases}\left|\{\alpha~{}~{}\text{s.t.}~{}~{}|\alpha|=k,c(\alpha )=j\}\right|,&\text{if $k=n$}\\ \left|\{\alpha~{}~{}\text{s.t.}~{}~{}|\alpha|=k,N_{1+}(\centerdot\centerdot \centerdot\centerdot\centerdot\centerdot\center\dot\centerdot\centerdot\center \dot\center\dot\center\dot\center\dot\center\dot\center\dot\center\dot \center\dot\center\dot\center\dot\center\dot\center\dot\center\dot \center\dot\center\dot\center\dot\center\dot\center\dot\center \dot\center\dot\center\dot\center\dot\center\dot\center\dot \center\dot\center\dot\center\dot\center\dot\center\dot\center \dot\center\dot\center\dot\center\dot\center\dot\center\dot \center\dot\center\dot\center\dot\center\dot\center\dot\right\} \end{cases} \tag{3}\]

where \(N_{1+}(\centerdot\centerdot\centerdot\centerdot\dot\center\dot\center \dot\center\dot\center\dot\center\dot\center\dot\center\dot\center \dot\center\dot\center\dot\center\dot\center\dot\center\dot\center \dot\center\dot\center\dot\center\dot\center\dot\center\dot \center\dot\center\dot\center\dot\center\dot\center\dot \center\dot\center\dot\center\dot\center\dot\center\dot \center\dot\center\dot\center\dot\center\dot\center\dot \center\dot\center\dot\center\dot\center\dot\center\dot \center\dot\center\dot\center\dot\center\dot\center\dot\) \(\center\dot\dot\center\dot\center\dot\center\dot\) \(\dot\center\dot\center\dot\dot\center\dot\) \(\dot\center\dot\center\dot\dot\center\dot\) \(\dot\center\dot\dot\center\dot\) \(\dot\center\dot\dot\center\dot\) \(\dot\center\dot\dot\center\dot\) \(\dot\center\dot\dot\) \(\dot\center\dot\) \(\dot\center\dot\dot\) \(\dot\center\dot\) \(\dot\center\dot\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\) \(\dot\center\dot\) \(\dot\) \(\dot\) \(\dot\center\dot\) 

defined with modifications applied to the discounting mechanism in order to make them sensitive to the _existing_ mass \(j\). The discounts are estimated as:

\[D^{j}_{k}=\begin{cases}0,&\text{if $j=0$}\\ j-(j+1)\frac{e_{2n(3)}}{e_{3}(3)}\frac{n_{1}(k)}{n_{1}(k)+2e_{2}(k)},&\text{if $j<B$}\\ B-(B+1)\frac{e_{2n(3)}}{n_{1}(k)+2e_{2}(k)},&\text{if $j\geq B$}\end{cases} \tag{4}\]

where \(n_{j}(k)\) is defined in Eq. (3), and \(B=3\). This leads to three discount parameters \(\{D^{1}_{k},D^{2}_{k},D^{3+}_{k}\}\) for each recursion level, and allows for discount values to be as large as \(c(w^{i}_{i-n+1})\). mkn is widely accepted as the state-of-the-art \(n\)-gram LM and is very frequently confused with kn in the literature.

Generalized Modified Kneser-Ney (gkn)In conditions where statistical sparsity is more severe a more refined approach to model the distributions is necessary. Motivated by this, Shareghi et al. (2016a) provided the mathematical proof (based on leave-one-out log-likelihood) of the discount bounds used in mkn and proposed a natural extension of its discount binning to \(B=10\). This was shown to be effective for further perplexity reduction in out-of-domain setting where OOV ratio is naturally high.

Bayesian Kneser-Ney (bkn)The Bayesian generalization of kn is the Hierarchical Pitman-Yor Process (hypp) language model Teh (2006). This can be interpreted as a richer parameterization of kn and mkn, where the additional parameters (introduced shortly) allow the model to have more flexibility in capturing the desired distribution.

The Pitman-Yor Process Pitman et al. (1997), pyp\((D,\theta,H)\), is a distribution defined over probability vectors (each draw from a pyp is a multinomial distribution) and has three parameters: a base probability vector \(H\) which is the expected value of a draw from a pyp, the concentration parameter \(-D<\theta\) which controls the variation of draws from a pyp around \(H\), and the discount parameter \(0\leq D<1\) which allows the drawn vectors to capture the power-law behavior.

In the LM context, given a sequence of words \(w_{i-n+1}\), a draw from a pyp is a multinomial distribution over the words following this sequence, denoted by \(G_{w_{i-n+1}}\). This distribution can be captured by a vector of two counts \(\{u^{w_{i}}_{i_{w-n+1}},u^{w_{i}}_{i_{w-n+1}}\}^{w_{i}\in\sigma}\) which defines a partitioning arrangement of \(G_{w_{i-n+1}}\), while different partitioning correspond to different multinomial draws from pyp. Here, \(u^{w_{i}}_{i-n+1}\) is the total number of evidence 2 for word \(w_{i}\) after the context \(w_{i-n+1}\), \(t^{w_{i}}_{i-n+1}\) is the total number of partitions dedicated to \(w_{i}\) constrained by \(0\leq t^{w_{i}}_{i-n+1}\leq n^{w_{i}}_{i-n+1}\), and \(\sigma\) is the vocabulary.

Footnote 2: This is equal to \(c(w^{i}_{i-n+1})\) at the top level of recursion, hence not mentioned as a part of \(\Theta\) in Table 1.

The hypp ties ppy distributions through their base and offers the statistical mean to smooth over infinitely long contexts Wood et al. (2011). For instance, pyp\((D_{u},\theta_{u},\texttt{pyp}(D_{\pi(u)},\theta_{\pi(u)},H))\) is a two-level hppy where a draw from a _child_ distribution \(u\), is smoothed by a draw from its _parent_\(\pi(u)\). Here, \(\pi(u)\) is \(u\) with its most earliest word dropped (e.g., \(u=abc\), \(\pi(u)=bc\)).

kn can be seen as a special case for bkn, when the concentration is 0, and \(\{u^{w_{i}}_{i_{w-n+1}}=1\}^{w_{i}\in\sigma}\). bkn can be considered as a richer parameterization of mkn: The product \(D_{w_{i-n+1}}t^{w_{i}}_{w_{i-n+1}}\) (see Table 1) allows \(\geq 1\) discounting, simulating the discount range of mkn, while an additional parameter \(\theta_{w_{i-n+1}}\) permits further adjustments of the distribution. bkn is shown to outperform mknSharegih et al. (2017) but relies on expensive parameter sampling.

Out-Vocabulary (oov)To complete the definitions we now explain how unseen words or contexts are handled during the computation without resorting to pruning or closed vocabulary setting. This treatment is the same for both non-Bayesian and Bayesian methods described in this paper. Let us consider the generic interpolation form of Eq. (1) and the maximum likelihood estimation of the \(\beta(.)\) term (hence \(\Theta\) is dropped) in the top level of the interpolation, \(\beta(w_{i}|w^{i-1}_{i-n+1})=\frac{c(w^{i}_{i-n+1})}{c(w^{i-1}_{i-n+1})}\). Regardless of the level of interpolation and the paradigm used for computing the \(\beta(.)\) term, \(\beta(.)\) and \(\gamma(.)\) always share the same denominator (normalizing factor).

An unseen word can appear as the target word \(w_{i}\), which results in \(\beta(.)=0\) as \(c(w^{i}_{i-n+1})=0\). It can also appear as a part of the prediction context \(w^{i-1}_{i-n+1}\) in which case both the \(\beta(.)\) and \(\gamma(.)\) terms will be undefined (and ignored) as the denominator \(c(w^{i-1}_{i-n+1})=0\). This procedure is applied to all levels of interpolation without loss of generality, and as can be seen it only relies on the basic mathematical property of the involved computations rather than any other presumptions about data preprocessing or vocabulary. 3 

Experiments and Results

As our main large-scale experiment we use a typologically diverse set of 50 languages. These LM datasets cover many languages which are challenging in terms of data size, as well as the type-token ratio. In a less challenging setup, we experiment on 3 languages with larger training data from the Europarl Koehn (2005) to compare the two classes of LMs based on perplexity reduction and training time. For full data statistics, actual sampling of languages and data curation see Gerz et al. (2018a,b).

The common practice of setting a frequency threshold and mapping training data unigrams (\((k=1)\)-gram) to an UNK token degrades the performance of \(n\)-gram LMs by discarding a range of discount parameters: e.g., using threshold (\(<3\)) results in \(n_{1}(1),n_{2}(1)=0\), both included in Eq. (2) and Eq. (4), increases the average perplexity score of 5-gram kn in our experiments by 11%. Motivated by its significance, we base our comparison on reported results by Gerz et al. (2018a): they deal with the task in the full vocabulary setting Adams et al. (2017) with word-level predictions, and follow a relatively comparable treatment of unseen words with both \(n\)-gram and neural LM families (although not identical) at test time without enforcing any threshold over the training data.

The benchmarked neural LMs include three models with word-level predictions: a standard lstm Zaremba et al. (2014), a Char-cnn-lstm (denoted as cnn) Kim et al. (2016) which incorporates character-level information in the input, and the Attract-Preserve model (denoted as ap) Gerz et al. (2018a) which injects further subword-level information. All benchmarked \(n\)-gram LMs are 5-grams, with the exception of bkn which is an \(\infty\)-gram model trained via 5 samples4 following the recipe of Shareghi et al. (2017). gkn results are based on \(B=5\), tuned on a development set.

Footnote 4: Marginal improvements achieved with more sampling.

Results and Discussion.The main results on the 50-lamaguages benchmark are summarized in Table 2. The results for the more recent \(n\)-gram LMs indicate that these \(n\)-gram models are highly competitive with neural LMs in this challenging resource-lean setup.5 For instance, for 26/50 languages all \(n\)-gram models outperform a regular LSTM, while gkn and bkn extend the lead to 42/50 

and 48/50 languages, respectively. For certain morphologically rich languages (e.g., Tamil, Mongolian, Hebrew), the _n_-gram LMs are able to outscore even character-aware neural LMs. On average we observe _n_-grams succeed, especially the bkn model, for introflexive and agglutinative languages which are known to have productive morphological systems: as reported in Table 2, they have the higher OOV ratio compared to the other language families. Overall, the best performing _n_-gram model, bkn, outperforms both the lstm (42% reduction in perplexity) and cnn models (3% reduction in perplexity), while falling behind ap by 8%.

These results highlight that _n_-gram models can serve as strong baselines for such morphologically rich languages with high OOV rates and type-to-token ratios, which are especially problematic in scarce data setups. Additionally, they suggest that more sophisticated _n_-gram variants such as gkn or bkn should be used to provide adequate comparison points with _n_-gram LMs than the commonly used kn or mkn.

As expected, experiments on 10\(\times\) larger Europarl datasets for 3 languages show that neural models outperform _n_-gram models in less challenging data-intensive scenarios. However, training on large datasets comes at the expense of training efficiency for neural models: e.g., according to Figure 1 training non-Bayesian _n_-grams is around 15,000\(\times\) quicker than training neural models. We leave a full-fledged investigation on the relation of training corpus size and efficacy of _n_-gram vs. neural LM training for future work. In addition, motivated by these preliminary insights, we advocate investing further efforts in future work into coupling the ideas behind _n_-gram and neural LMs towards improved language modeling.

## 4 Conclusion

We provided an overview of previous work and very recent progress in _n_-gram LMs. The recent developments, when tested on a challenging set of 50 languages, demonstrated superior or highly competitive performance compared with neural LMs, while being substantially cheaper to train. We also shed light on a common issue in the experimental setups, concerning OOV or rare words handling, when comparing _n_-grams and neural LMs.

While being non-trivial, investigating any correlation between cheap-to-compute heuristics (e.g., basic data statistics) and the choice of the most suitable model for a given dataset is worth exploring. Also, motivated by our findings, we will work on utilizing continuous space representations as side information in sampling the parameters of bkn, i.e. similar to Zhao et al. (2018), which potentially can reduce the gap between bkn and neural models.

## Acknowledgments

This work is supported by the ERC Consolidator Grant LEXICAL (648909). The authors would like to thank the anonymous reviewers for their helpful suggestions. The first author would like to also thank the members of the Language Technology Lab for their comments on the presentation of this work.

## References

* Adams et al. (2017) Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In _EACL_, pages 937-947.
* Chen and Goodman (1999) Stanley F. Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. _Computer Speech & Language_, 13(4):359-393.
* Chen et al. (2016) Wenlin Chen, David Grangier, and Michael Auli. 2016. Strategies for training large vocabulary neural language models. In _ACL_, pages 1975-1985.
* Cotterell et al. (2018) Ryan Cotterell, Sebastian J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In _NAACL_, pages 536-541.
* Gerz et al. (2018a) Daniela Gerz, Ivan Vulic, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018a. Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction. _TACL_, 6:451-465.

Figure 1: Time-Perplexity for _n_-gram and neural LMs across 3 languages fi, cs, nl. Timings done on a single core of AMD Ryzen 1900X for _n_-grams, and on a Nvidia TITAN X Pascal GPU for neural models.

 

* Gerz et al. (2018) Daniela Gerz, Ivan Vulic, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018b. On the relation between linguistic typology and (limitations of) multilingual language modeling. In _EMNLP_, pages 316-327.
* Heafield (2013) Kenneth Heafield. 2013. _Efficient Language Modeling Algorithms with Applications to Statistical Machine Translation_. Ph.D. thesis, Carnegie Mellon University.
* Jozefowicz et al. (2016) Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. _arXiv preprint arXiv:1602.02410_.
* Kawakami et al. (2017) Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in open-vocabulary neural language modeling. In _ACL_, pages 1492-1502.
* Kim et al. (2016) Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In _AAAI_, pages 2741-2749.
* Kneser and Ney (1995) Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In _ICASSP_, pages 181-184.
* Koehn (2005) Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In _MT Summit_.
* Liu et al. (2018) Xuan Liu, Di Cao, and Kai Yu. 2018. Binarized LSTM language model. In _NAACL_, pages 2113-2121.
* Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. _Computational Linguistics_, 19(2):313-330.
* Melis et al. (2018) Gabor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In _ICLR_.
* Mikolov et al. (2010) Tomas Mikolov, Martin Karafat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _INTERSPEECH_, pages 1045-1048.
* Neubig and Dyer (2016) Graham Neubig and Chris Dyer. 2016. Generalizing and hybridizing count-based and neural language models. In _EMNLP_, pages 1163-1172.
* Patwary et al. (2018) Mostofa Patwary, Milind Chabbi, Heewoo Jun, Jiaji Huang, Gregory Diamos, and Kenneth Church. 2018. Language modeling at scale. _arXiv preprint arXiv:1810.10045_.
* Pitman et al. (1997) Jim Pitman, Marc Yor, et al. 1997. The two-parameter poisson-dirichlet distribution derived from a stable subordinator. _The Annals of Probability_, 25(2):855-900.
* Puri et al. (2018) Raul Puri, Robert Kirby, Nikolai Yakovenko, and Bryan Catanzaro. 2018. Large scale language modeling: Converging on 40GB of text in four hours. _arXiv preprint arXiv:1808.01371_.
* Shareghi (2017) Ehsan Shareghi. 2017. _Scalable Non-Markovian Sequential Modelling for Natural Language Processing_. Ph.D. thesis, Monash University.
* Shareghgi et al. (2016a) Ehsan Shareghi, Trevor Cohn, and Gholamreza Hafari. 2016a. Richer interpolative smoothing based on modified Kneser-Ney language modeling. In _EMNLP_, pages 944-949.
* Shareghi et al. (2017) Ehsan Shareghi, Gholamreza Haffari, and Trevor Cohn. 2017. Compressed nonparametric language modelling. In _IJCAI_, pages 2701-2707.
* Shareghi et al. (2016b) Ehsan Shareghi, Matthias Petri, Gholamreza Haffari, and Trevor Cohn. 2016b. Fast, small and exact: Infinite-order language modelling with compressed suffix trees. _TACL_, pages 477-490.
* Tang and Lin (2018) Raphael Tang and Jimmy Lin. 2018. Progress and tradeoffs in neural language models. _arXiv preprint arXiv:1811.00942_.
* Teh (2006) Yee Whye Teh. 2006. A Bayesian interpretation of interpolated Kneser-Ney. Technical report, NUS School of Computing.
* Wood et al. (2011) Frank Wood, Jan Gasthaus, Cedric Archamb Beau, Lancelot James, and Yee Whye Teh. 2011. The sequence memoizer. _Communications of the ACM_, 54(2):91-98.
* Yang et al. (2018) Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. In _ICLR_.
* Zaremba et al. (2014) Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. _arXiv preprint arXiv:1409.2329_.
* Zhao et al. (2018) He Zhao, Lan Du, Wray L. Buntine, and Mingyuan Zhou. 2018. Inter and intra topic structure learning with word embeddings. In _ICML_, pages 5887-5896.