Venue: NAACL 2024

# The Role of \(n\)-gram Smoothing in the Age of Neural Networks

Luca Malaguti Andrius Buinovskij Anej Svete Clara Meister Afra Amini Ryan Cotterell lmalagutti@inf.ethz.ch andriusb@student.ethz.ch  {asvete, meistecl, aamini, ryan.cotterell}@inf.ethz.ch

###### Abstract

For nearly three decades, language models derived from the \(n\)-gram assumption held the state of the art on the task. The key to their success lay in the application of various smoothing techniques that served to combat overfitting. However, when neural language models toppled \(n\)-gram models as the best performers, \(n\)-gram smoothing techniques became less relevant. Indeed, it would hardly be an understatement to suggest that the line of inquiry into \(n\)-gram smoothing techniques became dormant. This paper re-opens the role classical \(n\)-gram smoothing techniques may play in the age of neural language models. First, we draw a formal equivalence between label smoothing, a popular regularization technique for neural language models, and add-\(\lambda\) smoothing. Second, we derive a generalized framework for converting _any_\(n\)-gram smoothing technique into a regularizer compatible with neural language models. Our empirical results find that our novel regularizers are comparable to and, indeed, sometimes outperform label smoothing on language modeling and machine translation.

1
Footnote 1: An alphabet is a finite, non-empty set.

2Autoregressivization is without loss of generality [Cotterell et al., 2023, Theorem 2.4.2]

[https://github.com/rycolab/](https://github.com/rycolab/) ngram_regularizers

## 1 Introduction

Let \(\Sigma\) be an **alphabet**.1 A **language model** is a probability distribution \(p\) over \(\Sigma^{*}\), the set of all strings \(\mathbf{x}=x_{1}\cdots x_{T}\) with symbols \(x_{t}\) drawn from \(\Sigma\). A fundamental task in natural language processing (NLP) is to estimate a language model--often from a parametric family--that places a high probability on held-out, human-generated text. A common design choice is to construct a locally normalized language model, i.e., one which factorizes autoregressively2 as \(p(\mathbf{x})=p(\textsc{Eos}\mid\mathbf{x})\prod_{t=1}^{|\mathbf{x}|}p(x_{t}\mid\mathbf{x}_{< t})\), where \(\textsc{Eos}\not\in\Sigma\) is a distinguished end-of-string symbol and \(\mathbf{x}_{<t}\stackrel{{\text{def}}}{{=}}x_{1}\cdots x_{t-1}\) is a prefix of \(\mathbf{x}\).

Footnote 1: footnotemark:

For years, the best parametric families of language models for this estimation task applied the \(n\)-gram assumption, detailed below.

**Assumption 1.1** (\(n\)-gram assumption).: _A language model obeys the \(n\)-**gram assumption** if the following conditional independence holds_

\[\begin{split} p(x_{t}\mid\mathbf{x}_{<t})&\overset{ \text{def}}{=}p(x_{t}\mid x_{1}\cdots x_{t-1})\\ &=p(x_{t}\mid x_{t-n+1}\cdots x_{t-1})\\ &\overset{\text{def}}{=}p\left(x_{t}\mid\mathbf{x}_{t}^{n}\right)\end{split} \tag{1}\]

_where \(x_{t}\) for \(t<1\) is treated as a distinguished padding symbol \(\textsc{bos}\not\in\Sigma\).3 We will call \(\mathbf{x}_{t}^{n}\stackrel{{\text{def}}}{{=}}x_{t-n+1}\cdots x_{t-1}\) the **history** of \(x_{t}\). We use \(\mathbf{x}^{n}\) for histories where the time step \(t\) is irrelevant._

Footnote 3: The symbol bos stands for beginning of string.

Assumption 1.1 was, historically, considered a practically effective manner to fight the curse of dimensionality, despite its inability to attend to contexts longer than \(n-1\) words and thus capture long-range dependencies.4

Figure 1: An illustration of the introduced framework. With maximum-likelihood estimation (MLE), a language model \(q_{\mathbf{\theta}}\) is trained to match \(p_{\mathcal{D}}\), the empirical distribution induced by a dataset \(\mathcal{D}\). However, we can also modify (_smooth_) \(p_{\mathcal{D}}\) into \(\tilde{p}_{\mathcal{D}}^{n}\) and train a language model \(\tilde{q}_{\mathbf{\theta}}\) on \(\tilde{p}_{\mathcal{D}}^{n}\). We show that the latter can be thought of as training \(\tilde{q}_{\mathbf{\theta}}\) with a regularized maximum-likelihood objective.



The maximum-likelihood estimator of a model \(q_{\mathrm{MLE}}^{n}\) under the \(n\)-gram assumption is straightforward to derive.5 Indeed, one can express it simply using counts of sub-string occurrences:

Footnote 5: We discuss the MLE of language models in detail in §2.1.

\[q_{\mathrm{MLE}}^{n}\left(x~{}|~{}\mathbf{x}^{n}\right)=\frac{\#(\mathbf{ x}^{n}x)}{\#(\mathbf{x}^{n})} \tag{2}\]

where, intuitively, \(\#(\mathbf{y})\) denotes the number of times substring \(\mathbf{y}\) occurs in the training dataset.6 Critically, the simplicity enforced by Assumption 1.1 alone is _not_ enough to prevent overfitting for reasonably sized \(n\): a minimally parameterized \(n\)-gram model has \(\mathcal{O}\left(|\Sigma|^{n}\right)\) free parameters, one for each \(n\)-gram. Therefore, the maximum-likelihood solution of an \(n\)-gram language model overfits on its training data by assigning probability \(0\) to any string containing an \(n\)-gram that does not occur in the training dataset, which is undesirable. To solve this issue, in addition to making the \(n\)-gram assumption, modelers applied a variety of **smoothing** techniques to regularize the estimation of \(n\)-gram probabilities and obtain smoothed probabilities \(\widetilde{q}_{\mathrm{MLE}}^{n}\).

Footnote 6: We formally introduce the counting function \(\#\) in §2.1 accounting for the presence of eos at the end of each string.

One of the simplest \(n\)-gram smoothing techniques is known as **add-\(\lambda\) smoothing**,7 which can be informally described as hallucinating \(n\)-grams in the training dataset that occur at frequency \(\lambda\). In other words, the counts of all \(n\)-grams--including those that are unobserved in the dataset--are augmented by \(\lambda\):

Footnote 7: In the special case that \(\lambda=1\), this technique is generally referred to as Lidstone smoothing.

\[\tilde{q}_{\mathrm{MLE}}^{n}\left(x~{}|~{}\mathbf{x}^{n}\right)\stackrel{{ \text{def}}}{{=}}\frac{\#(\mathbf{x}^{n}x)+\lambda}{\#(\mathbf{x}^{n})+(| \Sigma|+1)\,\lambda}. \tag{3}\]

In the context of NLP, add-\(\lambda\) smoothing has often served a pedagogical purpose and is thus commonly taught, but its efficacy in practice was considered limited Eisner (2023). However, researchers developed more sophisticated related smoothing methods Jelinek (1980); Katz (1987); Ney et al. (1995) that were useful in practice. Ney et al. (1995), for example, was long considered the best available method Chen and Goodman (1999) and its popularity inspired a principled Bayesian interpretation Teh (2006). Interestingly though, add-\(\lambda\) smoothing still has a place in today's literature; as we prove in SS2, it is identical to a regularization technique called label smoothing, common in the training of (conditional) neural language models Pereyrera et al. (2017); Meister et al. (2020), and particularly common in machine translation Costa-jussa et al. (2022).

Footnote 7: The special case that \(\lambda=1\), this technique is generally referred to as Lidstone smoothing.

In this context, we present the primary theoretical research question addressed in our paper. If add-\(\lambda\) smoothing is equivalent to label smoothing--an effective regularization technique for training today's neural language models--can we reverse-engineer further regularization methods starting from other \(n\)-gram smoothing methods? Given that add-\(\lambda\) smoothing was not known to perform well compared to other smoothing techniques in the context of \(n\)-gram language models, it is natural to suspect that reverse-engineered regularizers based on empirically more successful smoothing techniques may indeed outperform label smoothing.

To this end, we derive a straightforward relationship between training on a smoothed empirical distribution and additive regularization methods, e.g., entropy regularization. Using that relationship, we further show that _any_ smoothing method can be reformulated as an additive regularization of the standard maximum-likelihood objective, and, hence, we provide an explicit way to construct the regularizer that can be applied to the training of any (neural) language model, helping connect classical smoothing methods and modern language models by introducing a way to incorporate smoothing techniques into the training of neural language models. We complement our theoretical analysis by empirically verifying the validity of our proposed methods on two small-scale datasets and observe that some regularizers based on more complex \(n\)-gram smoothing techniques do indeed perform better than label smoothing on language modeling and machine translation.

## 2 Label Smoothing and add-\(\lambda\) Smoothing

In this section, we derive a formal relationship between the add-\(\lambda\) smoothing applied to \(n\)-gram models and label smoothing. The relationship is based on a regularizer that, when applied during MLE, simulates add-\(\lambda\) smoothing exactly. Additionally, we contend that, in this sense, label smoothing _generalizes_ add-\(\lambda\) smoothing from a method applicable only to \(n\)-gram language models to one that can be used with neural language models.8 To the best of the authors' 

knowledge, this derivation and the relationship it exposes are novel. We re-use this meta paradigm in SS4 to derive novel regularizers, which we then compare experimentally to label smoothing.

Footnote 8: Note that, for complete generality, \(\#\) should also take the dataset \({\cal D}\) as argument. For conciseness, we leave this parameter implicit.

### Preliminaries

Some Notation.We define \(\overline{\Sigma}\stackrel{{\text{def}}}{{=}}\Sigma\cup\{\texttt{ eos}\}\). To make it possible to always condition on exactly \(n-1\) symbols, whenever the history is shorter than \(n-1\) symbols, we prepend a string with the appropriate number of bos symbols. Thus, we define \(\Sigma_{\textsc{bos}}^{n-1}\stackrel{{\text{def}}}{{=}}\bigcup_{ \ell=1}^{m-1}\left\{\textsc{bos}\right\}^{\ell}\times\Sigma^{n-1-\ell}\) (the set of all possible histories) and \(\Sigma_{\textsc{bos}}^{*}\stackrel{{\text{def}}}{{=}}\Sigma_{ \textsc{bos}}^{n-1}\cup\bigcup_{\ell=n}^{\infty}\Sigma^{\ell}\) as the set of all strings that are either prefixed by bos or contain more than \(n-1\) symbols.

Maximum-likelihood Estimation.We now introduce maximum-likelihood estimation in the context of language modeling. Suppose we observe a collection of samples \({\cal D}=\{\mathbf{x}^{(m)}\}_{m=1}^{M}\) where \(\mathbf{x}^{(m)}\sim p\) and \(p\) is the distribution over strings that we are trying to model. Let \(p_{\mathcal{D}}\) be the empirical distribution induced by \({\cal D}\), i.e., the probability distribution defined as

\[p_{\mathcal{D}}(\mathbf{x})=\frac{1}{M}\sum_{m=1}^{M}\mathbbm{1}\{\mathbf{x}=\mathbf{x}^{( m)}\}. \tag{4}\]

Choosing a model \(q_{\mathbf{\theta}}\) that minimizes the forward KL divergence \(D_{\mathrm{KL}}(p_{\mathcal{D}}~||~{}q_{\mathbf{\theta}})\) is known as maximum-likelihood estimation. Under regularity conditions Le Cam (1952), MLE is consistent, i.e., in the limit of infinite data, we arrive at the true parameters of the data-generating distribution if the data-generating distribution indeed came from the model's parametric family.

Counting Substrings in a Dataset.As discussed in SS1, estimation of \(n\)-gram language models relies on _counting_ the occurrences of various substrings in a dataset. Given the dataset \({\cal D}=\{\mathbf{x}^{(m)}\}_{m=1}^{M}\), we define the counting function \(\#\) as

\[\#(\mathbf{x})\stackrel{{\text{def}}}{{=}}\sum_{m=1}^{ M}\sum_{t=1}^{|\mathbf{x}|+1}\sum_{s=t+1}^{|\mathbf{x}|+1}\mathbbm{1}\left\{\mathbf{ x}=\mathbf{x}_{t:s}^{(m)}\right\} \tag{5a}\] \[\#(\mathbf{x}\texttt{eos})\stackrel{{\text{def}}}{{=}} \sum_{m=1}^{M}\sum_{t=1}^{|\mathbf{x}|+1}\mathbbm{1}\left\{\mathbf{x}=\mathbf{x}_{t:}^{( m)}\right\}, \tag{5b}\]

where \(\mathbf{x}_{t:s}\stackrel{{\text{def}}}{{=}}x_{t}\cdots x_{s-1}\) and \(\mathbf{x}_{t:}\stackrel{{\text{def}}}{{=}}\mathbf{x}_{t:|\mathbf{x}|+1}\). As hinted at in SS1, \(\#\) counts the number of times the string \(\mathbf{x}\) appears as a substring of a string in \({\cal D}\).9 This is to be distinguished from simply counting the number of occurrences of \(\mathbf{x}\) in \({\cal D}\).

Footnote 9: Note that, for complete generality, \(\#\) should also take the dataset \({\cal D}\) as argument. For conciseness, we leave this parameter implicit.

Empirical Distributions.Using the counting functions defined in Eq. (5), we define two empirical probability distributions. The **autoregressive empirical probability** distribution is defined as

\[p_{\mathcal{D}}(y~{}|~{}\mathbf{x})\stackrel{{\text{ def}}}{{=}}\frac{\#(\mathbf{x}y)}{\#(\mathbf{x})}, \tag{6}\]

where \(\mathbf{x}_{<t}\in\Sigma^{*}\) and \(y\in\overline{\Sigma}\). Eq. (6) is the autoregressive decomposition of Eq. (4). Then, the **autoregressive empirical \(n\)-gram probability** distribution is defined as follows

\[p_{\mathcal{D}}^{n}(y~{}|~{}\mathbf{x}^{n})\stackrel{{\text{def}}}{{= }}\frac{\#(\mathbf{x}^{n}y)}{\#(\mathbf{x}^{n})}, \tag{7}\]

where \(\mathbf{x}^{n}\in\Sigma_{\textsc{bos}}^{n-1}\) and \(y\in\overline{\Sigma}\). Note that the autoregressive empirical \(n\)-gram probabilities are equivalent to the maximum-likelihood estimator of an \(n\)-gram model \(q_{\rm MLE}^{n}\), as seen in Eq. (2).

Prefix Probabilities.Prefix probabilities Cotterell et al. (2023), SS2.4.2) are useful quantities in language modeling and are found both in the language model's autoregressive factorization as well as in maximum-likelihood estimation Jelinek and Lafferty (1991); Nowak and Cotterell (2023). We give a formal definition below.

**Definition 2.1**.: _We define the prefix probability function \(\pi\) of a language model \(p\) over \(\Sigma^{*}\) as_

\[\pi(\mathbf{x})\stackrel{{\text{def}}}{{=}}\sum_{\mathbf{y}\in\Sigma^{*} }p(\mathbf{y})\mathbbm{1}\{\mathbf{x}\preceq\mathbf{y}\}=\sum_{\mathbf{y}\in\Sigma^{*}}p(\mathbf{ x}\mathbf{y}), \tag{8}\]

_where \(\mathbf{x}\preceq\mathbf{y}\) indicates that \(\mathbf{x}\) is a prefix of \(\mathbf{y}\)._

We now relate the MLE of a language model to the matching of _next-symbol_ conditional distributions. It will later allow us to reason about the relationship between smoothing and regularized maximum-likelihood estimation.

**Theorem 2.2**.: _Let \(p\) and \(q\) be two language models over \(\Sigma\) and \(\pi\) the prefix probability function of \(p\). Furthermore, we assume that \(\operatorname{H}(p,q)<\infty\). Then, the following equality holds_

\[D_{\mathrm{KL}}(p~{}||~{}q)\\ =\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})D_{\mathrm{KL}}(p(\cdot~{}|~{ }\mathbf{x})~{}||~{}q(\cdot~{}|~{}\mathbf{x})).\] (9) 

Proof.: App. A.1.2. 

We also consider the following corollary for \(p=p_{\mathcal{D}}\) and an \(n\)-gram language model \(q\).

**Corollary 2.3**.: _Let \(p_{\mathcal{D}}\) be an empirical distribution induced by a dataset \(\mathcal{D}\). Let \(q\) be an \(n\)-gram language model. Then, it holds that:_

\[D_{\mathrm{KL}}\left(p_{\mathcal{D}}~{}||~{}q\right) \tag{10}\] \[\propto\sum_{\mathbf{x}^{n}\in\Sigma_{\text{nos}}^{n-1}}\hskip-56.905512pt \#(\mathbf{x}^{n})D_{\mathrm{KL}}\left(p_{\mathcal{D}}(\cdot~{}|~{}\mathbf{x}^{n})~{}||~ {}q(\cdot~{}|~{}\mathbf{x}^{n})\right).\]

Proof.: App. A.2. 

Crucially, the \(n\)-gram assumption allows us to reduce the infinite sum over \(\Sigma^{*}\) in the definition of \(D_{\mathrm{KL}}\left(p_{\mathcal{D}}~{}||~{}q_{\mathrm{MLE}}^{n}\right)\) to one over \(\Sigma_{\text{ROS}}^{n-1}\) due to the resulting conditional independence of \(x_{t}\) and \(x_{1}\cdots x_{t-n}\) given \(\mathbf{x}_{t}^{n}=x_{t-n+1}\cdots x_{t-1}\). Further, it allows us to restrict the summation to the \(n\)-grams that are present in the training corpus.

### Label Smoothing of \(n\)-gram LMs

Let \(q_{\mathbf{\theta}}\) be a language model parametrized by parameters \(\mathbf{\theta}\in\Theta\). We further assume that \(q_{\mathbf{\theta}}\) is a differentiable function in \(\mathbf{\theta}\) and that \(\Theta\) is compact. Optimizing the objective in Eq. (10) gives us the maximum-likelihood estimate of model parameters \(\mathbf{\theta}\). To prevent overfitting and improve generalization abilities, this estimation can be regularized.

**Principle 2.4**.: _The principle of **regularization states that we should add an inductive bias to the parameter estimation procedure, the goal of which is to help the model generalize to unseen data at the expense of its ability to better fit the training data._

By Principle 2.4, we can intuitively see that smoothing techniques are a form of regularization. However, they form regularization that is defined procedurally in terms of the manipulation of count-based estimates. **Label smoothing**, on the other hand, is defined as an additive augmentation of the _training objective_--it represents the addition of the following regularizer to the training objective

\[\mathcal{R}_{\mathrm{LS}}(\mathbf{\theta},\mathbf{x})=D_{\mathrm{KL}}\left(u~{}||~{} q_{\mathbf{\theta}}(\cdot~{}|~{}\mathbf{x})\right), \tag{11}\]

where \(u=1/|\overline{\Sigma}|\cdot\mathbf{1}\) is the uniform distribution over \(\overline{\Sigma}\). In words, label smoothing regularizes the maximum-likelihood objective toward a uniform distribution over the next symbol. In the case of an \(n\)-gram model, we have the regularized objective

\[\sum_{\mathbf{x}^{n}\in\Sigma_{\text{ROS}}^{n-1}}\hskip-56.905512pt \#(\mathbf{x}^{n})\Big{[}D_{\mathrm{KL}}\Big{(} p_{\mathcal{D}}(\cdot~{}|~{}\mathbf{x}^{n})~{}||~{}q_{\mathbf{\theta}}(\cdot~{}|~{} \mathbf{x}^{n})\Big{)}\] \[+\gamma\mathcal{R}_{\mathrm{LS}}(\mathbf{\theta},\mathbf{x}^{n})\Big{]}. \tag{12}\]

The optimum of Eq. (12), \(\tilde{q}_{\mathbf{\theta}}\), is then what we refer to as the label-smoothed version of the maximum-likelihood solution.10

Footnote 10: Throughout the paper, we will use the notation \(\hat{q}\) for the smoothed version of the distribution \(q\).

In SS1, we introduced add-\(\lambda\) smoothing of \(n\)-_gram_ language models as a way to improve their generalization. However, regularization of the form Eq. (12) can be applied to _any_ language model \(q_{\mathbf{\theta}}\) whose parameters are learned through standard maximum-likelihood estimation--usually via gradient descent. Attractively, we can show that, if \(q_{\mathbf{\theta}}\) is a \(n\)-gram language model, regularization from Eq. (12) is equivalent to add-\(\lambda\) smoothing of \(n\)-gram counts in the sense that its optimum recovers the same model. We believe this to be the first formal connection made between add-\(\lambda\) smoothing and label smoothing of \(n\)-gram language models.11

Footnote 11: The restriction to \(n\)-gram language models is natural since the simple nature of \(n\)–gram language models permits the augmentation with hallucinated substring counts. Later in the paper, we show how this can be translated to neural language models by pre-processing the empirical data distribution.

**Theorem 2.5**.: _Estimating an \(n\)-gram model under regularized MLE with regularizer \(\mathcal{R}_{\mathrm{LS}}\) with strength parameter \(\gamma\) is equivalent to estimating an \(n\)-gram model and applying add-\(\lambda\) smoothing with \(\lambda=\frac{\gamma}{|\Sigma|+1}\)._

Proof.: App. A.2 

Theorem 2.5 establishes an interpretable connection between a _smoothing technique_--in this case, add-\(\lambda\) smoothing, which can only be performed in the context of \(n\)-gram language models--and an _additive regularizer_, which can be applied to general language models. While additive regularizers are not common in the context of \(n\)-gram language models, where an augmentation of counts is usually more appropriate, this framing will facilitate the connection to more modern neural language models, as we showcase in SS4.

## 3 Smoothing \(n\)-Gram Counts

In the context of \(n\)-gram models, smoothing procedures generally modify the count-based MLE computation to address the fact that not all \(n\)-grams oc 

cur in the training data. We follow Chen and Goodman (1999) and review four well-known smoothing techniques of \(n\)-gram language models before connecting them to a generalized framework of regularization in SS4.

### Good-Turing (1953)

Good-Turing (GT) smoothing is one of the earliest methods devised to compute a smoothed \(n\)-gram model \({\tilde{q}_{\rm MLE}^{n}}\) from a \(n\)-gram model \(q_{\rm MLE}^{n}\). GT smoothing assigns cumulative probability mass to \(n\)-grams that appear \(i\) times in the training data to be equal to the total probability mass of \(n\)-grams that appear \(i+1\) times in the training data. To do so, adjusted \(n\)-gram counts \(\#_{\rm GT}({\mathbf{x}}^{n}x)\) are computed as

\[\#_{\rm GT}({\mathbf{x}}^{n}x)=(\#({\mathbf{x}}^{n}x)+1)\frac{r_{\#({\mathbf{x}}^{n}x)+1}}{ r_{\#({\mathbf{x}}^{n}x)}}, \tag{13}\]

where \(r_{\#({\mathbf{x}}^{n}x)}\) is the total number of \(n\)-grams that occur \(\#({\mathbf{x}}^{n}x)\) times in the training data, i.e., \(r_{i}\stackrel{{\text{\tiny def}}}{{=}}\sum_{{\mathbf{x}}^{n}\in\Sigma _{n03}^{n-1}}\mathbbm{1}\{\#({\mathbf{x}}^{n}x)=i\}\). The probability of \({\mathbf{x}}^{n}\) is then defined as

\[{\tilde{q}_{\rm GT}^{n}}(x\mid{\mathbf{x}}^{n})=\frac{\#_{\rm GT}({\mathbf{x}}^{n}x) }{\sum_{i=1}^{\infty}ir_{i}}, \tag{14}\]

where the denominator in Eq. (14) is equivalent to the total number of tokens in \({\cal D}\). Note that any symbol whose successive count of counts is null is also assigned a null smoothed count. To avoid this issue, Gale and Sampson (1995) propose to interpolate the missing counts through linear regression and use the regressed counts to compute the smoothed probabilities.

### Jelinek-Mercer (1980)

Jelinek-Mercer (JM) smoothing relies on interpolation between higher-order and lower-order \(n\)-gram models to smooth \(q_{\rm MLE}^{n}\). The interpolation is applied recursively according to the following convex combination

\[\begin{split}{\tilde{q}_{\rm JM}^{n}}(x\mid{\mathbf{x}}^{n})& =\lambda_{n}q_{\rm MLE}^{n}(x\mid{\mathbf{x}}^{n})\\ &+(1-\lambda_{n}){\tilde{q}_{\rm JM}^{n-1}}(x\mid{\mathbf{x}}^{n-1}) .\end{split} \tag{15}\]

The recursion can be grounded either at the unigram level or with a uniform distribution over \(\overline{\Sigma}\).

### Katz (1987)

Katz smoothing relies on smoothed counts to compute its smoothed probabilities. These counts are computed as follows

\[\#_{\rm K}({\mathbf{x}}^{n}x)\stackrel{{\text{\tiny def}}}{{=}} \left\{\begin{array}{l}\#_{\rm K}({\mathbf{x}}^{n}x)\\ \mbox{\bf if $\#_{\rm K}({\mathbf{x}}^{n}x)>k$}\\ d_{\#_{\rm K}({\mathbf{x}}^{n}x)\#_{\rm K}({\mathbf{x}}^{n}x)}\\ \mbox{\bf if $0<\#_{\rm K}({\mathbf{x}}^{n}x)\leq k$}\\ \alpha({\mathbf{x}}^{n})q_{\rm MLE}^{n}({\mathbf{x}}^{n-1}x)\\ \mbox{\bf otherwise}~{},\end{array}\right. \tag{16}\]

where \(k\) is a hyperparameter whose value is usually assigned to a high-range, single-digit integer. For large counts, smoothed counts are equivalent to the empirical \(n\)-gram counts as the latter are assumed to be reliable. Small non-zero counts, however, are discounted using count-specific discount factors \(d_{\#_{\rm K}}\), which are derived from the Good-Turing counts in Eq. (13) and computed as

\[d_{\#_{\rm K}({\mathbf{x}}^{n}x)}=\frac{\#_{\rm GT}({\mathbf{x}}^{n}x)-\frac{(k+1)r_{ k+1}}{r_{1}}}{1-\frac{(k+1)r_{k+1}}{r_{1}}}. \tag{17}\]

The total amount obtained by discounting is then redistributed to the \(n\)-grams with null counts, weighted by the probability of the lower-order \(n\)-gram and according to the normalization factor

\[\alpha({\mathbf{x}}^{n})=\frac{1-\sum_{x:\#({\mathbf{x}}^{n}x)>0}{\tilde{q}_{\rm K}^{ n}(x\mid{\mathbf{x}}^{n})}}{1-\sum_{x:\#({\mathbf{x}}^{n}x)>0}{q_{\rm MLE}^{n}({\mathbf{ x}}^{n-1}x)}}. \tag{18}\]

Finally, smoothed probabilities are computed by normalizing the smoothed counts according to the following formula

\[{\tilde{q}_{\rm K}^{n}}(x\mid{\mathbf{x}}^{n})=\frac{\#_{\rm K}({\mathbf{x}}^{n}x)}{ \sum_{y\in\overline{\Sigma}}\#_{\rm K}({\mathbf{x}}^{n}y)}. \tag{19}\]

### Kneser-Essen-Ney (1995)

Kneser-Essen-Ney (KEN) smoothing is similar to Katz smoothing in that it also computes higher-order \(n\)-gram probabilities as a function of lower-order \(n\)-gram probabilities. However, in contrast to other smoothing methods, KEN smoothing does not construct \(n\)-gram probability distributions using simple counts, but rather using _type counts_. The type count of an \(n\)-gram is defined as the number of _distinct_ histories that the \(n\)-gram follows, rather than the absolute number of its occurrences in the data. Formally, the type count function \(\#_{\rm T}^{n}\) of order \(n\) is defined as

\[\#_{\rm T}^{n}({\mathbf{x}}^{n},x)\stackrel{{\text{\tiny def}}}{{=}} \begin{cases}1&\mbox{\bfif $\#({\mathbf{x}}^{n}x)>0$}\\ 0&\mbox{\bfotherwise}.\end{cases}\] (20) 

We further define \(\#_{\text{T}}^{n}(\bullet,x)\), \(\#_{\text{T}}^{n}(\mathbf{x}^{n},\bullet)\) and \(\#_{\text{T}}^{n}(\bullet,\bullet)\) as the type count function with bulleted arguments summed out. Type counts allow us to reduce the probability assigned to \(n\)-grams that occur many times in the data, but whose constituent \((n-1)\)-grams have low probability. A common illustrative example in the literature Chen and Goodman (1999) is the bigram _San Francisco_. If the term _San Francisco_ appears frequently in a dataset, then the unigram probability assigned to _Francisco_ by smoothing methods that rely on lower-order \(n\)-gram distributions to compute higher-order \(n\)-gram distributions will be quite high. However, this is arguably undesirable in many situations, since the unigram _Francisco_ does not often appear after words other than _San_.

At the unigram level, the probability estimates of a KEN-smoothed distribution are computed using type counts for unigrams and bigrams as

\[\tilde{q}_{\mathrm{KEN}}^{1}(x)=\frac{\#_{\text{T}}^{1}(\bullet,x)}{\#_{ \text{I}}(\bullet,\bullet)}. \tag{21}\]

These probabilities are then used to ground the recursion that computes the smoothed probabilities \(\tilde{q}_{\mathrm{KEN}}^{n}\) for higher-order \(n\)-grams according to the following formula

\[\begin{split}&\tilde{q}_{\mathrm{KEN}}^{n}(x\mid\mathbf{x}^{n})= \frac{\max\{\#(\mathbf{x}^{n}x)-D,0\}}{\sum_{y\in\Sigma}\#(\mathbf{x}^{n}y)}\\ &\qquad+\frac{D\cdot\#_{\text{T}}^{n}(\mathbf{x}^{n}\bullet)\cdot \tilde{q}_{\mathrm{KEN}}^{n-1}(x\mid\mathbf{x}^{n-1})}{\sum_{y\in\Sigma}\#(\mathbf{x}^{ n}y)}.\end{split} \tag{22}\]

## 4 A Generalized Framework

In SS2, we evince a connection between add-\(\lambda\) smoothing and regularization of the maximum-likelihood objective. However, the derivation we formalize (cf. App. A.1.2) is tedious and long. Moreover, it exploits several specific properties of add-\(\lambda\) smoothing. Performing such a derivation for each smoothing technique individually would be laborious and further, it would hinder building intuitions about the relationships between different methods. Luckily, we can introduce a more general framework. Specifically, in this section, we propose a framework that allows us to formulate equivalent regularizers for _any_ smoothing technique and apply them to the training of neural language models.

### \(n\)**-Gram Smoothing as Regularization**

Without further ado, we now introduce our framework for connecting the smoothing of \(n\)-gram language models to the regularization of the maximum-likelihood objective. This allows us to expand the notion of \(n\)-gram smoothing to neural language models. To this end, we first revisit MLE.

One way of framing MLE is using the KL divergence. Specifically, given an empirical distribution \(p_{\mathcal{D}}\), the principle of MLE dictates that we should choose a model \(q_{\boldsymbol{\theta}}\) such that \(D_{\mathrm{KL}}\left(p_{\mathcal{D}}\mid|~{}q_{\boldsymbol{\theta}}\right)=0\). In comparison, \(n\)-gram smoothing techniques are often not defined so declaratively. Instead, they are presented as procedures that directly modify the empirical counts derived from a large dataset (e.g., Eq. (3) in the simple case of add-\(\lambda\) smoothing). The crucial observation in this work is that we can treat \(n\)-gram smoothing as a two-step process. First, we view the smoother as a map \(p_{\mathcal{D}}^{n}\mapsto\tilde{p}_{\mathcal{D}}^{n}\) that outputs a smoothed empirical \(n\)-gram distribution. Then, we choose the \(q_{\boldsymbol{\theta}}\) that minimizes \(D_{\mathrm{KL}}\left(\tilde{p}_{\mathcal{D}}^{n}\mid\mid q_{\boldsymbol{ \theta}}\right)\) where we have replaced \(p_{\mathcal{D}}\) with \(\tilde{p}_{\mathcal{D}}^{n}\).

In that context, the question we ask is: Rather than minimizing \(D_{\mathrm{KL}}\left(\tilde{p}_{\mathcal{D}}^{n}\mid q_{\boldsymbol{\theta}}\right)\), can we always find a regularizer \(\mathcal{R}(\boldsymbol{\theta})\) such that \(D_{\mathrm{KL}}\left(\tilde{p}_{\mathcal{D}}^{n}\mid\mid q_{\boldsymbol{ \theta}}\right)=D_{\mathrm{KL}}\left(p_{\mathcal{D}}\mid\mid q_{\boldsymbol{ \theta}}\right)+\mathcal{R}(\boldsymbol{\theta})\)? Such a result would be a natural generalization of the add-\(\lambda\) case, discussed in Theorem 2.5 that would apply to any \(n\)-gram smoothing techniques, including all of those presented in SS3.

### Smoothing as Regularization

Now we turn to the primary question of this paper. How do we construct a regularizer that corresponds to an _arbitrary_\(n\)-gram smoothing technique? We begin by defining the following two probability distributions that together capture the difference between the empirical distribution and the smoothed empirical \(n\)-gram distribution:

\[p_{+}(\mathbf{x}) \stackrel{{\mbox{\tiny def}}}{{=}}\frac{1}{Z_{+}} \max(0,\tilde{p}_{\mathcal{D}}^{n}(\mathbf{x})-p_{\mathcal{D}}(\mathbf{x})) \tag{23a}\] \[p_{-}(\mathbf{x}) \stackrel{{\mbox{\tiny def}}}{{=}}\frac{1}{Z_{-}} \max(0,p_{\mathcal{D}}(\mathbf{x})-\tilde{p}_{\mathcal{D}}^{n}(\mathbf{x})), \tag{23b}\]

where the normalization constants are defined as

\[Z_{+} \stackrel{{\mbox{\tiny def}}}{{=}}\sum_{\mathbf{x}\in \Sigma^{*}}\max(0,\tilde{p}_{\mathcal{D}}^{n}(\mathbf{x})-p_{\mathcal{D}}(\mathbf{x})) \tag{24a}\] \[Z_{-} \stackrel{{\mbox{\tiny def}}}{{=}}\sum_{\mathbf{x}\in \Sigma^{*}}\max(0,p_{\mathcal{D}}(\mathbf{x})-\tilde{p}_{\mathcal{D}}^{n}(\mathbf{x})). \tag{24b}\]

This results in the following simple decomposition:

\[\tilde{p}_{\mathcal{D}}^{n}(\mathbf{x})=p_{\mathcal{D}}(\mathbf{x})+Z_{+}p_{+}(\mathbf{x})- Z_{-}p_{-}(\mathbf{x}). \tag{25}\]

Why does the above formulation help? Fundamental to our derivation in SS2 was the idea that we could 

think of add-\(\lambda\) smoothing as adding a regularization term to the maximum-likelihood objective that penalizes diverging from a simple distribution--in the case of add-\(\lambda\) smoothing, the uniform distribution over \(\overline{\Sigma}\). Similarly, the decomposition of \(\tilde{p}_{\mathcal{D}}^{n}\) given in Eq. (25) facilitates the interpretation of training a language model on \(\tilde{p}_{\mathcal{D}}^{n}\) as training \(p_{\mathcal{D}}\) with the addition of regularization. Concretely, we define the following regularizer

\[\begin{split}\mathcal{R}(\mathbf{\theta})&\stackrel{{ \text{def}}}{{=}}Z_{+}D_{\mathrm{KL}}(p_{+}~{}||~{}q_{\mathbf{\theta}})\\ &+Z_{-}D_{\mathrm{KL}}(p_{-}~{}||~{}q_{\mathbf{\theta}}).\end{split} \tag{26}\]

Now, the relation between estimating an \(n\)-gram model with a smoothing technique and using the regularizer formalized in Eq. (26) is given by the following theorem.

**Theorem 4.1**.: _Let \(p_{\mathcal{D}}\) be the empirical distribution induced by the dataset \(\mathcal{D}\) and \(\tilde{p}_{\mathcal{D}}^{n}\) a smoothed empirical \(n\)-gram distribution. For \(\gamma=1\), the following holds_

\[\begin{split} D_{\mathrm{KL}}(\tilde{p}_{\mathcal{D}}^{n}& ~{}||~{}q_{\mathbf{\theta}})\\ &~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} =D_{\mathrm{KL}}(p_{\mathcal{D}}~{}||~{}q_{\mathbf{\theta}})+\gamma\mathcal{R}( \mathbf{\theta})+C,\end{split} \tag{27}\]

_where \(C\) is constant with respect to \(q_{\mathbf{\theta}}\)._

Proof.: App. A.2 

Theorem 4.1 formalizes how training on the smoothed distribution \(\tilde{p}_{\mathcal{D}}^{n}\) computed by smoothing the \(n\)-gram counts affects the maximum-likelihood objective. It brings us to an interesting observation about smoothing methods _in general_--they can all be formalized as solutions to a regularized maximum-likelihood objective. Inspecting Eq. (27), we see that, crucially, only the first term depends on the _original_ empirical distribution \(p_{\mathcal{D}}\)--indeed, it represents the original maximum-likelihood objective. The other two terms depend both on the empirical data distribution as well as its smoothed variant. We can therefore interpret Eq. (27) as a regularized loss where the last two terms correspond to the equivalent regularizer of the smoothing method used to construct \(\tilde{p}_{\mathcal{D}}^{n}\). In practice, we might want to _modulate_ the strength of the regularization towards the smoothed distribution \(\tilde{p}_{\mathcal{D}}^{n}\). We can achieve such an effect through an additional hyperparameter \(\gamma\), by which we multiply our regularizer in Eq. (27) to control its influence. The regularized loss can be decomposed further by _splitting_ the \(\gamma\) hyperparameter into two terms \(\gamma_{+}\) and \(\gamma_{-}\) and applying them separately to the positive and negative terms of the regularizer \(\mathcal{R}\)

\[\begin{split} D_{\mathrm{KL}}(p_{\mathcal{D}}~{}||~{}q_{\mathbf{ \theta}})+&\gamma_{+}Z_{+}D_{\mathrm{KL}}(p_{+}~{}||~{}q_{\mathbf{ \theta}})\\ +&\gamma_{-}Z_{-}D_{\mathrm{KL}}(p_{-}~{}||~{}q_{\mathbf{ \theta}}),\end{split} \tag{28}\]

where Eq. (28) is equivalent to \(D_{\mathrm{KL}}(\tilde{p}_{\mathcal{D}}^{n}~{}||~{}q_{\mathbf{\theta}})\) when both \(\gamma_{+}\) and \(\gamma_{-}\) are equal to \(1\).

Our generalized framework, therefore, presents a novel way of constructing regularizers to be used in the language modeling objective based on insights from classical methods for smoothing \(n\)-gram language models. Importantly, it provides a direct mechanism by which smoothing-based regularization can be applied to any language model \(q_{\mathbf{\theta}}\). In the following section, we use this framework to explore the empirical effects of using regularizers constructed from smoothing methods (cf. SS3) in the training of neural language models.

Runtime Analysis.The distributions \(p_{+}\) and \(p_{-}\) require \(\mathcal{O}\left(|\Sigma|^{n}\right)\) space to represent where \(n\) is the \(n\)-gram order, i.e., the space complexity is of the order of the number of \(n\)-gram contexts in the model. While the exponential increase in \(n\) of \(\mathcal{O}\left(|\Sigma|^{n}\right)\) is one of the main limitations for scalability of \(n\)-gram models, our method does not require increasing \(n\) to large values, as it leverages (smoothed) \(n\)-gram models only in the construction of a regularizer for the training of a much larger neural model. Further, the scalability issues of \(n\)-gram models can be circumvented by using bespoke data structures Liu et al. (2024).

## 5 Experiments

### Setup

We validate our proposed regularization framework on two tasks: language modeling and machine translation. We rely on the small-scale WikiText-2 Merity et al. (2017) and IWSLT-14 Cettolo et al. (2014) data sets, respectively, and compare the performance of standard MLE and label smoothing to the performance obtained by using regularizers based on the smoothing methods illustrated in SS3. For both tasks, we perform our experiments via the fairseq library Ott et al. (2019) on Transformer-based Vaswani et al. (2017) language models.

Our implementation of (Simple) Good-Turing smoothing in fairseq builds on an open-source implementation,12 while we leverage the efficient implementation of Kneser-Essen-Ney smoothing 

available through the KenLM(Heafield, 2011; Heafield et al., 2013) library.13 The remaining smoothing methods were implemented natively in fairseq. Note that, as all data sets are small in scale, we limit the maximum \(n\)-gram order to 2 (i.e., bigrams) for all smoothing methods. We use dropout for all experiments fixing the dropout probability to \(0.1\) and \(0.3\) for language modeling and machine translation, respectively. For all smoothing techniques, we set \(\Gamma~{}\stackrel{{\text{def}}}{{=}}~{}\{0.005,0.01,0.05,0.1,0.5\}\) and grid search regularization hyperparameter pairs \(\gamma_{+},\gamma_{-}\in\Gamma\times\Gamma\). For smoothing methods that have additional hyperparameters, we extend the grid search described above to include them. We provide the complete list of method-specific hyperparameter values in Tab. 3. Additional dataset details are provided in Tab. 4.

Footnote 13: github.com/kpu/kenlm

### Language Modeling

For language modeling, we evaluate the performance of our regularizers on the raw version of the WikiText-2 dataset Merity et al. (2017) which we preprocess to remove all empty samples. We tokenize the data using BPE Sennrich et al. (2016) with \(16{,}000\) merge operations through the subword-nmt library.14 For modeling, we use the decoder-only Transformer architecture denoted as transformer-lm in fairseq while adopting standard hyperparameter settings as suggested by fairseq15 to encourage reproducibility. We train all models using early stopping and take as the best-performing models the ones with the lowest perplexity on the validation set. For each method, the best-performing hyperparameter setting is then trained over 5 different seeds. We summarize the results for the best-performing hyperparameter settings in Tab. 1. We find that all regularized objectives outperform the unregularized baseline, with Jelinek-Mercer obtaining the lowest perplexity. We test for mean separation using the Wilcoxon rank-sum test finding that all smoothing methods obtain statistically significant improvements over the unregularized baseline. Perplexity scores for the best-performing runs are shown in App. C together with \(p\)-value under a paired permutation test.

Footnote 14: github.com/rsenrich/subword-nmt

Footnote 15: github.com/facebookresearch/fairseq/tree/main/examples/language_model

### Machine Translation

We evaluate the performance of our proposed regularizers on machine translation on the German-to-English task of the IWLST-14 dataset. In the translation setting, we limit the application of smoothing only to distributions over the vocabulary of the target language. We preprocess the data set by following the processing script provided by fairseq16 and tokenize the dataset with BPE using \(10{,}000\) merge operations for both languages. As our model, we use the small-sized transformer_iwslt_de_en encoder-decoder Transformer and its corresponding standard training hyperparameters.17 We repeat the same grid search procedure over regularization hyperparameters as

\begin{table}
\begin{tabular}{l|c} \hline Smoothing Method & ppl \(\downarrow\) \\ \hline None & \(147.12\pm 0.34\) \\ add-\(\lambda\) (\(\gamma_{+}=0.1,\gamma_{-}=0.05,\gamma_{LS}=0.01\)) & \(142.10^{\dagger}\pm 0.65\) \\ GT (\(\gamma_{+}=0.1,\gamma_{-}=0.05\)) & \(141.93^{\dagger}\pm 0.73\) \\ JM (\(\gamma_{+}=0.1,\gamma_{-}=0.5,\lambda_{1}=0.75\)) & **137.41\({}^{\dagger}\pm 0.40\)** \\ Katz (\(\gamma_{+}=0.1,\gamma_{-}=0.01,k=5\)) & \(142.69^{\dagger}\pm 0.54\) \\ KEN (\(\gamma_{+}=0.1,\gamma_{-}=0.1\)) & \(142.30^{\dagger}\pm 0.29\) \\ \hline \end{tabular}
\end{table}
Table 1: Perplexity on WikiText-2 test set. Included are performances of models trained with no regularization (None), and with various smoothing methods. Reported perplexities are mean values for \(5\) independently trained models, together with their standard errors. The best-performing method is in bold, while the second-best is underlined. \(\dagger\) indicates statistical significance with respect to the unregularized baseline with \(p<0.05\).

\begin{table}
\begin{tabular}{l|l} \hline Smoothing Method & BLEU \(\uparrow\) \\ \hline None & \(32.86\pm 0.04\) \\ add-\(\lambda\) (\(\gamma_{+}=0.1,\gamma_{-}=0.01,\gamma_{IS}=0.01\)) & \(33.23^{\dagger}\pm 0.03\) \\ GT (\(\gamma_{+}=0.05,\gamma_{-}=0.5\)) & \(33.37^{\dagger}\pm 0.01\) \\ JM (\(\gamma_{+}=0.1,\gamma_{-}=0.5,\lambda_{1}=0.5\)) & \(\textbf{33.67}^{\dagger}\pm 0.05\) \\ Katz (\(\gamma_{+}=0.1,\gamma_{-}=0.1,k=5\)) & \(33.23^{\dagger}\pm 0.02\) \\ KEN (\(\gamma_{+}=0.1,\gamma_{-}=0.1\)) & \(33.38^{\dagger}\pm 0.03\) \\ \hline \end{tabular}
\end{table}
Table 2: BLEU on test set of IWSLT-14 DE-EN. Different regularized methods are compared to no regularization (None). Reported values are means over \(5\) independently trained models together with their standard errors. The best-performing method is in bold, while the second-best is underlined. \(\dagger\) indicates statistical significance with respect to the unregularized baseline with \(p<0.05\).

 

previously outlined, and use BLEU Papineni et al. (2002) on the validation set to determine the best-performing model checkpoints. To decode text from the model, we use beam search with a beam size of \(5\). We evaluate the generated translations with sacreBLEU Post (2018).19 Tab. 2 contains our results. All smoothing methods improve over the baseline (no regularizer), and Jelinek-Mercer smoothing is the best-performing technique. We repeat the mean separation tests outlined in the language modeling subsection and find that all smoothing methods obtain statistically significant improvements over the unregularized baseline. In App. C we additionally show the results of the best-performing models for each method and test their significance using paired bootstrap resampling Koehn (2004). Further, in App. C.1 we present the results of a preliminary evaluation of our methods on the English-to-German task of the larger WMT14 machine translation dataset.

Footnote 19: Sätthub.com/mjpost/sacrebleu

Footnote 20: SacreBLEU signature: nrefs:1|case::mixed|eff:nob|tok:13a|smooth:exp|version:2.3.2

## 6 Related Work

Hybrid Neural and \(n\)-gram Models.The relationship between neural networks and \(n\)-gram models has been explored in previous work. For instance, Bengio et al. (2000) famously introduced a neural parameterization of an \(n\)-gram model, achieving state-of-the-art results at the time. More recently, Sun and Iyyer (2021) scaled Bengio et al.'s (2000) model on modern hardware and demonstrated a small performance increase on language modeling over a Transformer model using a hybrid \(n\)-gram-Transformer model. Schwenk (2007) explored interpolating neural and \(n\)-gram language models. Neubig and Dyer (2016) expanded Schwenk's (2007) approach by exploring various ways to combine neural and \(n\)-gram language models.

Regularization.On the topic of smoothing-based regularization, Lee et al. (2022) propose dynamically adjusting the strength of label smoothing regularization based on the entropy of the model distribution and using an earlier version of the model as a regularizer. In a similar vein, Baziotis et al. (2020) propose using a monolingual language model as a regularizer for a translation model. The idea is that monolingual data is far more abundant than bilingual data, so a language model of the target language is used to guide the target distribution of the translation model. Peters and Martins (2021) generalize label smoothing to the broader family of Fenchel-Young losses, making it applicable to entmax-based models, while Meister et al. (2020) generalize label smoothing to a set of entropy-based regularizers.

## 7 Conclusion

In this work, we re-imagine the application of classical \(n\)-gram smoothing techniques in the context of modern neural NLP models. For several of these historic methods, we derive equivalent, differentiable regularizers that can be added to neural models' training objectives. We present these results within a generalized framework that allows for insights about the smoothing methods themselves and their relationships to each other. We apply these smoothing methods in the training of neural language models and machine translation models. We find that our smoothing-based regularizers outperform label smoothing and standard MLE in language modeling, while some methods also achieve competitive results with label smoothing for machine translation.

### Limitations

We present results only for English (for language modeling) and between German and English (for machine translation). Most experiments are limited to small datasets for both language modeling and machine translation. Future work could verify how scaling the amount of data impacts present results and whether the observed performance improvements are also achievable in a wider set of languages. While in some experimental settings, we observed performance improvements for some smoothing methods, the additional computational

\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline \hline Method & Hyperparameters \\ \hline add-\(\lambda\) & \(\gamma_{\mathrm{LS}}\in\left\{0.01,0.05,0.1\right\}\) \\ GT & None \\ JM & \(\lambda_{1}\in\left\{0.25,0.5,0.75\right\}\) \\ Katz & \(k\in\left\{5,7,10\right\}\) \\ KEN & None \\ \hline \hline \end{tabular}
\end{table}
Table 3: Method-specific hyperparameters on which a grid search was performed for both tasks. Note that in Jelinek–Mercer, \(\lambda_{2}\) is obtained following its normalization constraint.

 

complexity required by their use may not be a worthwhile trade-off for their performance benefits.

## Ethics Statement

This paper is theoretical in nature, as it aims to shed light on the relationship between \(n\)-gram smoothing methods and language model regularization. For this reason, the authors foresee no ethical concerns with the research presented in this paper.

## Acknowledgements

We thank Li Du for helpful feedback on a draft of this paper. Anej Svate and Afra Amini are supported by the ETH AI Center Doctoral Fellowship.

## References

* Abney et al. [1999] Steven Abney, David McAllester, and Fernando Pereira. 1999. Relating probabilistic grammars and automata. In _Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computational Linguistics_, page 542-549, USA. Association for Computational Linguistics.
* Baziotis et al. [2020] Christos Baziotis, Barry Haddow, and Alexandra Birch. 2020. Language model prior for low-resource neural machine translation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_, pages 7622-7634, Online. Association for Computational Linguistics.
* Bengio et al. [2000] Yoshua Bengio, Rejean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. In _Advances in Neural Information Processing Systems_, volume 13. MIT Press.
* Cettolo et al. [2014] Mauro Cettolo, Jan Niehues, Sebastian Stuker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th IWSLT evaluation campaign. In _Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign_, pages 2-17, Lake Tahoe, California.
* Chelba and Jelinek [1998] Ciprian Chelba and Frederick Jelinek. 1998. Exploiting syntactic structure for language modeling. In _36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1_, pages 225-231, Montreal, Quebec, Canada. Association for Computational Linguistics.
* Chen and Goodman [1999] Stanley F. Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. _Computer Speech & Language_, 13(4):359-394.
* Chomsky [1957] Noam Chomsky. 1957. _Syntactic Structures_. De Gruyter Mouton, Berlin, Boston.
* Costa-jussa et al. [2022] Marta R. Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elaine Kalbassi, Janice Lam, Daniel Licht, Jean Mainillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangtijn Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shrutoi Bhosalse, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanu Goswami, Francisco Guzman, Philipp Koehin, Alexandre Mourachko, Christopher Rogers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. _arXiv preprint arXiv:2207.04672_.
* Cotterell [2023] Ryan Cotterell. 2023. Natural language processing assignment 4: CKY. Homework Assignment. ETH Zurich, Natural Language Processing, 252-3005-00L, Fall 2023.
* Cotterell et al. [2023] Ryan Cotterell, Anej Svate, Clara Meister, Tianyu Liu, and Li Du. 2023. Formal aspects of language modeling. _CoRR_, abs/2311.04329.
* Eisner [2023] Jason Eisner. 2023. Natural language processing homework 3: Smoothed language modeling. Homework Assignment. Johns Hopkins University, Natural Language Processing, 601.465/665, Fall 2023.
* Folland [1999] Gerald B. Folland. 1999. _Real Analysis: Modern Techniques and Their Applications_, 2nd edition. Wiley.
* Gale and Sampson [1995] William A. Gale and Geoffrey Sampson. 1995. Goodturing frequency estimation without tears. _Journal of Quantitative Linguistics_, 2(3):217-237.
* Good [1953] I. J. Good. 1953. The population frequencies of species and the estimation of population parameters. _Biometrika_, 40(3-4):237-264.
* Heafield [2011] Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In _Proceedings of the Sixth Workshop on Statistical Machine Translation_, pages 187-197, Edinburgh, Scotland. Association for Computational Linguistics.
* Heafield et al. [2013] Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 690-696, Sofia, Bulgaria. Association for Computational Linguistics.
* Jager and Rogers [2012] Gerhard Jager and James Rogers. 2012. Formal language theory: Refining the Chomsky hierarchy. _Philosophical Transactions of the Royal Society B: Biological Sciences_, 367(1598):1956-1970.
* Jelinek [1980] Frederick Jelinek. 1980. Interpolated estimation of markov source parameters from sparse data. In _Workshop on Pattern Recognition in Practice_ 

Frederick Jelinek and John D. Lafferty. 1991. Computation of the probability of initial substring generation by stochastic context-free grammars. _Computational Linguistics_, 17(3):315-353.
* Katz [1987] S. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. _IEEE Transactions on Acoustics, Speech, and Signal Processing_, 35(3):400-401.
* Koehn [2004] Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
* Le Cam [1952] Lucien M. Le Cam. 1952. _On Some Asymptotic Properties of Maximum Likelihood Estimates and Related Bayes' Estimates_. University of California: Publ. in statistics. University of California, Berkeley.
* Lee et al. [2022] Dongkyu Lee, Ka Chun Cheung, and Nevin Zhang. 2022. Adaptive label smoothing with self-knowledge in natural language generation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9781-9792, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Liu et al. [2024] Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024. Infini-gram: Scaling unbounded \(n\)-gram language models to a trillion tokens. _arXiv preprint arXiv:2401.17377_.
* Meister et al. [2020] Clara Meister, Elizabeth Salesky, and Ryan Cotterell. 2020. Generalized entropy regularization or: There's nothing special about label smoothing. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6870-6886, Online. Association for Computational Linguistics.
* Merity et al. [2017] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In _5th International Conference on Learning Representations_, Toulon, France.
* Neubig and Dyer [2016] Graham Neubig and Chris Dyer. 2016. Generalizing and hybridizing count-based and neural language models. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1163-1172, Austin, Texas. Association for Computational Linguistics.
* Ney et al. [1995] Hermann Ney, Ute Essen, and Reinhard Kneser. 1995. On the estimation of small probabilities by leaving-one-out. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 17(12):1202-1212.
* Nowak and Cotterrell [2023] Franz Nowak and Ryan Cotterrell. 2023. A fast algorithm for computing prefix probabilities. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 57-69, Toronto, Canada. Association for Computational Linguistics.
* Ott et al. [2019] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auil. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
* Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Pereyra et al. [2017] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In _5th International Conference on Learning Representations_, Toulon, France.
* Peters and Martins [2021] Ben Peters and Andre F. T. Martins. 2021. Smoothing and shrinking the sparse Seq2Seq search space. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, Online. Association for Computational Linguistics.
* Post [2018] Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
* Schwenk [2007] Holger Schwenk. 2007. Continuous space language models. _Computer Speech & Language_, 21(3):492-518.
* Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Sun and Iyyer [2021] Simeng Sun and Mohit Iyyer. 2021. Revisiting simple neural probabilistic language models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5181-5188, Online. Association for Computational Linguistics.
* Teh [2006] Yee Whye Teh. 2006. A Bayesian interpretation of interpolated Kneser-Ney. Technical report, National University of Singapore.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreti, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.

 

Proofs

This section contains the proofs of all the theorems in the main text. We begin by discussing prefix probabilities in App. A.1 as a tool for analyzing the relationship between the full Kullback-Leibler divergence \(D_{\mathrm{KL}}\left(p~{}||~{}q\right)\) and the divergences between the conditional probabilities \(D_{\mathrm{KL}}\left(p\left(\cdot~{}|~{}\mathbf{x}\right)\,||~{}q\left(\cdot~{}|~{ }\mathbf{x}\right)\right)\) for \(\mathbf{x}\in\Sigma^{*}\). We then move on to the proofs of the results characterizing the aforementioned relationship in App. A.2.

### Prefix Probabilities

#### a.1.1 Introductory Notes on Prefix Probabilities

In this section, we provide supplementary commentary on prefix probabilities. First, proving the second equality in Eq.8 is a useful exercise; indeed, the last author often assigns the task to his students [30]. Second, it is important to keep in mind that while \(\pi(\mathbf{x})\) is the probability of a certain event--namely, the event that a string _starts_ with the prefix \(\mathbf{x}\), \(\pi\) itself is not a valid probability distribution, i.e., \(\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})\neq 1\). Indeed, \(\pi\) may not even be normalizable, i.e., we may have that \(\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})\to\infty\). This property should make intuitive sense: By the definition of \(\pi\), we count the probability of certain events under \(p\) in our computation of prefix probabilities under \(\pi\)_multiple times_. For example, in the case that \(\Sigma=\{\mathsf{a}\}\), \(p(\mathsf{a})\) counts towards both \(\pi(\mathsf{a})\) and \(\pi(\mathsf{aa})\). Since \(p\) is a valid probability distribution, i.e., its probabilities sum to 1, then our prefix probabilities will often sum to \(>1\).

In the special case of empirical distributions, the prefix probability of a substring is proportional to the number of times the substring appears in the dataset \(\mathcal{D}\). Concretely, we have that

\[\pi_{\mathcal{D}}(\mathbf{x})\propto\text{number of strings in $\mathcal{ D}$ starting with $\mathbf{x}$}, \tag{29}\]

which means that

\[\sum_{\mathbf{y}\in\Sigma^{*}}\pi_{\mathcal{D}}(\mathbf{y}\mathbf{x}) \propto\sum_{\mathbf{y}\in\Sigma^{*}}\text{number of strings in $\mathcal{D}$ starting with $\mathbf{y}\mathbf{ x}$} \tag{30a}\] \[\propto\#(\mathbf{x}). \tag{30b}\]

That is, the number of occurrences of \(\mathbf{x}\) in \(\mathcal{D}\) is proportional to \(\sum_{\mathbf{y}\in\Sigma^{*}}\pi_{\mathcal{D}}(\mathbf{yx})\). Why is Eq.30 true? Because every time we observe a \(\mathbf{x}\) in the training dataset it must have some prefix that starts with the beginning of a string.

#### a.1.2 Prefix Probabilities and Local Kullback-Leibler Divergences

We now move on to proving a crucial component of analyzing the relationship between dataset smoothing and regularized training--the relationship between the global Kullback-Leibler divergence and the local Kullback-Leibler divergences of the next-symbol conditional probabilities. Intuitively, we show that the global Kullback-Leibler divergence \(D_{\mathrm{KL}}(p~{}||~{}q)\) can be written as a prefix-probability-weighted sum of local next-symbol probability distributions \(D_{\mathrm{KL}}\left(p\left(\cdot~{}|~{}\mathbf{x}\right)\,||~{}q\left(\cdot~{}|~ {}\mathbf{x}\right)\right)\) for \(\mathbf{x}\in\Sigma^{*}\). This result, which we believe to be novel, is formally captured by Theorem 2.2.

**Theorem 2.2**.: _Let \(p\) and \(q\) be two language models over \(\Sigma\) and \(\pi\) the prefix probability function of \(p\). Furthermore, we assume that \(\operatorname{H}(p,q)<\infty\). Then, the following equality holds_

\[D_{\mathrm{KL}}( p~{}||~{}q)\] \[=\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})D_{\mathrm{KL}}(p(\cdot~{}|~{} \mathbf{x})~{}||~{}q(\cdot~{}|~{}\mathbf{x})). \tag{9}\]

Proof.: Let \(\Sigma\) be an alphabet and let \(p\) and \(q\) be distributions over \(\Sigma^{*}\). We make use of the following definition. Let \(\mathbf{x}\in\Sigma^{*}\) be a string and \(T\in\mathbb{N}_{\geq 0}\). The bounded prefix probability of \(\mathbf{x}\) is defined as

\[\pi_{T}(\mathbf{x})\stackrel{{\textup{def}}}{{=}}\sum_{\mathbf{y}\in \Sigma^{*}}\mathbbm{1}\{\mathbf{x}\preceq\mathbf{y}\}p(\mathbf{y}~{}|~{}T),\] (31) 

where \({\mathbf{x}}\preceq{\mathbf{y}}\) indicates that \({\mathbf{x}}\) is a prefix of \({\mathbf{y}}\), \(p({\mathbf{y}}~{}|~{}T)\) is the conditional of language model \(p\) to strings of length \(T\), and \(p(T)=\sum_{{\mathbf{x}}\in\Sigma^{T}}p({\mathbf{x}})\). Note that the following equality relates \(\pi\) and \(\pi_{T}\)

\[\pi({\mathbf{x}})=\sum_{T=0}^{\infty}p(T)\pi_{T}({\mathbf{x}}). \tag{32}\]

To prove Eq. (9), we split \(D_{\mathrm{KL}}\) into a cross-entropy and an entropy term as follows

\[D_{\mathrm{KL}}(p~{}||~{}q)\stackrel{{\text{ def}}}{{=}}\sum_{{\mathbf{x}}\in\Sigma^{*}}p({\mathbf{x}})\log\left(\frac{p({\mathbf{x}})}{ q({\mathbf{x}})}\right) \tag{33a}\] \[=-\sum_{{\mathbf{x}}\in\Sigma^{*}}p({\mathbf{x}})\log(q({\mathbf{x}}))+\sum_ {{\mathbf{x}}\in\Sigma^{*}}p({\mathbf{x}})\log(p({\mathbf{x}}))\] (33b) \[=\operatorname{H}(p,q)-\operatorname{H}(p). \tag{33c}\]

We show the equivalence of cross-entropy, as entropy is the special case when \(\operatorname{H}(p,p)\).20 Starting with the cross-entropy we have

Footnote 20: In what follows, we will frequently interchange infinite sums. Since the terms involved are either all \(\geq 0\) or all \(\leq 0\), Tonelli’s theorem guarantees that such interchanges are valid (Fölland, 1999, Theorem 2.37.a applied to discrete measures).

\[\operatorname{H}(p,q)\stackrel{{\text{def}}}{{=}}- \sum_{{\mathbf{x}}\in\Sigma^{*}}p({\mathbf{x}})\log q({\mathbf{x}})\] (34a) \[=-\sum_{{\mathbf{x}}\in\Sigma^{*}}p({\mathbf{x}})\left[\log\left(q( \textsc{eos}~{}|~{}{\mathbf{x}})\prod_{t=1}^{T}q(x_{t}~{}|~{}{\mathbf{x}}_{<t})\right)\right]\] (34b) \[=-\sum_{{\mathbf{x}}\in\Sigma^{*}}\sum_{T=0}^{\infty}p(T)p({\mathbf{x}}~{ }|~{}T)\left[\log\left(q(\textsc{eos}~{}|~{}{\mathbf{x}})\prod_{t=1}^{T}q(x_{t}~{}| ~{}{\mathbf{x}}_{<t})\right)\right]\] (34c) \[=-\sum_{T=0}^{\infty}p(T)\sum_{{\mathbf{x}}\in\Sigma^{*}}p({\mathbf{x}}~{ }|~{}T)\left[\log\left(q(\textsc{eos}~{}|~{}{\mathbf{x}})\prod_{t=1}^{T}q(x_{t}~{}| ~{}{\mathbf{x}}_{<t})\right)\right]\] (34d) \[=-\sum_{T=0}^{\infty}p(T)\Bigg{[}\sum_{{\mathbf{x}}\in\Sigma^{*}}p({ \mathbf{x}}~{}|~{}T)\log q(\textsc{eos}~{}|~{}{\mathbf{x}})\] (34e) \[\qquad\qquad\quad+\sum_{t=1}^{T}\sum_{{\mathbf{x}}\in\Sigma^{*}}p({ \mathbf{x}}~{}|~{}T)\log q(x_{t}~{}|~{}{\mathbf{x}}_{<t})\Bigg{]}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~ 

\[=-\sum_{T=0}^{\infty}p(T)\left[\sum_{\mathbf{x}\in\Sigma^{*}}p(\mathbf{x} \mid T)\log q(\textsc{Eos}\mid\mathbf{x})+\sum_{x\in\Sigma}\sum_{t=1}^{T}\sum_{\mathbf{ x}<t\in\Sigma^{t-1}}\log q(x\mid\mathbf{x}_{<t})\pi_{T}(\mathbf{x}_{<t}x)\right] \tag{34j}\] \[=-\sum_{T=0}^{\infty}p(T)\left[\sum_{\mathbf{x}\in\Sigma^{*}}p(\mathbf{x} \mid T)\log q(\textsc{Eos}\mid\mathbf{x})+\sum_{x\in\Sigma}\sum_{\mathbf{x}\in\Sigma^{C }}\log q(x\mid\mathbf{x})\pi_{T}(\mathbf{x}x)\right]\] (34k) \[=-\sum_{T=0}^{\infty}p(T)\left[\sum_{\mathbf{x}\in\Sigma^{*}}p(\mathbf{x} \mid T)\log q(\textsc{Eos}\mid\mathbf{x})+\sum_{x\in\Sigma}\sum_{\mathbf{x}\in\Sigma^{* }}\log q(x\mid\mathbf{x})\pi_{T}(\mathbf{x}x)\right]\] (34l) \[=-\sum_{\mathbf{x}\in\Sigma^{*}}\sum_{T=0}^{\infty}p(T)p(\mathbf{x} \mid T)\log q(\textsc{Eos}\mid\mathbf{x})+\sum_{x\in\Sigma}\sum_{\mathbf{x}\in\Sigma^{ *}}\sum_{T=0}^{\infty}p(T)\pi_{T}(\mathbf{x}x)\log q(x\mid\mathbf{x})\] (34m) \[=-\sum_{\mathbf{x}\in\Sigma^{*}}p(\mathbf{x})\log q(\textsc{Eos}\mid\mathbf{ x})-\sum_{x\in\Sigma}\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x}x)\log q(x\mid\mathbf{x})\] (34n) \[=-\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})p(\textsc{Eos}\mid\mathbf{x}) \log q(\textsc{Eos}\mid\mathbf{x})-\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})\sum_{x \in\Sigma}p(x\mid\mathbf{x})\log q(x\mid\mathbf{x})\] (34o) \[=-\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})\sum_{x\in\bar{\Sigma}}p(x \mid\mathbf{x})\log q(x\mid\mathbf{x})\] (34p) \[=\sum_{\mathbf{x}\in\Xi^{*}}\pi(\mathbf{x})\mathrm{H}\left(p(\cdot\mid \mathbf{x}),q(\cdot\mid\mathbf{x})\right). \tag{34q}\]

Now, we substitute Eq. (34q) into the following equation

\[D_{\rm KL}(p\mid\mid q)=\sum_{\mathbf{x}\in\Sigma^{*}}p(\mathbf{x})\log \left(\frac{p(\mathbf{x})}{q(\mathbf{x})}\right) \tag{35a}\] \[=\mathrm{H}(p,q)-\mathrm{H}(p)\] (35b) \[=\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})\mathrm{H}\left(p(\cdot \mid\mathbf{x}),q(\cdot\mid\mathbf{x})\right)-\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x}) \mathrm{H}\left(p(\cdot\mid\mathbf{x})\right)\] (35c) \[=\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})\left(-\sum_{x\in\overline{ \Sigma}}p(x\mid\mathbf{x})\log q(x\mid\mathbf{x})\right)-\sum_{\mathbf{x}\in\Sigma^{*}} \pi(\mathbf{x})\left(-\sum_{x\in\overline{\Sigma}}p(x\mid\mathbf{x})\log p(x\mid\mathbf{x })\right)\] (35d) \[=\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})\left(\sum_{x\in\overline{ \Sigma}}p(x\mid\mathbf{x})\log\frac{p(x\mid\mathbf{x})}{q(x\mid\mathbf{x})}\right)\] (35e) \[=\sum_{\mathbf{x}\in\Sigma^{*}}\pi(\mathbf{x})D_{\rm KL}(p(\cdot\mid\mathbf{x })\mid||\;q(\cdot\mid\mathbf{x})). \tag{35f}\]

Note that because we have assumed that \(\mathrm{H}(p,q)<\infty\) and \(\mathrm{H}(p)\leq\mathrm{H}(p,q)\), we have that both additive terms in the KL divergence are finite. This is sufficient to avoid \(\infty-\infty\) and the unpleasantries that follow. 

### Smoothing and Regularization

**Corollary 2.3**.: _Let \(p_{\mathcal{D}}\) be an empirical distribution induced by a dataset \(\mathcal{D}\). Let \(q\) be an \(n\)-gram language model. Then, it holds that:_

\[D_{\rm KL}\left(p_{\mathcal{D}}\mid\mid q\right) \tag{10}\] \[\propto\!\!\sum_{\mathbf{x}^{n}\in\Sigma_{\rm ROS}^{n-1}}\!\!\#(\mathbf{x} ^{n})D_{\rm KL}\left(p_{\mathcal{D}}(\cdot\mid\mathbf{x}^{n})\mid\mid q(\cdot\mid \mathbf{x}^{n})\right).\]

Proof.: Then, we have

\[D_{\rm KL}\left(p_{\mathcal{D}}\mid\mid q_{\rm MLE}^{n}\right)=\sum_{\mathbf{x}\in \Sigma^{*}}\pi_{\mathcal{D}}(\mathbf{x})D_{\rm KL}\left(p_{\mathcal{D}}(\cdot\mid \mathbf{x})\mid\mid q(\cdot\mid\mathbf{x})\right)\] (Theorem 2.2 ) 

\[= \sum_{\mathbf{x}^{n}\in\Sigma_{\text{pos}}^{n-1}}\sum_{x\in\overline{ \Sigma}}\left[\#(\mathbf{x}^{n})p_{\mathcal{D}}^{n}\,(x\mid\mathbf{x}^{n})\leq q\,(x \mid\mathbf{x}^{n})\right]+\text{const.} \tag{38a}\] \[= \sum_{\mathbf{x}^{n}\in\Sigma_{\text{pos}}^{n-1}}\sum_{x\in\overline{ \Sigma}}\left[\#(\mathbf{x}^{n})p_{\mathcal{D}}^{n}\,(x\mid\mathbf{x}^{n})+\gamma u\,( x\mid\mathbf{x}^{n})\log q\,(x\mid\mathbf{x}^{n})\right]+\text{const.}\] (38b) \[= \sum_{\mathbf{x}^{n}\in\Sigma_{\text{pos}}^{n-1}}\sum_{x\in\overline{ \Sigma}}\left[\#(\mathbf{x}^{n})\frac{\#(\mathbf{x}^{n}x)}{\#(\mathbf{x}^{n})}+\frac{ \gamma}{|\Sigma|+1}\right]\log q\,(x\mid\mathbf{x}^{n})+\text{const.}\] (38c) \[= \sum_{\mathbf{x}^{n}\in\Sigma_{\text{pos}}^{n-1}}\sum_{x\in\overline{ \Sigma}}\left[\#(\mathbf{x}^{n})p_{\mathcal{D}}^{n}\,(x\mid\mathbf{x}^{n})+\gamma u\, (x\mid\mathbf{x}^{n})\right]\log q\,(x\mid\mathbf{x}^{n})+\text{const.}\] (38d) \[= \sum_{\mathbf{x}^{n}\in\Sigma_{\text{pos}}^{n-1}}\sum_{x\in\overline{ \Sigma}}\left[\#(\mathbf{x}^{n})\frac{\#(\mathbf{x}^{n}x)}{\#(\mathbf{x}^{n})}+\frac{ \gamma}{|\Sigma|+1}\right]\log q\,(x\mid\mathbf{x}^{n})+\text{const.}\] (38e) \[= \sum_{\mathbf{x}^{n}\in\Sigma_{\text{pos}}^{n-1}}\sum_{x\in\overline{ \Sigma}}\left[\#(\mathbf{x}^{n})p_{\mathcal{D}}^{n}\,(x\mid\mathbf{x}^{n})+\gamma u\, (x\mid\mathbf{x}^{n})\right]\log q\,(x\mid\mathbf{x}^{n})+\text{const.}\] (38f) \[= \sum_{\mathbf{x}^{n}\in\Sigma_{\text{pos}}^{n-1}}\sum_{x\in\overline{ \Sigma}}\left[\#(\mathbf{x}^{n})\frac{\#(\mathbf{x}^{n}x)}{\#(\mathbf{x}^{n})}+\frac{ \gamma}{|\Sigma|+1}\right]\log q\,(x\mid\mathbf{x}^{n})+\text{const.}\] (38g) \[= \sum_{\mathbf{x}^{n}\in\Sigma_{\text{pos}}^{n-1}}\sum_{x\in\overline{ \Sigma}}\left[\#(\mathbf{x}^{n}x)+\frac{\gamma}{|\Sigma|+1}\right]\log q\,(x\mid \mathbf{x}^{n})+\text{const.}, \tag{38h}\]

where the constant terms are independent of \(q\). Next, note that we can optimize each \(q\,(x\mid\mathbf{x}^{n})\) independently, i.e., we can find the distribution \(q\,(x\mid\mathbf{x}^{n})\) that minimizes the following expression

\[\left[\#(\mathbf{x}^{n}x)+\frac{\gamma}{|\Sigma|+1}\right]\log q\,(x\mid\mathbf{x}^{ n})\,,\] (39) 

under the constraint that \(\sum_{x\in\overline{\Sigma}}q(x~{}|~{}\mathbf{x}^{n})=1\) and \(q(x~{}|~{}\mathbf{x}^{n})\geq 0\), \(\forall x\in\overline{\Sigma}\) independently. It is a standard result that the minimizing \(q\left(\cdot~{}|~{}\mathbf{x}^{n}\right)\) for any \(\mathbf{x}^{n}\in\Sigma_{\text{hos}}^{n-1}\) is given by

\[\tilde{q}_{\mathrm{MLE}}^{n}\left(\cdot~{}|~{}\mathbf{x}^{n}\right)=\frac{\#(\mathbf{x }^{n}x)+\frac{\gamma}{|\Sigma|+1}}{\#(\mathbf{x}^{n})+\gamma}\propto\bigg{[}\#( \mathbf{x}^{n}x)+\frac{\gamma}{|\Sigma|+1}\bigg{]}, \tag{40}\]

in which we recognize the \(\lambda=\frac{\gamma}{|\Sigma|+1}\) add-\(\lambda\) smoothed maximum-likelihood solution \(\tilde{q}_{\mathrm{MLE}}^{n}\) from Eq. (37). 

**Theorem 4.1**.: _Let \(p_{\mathcal{D}}\) be the empirical distribution induced by the dataset \(\mathcal{D}\) and \(\tilde{p}_{\mathcal{D}}^{n}\) a smoothed empirical \(n\)-gram distribution. For \(\gamma=1\), the following holds_

\[\begin{split} D_{\mathrm{KL}}(\tilde{p}_{\mathcal{D}}^{n}& ||~{}q_{\mathbf{\theta}})\\ &=D_{\mathrm{KL}}(p_{\mathcal{D}}~{}||~{}q_{\mathbf{\theta}})+\gamma \mathcal{R}(\mathbf{\theta})+C,\end{split} \tag{27}\]

_where \(C\) is constant with respect to \(q_{\mathbf{\theta}}\)._

Proof.: The definitions of \(p_{+},p_{-},Z_{+},Z_{-}\) from Eq. (23a), (23b), (24a) and (25), respectively, results in the following simple decomposition:

\[\tilde{p}_{\mathcal{D}}^{n}(\mathbf{x})=p_{\mathcal{D}}(\mathbf{x})+Z_{+} p_{+}(\mathbf{x})-Z_{-}p_{-}(\mathbf{x}). \tag{41}\]

Then, we proceed with some basic manipulations

\[D_{\mathrm{KL}}(\tilde{p}_{\mathcal{D}}^{n}~{}||~{}q_{\mathbf{\theta }})\stackrel{{\mbox{\tiny def}}}{{=}}\operatorname*{H}(\tilde{p} _{\mathcal{D}}^{n},q_{\mathbf{\theta}})-\underbrace{\operatorname*{H}(\tilde{p}_{ \mathcal{D}}^{n})}_{\stackrel{{\mbox{\tiny def}}}{{\llleftarrow}} C} \tag{42a}\] \[=\operatorname*{H}(\tilde{p}_{\mathcal{D}}^{n},q_{\mathbf{\theta}})+C \quad\text{(independence of H}(\tilde{p}_{\mathcal{D}}^{n})\text{ with respect to $q_{\mathbf{\theta}}$)}\] (42b) \[=\operatorname*{H}(p_{\mathcal{D}}+Z_{+}p_{+}(\mathbf{x})+Z_{-}p_{-} (\mathbf{x}),q_{\mathbf{\theta}})+C\quad\text{(definitions of $p_{+},p_{-},Z_{+},Z_{-}$)}\] (42c) \[=\operatorname*{H}(p_{\mathcal{D}},q_{\mathbf{\theta}})+Z_{+} \operatorname*{H}(p_{+},q_{\mathbf{\theta}})+Z_{-}\operatorname*{H}(p_{-},q_{\mathbf{ \theta}})+C\quad\text{(linearity of cross-entropy)}\] (42d) \[=D_{\mathrm{KL}}(p_{\mathcal{D}}~{}||~{}q_{\mathbf{\theta}})+ \underbrace{Z_{+}D_{\mathrm{KL}}(p_{+}~{}||~{}q_{\mathbf{\theta}})+Z_{-}D_{ \mathrm{KL}}(p_{-}~{}||~{}q_{\mathbf{\theta}})}_{\stackrel{{\mbox{ \tiny def}}}{{\llleftarrow}}\mathcal{R}(q_{\mathbf{\theta}})}+C\quad\text{( definition of $\mathcal{R}$)}\] (42e) \[=D_{\mathrm{KL}}(p_{\mathcal{D}}~{}||~{}q_{\mathbf{\theta}})+ \underbrace{\gamma}_{=1}\mathcal{R}(q_{\mathbf{\theta}})+C. \tag{42f}\]

This proves the result. 

## Appendix B Experimental Details

Experiments on WikiText-2 and IWSLT-14 were run on a shared cluster on NVIDIA Quadro RTX 6000 GPUs. The Transformer models used for language modeling and machine translation have \(58,\!145,\!792\) and \(39,\!469,\!056\) parameters, respectively 



## Appendix C Additional Experimental Results

Tab. 5 and Tab. 6 show the performance of the best-performing seed for each model. JM smoothing remains the best-performing technique on both datasets. For language modeling, we test for statistical significance using a paired permutation test over sentence-level log-likelihoods, using the mean of the observation differences as the test statistic. For machine translation, we test for statistical significance using paired bootstrap resampling as implemented in sacreBLEU. Different tests are performed using either the unregularized results or the add-\(\lambda\) results as the baseline. In Tab. 5 and Tab. 6, the first symbol in \({\dagger}/{\dagger}\) refers to statistical significance with respect to the unregularized model, while the second refers to statistical significance over the add-\(\lambda\) results. \({\dagger}\) indicates \(p<0.05\); \({\ddagger}\) indicates \(p<0.01\). In language modeling, all regularization methods perform significantly better than no regularization. However, only GT and JM smoothing perform better than add-\(\lambda\) smoothing. For machine translation, we see that all regularized methods except for Katz smoothing perform significantly better than the unregularized baseline model, while only JM smoothing performs significantly better than add-\(\lambda\) smoothing.

### WMT-14 results

To evaluate the performance of our regularizers on larger datasets, we perform a preliminary evaluation of our methods on the WMT-14 machine translation dataset. We download and preprocess the data following a script provided by fairseq.22 Experiments on WMT were run on two NVIDIA Tesla V100 GPUs. Given our computational constraints, we limit our experiments to our best-performing smoothing method (Jelinek-Mercer), add-\(\lambda\) smoothing, and the unregularized baseline. We also use the same hyperparameter

\begin{table}
\begin{tabular}{l|c} \hline \hline Smoothing Method & BLEU \\ \hline None & \(33.11\) \\ add-\(\lambda\) (\(\gamma_{+}=0.1,\gamma_{-}=0.01,\gamma_{\text{LS}}=0.01\)) & \(33.41^{\ddagger}\) \\ \hline GT (\(\gamma_{+}=0.05,\gamma_{-}=0.5\)) & \(33.44^{\ddagger}\) \\ JM (\(\gamma_{+}=0.1,\gamma_{-}=0.5,\lambda_{1}=0.5\)) & \(33.97^{\ddagger/\ddagger}\) \\ Katz (\(\gamma_{+}=0.1,\gamma_{-}=0.1,k=5\)) & \(33.31\) \\ KEN (\(\gamma_{+}=0.1,\gamma_{-}=0.1\)) & \(33.58^{\ddagger}\) \\ \hline \end{tabular}
\end{table}
Table 6: BLEU on test set of IWSLT-14 DE-EN. Different regularized methods are compared to no regularization (None). We report the best BLEU score over \(5\) independently trained models.

\begin{table}
\begin{tabular}{l|l|l|l|l|l} \hline \hline Dataset & Split & Language & Vocabulary size & Samples & Number of tokens \\ \hline WikiText-2 & Train & English & \(16{,}932\) & \(23{,}767\) & \(2{,}389{,}674\) \\ WikiText-2 & Validation & English & \(16{,}932\) & \(2461\) & \(255{,}327\) \\ WikiText-2 & Test & English & \(16{,}932\) & \(2891\) & \(292{,}710\) \\ \hline IWSLT-14 & Train & English & \(6628\) & \(160{,}239\) & \(3{,}788{,}875\) \\ IWSLT-14 & Validation & English & \(6628\) & \(7283\) & \(171{,}339\) \\ IWSLT-14 & Test & English & \(6628\) & \(6750\) & \(150{,}178\) \\ \hline IWSLT-14 & Train & German & \(8844\) & \(160{,}239\) & \(3{,}875{,}352\) \\ IWSLT-14 & Validation & German & \(8844\) & \(7283\) & \(175{,}309\) \\ IWSLT-14 & Test & German & \(8844\) & \(6750\) & \(155{,}088\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Dataset details

\begin{table}
\begin{tabular}{l|l} \hline \hline Smoothing Method & ppl \(\downarrow\) \\ \hline None & \(144.67\) \\ add-\(\lambda\) (\(\gamma_{+}=0.1,\gamma_{-}=0.05,\gamma_{\text{LS}}=0.01\)) & \(138.72^{\ddagger}\) \\ \hline GT (\(\gamma_{+}=0.1,\gamma_{-}=0.05\)) & \(137.00^{\ddagger/\ddagger}\) \\ JM (\(\gamma_{+}=0.1,\gamma_{-}=0.5,\lambda_{1}=0.75\)) & \(134.63^{\ddagger/\ddagger}\) \\ Katz (\(\gamma_{+}=0.1,\gamma_{-}=0.01,k=5\)) & \(139.44^{\ddagger}\) \\ KEN (\(\gamma_{+}=0.1,\gamma_{-}=0.1\)) & \(140.71^{\ddagger}\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Perplexity on WikiText-2 test set. Included are performances of models trained with no regularization (None), and with various smoothing methods. We report the best perplexity over \(5\) independently trained models.



set as in the IWSLT-14 experiments for all methods. Tab. 7 shows the results of our experiments. JM smoothing obtains the best performance out of the three methods. Using paired bootstrap resampling, we assessed that the improvements of JM smoothing are statistically significant over the unregularized baseline, while they are not significant over the results obtained using add-\(\lambda\) smoothing (\(p=0.07\)). A regularization hyperparameter search on the new dataset might yield larger performance improvements for all smoothing-based methods.