Venue: ACL 2020

# Generalized Entropy Regularization or:

There's Nothing Special about Label Smoothing

Clara Meister\({}^{\text{\textcircled{T}}}\) Elizabeth Salesky\({}^{\text{\textcircled{S}}}\) Ryan Cotterell\({}^{\text{\textcircled{O,$\Sigma$}}}\)

\({}^{\text{\textcircled{I}}}\)ETH Zurich Johns Hopkins University University University of Cambridge

clara.meister@inf.ethz.ch esalesky@jhu.edu ryan.cotterell@inf.ethz.ch

###### Abstract

Prior work has explored directly regularizing the output distributions of probabilistic models to alleviate peaky (i.e. over-confident) predictions, a common sign of overfitting. This class of techniques, of which label smoothing is one, has a connection to entropy regularization. Despite the consistent success of label smoothing across architectures and datasets in language generation tasks, two problems remain open: (1) there is little understanding of the underlying effects entropy regularizers have on models, and (2) the full space of entropy regularization techniques is largely unexplored. We introduce a parametric family of entropy regularizers, which includes label smoothing as a special case, and use it to gain a better understanding of the relationship between the entropy of a trained model and its performance on language generation tasks. We also find that variance in model performance can be explained largely by the resulting entropy of the model. Lastly, we find that label smoothing provably does not allow for sparse distributions, an undesirable property for language generation models, and therefore advise the use of other entropy regularization methods in its place. Our code is available online at [https://github.com/rycolab/entropyRegularization](https://github.com/rycolab/entropyRegularization).

+
Footnote †: July 5 - 10, 2020. ©2020 Association for Computational Linguistics

## 1 Introduction

When training large neural networks with millions of parameters, regularization of some form is needed to prevent overfitting, even when large amounts of data are used; models for language generation are no exception. In probabilistic modeling, e.g. when the final layer of the neural network is a softmax, overfitting often manifests itself in overconfident placement of most of the probability mass on a few candidates, resulting in peaky (low-entropy) probability distributions over the vocabulary. Specifically for language generation tasks, this behavior leads to the output of repetitive or frequently occurring but unrelated text, which is detrimental to the generalization abilities of the model Chorowski and Jaitly (2017); Holtzman et al. (2020). A natural regularizer to consider is, therefore, one that penalizes overconfidence, encouraging higher entropy in the learned distribution. Indeed, the literature has ascribed gains of \(\approx 1\) bleu point in machine translation to label smoothing, one such technique Chen et al. (2018).

Despite the clear relationship between low entropy and overfitting, only a handful of distinct entropy regularizers have been explored. To fill this gap, we introduce **generalized entropy regularization** (GER), a unified framework for understanding and exploring a broad range of entropy-inducing regularizers. GER is based on the skew-Jensen family of divergences J\({}_{\alpha,G}\)Nielsen and Boltz (2011) and thus may be generalized to any Bregman divergence through the choice of generator function \(G\). For the negative entropy generator function, GER recovers label smoothing Szegedy et al. (2015) as \(\alpha\to 1\), and the confidence penalty Pereyra et al. (2017) as \(\alpha\to 0\). We provide formal properties of GER in SS3, proving these special-case equivalences among other characteristics of GER. We then use GER to examine the relationship between entropy and the evaluation metrics in two language generation tasks: neural machine translation (NMT) and abstractive summarization.

GER encompasses a large family of regularizers, which allows us to directly compare label smoothing to other forms of entropy regularization. By studying the relationship between different regularizers on the performance of natural language generation (NLG) systems, we can better understand not just _when_ but also _why_ label smoothing aids language generation tasks. Through our analysis, we gain the following insights:

* With tuning of the regularizer's coefficient, _any_ choice of \(\alpha\) can yield similar performance, i.e. there is nothing special about label smoothing. In fact, our results suggest that la

bel smoothing (\(\alpha\to 1\)) makes it more difficult to tune the regularizer's coefficient.
2. Label smoothing assigns infinite cost to sparse output distributions, which may be an undesirable behavior for language generation tasks.
3. There is a strong (quadratic) relationship between a model's performance on the evaluation metric and its (average) entropy, offering a hint as to why these regularizers are so effective for NLG.

In summary, entropy-inducing regularizers are a boon to probabilistic NLG systems, which benefit from higher entropy output distributions. Label smoothing works _because_ it forces the model towards a higher-entropy solution, but we recommend the confidence penalty and other entropy regularizers (\(\alpha<1\)) for reasons (i) and (ii) above.

## 2 Preliminaries

In this work, we consider conditional probability models \(p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\) for natural language generation; such models assign probability to a target sequence \(\mathbf{y}\in\mathcal{Y}\) given a source sequence \(\mathbf{x}\). Specifically, our target sequence \(\mathbf{y}=\langle y_{1},\ldots,y_{n}\rangle\) of arbitrary length \(n\) is a sequence of target words1\(y_{i}\) from our vocabulary \(Y\). The set of all complete target sequences, which are padded with distinguished beginning- and end-of-sentence symbols, bos and eos, is then defined as \(\mathcal{Y}:=\{\texttt{bos}\circ\mathsf{y}\circ\texttt{eos}~{}|~{}\mathbf{y} \in Y^{*}\}\). For language generation tasks, \(p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\) is typically a neural network with parameters \(\boldsymbol{\theta}\); this network is often trained to approximate \(\tilde{p}(\mathbf{y}~{}|~{}\mathbf{x})\), the empirical distribution (i.e. the distribution of the data). Here, we focus on locally normalized models; in such models \(p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\) is factored as:

Footnote 1: Targets \(y_{i}\) may also be characters or subwords; our experiments use byte-pair encoding Sennrich et al. (2016)

\[p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})=p_{\theta}(y_{1}~{}|~{}\mathbf{x}) \cdots p_{\theta}(y_{n}~{}|~{}\mathbf{x},\mathbf{y}_{<n}) \tag{1}\]

where \(p_{\theta}(y_{i}~{}|~{}\mathbf{x},\mathbf{y}_{<i})\) is defined by a softmax over the output of the final fully connected layer of the network. Generation is performed using greedy search, beam search or a sampling scheme. Of the candidate sequences generated, the one with the highest probability under the model \(p_{\theta}\) is returned as the model's prediction.

One way of selecting the parameters \(\boldsymbol{\theta}\) is to minimize the KL-divergence between the empirical distribution and the model. This yields the cross-entropy loss (plus an additive constant):2

Footnote 2: \(\mathrm{H}(p,q):=-\sum_{z\in\mathcal{Z}}p(z)\log q(z)\) is cross-entropy and \(\mathrm{H}(p):=\mathrm{H}(p,p)=-\sum_{z\in\mathcal{Z}}p(z)\log p(z)\) is the Shannon entropy, for which \(\log=\log_{2}\) and \(\mathcal{Z}=\operatorname{supp}(p)\).

\[\mathcal{L}(\boldsymbol{\theta}) =\mathrm{KL}(\tilde{p}~{}||~{}p_{\theta}) \tag{2}\] \[=\underbrace{\mathrm{H}(\tilde{p},p_{\theta})}_{\text{cross-entropy loss}}~{}- \underbrace{\mathrm{H}(\tilde{p})}_{\text{constant w.r.t. $\theta$}} \tag{3}\]

However, fitting a model that perfectly approximates the empirical distribution is, in general, fraught with problems Hastie et al. (2001). The goal of learning is to generalize beyond the observed data. Exactly fitting the empirical distribution, often termed overfitting, is therefore not an ideal goal and for language generation models specifically, does not go hand-in-hand with the ability of a model to generate desirable text Bengio et al. (2015). Consequently, it is advisable to minimize a regularized objective to prevent overfitting:

\[\mathcal{L}(\boldsymbol{\theta})+\beta\,\mathcal{R}(\boldsymbol{\theta}) \tag{4}\]

where \(\mathcal{R}(\boldsymbol{\theta})\) is a regularizer defined over the model with "strength" coefficient \(\beta>0\).

### Entropy Regularization

Overfitting can manifest itself as peakiness in \(p_{\theta}\)Williams and Peng (1991); Mnih et al. (2016); Pereyra et al. (2017). In other words, \(p_{\theta}\) overconfidently places most of the probability mass on very few candidates. While this overconfidence improves training loss, it hurts generalization. Entropy regularization is one technique that directly combats such overconfidence by encouraging more entropic (less peaky) distributions.

The entropy of the model \(p_{\theta}\) is defined as

\[\mathrm{H}\left(p_{\theta}\right):=-\sum_{\mathbf{y}\in\mathcal{Y}}p_{\theta }(\mathbf{y})\log p_{\theta}(\mathbf{y}) \tag{5}\]

where we remove dependence on \(\mathbf{x}\) for notational simplicity. However, the sum in eq. (5) over \(\mathcal{Y}\) generally renders its computation intractable.3 Instead, regularization is performed on the conditional distribution over \(Y\cup\{\texttt{eos}\}\) at each time step, which can be interpreted as an approximation of the true model entropy. For ease of notation, we define a higher-order function \(D_{f}\) over our training corpus \(\mathcal{C}\) consisting of \(\langle\mathbf{x},\mathbf{y}\rangle\) pairs that maps a function 

over distributions \(p,q\) as follows below:

\[D_{f}(p~{}||~{}q)= \tag{6}\] \[\sum_{(\mathbf{x},\mathbf{y})\in\mathcal{C}}\sum_{t=1}^{|\mathbf{ y}|}f(p(\cdot~{}|~{}\mathbf{x},\mathbf{y}_{<t})~{}||~{}q(\cdot~{}|~{}\mathbf{ x},\mathbf{y}_{<t}))\]

The function \(D_{f}\) allows us to describe in notation how entropy regularization is typically employed in the training of language generation systems.4

Footnote 4: Note that the standard loss function in eq.3 can be written in this form when computed over \(\mathcal{C}\), i.e. \(\mathrm{KL}(\tilde{p}~{}||~{}p_{\theta})=D_{\mathrm{KL}}(\tilde{p}~{}||~{}p_{ \theta})\), since the reference \(\mathbf{y}\) is the only value in \(\operatorname*{supp}(\tilde{p})\).

Label Smoothing.Label smoothing, first introduced as a regularizer for neural networks by Szegedy et al. (2015), is so named because the technique smooths hard target distributions. One such distribution, the empirical distribution, is encoded as a set of one-hot vectors (hard targets) where for each data point, the correct label (e.g., vocabulary index of a word) has value \(1\) and all other labels have value \(0\). Label smoothing with strength coefficient \(\gamma\) is an add-\(\gamma\) smoothing scheme on the distribution over labels at every time step. Interestingly, minimizing the cross entropy between this modified distribution and the model \(p_{\theta}\) is equivalent to adding the weighted KL divergence between the uniform distribution and the model \(p_{\theta}\) in our original objective function with the same strength coefficient:

\[\mathcal{L}(\boldsymbol{\theta})^{LS}_{\gamma}:=(1-\gamma)\mathcal{L}( \boldsymbol{\theta})+\gamma\,D_{\mathrm{KL}}(u~{}||~{}p_{\theta}) \tag{7}\]

While the loss function is often scaled as above, it is nonetheless equivalent to \(\mathcal{L}(\boldsymbol{\theta})^{LS}_{\beta}=\mathcal{L}(\boldsymbol{\theta })+\beta\,D_{\mathrm{KL}}(u~{}||~{}p_{\theta})\);5 we use this form for consistency.

Footnote 5: up to multiplicative factor \((1-\gamma)\) when \(\beta=\gamma/(1-\gamma)\)

Confidence Penalty.The confidence penalty, empirically explored in the supervised learning setting by Pereyra et al. (2017), aims to penalize a low-entropy model. This is done by subtracting a weighted term for the entropy of the model's prediction \(p_{\theta}(\cdot)\) from the loss function, thereby encouraging a more entropic model. This is equivalent to adding the KL divergence between the model \(p_{\theta}\) and the uniform distribution:

\[\mathcal{L}(\boldsymbol{\theta})^{CP}_{\beta}:=\mathcal{L}(\boldsymbol{\theta })+\beta\,D_{\mathrm{KL}}(p_{\theta}~{}||~{}u) \tag{8}\]

While Pereyra et al. (2017) found that label smoothing performed better than the confidence penalty for NMT, they only searched coarsely over a small range of \(\beta\)'s for both regularizers. Our findings in SS4 suggest an alternate conclusion.

## 3 Generalized Entropy Regularization

The positive effect of both label smoothing and the confidence penalty on model performance in language generation tasks motivates further exploration of entropy-promoting regularizers. To this end, we construct a parameterized family of regularizers with label smoothing and the confidence penalty as special cases. We discuss the formal properties of a subset of this family, providing upper and lower bounds for it. We show divergence only occurs in one case for this subset (\(\alpha\to 1\)), which directly implies that no sparse solution exists when label smoothing is used as a regularizer.

### A Family of Entropy Regularizers

We derive a family of regularizers from the skew-Jensen divergence J\({}_{\alpha,G}\)Nielsen and Boltz (2011), which is defined below as:

\[\text{J}_{\alpha,G}(q~{}||~{}p_{\theta}):= \frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha)G(q)+\alpha G(p_{ \theta})\] \[-G((1-\alpha)q+\alpha p_{\theta})\Big{)} \tag{9}\]

for a strictly convex generator function \(G:\Omega\rightarrow\mathbb{R}\) and \(\alpha\in(0,1)\) where \(\Omega\) is a closed convex set. In this paper, we restrict \(\Omega\) to be the \((|Y|+1)\)-simplex. Note that \(\text{J}_{\alpha,G}(q~{}||~{}p_{\theta})\neq\text{J}_{\alpha,G}(p_{\theta}~{}|| ~{}q)\) in general, although this is true for some choices of \(G\) and \(\alpha\).

We define the generalized entropy regularizer as \(\mathcal{R}(\boldsymbol{\theta})=D_{\text{J}_{\alpha,G}}(u~{}||~{}p_{\theta})\) where \(u\) is the uniform

\begin{table}
\begin{tabular}{l c l} \hline
**Training Method** & **Loss Function** & **Alternate Formulation** \\ \hline Cross Entropy & \(\mathcal{L}(\boldsymbol{\theta})=\mathrm{H}(\tilde{p},p_{\theta})\) & \(=\mathrm{KL}(\tilde{p}~{}||~{}p_{\theta})+\mathrm{H}(\tilde{p})\) \\ Confidence Penalty, \(D_{\text{J}_{0}}\) & \(\mathcal{L}_{\mathcal{CP}}(\boldsymbol{\theta})=\mathcal{L}(\boldsymbol{ \theta})+\beta\,D_{\mathrm{KL}}(p_{\theta}~{}||~{}u)\) & \(=\mathcal{L}(\boldsymbol{\theta})-\beta\,D_{\mathrm{H}}(p_{\theta})+C\) \\ Label Smoothing, \(D_{\text{J}_{1}}\) & \(\mathcal{L}_{\textit{IS}}(\boldsymbol{\theta})=\mathcal{L}(\boldsymbol{\theta })+\beta\,D_{\mathrm{KL}}(u~{}||~{}p_{\theta})\) & \(=\mathcal{L}(\boldsymbol{\theta})+\beta\,D_{\mathrm{H}}(u,p_{\theta})+C\) \\ Generalized Entropy Regularization, \(D_{\text{J}_{\alpha}}\) & \(\mathcal{L}_{\textit{GEER}}(\boldsymbol{\theta})=\mathcal{L}(\boldsymbol{ \theta})+\beta\,D_{\text{J}_{a}}(u~{}||~{}p_{\theta})\) & — \\ \hline \end{tabular}
\end{table}
Table 1: Loss functions and their alternate formulations for different training methods; the latter three are entropy regularization techniques that augment the standard loss function in row 1. \(C\) denotes a constant with respect to \(\boldsymbol{\theta}\).

 

distribution.6 These regularizers promote entropy because they push the model \(p_{\mathfrak{e}}\) towards \(u\), which is the maximum-entropy distribution with an entropy of \(\log(|Y|+1)\). Throughout the rest of this paper, we primarily use the generator function7\(G(p)=-\mathrm{H}(p)\). We use \(\mathrm{J}_{\alpha}\) as shorthand for \(\mathrm{J}_{\alpha,-\mathrm{H}}\).

Footnote 6: Distributions other than \(u\) may also be used. See §5.

We note \(\mathrm{J}_{\alpha}\) is equivalent to quadruple the Jensen-Shannon (JS) divergence and asymptotically approaches the Kullback-Leibler (KL) divergence for certain values of \(\alpha\). Specifically, we have:

\[\lim_{\alpha\to 0}\mathrm{J}_{\alpha}(q~{}||~{}p_{\mathfrak{e}}) =\operatorname{KL}(p_{\mathfrak{e}}~{}||~{}q) \tag{10}\] \[\lim_{\alpha\to 1}\mathrm{J}_{\alpha}(q~{}||~{}p_{\mathfrak{ e}}) =\operatorname{KL}(q~{}||~{}p_{\mathfrak{e}})\] (11) \[\quad\mathrm{J}_{1/2}(q~{}||~{}p_{\mathfrak{e}}) =4\cdot\mathrm{JS}(q~{}||~{}p_{\mathfrak{e}}) \tag{12}\]

We prove these relationships in App. A and App. B. For ease, we define \(\mathrm{J}_{1}:=\lim_{\alpha\to 1}\mathrm{J}_{\alpha}\) and \(\mathrm{J}_{0}:=\lim_{\alpha\to 0}\mathrm{J}_{\alpha}\). We note the following two equivalences for these special cases.

**Proposition 1**.: \(\nabla_{\boldsymbol{\theta}}\mathrm{J}_{1}(u~{}||~{}p_{\mathfrak{e}})=\nabla_ {\boldsymbol{\theta}}\mathrm{H}(q,p_{\mathfrak{e}})\)_. In words, the gradient of the loss with GER as \(\alpha{\rightarrow}1\) is equivalent to the gradient of the loss augmented with **label smoothing**._

**Proposition 2**.: \(\nabla_{\boldsymbol{\theta}}\mathrm{J}_{0}(u~{}||~{}p_{\mathfrak{e}})=\nabla_ {\boldsymbol{\theta}}\mathrm{H}(p_{\mathfrak{e}})\)_. In words, the gradient of the loss with GER as \(\alpha\to 0\) is equivalent to the gradient of the loss augmented with the **confidence penalty**._

See App. C and App. D for proofs.

### Formal Properties of \(\mathrm{J}_{\alpha}\)

When fitting a model \(p_{\mathfrak{e}}\), we generally optimize the inclusive \(\operatorname{KL}\), i.e. \(\operatorname{KL}(\tilde{p}~{}||~{}p_{\mathfrak{e}})\), so that, among other reasons, \(p_{\mathfrak{e}}\) has support everywhere that \(\tilde{p}\) has support. However, it is unclear what relationships we want to encourage between the model \(p_{\mathfrak{e}}\) and the uniform distribution \(u\) during regularization as complete support of \(u\) implies no word can ever have non-zero probability. Here we explore formal properties of \(\mathrm{J}_{\alpha}\) as a regularizer to gain insight into how, as a function of \(\alpha\), these regularizers affect the learned distribution.

Magnitude.Figure 1 shows the different divergence measures between \(u\) and \(p_{\mathfrak{e}}\). We see that \(\mathrm{J}_{1}=\operatorname{KL}(u~{}||~{}p_{\mathfrak{e}})\)**(label smoothing)** is much larger than \(\mathrm{J}_{0}=\operatorname{KL}(p_{\mathfrak{e}}~{}||~{}u)\)**(confidence penalty)** at values of \(p_{\mathfrak{e}}\) farther from \(u\). This indicates that \(\mathrm{J}_{1}\) would be a stronger regularizer than \(\mathrm{J}_{<1}\), i.e. penalize values of \(p_{\mathfrak{e}}\) far from \(u\) more heavily, given the same strength coefficient \(\beta\). Note that it is not always the case that \(\mathrm{J}_{<1}(u~{}||~{}p)\leq\mathrm{J}_{1}(u~{}||~{}p)\) for fixed \(p\). We can, however, bound \(\mathrm{J}_{\alpha}\) from above and below by other quantities.

**Proposition 3**.: _The divergence \(\mathrm{J}_{\alpha}(u~{}||~{}p)\) is not a monotonic function of \(\alpha\) for all distributions \(p\)._

A proof by counter example is shown in Figure 2.

**Proposition 4**.: _For fixed \(p\), \(\mathrm{J}_{\alpha}\) has bounds: \(0\leq\mathrm{J}_{\alpha}(u~{}||~{}p)\leq\operatorname{KL}(u~{}||~{}p)+ \operatorname{KL}(p~{}||~{}u)\)._

See App. E for a proof.

Sparsity.Sparsity is generally a desirable trait in probabilistic models; specifically for structured prediction, it leads to improvements in performance and interpretability Martins et al. (2011); Niculae 

et al., 2018). For example, Martins and Astudillo (2016) showed the benefits of using sparsemax, which induces sparsity in an output distribution or attention layer, for natural language inference tasks. There are also intuitive reasons for allowing \(p_{\theta}\) to be sparse. Part of modeling language generations tasks is learning when particular sequences _cannot_, or at least should not, occur (e.g. are grammatically or syntactically incorrect). In these cases, a model should be able to assign 0 probability mass to that sequence. However, there is no sparse optimal solution \(p_{\theta}\) when using label smoothing as the label smoothing loss function becomes divergent if \(p_{\theta}\) does not assign probability mass \(\forall y\in\text{supp}(u)\).

**Proposition 5**.: J\({}_{\alpha}(u~{}||~{}p)\) _is finite for any \(p\in\Omega\) and any \(\alpha<1\). As \(\alpha\to 1\), \(\mathrm{J}_{\alpha}(u~{}||~{}p)\) diverges iff \(\exists y\in\operatorname*{supp}(u)\) for which \(p(y)=0\)._

See App. F for a proof.

## 4 Experiments

We evaluate our family of entropy regularizers on two language generation tasks: machine translation and abstractive summarization. We then analyze trends in model performance as a function of \(\alpha\) and model entropy9 and explore how this entropy affects other properties of language generation models. In the following experiments, each model is trained using eq. (4) where \(\mathcal{R}(\boldsymbol{\theta})=D_{\mathrm{J}_{\alpha}}(\tilde{p}~{}||~{} p_{\theta})\). We conduct searches over \(\alpha\) and \(\beta\) using Bayesian optimization (Snoek et al., 2012) to find the combination of regularizer \(D_{\mathrm{J}_{\alpha}}\) and strength coefficient \(\beta\) that lead to the lowest loss on the development set for the respective task.10 We additionally do a more fine-grained grid search over \(\beta\) for \(\mathrm{J}_{0}\) (confidence penalty) and \(\mathrm{J}_{1}\) (label smoothing) for completeness. All other model hyperparameters are held constant. We run experiments on multiple architectures and across several data sets to ensure trends are general.

Footnote 7: We have \(\alpha\approx 1\) as an exception; the standard deviation is slightly higher for larger values of \(\beta\).

### Neural Machine Translation

We explore performance of the regularizer \(D_{\mathrm{J}_{\alpha}}\) on NMT systems using three language pairs and corpora of two different sizes on the following tasks: WMT'14 German-to-English (De-En) (Bojar et al., 2014), IWSLT'14 German-to-English (De-En) (Cettolo et al., 2012), and Multitarget TED Talks Task (MTTT) French-to-English (Fr-En) and Japanese-to-English (Ja-En) tasks (Duh, 2018). For the larger WMT data set, we train fewer models using coarser-grained \(\alpha\) and \(\beta\) ranges. We perform experiments for both Transformers (Vaswani et al., 2017) and convolutional sequence-to-sequence models (Gehring et al., 2017).

For reproducibility and comparability, we use the data pre-processing scripts provided by fairseq Ott et al. (2019) and follow recommended hyperparameter settings from previous work (Vaswani et al., 2017; Gehring et al., 2017) for baseline models. We use SacreBLEU Post (2018) to calculate bleu scores Papinieni et al. (2002). Specific data pre-processing steps and model hyperparameter details are provided in App. G. Decoding is performed with length-normalized beam search with a beam size of 5 unless otherwise stated. Early stopping was used during training; model parame

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**WMIT’14 De-En**} & \multicolumn{3}{c}{**IWSLT’14 De-En**} & \multicolumn{3}{c}{**MTTT Fr-En**} \\  & \(\alpha\) & \(\beta\) & H & bleu & \(\alpha\) & \(\beta\) & H & bleu & \(\alpha\) & \(\beta\) & H & bleu \\ \hline _No Regularization_ & – & 0 & 0.11 & 31.1 & – & 0 & 0.1 & 35.7 & – & 0 & 0.15 & 35.2 \\ Label Smoothing \(D_{\mathrm{J}_{1}}\) (\(\gamma\)=0.1) & 1 & 0.11 & 0.23 & 31.3 +0.2 & 1 & 0.11 & 0.18 & 36.9 +1.2 & 1 & 0.11 & 0.18 & 36.5 +0.8 \\ Label Smoothing \(D_{\mathrm{J}_{1}}\) & 1 & 0.35 & 0.38 & 31.7 +0.6 & 1 & 0.50 & 0.40 & 37.2 +1.5 & 1 & 0.693 & 0.47 & 37.5 +2.3 \\ Confidence Penalty \(D_{\mathrm{J}_{0}}\) & 0 & 0.28 & 0.55 & 31.6 +0.5 & 0 & 0.76 & 0.81 & 37.5 +1.8 & 0 & 0.95 & 0.86 & 37.4 +2.2 \\ GER \(D_{\mathrm{J}_{\alpha}}\) & 0.7 & 0.65 & 0.47 & 32.0 +0.9 & 0.5 & 1.00 & 0.56 & 37.5 +1.8 & 0.85 & 0.52 & 0.37 & 37.6 +2.4 \\ \hline \hline \end{tabular}
\end{table}
Table 2: bleu scores and normalized entropy \(\hat{\mathrm{H}}(p_{\theta})\) on the test sets for WMT’14 De-En, WMT’14 De-En, and MTTFT Fr-En. Results include baseline models with no (entropy) regularization and standard label smoothing with \(\gamma\)=\(0.1\) (equivalent to \(\beta\approx 0.11\)). We report scores from the best model found (on validation set) for \(D_{\mathrm{J}_{0}}\), \(D_{\mathrm{J}_{1}}\), and \(D_{\mathrm{J}_{\alpha}}\) over all \(\alpha,\beta\) pairs. bleu standard deviation across random seeds was typically \(<0.1\) and always \(<0.16\).8 Results for MTTF Ja-En and convolutional architectures can be found in App. H.

 

ters were taken from the checkpoint with the best validation set bleu.

Results of our experiments are shown in Table 2 and Figure 3. We see the same relation between model entropy and bleu with both Transformer and convolutional architectures and between different language pairs. We show results for the Transformer architectures inline as they are the current standard for many NLP tasks; results for convolutional architectures are in App. H. Our results show better performance is achieved with values of \(\alpha\) and \(\beta\) other than those that correspond to label smoothing with \(\gamma=0.1\), which is the commonly used value for the strength coefficient Vaswani et al. (2017); Edunov et al. (2018). Moreover, the relationship between model entropy and evaluation performance is strong, following the same trend for all values of \(\alpha\), which suggests tuning a model for a specific entropy rather than \(\alpha,\beta\) may be a better method in practice. We discuss trends in SS4.3.

### Abstractive Summarization

We fine-tune BART Lewis et al. (2019) on the CNN/DailyMail abstractive summarization task Hermann et al. (2015) with regularizer \(D_{\mathrm{J}_{\alpha}}\). Data pre-processing and other hyperparameter settings follow Lewis et al. (2019). Results in Table 3 show that optimal values of Rouge-L Lin (2004), the evaluation metric, can be achieved by regularizing with \(D_{\mathrm{J}_{\alpha}}\) for different values of \(\alpha\). Notably, the entropy is virtually the same for the models that achieve top performance, demonstrating the closer relationship of performance with model entropy than with \(\alpha\), discussed further in SS4.3.

### Significance of \(\alpha\) and Model Entropy

We look at the strength of the relationship between the evaluation metrics and both \(\alpha\) and the model's entropy. Figure 3 shows a quadratic relationship between model entropy and bleu. On the other hand, the relationship between \(\alpha\) (coloring of points) and bleu is not an obvious one; the best performing models are regularized with various values of \(\alpha\).

As correlation only tells us about linear relationships, we report mutual information to measure the strength of the relationship between \(\alpha\), model entropy, and bleu. Mutual information shows the proportion of entropy of a variable that is "explained" by another and is often used as a generalized correlation measure i.e. for nonlinear relationships Song et al. (2012). We see in Figure 4 that model entropy has a much stronger relationship with bleu than \(\alpha\). Indeed, the normalized mutual information (NMI) between \(\alpha\) and bleu is \(\approx 0.05\) compared to \(\approx 0.25\) between model entropy and bleu--implying that any flavor of entropy regularization can lead to similar performance.

While the relationship between \(\alpha\) and bleu is

Figure 3: Model entropy \(\hat{\mathrm{H}}(p_{\boldsymbol{\theta}})\) vs. bleu on IWSLT’14 German to English (De-En) and Multitarget TED Talks Task French to English (Fr-En) using a Transformer architecture; each point is a fully trained model, regularized with \(D_{\mathrm{J}_{\alpha}}\) for varying \(\alpha\) and \(\beta\). Label smoothing at standard \(\gamma=0.1\) and no (entropy) regularization are marked.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline  & \(\alpha\) & \(\beta\) & \(\hat{\mathrm{H}}(p_{\boldsymbol{\theta}})\) & Rouge-L \\ \hline _No Regularization_ & – & – & 0.08 & 40.5 \\ Confidence Penalty \(D_{\mathrm{J}_{0}}\) & 0 & 0.15 & 0.19 & 40.9 **+0.4** \\ Label Smoothing \(D_{\mathrm{J}_{1}}\) & 1 & 0.1 & 0.2 & 40.9 **+0.4** \\ GER \(D_{\mathrm{J}_{\alpha}}\) & 0.5 & 0.35 & 0.19 & 40.8 **+0.3** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Rouge-L on test set for CNN/DailyMail abstractive summarization task. Note that we replicate their reported result (achieved with label smoothing).

 

weak, it is still statistically significant. Some evidence for this exists in Figure 3 where a closer examination reveals that each level of \(\alpha\) has a similar quadratic trend, albeit with a different offset. Specifically, the performance of models trained with \(D_{\mathrm{J}_{\alpha}}\) for \(\alpha\in[0.75,1]\) (which includes label smoothing) starts to degrade at lower levels of entropy than models trained with \(D_{\mathrm{J}_{\alpha}}\) for \(\alpha\in[0,0.25]\) (confidence penalty). As quantitative validation of this observation, we (i) run a conditional independence test to see whether bleu and \(\alpha\) are conditionally independent given model entropy and (ii) look at the range of \(\beta\) for which \(D_{\mathrm{J}_{\alpha}}\) leads to good performance for different \(\alpha\).

Conditional Independence.If \(\alpha\) and bleu are conditionally independent it implies that the value of \(\alpha\) does not supply any additional information about the value bleu given model entropy, i.e. \(\alpha\) does not matter when using the regularizer \(D_{\mathrm{J}_{\alpha}}\). We use a Monte Carlo permutation test where the null hypothesis is that no relationship between \(\alpha\) and bleu exists.11 However, this test rejects the null hypothesis with \(p\)-value \(<0.05\), supporting the alternate hypothesis that \(\alpha\) and bleu are _not_ conditionally independent.

Footnote 11: The underlying distributions of random variables are assumed to be Gaussian. See Legendre (2000) for more details.

Tuning \(\beta\).On the tasks for which we trained \(>60\) models, we take the subset of models for which performance is within \(\approx 1\%\) (\(<0.4\text{~{}bleu}\)) of the best overall model. We then look at the range of \(\beta\) used with the regularizer \(D_{\mathrm{J}_{\alpha}}\) for these models. The range of \(\beta\) that meets the above criterion is much larger for \(\alpha\) close to \(0\) than for for \(\alpha\) close to \(1\) (see Figure 5). We contend this implies that \(D_{\mathrm{J}_{\alpha}}\) is easier to tune (i.e. it is more robust) for \(\alpha\approx 0\) while for \(\alpha\approx 1\), \(D_{\mathrm{J}_{\alpha}}\) is relatively sensitive to \(\beta\).

### Sparsity

We take a subset of models trained with regularizers \(D_{\mathrm{J}_{0}}\) and \(D_{\mathrm{J}_{1}}\) and examine the sparsity of \(p_{\bullet}\). Results in Table 4 support our formal analysis regarding the sparsity of \(D_{\mathrm{J}_{0}}\) and \(D_{\mathrm{J}_{1}}\) in SS3.2; \(D_{\mathrm{J}_{1}}\) steeply penalizes sparsity while \(D_{\mathrm{J}_{\alpha}}\) for \(\alpha<1\) allows words to be assigned probability \(\approx 0\).

### Sequence Likelihood

We look at how the probability (under \(p_{\bullet}\)) of the reference sequence on the test set changes with model entropy. While higher entropy in models trends positively with downstream evaluation metrics (Figure 3), experiments show they often lead to lower log-likelihood of the reference sequence. Both of these observations have been made for models trained with label smoothing in previous works Ott et al. (2018); Muller et al. (2019). However, log-likelihood alone does not tell a complete story. During decoding, we search for the

\begin{table}
\begin{tabular}{l c c} \hline \hline  & \multicolumn{2}{c}{Sparsity Threshold} \\  & \(e^{-10}\) & \(e^{-15}\) \\ \cline{2-3} Label Smoothing \(D_{\mathrm{J}_{1}}\) & \(38\%\pm 0.01\%\) & \(0.0\%\pm 5\text{e-}5\%\) \\ Confidence Penalty \(D_{\mathrm{J}_{0}}\) & \(54\%\pm 5\text{e-}3\%\) & \(0.7\%\pm 4\text{e-}4\%\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Percentage of words with \(<\epsilon\) probability mass at different values of \(\epsilon\) (below which we consider as functionally \(0\)) for models trained with \(D_{\mathrm{J}_{1}}\) and \(D_{\mathrm{J}_{0}}\). To control for entropy, all models used in the calculation have entropy within the same \(1\%\).

 

most probable sequence _relative_ to other candidate sequences. This implies that a more relevant calculation would be that of the overall ranking in \({\cal Y}\) of the reference sequence or of the log-likelihood of the reference sequence relative to the most probable sequence. Since the former is typically impossible to calculate exactly due to the size of \({\cal Y}\), we approximate it by looking at the average ranking in \(Y\) of each word in the reference sequence.

In Figure 6, we see that higher-entropy models generally rank the reference sequence lower than lower-entropy models; this result is surprising because higher-entropy models generally perform better on downstream evaluation metrics, e.g. bleu. Notably, this decrease in ranking is less prominent for models regularized with \(\alpha\approx 0\). In Figure 8, we see that while lower-entropy models place more probability mass on the reference sequence, the reference sequence is still far from probable compared to the decoded sequence. However, the ratio of log-likelihoods of the reference to the decoded sequence is larger for high-entropy models, which shows that, in this context, the reference sequence has higher _relative_ log-likelihood under higher-entropy models.

### Decoding

In language generation tasks, estimated distributions are fed to decoding algorithms to create sequence predictions. To fully understand how model entropy affects performance for these tasks, we must explore the potential interactions between model entropy and the decoding strategy.

Chorowski and Jaitly (2017) saw that with label smoothing, prediction accuracy improved and so using a wider beam during beam search did not give further improvements; however, our results suggest otherwise. As shown in Figure 7, the trend in bleu vs. model entropy stays remarkably constant for beam search as the beam width is varied, including for greedy decoding (beam size of 1). Perhaps unsurprisingly though, higher entropy is detrimental to the performance of decoding with random sampling (with temperature \(T=1\)). However, this phenomenon could potentially be remedied by decreasing the temperature during decoding, a common practice for avoiding sampling from the tail of the distribution Kirkpatrick et al. (1983).

## 5 Discussion

Our experiments show entropy regularization has a number of beneficial effects on natural language generation models. Clearly, low-entropy predictions, which are more aligned with the empirical distribution (Figure 8), are a sign of overfitting in a model since they lead to poor generalization abilities (Figure 3). In other words, we observe that closely approximating the empirical distribution is at odds with a well calibrated model, i.e. a model \(p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\) that matches the true, underlying probabilities \(p(\mathbf{y}~{}|~{}\mathbf{x})\).12 Entropy regularization appears to alleviate this problem; namely, for more regularized models, Figure 3 shows increased evaluation metric scores and Figure 8 demonstrates an increase in the log-likelihood of the reference sequence relative to the highest probability sequence.

Footnote 12: This is different than the empirical distribution \(\tilde{p}(\mathbf{y}~{}|~{}\mathbf{x})\).

Figure 6: Average ranking in \(p_{\theta}\) of words in the reference sequence on the test set for IWSLT ’14 (De-En) plotted against model entropy. Overall trends show a decrease in the ranking of the reference for models with more entropy regularization. Notably, the reference is generally ranked higher for models regularized with \(D_{\mathrm{J_{a}}}\) for \(\alpha\approx 0\) than for \(\alpha\in[0.25,1)\).

 

Decoding.Overconfident predictions inhibit the ability to recover after a poor choice of words during decoding; Chorowski and Jaitly (2017) suggest that higher-entropy models \(p_{\boldsymbol{\mathsf{s}}}\), like the ones resulting from regularization with label smoothing, would alleviate this problem. Results throughout this paper support this hypothesis not just for label smoothing, but for the \(D_{\mathrm{J}_{\alpha}}\) family of entropy regularizers as well.

Choosing the baseline distribution.Throughout this work, we use the uniform distribution \(u\) as our **baseline distribution** for the regularizer \(D_{\mathrm{J}_{\alpha}}\). However, one could also use some other distribution defined over the vocabulary such as the unigram (Chorowski and Jaitly, 2017) or a function of word embedding distance with the target word (Kumar and Tsvetkov, 2019; Li et al., 2020). Both have proven to be more effective than \(u\) when used with label smoothing and the confidence penalty. However, using distributions other than \(u\) with \(D_{\mathrm{J}_{\alpha}}\) leads to indirect forms of entropy regularization. Specifically, the mathematical relationship to entropy regularization becomes more convoluted. Therefore, we leave the application of GER to other distributions as a topic for future work.

## 6 Related Work

Entropy regularization has a long history in reinforcement learning (Williams and Peng, 1991; Mnih et al., 2016; Fox et al., 2016; Haarnoja et al., 2018) where it has provided substantial improvements in exploration. Such methods have since been adapted for supervised learning where they have proven to be reliable forms of regularization for various probabilistic modeling tasks (Grandvalet and Bengio, 2005; Smith and Eisner, 2007).

More recently, interpolating between exclusive and inclusive KL divergences has been explored in NMT by Xiao et al. (2019). However, this method was used for the objective function (i.e. between \(\tilde{p}\) and \(p_{\boldsymbol{\mathsf{s}}}\)) and not as a regularization technique (i.e. between a baseline distribution \(q\) and \(p_{\boldsymbol{\mathsf{s}}}\)). Li et al. (2020) construct a baseline distribution \(q\) as a function of word embedding distances to to use in place of the uniform distribution \(u\) in the label smoothing equation. This work is complementary to ours, as \(q\) can similarly be used in place of \(u\) with GER. Finally, our work is closest to that of Muller et al. (2019), which attempts to find the circumstances under which label smoothing has a positive effect on model performance. However, they do not explore entropy regularization on the whole nor do they attempt to provide an explanation for why label smoothing works. We attempt to answer the "why" question through a quantitative analysis of label smoothing and empirical exploration of the relationship between model entropy and performance.

## 7 Conclusion

We discuss the properties of generalized entropy regularization and provide empirical results on two language generation tasks. We find entropy regularization leads to improvements over baseline systems on evaluation metrics for all values of the parameter \(\alpha\) with our regularizer \(D_{\mathrm{J}_{a}}\). Theoretical and empirical evidence show label smoothing adds undesirable constraints to the model and is the hardest to tune of the regularizers tested. We therefore advocate the use of alternate forms of entropy regularization for language generation tasks.

## References

* Beirlant et al. (1997) J. Beirlant, E. Dudewicz, L. Gyor, and E. C. Meulen. 1997. Nonparametric entropy estimation: An overview. _International Journal of Mathematical and Statistical Sciences_, 6.
* Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In _Advances in Neural Information Processing Systems 28_, pages 1171-1179.
* Bojar et al. (2014) Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve

Figure 8: Average word probability of the reference and the most probable (for beam search with \(k=5\)) sequences plotted against model entropy on test set for IWSLT ’14 (De-En). The black line is a smoothed estimate of their ratio.

 

Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
* Cettolo et al. (2012) Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT\({}^{3}\): Web inventory of transcribed and translated talks. In _Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT)_, pages 261-268.
* Chen et al. (2018) Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_, pages 76-86.
* Chorowski and Jaitly (2017) Jan Chorowski and Navdeep Jaitly. 2017. Towards better decoding and language model integration in sequence to sequence models. In _Proceedings of INTERSPEECH_.
* Duh (2018) Kevin Duh. 2018. The multitarget TED talks task. [http://www.cs.jhu.edu/~kevinduh/a/multitarget-tedtalks/](http://www.cs.jhu.edu/~kevinduh/a/multitarget-tedtalks/).
* Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
* Fox et al. (2016) Roy Fox, Ari Pakman, and Naftali Tishby. 2016. Taming the noise in reinforcement learning via soft updates. In _Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence_, pages 202-211.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In _Proceedings of the 34th International Conference on Machine Learning_, volume 70, pages 1243-1252.
* Grandvalet and Bengio (2005) Yves Grandvalet and Yoshua Bengio. 2005. Semi-supervised learning by entropy minimization. In L. K. Saul, Y. Weiss, and L. Bottou, editors, _Advances in Neural Information Processing Systems 17_, pages 529-536. MIT Press.
* Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In _Proceedings of the 35th International Conference on Machine Learning_.
* Hastie et al. (2001) Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2001. _The Elements of Statistical Learning_. Springer Series in Statistics. Springer New York Inc., New York, NY, USA.
* Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In _Advances in Neural Information Processing Systems 28_, pages 1693-1701.
* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. _International Conference on Learning Representations_.
* Kirkpatrick et al. (1983) Scott Kirkpatrick, C. Daniel Gelatt, and Mario P. Vecchi. 1983. Optimization by simulated annealing. _Science_, 220(4598):671-680.
* Kumar and Tsvetkov (2019) Sachin Kumar and Yulia Tsvetkov. 2019. Von Mises-Fisher loss for training sequence to sequence models with continuous outputs. In _International Conference on Learning Representations_.
* Legendre (2000) Pierre Legendre. 2000. Comparison of permutation methods for the partial correlation and partial mantel tests. _Journal of Statistical Computation and Simulation_, 67(1):37-73.
* Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. _CoRR_, abs/1910.13461.
* Li et al. (2020) Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. 2020. Data-dependent Gaussian prior objective for language generation. In _International Conference on Learning Representations_.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Martins and Astudillo (2016) Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In _Proceedings of The 33rd International Conference on Machine Learning_, volume 48, pages 1614-1623.
* Martins et al. (2011) Andre F. T. Martins, Noah A. Smith, Pedro M. Q. Aguiar, and Mario A. T. Figueiredo. 2011. Structured sparsity in structured prediction. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 1500-1511.
* Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In _Proceedings of The 33rd International Conference on Machine Learning_, volume 48 of _Proceedings of Machine Learning Research_, pages 1928-1937, New York, New York, USA. PMLR 

Rafael Muller, Simon Kornblith, and Geoffrey E. Hinton. 2019. When does label smoothing help? In _Advances in Neural Information Processing Systems 32_, pages 4696-4705.
* Niculae et al. (2018) Vlad Niculae, Andre Martins, Mathieu Blondel, and Claire Cardie. 2018. SparseMAP: Differentiable sparse structured inference. In _Proceedings of the 35th International Conference on Machine Learning_, volume 80, pages 3799-3808.
* Nielsen and Boltz (2011) Frank Nielsen and Sylvain Boltz. 2011. The Burbea-Rao and Bhattacharyya centroids. _IEEE Transactions on Information Theory_, 57(8):5455-5466.
* Ott et al. (2018) Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In _International Conference on Machine Learning_.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of NAACL-HLT 2019: Demonstrations_, pages 48-53.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_, pages 311-318.
* Pereyra et al. (2017) Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In _Proceedings of the International Conference on Learning Representations_.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 1715-1725.
* Smith and Eisner (2007) David A. Smith and Jason Eisner. 2007. Bootstrapping feature-rich dependency parsers with entropic priors. In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_, pages 667-677, Prague, Czech Republic. Association for Computational Linguistics.
* Snoek et al. (2012) Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In _Advances in Neural Information Processing Systems 25_, pages 2951-2959.
* Song et al. (2012) Lin Song, Peter Langfelder, and Steve Horvath. 2012. Comparison of co-expression measures: mutual information, correlation, and model based indices. _BMC Bioinformatics_, 13(1):328.
* Szegedy et al. (2015) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. _2016 IEEE Conference on Computer Vision and Pattern Recognition_, pages 2818-2826.
* Vaswani et al. (2018) Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2Tensorfor neural machine translation. _CoRR_, abs/1803.0741.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30_, pages 5998-6008.
* Williams and Peng (1991) Ronald Williams and Jing Peng. 1991. Function optimization using connectionist reinforcement learning algorithms. _Connection Science_, 3:241-268.
* Xiao et al. (2019) Fengshun Xiao, Yingting Wu, Hai Zhao, Rui Wang, and Shu Jiang. 2019. Dual skew divergence loss for neural machine translation. _CoRR_, abs/1908.08399.

 

\(\alpha\)-Jensen to KL

For reference, we repeat eq. (9), the definition of the skew Jensen divergence for some strictly convex function \(G:\Omega\rightarrow{\mathbb{R}}\) and probability distributions \(p\), \(q\):

\[\mathrm{J}_{\alpha,G}(p~{}||~{}q):=\frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha) G(p)+\alpha G(q)-\nabla G((1-\alpha)p+\alpha q)\Big{)}\]

We can rewrite the \(\alpha\)-Jensen divergence with convex generator function \(G\) in terms of the Bregman divergence

\[\begin{split}\mathrm{J}_{\alpha,G}(p~{}||~{}q)&= \frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha)G(p)+\alpha G(q)-G((1-\alpha)p+ \alpha q)\Big{)}\\ &=\frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha)G(p)+\alpha G(q)-G( (1-\alpha)p+\alpha q)\\ &\qquad\underbrace{-\alpha(1-\alpha)\langle p-q,\nabla G((1- \alpha)p+\alpha q)\rangle-\alpha(1-\alpha)\langle q-p,\nabla G((1-\alpha)p+ \alpha q)\rangle}_{=0,\text{ note $p-q$ in first inner product and $q-p$ in second}}\Big{)}\\ &=\frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha)G(p)+\alpha G(q)-G( (1-\alpha)p+\alpha q)\\ &\qquad-\underbrace{(1-\alpha)\langle\alpha(p-q),\nabla G((1- \alpha)p+\alpha q)\rangle}_{\text{bring a inside the inner product since $ b(v,w)=\langle b\cdot v,w\rangle$}}\\ &\qquad-\underbrace{\alpha\langle(1-\alpha)(q-p),\nabla G((1- \alpha)p+\alpha q)\rangle}_{\text{likewise, bring $(1-\alpha)$ inside the inner product}}\Big{)}\\ &=\frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha)G(p)+\alpha G(q)-G( (1-\alpha)p+\alpha q)\\ &\qquad-(1-\alpha)\underbrace{p-((1-\alpha)p+\alpha q),\nabla G ((1-\alpha)p+\alpha q)\rangle}_{\text{distribute $\alpha$ and rewrite}}\\ &\qquad-\alpha\underbrace{(q-((1-\alpha)p+\alpha q),\nabla G(( 1-\alpha)p+\alpha q)\rangle}_{\text{distribute $(1-\alpha)$ and rewrite}}\Big{)}\\ &=\frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha)[G(p)-G((1-\alpha) p+\alpha q)-\langle p-((1-\alpha)p+\alpha q),\nabla G((1-\alpha)p+\alpha q)\rangle] \\ &\qquad\underbrace{+\alpha[G(q)-G((1-\alpha)p+\alpha q)-\langle q -((1-\alpha)p+\alpha q),\nabla G((1-\alpha)p+\alpha q)\rangle]}_{\text{ regroup terms based on multiplier (either $\alpha$ or $1-\alpha$) so we can rewrite equation as two Bregman divergences}\\ &=\frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha)D_{G}(p,(1-\alpha)p+ \alpha q)+\alpha D_{G}(q,(1-\alpha)p+\alpha q)\Big{)}\end{split}\]

We look at the behavior of \(D_{\mathrm{J}_{\alpha,G}}(p~{}||~{}q)\) as \(\alpha\to\{0,1\}\)

\[\lim_{\alpha\to 0}\frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha)D_ {G}(p,(1-\alpha)p+\alpha q)+\alpha D_{G}(q,(1-\alpha)p+\alpha q)\Big{)}\] \[=\lim_{\alpha\to 0}\frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha) \underbrace{D_{G}(p,p)}_{=0}+\alpha D_{G}(q,p)\Big{)}\] \[=\lim_{\alpha\to 0}\frac{1}{(1-\alpha)}D_{G}(q,p)\] 

If we expand \(D_{G}(q,p)\) using our generator function \(G(p)=\sum_{i}p(i)\log p(i)\), we get

\[D_{G}(q,p)\] \[=\sum_{i}q(i)\log q(i)-\sum_{i}p(i)\log p(i)-\langle q-p,log(p)-1\rangle\] \[=\sum_{i}q(i)\log q(i)-\sum_{i}p(i)\log p(i)+\sum_{i}p(i)\log p(i)- \sum_{i}q(i)\log p(i)\] \[\underbrace{-\sum_{i}p(i)+\sum_{i}q(i)}_{=0~{}\text{since}~{}q,p~{} \text{are both probability distributions summing to 1}}\] \[=\sum_{i}q(i)\log q(i)-\sum_{i}q(i)\log p(i)\] \[=\operatorname*{KL}(q~{}||~{}p)\]

Similarly, we can show \(\lim_{\alpha\to 1}\mathrm{J}_{\alpha}=\operatorname*{KL}(p~{}||~{}q)\)

## Appendix B \(\alpha\)-Jensen to Jensen-Shannon

The proof that the \(\alpha\)-Jensen divergence is proportional to the Jensen-Shannon divergence is quite straightforward. If we evaluate \(\mathrm{J}_{\alpha}(p~{}||~{}q)\) at \(G=x\log x\) and \(\alpha=\frac{1}{2}\)

\[\mathrm{J}_{\alpha}(p~{}||~{}q) =\frac{1}{\alpha(1-\alpha)}\Big{(}(1-\alpha)G(p)+\alpha G(q)-G((1 -\alpha)p+\alpha q)\Big{)}\] \[=4\cdot\Big{(}\frac{1}{2}G(p)+\frac{1}{2}G(q)-G(\frac{1}{2}p+ \frac{1}{2}q)\Big{)}\] \[=4\cdot\Big{(}\frac{1}{2}p\log(p)+\frac{1}{2}q\log(q)-\frac{p+q} {2}\log(\frac{p+q}{2})\Big{)}\] \[=4\cdot\Big{(}\frac{1}{2}(p\log(p)-p\log(\frac{p+q}{2}))+\frac{1 }{2}(q\log(q)-q\log(\frac{p+q}{2}))\Big{)}\] \[=4\cdot\left(\frac{1}{2}\mathrm{KL}(p~{}||~{}\frac{p+q}{2})+ \frac{1}{2}\mathrm{KL}(p~{}||~{}\frac{p+q}{2})\right)\] \[=4\cdot\mathrm{JS}(p~{}||~{}q)\]

## Appendix C Label Smoothing

For the case that \(\alpha\to 1\), \(p=u\), and \(q=p_{\bullet}\), we have

\[\lim_{\alpha\to 1}\mathrm{J}_{\alpha}(u~{}||~{}p_{\bullet}( \cdot~{}|~{}\mathbf{x})) =\operatorname*{KL}(u~{}||~{}p_{\bullet})\] \[=\sum_{y\in Y}u(y)\log\frac{u(y)}{p_{\bullet}(y~{}|~{}\mathbf{x})}\] \[=\sum_{y\in Y}u(y)\log u(y)-\sum_{y\in Y}u(y)\log p_{\bullet}(y~{} |~{}\mathbf{x})\] \[=\log|Y|-\sum_{y\in Y}u(y)\log p_{\bullet}(y~{}|~{}\mathbf{x})\] \[=-\sum_{y\in Y}u(y)\log p_{\bullet}(y~{}|~{}\mathbf{x})+N\] 

When \({\rm J}_{1}(u~{}||~{}p_{\theta}(\cdot~{}|~{}{\bf x}))\) is used as a regularizer for maximum likelihood training, we get the loss function

\[\mathcal{L}(\theta)\] \[=\mathrm{KL}\left(\tilde{p}(\cdot~{}|~{}{\bf x})~{}||~{}p_{\theta}( \cdot~{}|~{}{\bf x})\right)+\beta\cdot\mathrm{KL}\left(u(\cdot)~{}||~{}p_{ \theta}(\cdot~{}|~{}{\bf x})\right)\] \[=-\underbrace{\sum_{y\in Y}\Big{(}\tilde{p}(y~{}|~{}{\bf x})+ \beta\cdot u(y)\Big{)}\log p_{\theta}(y~{}|~{}{\bf x})}_{\text{unnormalized label- smoothed cross-entropy loss}}+N\]

where \(N\) is constant with respect to \(\boldsymbol{\theta}\).

## Appendix D Classical Entropy Regularization

For the case that \(\alpha\to 0\), \(p=u\), and \(q=p_{\theta}\), we have

\[\lim_{\alpha\to 0}{\rm J}_{\alpha}(q~{}||~{}p_{\theta}(\cdot~{}|~{}{\bf x })) =\mathrm{KL}(p_{\theta}~{}||~{}u)\] \[=\sum_{y\in Y}p_{\theta}(y~{}|~{}{\bf x})\log\frac{p_{\theta}(y~{ }|~{}{\bf x})}{u(y)}\] \[=\sum_{y\in Y}p_{\theta}(y~{}|~{}{\bf x})\log p_{\theta}(y~{}|~{} {\bf x})-\sum_{y\in Y}p_{\theta}(y~{}|~{}{\bf x})\log u(y)\] \[=-\mathrm{H}(p_{\theta}(y~{}|~{}{\bf x}))-\log\frac{1}{|Y|}\sum_{ y\in Y}p_{\theta}(y~{}|~{}{\bf x})\] \[=-\mathrm{H}(p_{\theta}(y~{}|~{}{\bf x}))-\log\frac{1}{|Y|}\] \[=-\mathrm{H}(p_{\theta}(y~{}|~{}{\bf x}))-N\]

When \({\rm J}_{0}(u~{}||~{}p_{\theta}(\cdot~{}|~{}{\bf x}))\) is used as a regularizer for maximum likelihood training, we get the loss function

\[\mathcal{L}(\theta)\] \[=\mathrm{KL}\left(\tilde{p}(\cdot~{}|~{}{\bf x})~{}||~{}p_{\theta }(\cdot~{}|~{}{\bf x})\right)+\beta\cdot\mathrm{KL}\left(p_{\theta}(\cdot~{}|~ {}{\bf x}~{}||~{}u(\cdot))\right)\] \[=\underbrace{\mathrm{KL}\left(\tilde{p}(\cdot~{}|~{}{\bf x})~{}|| ~{}p_{\theta}(\cdot~{}|~{}{\bf x})\right)-\beta\cdot H(p_{\theta}(y~{}|~{}{\bf x }))}_{\text{confidence penalty cross-entropy loss}}+\beta\cdot N\]

## Appendix E Bounds of \({\rm J}_{\alpha}\)

**Upper bound of \({\rm J}_{\alpha}\):**

First note that \(\mathrm{KL}(p~{}||~{}q)\) is convex in \(q\) when supp(\(p\)) \(\subseteq\) supp(\(q\)), which must be true since \((1-\alpha)u+\alpha p\) has support everywhere both \(p\) and \(u\) do for \(\alpha\in(0,1)\). Therefore for \(\alpha\in(0,1)\)

\[\mathrm{KL}(p~{}||~{}(1-\alpha)u+\alpha p) \leq(1-\alpha)\mathrm{KL}(p~{}||~{}u)+\alpha\mathrm{KL}(p~{}||~{}p)\] \[=(1-\alpha)\mathrm{KL}(p~{}||~{}u)\]

similarly,

\[\mathrm{KL}(u~{}||~{}(1-\alpha)u+\alpha p)\leq\alpha\mathrm{ KL}(u~{}||~{}p)\]

We then have: 

\[{\rm J}_{\alpha}(u~{}||~{}p) =\frac{\alpha}{\alpha(1-\alpha)}{\rm KL}(p~{}||~{}(1-\alpha)u+ \alpha p)+\frac{1-\alpha}{\alpha(1-\alpha)}{\rm KL}(u~{}||~{}(1-\alpha)u+ \alpha p)\] \[\leq\frac{\alpha(1-\alpha)}{\alpha(1-\alpha)}{\rm KL}(p~{}||~{} u)+\frac{\alpha(1-\alpha)}{\alpha(1-\alpha)}{\rm KL}(u~{}||~{}p)\] \[={\rm KL}(p~{}||~{}u)+{\rm KL}(u~{}||~{}p)\]

**Lower bound of \({\rm J}_{\alpha}\):**

The bound from below is trivial given the definition of \({\rm J}_{\alpha}\), however, it can more easily be seen by expressing \({\rm J}_{\alpha}\) as the sum of \({\rm KL}\) divergences as above:

\[{\rm J}_{\alpha}(u~{}||~{}p)=\frac{\alpha}{\alpha(1-\alpha)}{\rm KL}(p~{}||~ {}(1-\alpha)u+\alpha p)+\frac{1-\alpha}{\alpha(1-\alpha)}{\rm KL}(u~{}||~{}( 1-\alpha)u+\alpha p)\]

Since \(\alpha>0\) and necessarily \({\rm KL}(\cdot~{}||~{}\cdot)\geq 0\), the lower bound \(0\leq{\rm J}_{\alpha}(u~{}||~{}p)\) follows.

## Appendix F No Sparse Solution for \({\rm J}_{1}\)

Proof.: By definition, for any distribution \(p\) over a vocabulary \(Y\):

\[{\rm J}_{1}(u~{}||~{}p)=-\frac{1}{|Y|}\sum_{y\in Y}\log p(y)+\log|Y| \tag{13}\]

Thus, if \(p_{\theta}(y~{}|~{}{\bf x})\to 0\) for some \(y\in Y\) and some \({\bf x}\in\mathcal{X}\), we have \({\rm J}_{1}(u~{}||~{}p)={\rm KL}(u~{}||~{}p_{\theta})\rightarrow\infty\). This means that label smoothing enforces \(p_{\theta}\) has support everywhere \(u>0\), i.e. over all words \(y\in Y\). For any \(\alpha<1\), \({\rm J}_{\alpha}\) allows for sparse solutions since \(\lim_{x\to 0}x\log x=0\). 

## Appendix G Data Pre-Processing and Hyperparameter Settings

For training with convolutional architectures we set hyperparameters, e.g. dropout, learning rate, etc., following Gehring et al. (2017). On IWSLT'14 and MTTT tasks, we follow the recommended Transformer settings for IWSLT'14 in fairseq.13Hyperparameters for models trained on the WMT task are set following version 3 of the Tensor2Tensor toolkit (Vaswani et al., 2018). We use byte-pair encoding (BPE; Sennrich et al. 2016.) for all languages. Vocabulary sizes for WMT and IWSLT'14 are set from recommendations for the respective tasks in fairseq; for the MTTT tasks, vocabulary sizes are tuned on models with standard label smoothing regularization.

Footnote 13: [https://github.com/pytorch/fairseq/tree/master/examples/translation](https://github.com/pytorch/fairseq/tree/master/examples/translation)

Similarly, the CNN/DailyMail data set is pre-processed and uses BPE following the same steps as Lewis et al. (2019). Hyperparameters are the same as for their model fine-tuned on CNN/DailyMail. Details are available on the fairseq website.14

Footnote 14: [https://github.com/pytorch/fairseq/blob/master/examples/bart/README.cnn.md](https://github.com/pytorch/fairseq/blob/master/examples/bart/README.cnn.md)
 

Figure 10: Model entropy vs. bleu (validation set) on IWSLT’14 German to English (De-En) using a convolutional architecture and generator function \(G(z)=||z||_{2}^{2}\); see Figure 3 for additional information.

Figure 9: Model entropy vs. bleu (validation set) on Multitarget Ted Talks Task Japanese to English (Ja-En) using a Transformer architecture; see Figure 3 for additional information.

 



\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**WMT’14 De-En (Convolutional)**} & \multicolumn{3}{c}{**MTTT Ja-En (Transformer)**} \\  & \(\alpha\) & \(\beta\) & \(\hat{\mathrm{H}}(p_{\theta})\) & bleu & \(\alpha\) & \(\beta\) & \(\hat{\mathrm{H}}(p_{\theta})\) & bleu \\ \hline _No Regularization_ & - & 0 & 0.15 & 33.2 & - & 0 & 0.19 & 13.8 \\ Label Smoothing \(D_{J_{1}}(\gamma=0.1)\) & 1 & 0.11 & 0.25 & 34.1+**0.9** & 1 & 0.11 & 0.27 & 15.2 ± **1.4** \\ Label Smoothing \(D_{J_{1}}\) & 1 & 0.35 & 0.42 & 34.6+**1.4** & 1 & 0.96 & 0.61 & 16.2 +**2.4** \\ Confidence Penalty \(D_{J_{0}}\) & 0 & 0.60 & 0.79 & 34.7+**1.5** & 0 & 0.65 & 0.80 & 15.9 +**2.1** \\ GER \(D_{J_{a}}\) & 0.75 & 0.60 & 0.45 & 34.8+**1.6** & 0.42 & 1.7 & 0.76 & 15.9 +**2.1** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Test bleu for IWSLT’14 German-to-English using a convolutional architecture and for MTTT Japanese-to-English using a Transformer architecture; see Table 2 for additional information.