SlowGuess/ACL_2020_OCR huggingface dataset:
Clara Meister<sup>1</sup> Elizabeth Salesky<sup>3</sup> Ryan Cotterell<sup>0,1</sup>  
<sup>1</sup>ETH Zürich 3Johns Hopkins University <sup>0</sup>University of Cambridge clara.meister@inf.ethz.ch esalesky@jhu.edu ryan.cotterell@inf.ethz.ch
Prior work has explored directly regularizing the output distributions of probabilistic models to alleviate peaky (i.e. over-confident) predictions, a common sign of overfitting. This class of techniques, of which label smoothing is one, has a connection to entropy regularization. Despite the consistent success of label smoothing across architectures and datasets in language generation tasks, two problems remain open: (1) there is little understanding of the underlying effects entropy regularizers have on models, and (2) the full space of entropy regularization techniques is largely unexplored. We introduce a parametric family of entropy regularizers, which includes label smoothing as a special case, and use it to gain a better understanding of the relationship between the entropy of a trained model and its performance on language generation tasks. We also find that variance in model performance can be explained largely by the resulting entropy of the model. Lastly, we find that label smoothing provably does not allow for sparse distributions, an undesirable property for language generation models, and therefore advise the use of other entropy regularization methods in its place. Our code is available online at https://github.com/rycolab/entropyRegularization.
When training large neural networks with millions of parameters, regularization of some form is needed to prevent overfitting, even when large amounts of data are used; models for language generation are no exception. In probabilistic modeling, e.g. when the final layer of the neural network is a softmax, overfitting often manifests itself in overconfident placement of most of the probability mass on a few candidates, resulting in peaky (low-entropy) probability distributions over the vocabulary. Specifically for language generation tasks, this behavior leads to the output of repetitive or frequently occurring but unrelated text,
which is detrimental to the generalization abilities of the model (Chorowski and Jaitly, 2017; Holtzman et al., 2020). A natural regularizer to consider is, therefore, one that penalizes overconfidence, encouraging higher entropy in the learned distribution. Indeed, the literature has ascribed gains of  $\approx 1$  BLEU point in machine translation to label smoothing, one such technique (Chen et al., 2018).
Despite the clear relationship between low entropy and overfitting, only a handful of distinct entropy regularizers have been explored. To fill this gap, we introduce generalized entropy regularization (GER), a unified framework for understanding and exploring a broad range of entropy-inducing regularizers. GER is based on the skew-Jensen family of divergences  $\mathrm{J}_{\alpha ,G}$  (Nielsen and Boltz, 2011) and thus may be generalized to any Bregman divergence through the choice of generator function  $G$ . For the negative entropy generator function, GER recovers label smoothing (Szegedy et al., 2015) as  $\alpha \rightarrow 1$ , and the confidence penalty (Pereyra et al., 2017) as  $\alpha \rightarrow 0$ . We provide formal properties of GER in §3, proving these special-case equivalences among other characteristics of GER. We then use GER to examine the relationship between entropy and the evaluation metrics in two language generation tasks: neural machine translation (NMT) and abstractive summarization.
GER encompasses a large family of regularizers, which allows us to directly compare label smoothing to other forms of entropy regularization. By studying the relationship between different regularizers on the performance of natural language generation (NLG) systems, we can better understand not just when but also why label smoothing aids language generation tasks. Through our analysis, we gain the following insights:
(i) With tuning of the regularizer's coefficient, any choice of  $\alpha$  can yield similar performance, i.e. there is nothing special about label smoothing. In fact, our results suggest that la
6870
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6870-6886 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
bel smoothing  $(\alpha \rightarrow 1)$  makes it more difficult to tune the regularizer's coefficient.
(ii) Label smoothing assigns infinite cost to sparse output distributions, which may be an undesirable behavior for language generation tasks.
(iii) There is a strong (quadratic) relationship between a model's performance on the evaluation metric and its (average) entropy, offering a hint as to why these regularizers are so effective for NLG.
In summary, entropy-inducing regularizers are a boon to probabilistic NLG systems, which benefit from higher entropy output distributions. Label smoothing works because it forces the model towards a higher-entropy solution, but we recommend the confidence penalty and other entropy regularizers  $(\alpha < 1)$  for reasons (i) and (ii) above.
In this work, we consider conditional probability models  $p_{\theta}(\mathbf{y} \mid \mathbf{x})$  for natural language generation; such models assign probability to a target sequence  $\mathbf{y} \in \mathcal{V}$  given a source sequence  $\mathbf{x}$ . Specifically, our target sequence  $\mathbf{y} = \langle y_1, \ldots, y_n \rangle$  of arbitrary length  $n$  is a sequence of target words  $y_i$  from our vocabulary  $Y$ . The set of all complete target sequences, which are padded with distinguished beginning- and end-of-sentence symbols, BOS and EOS, is then defined as  $\mathcal{Y} := \{\mathrm{BOS} \circ \mathbf{y} \circ \mathrm{EOS} \mid \mathbf{y} \in Y^*\}$ . For language generation tasks,  $p_{\theta}(\mathbf{y} \mid \mathbf{x})$  is typically a neural network with parameters  $\theta$ ; this network is often trained to approximate  $\tilde{p}(\mathbf{y} \mid \mathbf{x})$ , the empirical distribution (i.e. the distribution of the data). Here, we focus on locally normalized models; in such models  $p_{\theta}(\mathbf{y} \mid \mathbf{x})$  is factored as:
$$
p _ {\theta} (\mathbf {y} \mid \mathbf {x}) = p _ {\theta} (y _ {1} \mid \mathbf {x}) \dots p _ {\theta} (y _ {n} \mid \mathbf {x}, \mathbf {y} _ {<   n}) \quad (1)
$$
where  $p_{\theta}(y_i \mid \mathbf{x}, \mathbf{y}_{<i})$  is defined by a softmax over the output of the final fully connected layer of the network. Generation is performed using greedy search, beam search or a sampling scheme. Of the candidate sequences generated, the one with the highest probability under the model  $p_{\theta}$  is returned as the model's prediction.
One way of selecting the parameters  $\theta$  is to minimize the KL-divergence between the empirical
distribution and the model. This yields the cross-entropy loss (plus an additive constant):2
$$
\begin{array}{l} \mathcal {L} (\boldsymbol {\theta}) = \mathrm {K L} (\tilde {p} | | p _ {\theta}) (2) \\ = \underbrace {\mathrm {H} (\tilde {p} , p _ {\theta})} _ {\text {c r o s s - e n t r o p y l o s s}} - \underbrace {\mathrm {H} (\tilde {p})} _ {\text {c o n s t a n t w . r . t .} \theta} (3) \\ \end{array}
$$
However, fitting a model that perfectly approximates the empirical distribution is, in general, fraught with problems (Hastie et al., 2001). The goal of learning is to generalize beyond the observed data. Exactly fitting the empirical distribution, often termed overfitting, is therefore not an ideal goal and for language generation models specifically, does not go hand-in-hand with the ability of a model to generate desirable text (Bengio et al., 2015). Consequently, it is advisable to minimize a regularized objective to prevent overfitting:
$$
\mathcal {L} (\boldsymbol {\theta}) + \beta \mathcal {R} (\boldsymbol {\theta}) \tag {4}
$$
where  $\mathcal{R}(\pmb{\theta})$  is a regularizer defined over the model with "strength" coefficient  $\beta > 0$ .
2.1 Entropy Regularization
Overfitting can manifest itself as peakiness in  $p_{\theta}$  (Williams and Peng, 1991; Mnih et al., 2016; Pereyra et al., 2017). In other words,  $p_{\theta}$  overconfidently places most of the probability mass on very few candidates. While this overconfidence improves training loss, it hurts generalization. Entropy regularization is one technique that directly combats such overconfidence by encouraging more entropic (less peaky) distributions.
The entropy of the model  $p_{\theta}$  is defined as
$$
\mathrm {H} \left(p _ {\theta}\right) := - \sum_ {\mathbf {y} \in \mathcal {Y}} p _ {\theta} (\mathbf {y}) \log p _ {\theta} (\mathbf {y}) \tag {5}
$$
where we remove dependence on  $\mathbf{x}$  for notational simplicity. However, the sum in eq. (5) over  $\mathcal{V}$  generally renders its computation intractable. Instead, regularization is performed on the conditional distribution over  $Y\cup \{\mathrm{EOS}\}$  at each time step, which can be interpreted as an approximation of the true model entropy. For ease of notation, we define a higher-order function  $D_{f}$  over our training corpus  $\mathcal{C}$  consisting of  $\langle \mathbf{x},\mathbf{y}\rangle$  pairs that maps a function  $f$
$\mathrm{^2H}(p,q)\coloneqq -\sum_{z\in \mathcal{Z}}p(z)\log q(z)$  is cross-entropy and  $\mathrm{H}(p)\coloneqq \mathrm{H}(p,p) = -\sum_{z\in \mathcal{Z}}p(z)\log p(z)$  is the Shannon entropy, for which  $\log = \log_2$  and  $\mathcal{Z} = \operatorname {supp}(p)$
3The notation used by Pereyra et al. (2017) is imprecise.
${}^{1}$  Targets  ${y}_{i}$  may also be characters or subwords; our experiments use byte-pair encoding (Sennrich et al., 2016)
6871
Training Method	Loss Function	Alternate Formulation
Cross Entropy	L(θ) = H(˜p,pθ)	= KL(˜p || pθ) + H(˜p)
Confidence Penalty, DJ0	LCP(θ) = L(θ) + β DKL(pθ || u)	= L(θ) - β DH(pθ) + C
Label Smoothing, DJ1	LLS(θ) = L(θ) + β DKL(u || pθ)	= L(θ) + β DH(u,pθ) + C
Generalized Entropy Regularization, DJα	LGER(θ) = L(θ) + β DJα(u || pθ)	-
Table 1: Loss functions and their alternate formulations for different training methods; the latter three are entropy regularization techniques that augment the standard loss function in row 1.  $C$  denotes a constant with respect to  $\theta$ .
over distributions  $p, q$  as follows below:
$$
D _ {f} (p \mid | q) = \tag {6}
$$
$$
\sum_ {\langle \mathbf {x}, \mathbf {y} \rangle \in \mathcal {C}} \sum_ {t = 1} ^ {| \mathbf {y} |} f (p (\cdot | \mathbf {x}, \mathbf {y} _ {<   t}) | | q (\cdot | \mathbf {x}, \mathbf {y} _ {<   t}))
$$
The function  $D_{f}$  allows us to describe in notation how entropy regularization is typically employed in the training of language generation systems.
Label Smoothing. Label smoothing, first introduced as a regularizer for neural networks by Szegedy et al. (2015), is so named because the technique smooths hard target distributions. One such distribution, the empirical distribution, is encoded as a set of one-hot vectors (hard targets) where for each data point, the correct label (e.g., vocabulary index of a word) has value 1 and all other labels have value 0. Label smoothing with strength coefficient  $\gamma$  is an add- $\gamma$  smoothing scheme on the distribution over labels at every time step. Interestingly, minimizing the cross entropy between this modified distribution and the model  $p_{\theta}$  is equivalent to adding the weighted KL divergence between the uniform distribution and the model  $p_{\theta}$  in our original objective function with the same strength coefficient:
$$
\mathcal {L} (\boldsymbol {\theta}) _ {\gamma} ^ {L S} := (1 - \gamma) \mathcal {L} (\boldsymbol {\theta}) + \gamma D _ {\mathrm {K L}} (u \| p _ {\theta}) \quad (7)
$$
While the loss function is often scaled as above, it is nonetheless equivalent to  $\mathcal{L}(\pmb {\theta})_{\beta}^{LS} = \mathcal{L}(\pmb {\theta}) + \beta D_{\mathrm{KL}}(u||p_{\theta})$  ; we use this form for consistency.
Confidence Penalty. The confidence penalty, empirically explored in the supervised learning setting by Pereyra et al. (2017), aims to penalize a low-entropy model. This is done by subtracting a weighted term for the entropy of the model's pre
diction  $p_{\theta}(\cdot)$  from the loss function, thereby encouraging a more entropic model. This is equivalent to adding the KL divergence between the model  $p_{\theta}$  and the uniform distribution:
$$
\mathcal {L} (\boldsymbol {\theta}) _ {\beta} ^ {C P} := \mathcal {L} (\boldsymbol {\theta}) + \beta D _ {\mathrm {K L}} \left(p _ {\theta} \mid | u\right) \tag {8}
$$
While Pereyra et al. (2017) found that label smoothing performed better than the confidence penalty for NMT, they only searched coarsely over a small range of  $\beta$ 's for both regularizers. Our findings in §4 suggest an alternate conclusion.
3 Generalized Entropy Regularization
The positive effect of both label smoothing and the confidence penalty on model performance in language generation tasks motivates further exploration of entropy-promoting regularizers. To this end, we construct a parameterized family of regularizers with label smoothing and the confidence penalty as special cases. We discuss the formal properties of a subset of this family, providing upper and lower bounds for it. We show divergence only occurs in one case for this subset  $(\alpha \to 1)$ , which directly implies that no sparse solution exists when label smoothing is used as a regularizer.
3.1 A Family of Entropy Regularizers
We derive a family of regularizers from the skew-Jensen divergence  $\mathrm{J}_{\alpha ,G}$  (Nielsen and Boltz, 2011), which is defined below as:
$$
\begin{array}{l} J _ {\alpha , G} (q \mid | p _ {\theta}) := \frac {1}{\alpha (1 - \alpha)} \left((1 - \alpha) G (q) + \alpha G (p _ {\theta}) \right. \\ - G \left(\left(1 - \alpha\right) q + \alpha p _ {\theta}\right)\left. \right) \tag {9} \\ \end{array}
$$
for a strictly convex generator function  $G:\Omega \to \mathbb{R}$  and  $\alpha \in (0,1)$  where  $\Omega$  is a closed convex set. In this paper, we restrict  $\Omega$  to be the  $(|Y| + 1)$ -simplex. Note that  $\mathrm{J}_{\alpha ,G}(q||p_{\theta})\neq \mathrm{J}_{\alpha ,G}(p_{\theta}||q)$  in general, although this is true for some choices of  $G$  and  $\alpha$ .
We define the generalized entropy regularizer as  $\mathcal{R}(\pmb{\theta}) = D_{\mathrm{J}_{\alpha,G}}(u||p_\theta)$  where  $u$  is the uniform
Note that the standard loss function in eq. (3) can be written in this form when computed over  $\mathcal{C}$ , i.e.  $\mathrm{KL}(\tilde{p} || p_{\theta}) = D_{\mathrm{KL}}(\tilde{p} || p_{\theta})$ , since the reference  $\mathbf{y}$  is the only value in  $\operatorname{supp}(\tilde{p})$ .
$^5$ up to multiplicative factor  $(1 - \gamma)$  when  $\beta = \gamma /(1 - \gamma)$
6872
Figure 1: Different divergence measures between  $u$ , the uniform distribution and  $p$ , a probability distribution over a Bernoulli random variable  $X$ . Note that the confidence penalty is equivalent to  $\mathrm{KL}(p\parallel u) = \mathrm{J}_0$  and label smoothing is equivalent to  $\mathrm{KL}(u\parallel p) = \mathrm{J}_1$  (see §3.1). We include entropy  $\mathrm{H}(p)$  and  $\mathrm{Eu}(u\parallel p) = \mathrm{J}_{\alpha ,G}(u\parallel p)$  for  $\alpha = 0.5$  and  $G(p) = ||p||_2^2$ .
distribution. These regularizers promote entropy because they push the model  $p_{\theta}$  towards  $u$ , which is the maximum-entropy distribution with an entropy of  $\log (|Y| + 1)$ . Throughout the rest of this paper, we primarily use the generator function  $G(p) = -\mathrm{H}(p)$ . We use  $\mathrm{J}_{\alpha}$  as shorthand for  $\mathrm{J}_{\alpha, -\mathrm{H}}$ .
We note  $\mathrm{J}_{\alpha}$  is equivalent to quadruple the Jensen-Shannon (JS) divergence and asymptotically approaches the Kullback-Leibler (KL) divergence for certain values of  $\alpha$ . Specifically, we have:
$$
\lim  _ {\alpha \rightarrow 0} \mathrm {J} _ {\alpha} (q | | p _ {\theta}) = \mathrm {K L} (p _ {\theta} | | q) \tag {10}
$$
$$
\lim  _ {\alpha \rightarrow 1} \mathrm {J} _ {\alpha} (q | | p _ {\theta}) = \mathrm {K L} (q | | p _ {\theta}) \tag {11}
$$
$$
J _ {1 / 2} (q \mid p _ {\theta}) = 4 \cdot J S (q \mid p _ {\theta}) \tag {12}
$$
We prove these relationships in App. A and App. B. For ease, we define  $\mathrm{J}_1 \coloneqq \lim_{\alpha \to 1} \mathrm{J}_{\alpha}$  and  $\mathrm{J}_0 \coloneqq \lim_{\alpha \to 0} \mathrm{J}_{\alpha}$ . We note the following two equivalences for these special cases.
Proposition 1.  $\nabla_{\theta}\mathrm{J}_1(u\parallel p_\theta) = \nabla_{\theta}\mathrm{H}(q,p_\theta)$ . In words, the gradient of the loss with GER as  $\alpha \rightarrow 1$  is equivalent to the gradient of the loss augmented with label smoothing.
Proposition 2.  $\nabla_{\theta}\mathrm{J}_0(u\parallel p_\theta) = \nabla_{\theta}\mathrm{H}(p_\theta)$ . In words, the gradient of the loss with GER as  $\alpha \rightarrow 0$  is equivalent to the gradient of the loss augmented with the confidence penalty.
See App. C and App. D for proofs.
Figure 2:  $\mathrm{J}_{\alpha}(u||p)$  as a function of  $\alpha$  for  $u$ , the uniform distribution, and  $p$ , a probability distribution over a 3-way categorical random variable, where for (a)  $p = (0.0001, 0.49995, 0.49995)$  (b)  $p = (0.15, 0.15, 0.7)$  and (c)  $p = (0.25, 0.25, 0.5)$ . There is no standard trend for  $\mathrm{J}_{\alpha}$  as purely a function of  $\alpha \in (0,1)$ .
3.2 Formal Properties of  $\mathrm{J}_{\alpha}$
When fitting a model  $p_{\theta}$ , we generally optimize the inclusive KL, i.e.  $\mathrm{KL}(\tilde{p} || p_{\theta})$ , so that, among other reasons,  $p_{\theta}$  has support everywhere that  $\tilde{p}$  has support. However, it is unclear what relationships we want to encourage between the model  $p_{\theta}$  and the uniform distribution  $u$  during regularization as complete support of  $u$  implies no word can ever have non-zero probability. Here we explore formal properties of  $\mathrm{J}_{\alpha}$  as a regularizer to gain insight into how, as a function of  $\alpha$ , these regularizers affect the learned distribution.
Magnitude. Figure 1 shows the different divergence measures between  $u$  and  $p_{\theta}$ . We see that  $\mathrm{J}_1 = \mathrm{KL}(u \parallel p_{\theta})$  (label smoothing) is much larger than  $\mathrm{J}_0 = \mathrm{KL}(p_{\theta} \parallel u)$  (confidence penalty) at values of  $p_{\theta}$  farther from  $u$ . This indicates that  $\mathrm{J}_1$  would be a stronger regularizer than  $\mathrm{J}_{<1}$ , i.e. penalize values of  $p_{\theta}$  far from  $u$  more heavily, given the same strength coefficient  $\beta$ . Note that it is not always the case that  $\mathrm{J}_{<1}(u \parallel p) \leq \mathrm{J}_1(u \parallel p)$  for fixed  $p$ . We can, however, bound  $\mathrm{J}_{\alpha}$  from above and below by other quantities.
Proposition 3. The divergence  $\mathrm{J}_{\alpha}(u||p)$  is not a monotonic function of  $\alpha$  for all distributions  $p$ .
A proof by counter example is shown in Figure 2.
Proposition 4. For fixed  $p$ ,  $\mathrm{J}_{\alpha}$  has bounds:  $0 \leq \mathrm{J}_{\alpha}(u \parallel p) \leq \mathrm{KL}(u \parallel p) + \mathrm{KL}(p \parallel u)$ .
See App. E for a proof.
Sparsity. Sparsity is generally a desirable trait in probabilistic models; specifically for structured prediction, it leads to improvements in performance and interpretability (Martins et al., 2011; Niculae
${}^{6}$  Distributions other than  $u$  may also be used. See §5.
<sup>7</sup>We also experiment with  $G(z) = ||z||_2^2$ .
6873
WMT'14 De-En	IWSLT'14 De-En	MTTT Fr-En
α	β	H	BLEU	α	β	H	BLEU	α	β	H	BLEU
No Regularization	-	0	0.11	31.1	-	0	0.1	35.7	-	0	0.15	35.2
Label Smoothing DJ1(γ=0.1)	1	0.11	0.23	31.3 +0.2	1	0.11	0.18	36.9 +1.2	1	0.11	0.18	36.5 +0.8
Label Smoothing DJ1	1	0.35	0.38	31.7 +0.6	1	0.50	0.40	37.2 +1.5	1	0.693	0.47	37.5 +2.3
Confidence Penalty JO0	0	0.28	0.55	31.6 +0.5	0	0.76	0.81	37.5 +1.8	0	0.95	0.86	37.4 +2.2
GER DJα	0.7	0.65	0.47	32.0 +0.9	0.5	1.00	0.56	37.5 +1.8	0.85	0.52	0.37	37.6 +2.4
Table 2: BLEU scores and normalized entropy  $\hat{\mathrm{H}} (p_{\theta})$  on the test sets for WMT'14 De-En, WMT'14 De-En, and MTTT Fr-En. Results include baseline models with no (entropy) regularization and standard label smoothing with  $\gamma = 0.1$  (equivalent to  $\beta \approx 0.11$ ). We report scores from the best model found (on validation set) for  $D_{\mathrm{J_0}}$ ,  $D_{\mathrm{J_1}}$ , and  $D_{\mathrm{J_\alpha}}$  over all  $\alpha ,\beta$  pairs. BLEU standard deviation across random seeds was typically  $< 0.1$  and always  $< 0.16$ . Results for MTTT Ja-En and convolutional architectures can be found in App. H.
et al., 2018). For example, Martins and Astudillo (2016) showed the benefits of using sparsemax, which induces sparsity in an output distribution or attention layer, for natural language inference tasks. There are also intuitive reasons for allowing  $p_{\theta}$  to be sparse. Part of modeling language generations tasks is learning when particular sequences cannot, or at least should not, occur (e.g. are grammatically or syntactically incorrect). In these cases, a model should be able to assign 0 probability mass to that sequence. However, there is no sparse optimal solution  $p_{\theta}$  when using label smoothing as the label smoothing loss function becomes divergent if  $p_{\theta}$  does not assign probability mass  $\forall y \in \operatorname{supp}(u)$ .
Proposition 5.  $\mathrm{J}_{\alpha}(u\parallel p)$  is finite for any  $p\in \Omega$  and any  $\alpha < 1$ . As  $\alpha \to 1$ ,  $\mathrm{J}_{\alpha}(u\parallel p)$  diverges iff  $\exists y\in \operatorname {supp}(u)$  for which  $p(y) = 0$ .
See App. F for a proof.
We evaluate our family of entropy regularizers on two language generation tasks: machine translation and abstractive summarization. We then analyze trends in model performance as a function of  $\alpha$  and model entropy and explore how this entropy affects other properties of language generation models. In the following experiments, each model is trained using eq. (4) where  $\mathcal{R}(\pmb{\theta}) = D_{\mathrm{J}_{\alpha}}(\tilde{p} || p_{\theta})$ . We conduct searches over  $\alpha$  and  $\beta$  using Bayesian optimization (Snoek et al., 2012) to find the combination of regularizer  $D_{\mathrm{J}_{\alpha}}$  and strength coefficient  $\beta$
that lead to the lowest loss on the development set for the respective task. $^{10}$  We additionally do a more fine-grained grid search over  $\beta$  for  $\mathrm{J}_0$  (confidence penalty) and  $\mathrm{J}_1$  (label smoothing) for completeness. All other model hyperparameters are held constant. We run experiments on multiple architectures and across several data sets to ensure trends are general.
We explore performance of the regularizer  $D_{\mathrm{J}_{\alpha}}$  on NMT systems using three language pairs and corpora of two different sizes on the following tasks: WMT'14 German-to-English (De-En) (Bojar et al., 2014), IWSLT'14 German-to-English (De-En) (Cettolo et al., 2012), and Multitarget TED Talks Task (MTTT) French-to-English (Fr-En) and Japanese-to-English (Ja-En) tasks (Duh, 2018). For the larger WMT data set, we train fewer models using coarser-grained  $\alpha$  and  $\beta$  ranges. We perform experiments for both Transformers (Vaswani et al., 2017) and convolutional sequence-to-sequence models (Gehring et al., 2017).
For reproducibility and comparability, we use the data pre-processing scripts provided by faireq (Ott et al., 2019) and follow recommended hyperparameter settings from previous work (Vaswani et al., 2017; Gehring et al., 2017) for baseline models. We use SacreBLEU (Post, 2018) to calculate BLEU scores (Papineni et al., 2002). Specific data pre-processing steps and model hyperparameter details are provided in App. G. Decoding is performed with length-normalized beam search with a beam size of 5 unless otherwise stated. Early stopping was used during training; model parame
8We have  $\alpha \approx 1$  as an exception; the standard deviation is slightly higher for larger values of  $\beta$ .