SlowGuess/ACL_2020_OCR huggingface dataset:
9Model entropy is estimated as an average of the entropies of distributions at each time step during decoding, i.e.  $\hat{\mathrm{H}} (p_{\theta}) = D_H(p_{\theta})$  . Entropy is normalized by the maximum possible entropy for the given vocabulary size  $(\log |Y|)$  in all figures and tables to control for the fact that languages have vocabularies of different sizes.
<sup>10</sup>We only report results with generator function  $G = -\mathrm{H}$  as results using  $G(z) = ||z||_2^2$  were consistently worse and often did not improve on the baseline; these results may be seen in App. H.
6874
Figure 3: Model entropy  $\hat{\mathrm{H}}(p_{\theta})$  vs. BLEU on IWSLT'14 German to English (De-En) and Multitarget TED Talks Task French to English (Fr-En) using a Transformer architecture; each point is a fully trained model, regularized with  $D_{\mathrm{J}_{\alpha}}$  for varying  $\alpha$  and  $\beta$ . Label smoothing at standard  $\gamma = 0.1$  and no (entropy) regularization are marked.
α	β	\( \hat{\mathrm{H}}(p_{\theta}) \)	ROUGE-L
No Regularization	-	-	0.08	40.5
Confidence Penalty \( D_{J_0} \)	0	0.15	0.19	40.9 +0.4
Label Smoothing \( D_{J_1} \)	1	0.1	0.2	40.9 +0.4
GER \( D_{J_\alpha} \)	0.5	0.35	0.19	40.8 +0.3
Table 3: ROUGE-L on test set for CNN/DailyMail abstractive summarization task. Note that we replicate their reported result (achieved with label smoothing).
ters were taken from the checkpoint with the best validation set BLEU.
Results of our experiments are shown in Table 2 and Figure 3. We see the same relation between model entropy and BLEU with both Transformer and convolutional architectures and between different language pairs. We show results for the Transformer architectures inline as they are the current standard for many NLP tasks; results for convolutional architectures are in App. H. Our results show better performance is achieved with values of  $\alpha$  and  $\beta$  other than those that correspond to label smoothing with  $\gamma = 0.1$ , which is the commonly used value for the strength coefficient (Vaswani et al., 2017; Edunov et al., 2018). Moreover, the relationship between model entropy and evaluation performance is strong, following the same trend for all values of  $\alpha$ , which suggests tuning a model for a specific entropy rather than  $\alpha, \beta$  may be a better method in practice. We discuss trends in §4.3.
4.2 Abstractive Summarization
We fine-tune BART (Lewis et al., 2019) on the CNN/DailyMail abstractive summarization task
(Hermann et al., 2015) with regularizer  $D_{\mathrm{J}_{\alpha}}$ . Data pre-processing and other hyperparameter settings follow Lewis et al. (2019). Results in Table 3 show that optimal values of ROUGE-L (Lin, 2004), the evaluation metric, can be achieved by regularizing with  $D_{\mathrm{J}_{\alpha}}$  for different values of  $\alpha$ . Notably, the entropy is virtually the same for the models that achieve top performance, demonstrating the closer relationship of performance with model entropy than with  $\alpha$ , discussed further in §4.3.
4.3 Significance of  $\alpha$  and Model Entropy
We look at the strength of the relationship between the evaluation metrics and both  $\alpha$  and the model's entropy. Figure 3 shows a quadratic relationship between model entropy and BLEU. On the other hand, the relationship between  $\alpha$  (coloring of points) and BLEU is not an obvious one; the best performing models are regularized with various values of  $\alpha$ .
As correlation only tells us about linear relationships, we report mutual information to measure the strength of the relationship between  $\alpha$ , model entropy, and BLEU. Mutual information shows the proportion of entropy of a variable that is "explained" by another and is often used as a generalized correlation measure i.e. for nonlinear relationships (Song et al., 2012). We see in Figure 4 that model entropy has a much stronger relationship with BLEU than  $\alpha$ . Indeed, the normalized mutual information (NMI) between  $\alpha$  and BLEU is  $\approx 0.05$  compared to  $\approx 0.25$  between model entropy and BLEU—implying that any flavor of entropy regularization can lead to similar performance.
While the relationship between  $\alpha$  and BLEU is
Figure 4: Entropy  $\mathrm{H}(\cdot)$ , Conditional Entropy  $\mathrm{H}(\cdot|\cdot)$  and Mutual Information  $\mathrm{I}(\cdot;\cdot)$  for BLEU with alpha  $(\alpha)$  and model entropy, respectively. Model entropy explains a greater portion of variability in BLEU than  $\alpha$  does. Non-parametric estimates are used for all values (Beirlant et al., 1997). Data from IWSLT'14 De-En Transformer models.
weak, it is still statistically significant. Some evidence for this exists in Figure 3 where a closer examination reveals that each level of  $\alpha$  has a similar quadratic trend, albeit with a different offset. Specifically, the performance of models trained with  $D_{\mathrm{J}_{\alpha}}$  for  $\alpha \in [0.75,1]$  (which includes label smoothing) starts to degrade at lower levels of entropy than models trained with  $D_{\mathrm{J}_{\alpha}}$  for  $\alpha \in [0,0.25]$  (confidence penalty). As quantitative validation of this observation, we (i) run a conditional independence test to see whether BLEU and  $\alpha$  are conditionally independent given model entropy and (ii) look at the range of  $\beta$  for which  $D_{\mathrm{J}_{\alpha}}$  leads to good performance for different  $\alpha$ .
Conditional Independence. If  $\alpha$  and BLEU are conditionally independent it implies that the value of  $\alpha$  does not supply any additional information about the value BLEU given model entropy, i.e.  $\alpha$  does not matter when using the regularizer  $D_{\mathrm{J}_{\alpha}}$ . We use a Monte Carlo permutation test where the null hypothesis is that no relationship between  $\alpha$  and BLEU exists. However, this test rejects the null hypothesis with  $p$ -value  $< 0.05$ , supporting the alternate hypothesis that  $\alpha$  and BLEU are not conditionally independent.
Tuning  $\beta$ . On the tasks for which we trained  $>60$  models, we take the subset of models for which performance is within  $\approx 1\% (< 0.4$  BLEU) of the best overall model. We then look at the range of  $\beta$  used with the regularizer  $D_{\mathrm{J}_{\alpha}}$  for these models. The range of  $\beta$  that meets the above criterion is
Figure 5: Each line represents the range of  $\beta$  for which  $D_{\mathrm{J}_{\alpha}}$  leads to performance within  $\approx 1\%$  ( $< 0.4$  BLEU) of the best overall model for the task. For  $\alpha$  close to 1, (which includes label smoothing)  $D_{\mathrm{J}_{\alpha}}$  has a smaller optimal range, and so is harder to tune.
Sparsity Threshold
e-10	e-15
Label Smoothing DJ1	38% ± 0.01%	0.0% ± 5e-5%
Confidence Penalty DJ0	54% ± 5e-3%	0.7% ± 4e-4%
Table 4: Percentage of words with  $< \epsilon$  probability mass at different values of  $\epsilon$  (below which we consider as functionally 0) for models trained with  ${D}_{{\mathrm{J}}_{1}}$  and  ${D}_{{\mathrm{J}}_{0}}$  . To control for entropy, all models used in the calculation have entropy within the same 1%.
much larger for  $\alpha$  close to 0 than for  $\alpha$  close to 1 (see Figure 5). We contend this implies that  $D_{\mathrm{J}_{\alpha}}$  is easier to tune (i.e. it is more robust) for  $\alpha \approx 0$  while for  $\alpha \approx 1$ ,  $D_{\mathrm{J}_{\alpha}}$  is relatively sensitive to  $\beta$ .
4.4 Sparsity
We take a subset of models trained with regularizers  $D_{\mathrm{J_0}}$  and  $D_{\mathrm{J_1}}$  and examine the sparsity of  $p_\theta$ . Results in Table 4 support our formal analysis regarding the sparsity of  $D_{\mathrm{J_0}}$  and  $D_{\mathrm{J_1}}$  in §3.2;  $D_{\mathrm{J_1}}$  steeply penalizes sparsity while  $D_{\mathrm{J_\alpha}}$  for  $\alpha < 1$  allows words to be assigned probability  $\approx 0$ .
4.5 Sequence Likelihood
We look at how the probability (under  $p_{\theta}$ ) of the reference sequence on the test set changes with model entropy. While higher entropy in models trends positively with downstream evaluation metrics (Figure 3), experiments show they often lead to lower log-likelihood of the reference sequence. Both of these observations have been made for models trained with label smoothing in previous works (Ott et al., 2018; Müller et al., 2019). However, log-likelihood alone does not tell a complete story. During decoding, we search for the
11The underlying distributions of random variables are assumed to be Gaussian. See Legendre (2000) for more details.
6876
Figure 6: Average ranking in  $p_{\theta}$  of words in the reference sequence on the test set for IWSLT '14 (De-En) plotted against model entropy. Overall trends show a decrease in the ranking of the reference for models with more entropy regularization. Notably, the reference is generally ranked higher for models regularized with  $D_{\mathrm{J}_{\alpha}}$  for  $\alpha \approx 0$  than for  $\alpha \in [0.25, 1)$ .
most probable sequence relative to other candidate sequences. This implies that a more relevant calculation would be that of the overall ranking in  $\mathcal{V}$  of the reference sequence or of the log-likelihood of the reference sequence relative to the most probable sequence. Since the former is typically impossible to calculate exactly due to the size of  $\mathcal{V}$ , we approximate it by looking at the average ranking in  $Y$  of each word in the reference sequence.
In Figure 6, we see that higher-entropy models generally rank the reference sequence lower than lower-entropy models; this result is surprising because higher-entropy models generally perform better on downstream evaluation metrics, e.g. BLEU. Notably, this decrease in ranking is less prominent for models regularized with  $\alpha \approx 0$ . In Figure 8, we see that while lower-entropy models place more probability mass on the reference sequence, the reference sequence is still far from probable compared to the decoded sequence. However, the ratio of log-likelihoods of the reference to the decoded sequence is larger for high-entropy models, which shows that, in this context, the reference sequence has higher relative log-likelihood under higher-entropy models.
4.6 Decoding
In language generation tasks, estimated distributions are fed to decoding algorithms to create sequence predictions. To fully understand how model entropy affects performance for these tasks, we must explore the potential interactions between model entropy and the decoding strategy.
Figure 7: BLEU scores on IWSLT'14 De-En validation set with the convolutional architecture by decoding strategy and model entropy. The trend in BLEU stays remarkably constant for beam search as the beam width is varied. Performance declines drastically for higher entropy models when random sampling is used. Color reflects average distance from baseline model.
Chorowski and Jaitly (2017) saw that with label smoothing, prediction accuracy improved and so using a wider beam during beam search did not give further improvements; however, our results suggest otherwise. As shown in Figure 7, the trend in BLEU vs. model entropy stays remarkably constant for beam search as the beam width is varied, including for greedy decoding (beam size of 1). Perhaps unsurprisingly though, higher entropy is detrimental to the performance of decoding with random sampling (with temperature  $T = 1$ ). However, this phenomenon could potentially be remedied by decreasing the temperature during decoding, a common practice for avoiding sampling from the tail of the distribution (Kirkpatrick et al., 1983).
Our experiments show entropy regularization has a number of beneficial effects on natural language generation models. Clearly, low-entropy predictions, which are more aligned with the empirical distribution (Figure 8), are a sign of overfitting in a model since they lead to poor generalization abilities (Figure 3). In other words, we observe that closely approximating the empirical distribution is at odds with a well calibrated model, i.e., a model  $p_{\theta}(\mathbf{y} \mid \mathbf{x})$  that matches the true, underlying probabilities  $p(\mathbf{y} \mid \mathbf{x})$ . Entropy regularization appears to alleviate this problem; namely, for more regularized models, Figure 3 shows increased evaluation metric scores and Figure 8 demonstrates an increase in the log-likelihood of the reference sequence relative to the highest probability sequence.
12This is different than the empirical distribution  $\tilde{p} (\mathbf{y}\mid \mathbf{x})$
6877
Figure 8: Average word probability of the reference and the most probable (for beam search with  $k = 5$ ) sequences plotted against model entropy on test set for IWSLT '14 (De-En). The black line is a smoothed estimate of their ratio.
Decoding. Overconfident predictions inhibit the ability to recover after a poor choice of words during decoding; Chorowski and Jaitly (2017) suggest that higher-entropy models  $p_{\theta}$ , like the ones resulting from regularization with label smoothing, would alleviate this problem. Results throughout this paper support this hypothesis not just for label smoothing, but for the  $D_{\mathrm{J}_{\alpha}}$  family of entropy regularizers as well.
Choosing the baseline distribution. Throughout this work, we use the uniform distribution  $u$  as our baseline distribution for the regularizer  $D_{\mathrm{J}_{\alpha}}$ . However, one could also use some other distribution defined over the vocabulary such as the unigram (Chorowski and Jaitly, 2017) or a function of word embedding distance with the target word (Kumar and Tsvetkov, 2019; Li et al., 2020). Both have proven to be more effective than  $u$  when used with label smoothing and the confidence penalty. However, using distributions other than  $u$  with  $D_{\mathrm{J}_{\alpha}}$  leads to indirect forms of entropy regularization. Specifically, the mathematical relationship to entropy regularization becomes more convoluted. Therefore, we leave the application of GER to other distributions as a topic for future work.
Entropy regularization has a long history in reinforcement learning (Williams and Peng, 1991; Mnih et al., 2016; Fox et al., 2016; Haarnoja et al., 2018) where it has provided substantial improvements in exploration. Such methods have since been adapted for supervised learning where they have proven to be reliable forms of regularization
for various probabilistic modeling tasks (Grandvalet and Bengio, 2005; Smith and Eisner, 2007).
More recently, interpolating between exclusive and inclusive KL divergences has been explored in NMT by Xiao et al. (2019). However, this method was used for the objective function (i.e. between  $\tilde{p}$  and  $p_{\theta}$ ) and not as a regularization technique (i.e. between a baseline distribution  $q$  and  $p_{\theta}$ ). Li et al. (2020) construct a baseline distribution  $q$  as a function of word embedding distances to use in place of the uniform distribution  $u$  in the label smoothing equation. This work is complementary to ours, as  $q$  can similarly be used in place of  $u$  with GER. Finally, our work is closest to that of Müller et al. (2019), which attempts to find the circumstances under which label smoothing has a positive effect on model performance. However, they do not explore entropy regularization on the whole nor do they attempt to provide an explanation for why label smoothing works. We attempt to answer the "why" question through a quantitative analysis of label smoothing and empirical exploration of the relationship between model entropy and performance.
We discuss the properties of generalized entropy regularization and provide empirical results on two language generation tasks. We find entropy regularization leads to improvements over baseline systems on evaluation metrics for all values of the parameter  $\alpha$  with our regularizer  $D_{\mathrm{J}_{\alpha}}$ . Theoretical and empirical evidence show label smoothing adds undesirable constraints to the model and is the hardest to tune of the regularizers tested. We therefore advocate the use of alternate forms of entropy regularization for language generation tasks.
J. Beirlant, E. Dudewicz, L. Gyor, and E. C. Meulen. 1997. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems 28, pages 1171-1179.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve
6878
Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT<sup>3</sup>: Web inventory of transcribed and translated talks. In Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), pages 261-268.
Mia Xu Chen, Orhan First, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 76-86.
Jan Chorowski and Navdeep Jaitly. 2017. Towards better decoding and language model integration in sequence to sequence models. In Proceedings of IN-TERSPEECH.
Kevin Duh. 2018. The multitarget TED talks task. http://www.cs.jhu.edu/~kevinduh/a/multitarget-tedtalks/.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
Roy Fox, Ari Pakman, and Naftali Tishby. 2016. Taming the noise in reinforcement learning via soft updates. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, pages 202-211.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1243-1252.
Yves Grandvalet and Yoshua Bengio. 2005. Semisupervised learning by entropy minimization. In L. K. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems 17, pages 529-536. MIT Press.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2001. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, pages 1693-1701.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. International Conference on Learning Representations.
Scott Kirkpatrick, C. Daniel Gelatt, and Mario P. Vecchi. 1983. Optimization by simulated annealing. Science, 220(4598):671-680.
Sachin Kumar and Yulia Tsvetkov. 2019. Von Mises-Fisher loss for training sequence to sequence models with continuous outputs. In International Conference on Learning Representations.
Pierre Legendre. 2000. Comparison of permutation methods for the partial correlation and partial mantel tests. Journal of Statistical Computation and Simulation, 67(1):37-73.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. CoRR, abs/1910.13461.
Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. 2020. Data-dependent Gaussian prior objective for language generation. In International Conference on Learning Representations.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
André Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of The 33rd International Conference on Machine Learning, volume 48, pages 1614-1623.
Andre F. T. Martins, Noah A. Smith, Pedro M. Q. Aguiar, and Mario A. T. Figueiredo. 2011. Structured sparsity in structured prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1500-1511.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928-1937, New York, New York, USA. PMLR.
6879
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. 2019. When does label smoothing help? In Advances in Neural Information Processing Systems 32, pages 4696-4705.
Vlad Niculae, André Martins, Mathieu Blondel, and Claire Cardie. 2018. SparseMAP: Differentiable sparse structured inference. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 3799-3808.
Frank Nielsen and Sylvain Boltz. 2011. The Burbearo and Bhattacharyya centroids. IEEE Transactions on Information Theory, 57(8):5455-5466.
Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, pages 48-53.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311-318.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In Proceedings of the International Conference on Learning Representations.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715-1725.
David A. Smith and Jason Eisner. 2007. Bootstrapping feature-rich dependency parsers with entropic priors. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 667-677, Prague, Czech Republic. Association for Computational Linguistics.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25, pages 2951-2959.
Lin Song, Peter Langfelder, and Steve Horvath. 2012. Comparison of co-expression measures: mutual information, correlation, and model based indices. BMC Bioinformatics, 13(1):328.