SlowGuess/NAACL_2021_OCR huggingface dataset:
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683-1692, Berlin, Germany. Association for Computational Linguistics.
Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3356-3362, Hong Kong, China. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826.
Constantino Tsallis. 1988. Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics, 52:479-487.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. NeurIPS.
Martin J Wainwright and Michael I Jordan. 2008. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305.
Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. 2020. On the inference calibration of neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3070-3079, Online. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3054-3059, Brussels, Belgium. Association for Computational Linguistics.
Kaisheng Yao and Geoffrey Zweig. 2015. Sequence-to-sequence neural net models for grapheme-to-phoneme conversion. In Sixteenth Annual Conference of the International Speech Communication Association.
Xiang Yu, Ngoc Thang Vu, and Jonas Kuhn. 2020. Ensemble self-training for low-resource languages: Grapheme-to-phoneme conversion and morphological inflection. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 70–78, Online. Association for Computational Linguistics.
For full generality, we consider label smoothing with an arbitrary distribution  $\pmb{r} \in \triangle^{|V|}$ , which may or not be the uniform distribution. We also consider an arbitrary gold distribution  $\pmb{q} \in \triangle^{|V|}$ , not necessarily a one-hot vector. Later we will particularize for the case  $\pmb{r} = \pmb{u} = [1 / |V|, \dots, 1 / |V|]$  and  $\pmb{q} = e_{y^*}$ , the case of interest in this paper.
For this general case, the Fenchel-Young label smoothing loss is defined analogously to (10) as
$$
L _ {\Omega , \epsilon , \boldsymbol {r}} (\boldsymbol {z}; \boldsymbol {q}) := L _ {\Omega} (\boldsymbol {z}, (1 - \epsilon) \boldsymbol {q} + \epsilon \boldsymbol {r}). \tag {14}
$$
A key quantity in this proof is the Bregman information induced by  $\Omega$  (a generalization of the Jensen-Shannon divergence, see Banerjee et al. (2005, §3.1) and Blondel et al. (2020, §3.3)):
$$
I _ {\Omega , \epsilon} (\boldsymbol {q}, \boldsymbol {r}) := - \Omega ((1 - \epsilon) \boldsymbol {q} + \epsilon \boldsymbol {r}) + (1 - \epsilon) \Omega (\boldsymbol {q}) + \epsilon \Omega (\boldsymbol {r}). \tag {15}
$$
Note that, from the convexity of  $\Omega$  and Jensen's inequality, we always have  $I_{\Omega ,\epsilon}(\pmb {q};\pmb {r})\geq 0$
Using the definition of Fenchel-Young loss (7), we obtain
$$
\begin{array}{l} L _ {\Omega , \epsilon , \boldsymbol {r}} (\boldsymbol {z}; \boldsymbol {q}) = \Omega^ {*} (\boldsymbol {z}) - \boldsymbol {z} ^ {\top} ((1 - \epsilon) \boldsymbol {q} + \epsilon \boldsymbol {r}) + \Omega ((1 - \epsilon) \boldsymbol {q} + \epsilon \boldsymbol {r}) \\ = (1 - \epsilon) \underbrace {\left(\Omega^ {*} (\boldsymbol {z}) - \boldsymbol {z} ^ {\top} \boldsymbol {q} + \Omega (\boldsymbol {q})\right)} _ {L _ {\Omega} (\boldsymbol {z}; \boldsymbol {q})} + \epsilon \underbrace {\left(\Omega^ {*} (\boldsymbol {z}) - \boldsymbol {z} ^ {\top} \boldsymbol {r} + \Omega (\boldsymbol {r})\right)} _ {L _ {\Omega} (\boldsymbol {z}; \boldsymbol {r})} + \\ \underbrace {\Omega ((1 - \epsilon) \boldsymbol {q} + \epsilon \boldsymbol {r}) - (1 - \epsilon) \Omega (\boldsymbol {q}) - \epsilon \Omega (\boldsymbol {r})} _ {:= - I _ {\Omega , \epsilon} (\boldsymbol {q}; \boldsymbol {r})} \\ = (1 - \epsilon) L _ {\Omega} (\boldsymbol {z}; \boldsymbol {q}) + \epsilon L _ {\Omega} (\boldsymbol {z}; \boldsymbol {r}) - I _ {\Omega , \epsilon} (\boldsymbol {q}; \boldsymbol {r}). \tag {16} \\ \end{array}
$$
Therefore, up to a constant term (with respect to  $z$ ) and a scalar multiplication by  $1 - \epsilon$ , the Fenchel-Young smoothing loss is nothing but the original Fenchel-Young loss regularized by  $L_{\Omega}(z; r)$ , with regularization constant  $\lambda = \frac{\epsilon}{1 - \epsilon}$ , as stated in Proposition 1. This generalizes the results of Pereyra et al. (2017) and Meister et al. (2020b), obtained as a particular case when  $\Omega$  is the negative Shannon entropy and  $L_{\Omega}$  is the Kullback-Leibler divergence.
We now derive the expression (11):
$$
\begin{array}{l} L _ {\Omega , \epsilon , \boldsymbol {r}} (\boldsymbol {z}, \boldsymbol {q}) = (1 - \epsilon) L _ {\Omega} (\boldsymbol {z}; \boldsymbol {q}) + \epsilon L _ {\Omega} (\boldsymbol {z}; \boldsymbol {r}) - I _ {\Omega , \epsilon} (\boldsymbol {q}; \boldsymbol {r}) \\ = (1 - \epsilon) L _ {\Omega} (\boldsymbol {z}; \boldsymbol {q}) + \epsilon \underbrace {\left(\Omega^ {*} (\boldsymbol {z}) - \boldsymbol {z} ^ {\top} \boldsymbol {r} + \Omega (\boldsymbol {r})\right)} _ {L _ {\Omega} (\boldsymbol {z}; \boldsymbol {q}) + \boldsymbol {z} ^ {\top} \boldsymbol {q} - \Omega (\boldsymbol {q}) - \boldsymbol {z} ^ {\top} \boldsymbol {r} + \Omega (\boldsymbol {r})} - I _ {\Omega , \epsilon} (\boldsymbol {q}; \boldsymbol {r}) \\ = L _ {\Omega} (\boldsymbol {z}; \boldsymbol {q}) + \epsilon (\boldsymbol {z} ^ {\top} \boldsymbol {q} - \boldsymbol {z} ^ {\top} \boldsymbol {r}) + \epsilon (\Omega (\boldsymbol {r}) - \Omega (\boldsymbol {q})) - I _ {\Omega , \epsilon} (\boldsymbol {q}; \boldsymbol {r}) \\ = L _ {\Omega} \left(\boldsymbol {z}; \boldsymbol {q}\right) + \epsilon \left(\boldsymbol {z} ^ {\top} \boldsymbol {q} - \boldsymbol {z} ^ {\top} \boldsymbol {r}\right) \underbrace {- \Omega (\boldsymbol {q}) + \Omega ((1 - \epsilon) \boldsymbol {q} + \epsilon \boldsymbol {r})} _ {: := C (\text {c o n s t a n t})}. \tag {17} \\ \end{array}
$$
If  $-\Omega(\boldsymbol{q}) \leq -\Omega(\boldsymbol{r})$  (i.e., if the regularizing distribution  $\boldsymbol{r}$  has higher generalized entropy than the model distribution  $\boldsymbol{q}$ , as is expected from a regularizer), then
$$
\begin{array}{l} C = - \Omega (\boldsymbol {q}) + \Omega ((1 - \epsilon) \boldsymbol {q} + \epsilon \boldsymbol {r}) \\ = - (1 - \epsilon) \Omega (\boldsymbol {q}) - \epsilon \Omega (\boldsymbol {q}) + \Omega ((1 - \epsilon) \boldsymbol {q} + \epsilon \boldsymbol {r}) \\ \leq - (1 - \epsilon) \Omega (\boldsymbol {q}) - \epsilon \Omega (\boldsymbol {r}) + \Omega ((1 - \epsilon) \boldsymbol {q} + \epsilon \boldsymbol {r}) \\ = - I _ {\Omega , \epsilon} (\boldsymbol {q}, \boldsymbol {r}) \\ \leq 0. \tag {18} \\ \end{array}
$$
Since the left hand side of (17) is by definition a Fenchel-Young loss, it must be non-negative. This implies that
$$
L _ {\Omega} (\boldsymbol {z}; \boldsymbol {q}) + \epsilon (\boldsymbol {z} ^ {\top} \boldsymbol {q} - \boldsymbol {z} ^ {\top} \boldsymbol {r}) \geq 0. \tag {19}
$$
In the conditions of the paper, we have  $\pmb{q} = \pmb{e}_{y^*}$  and  $\pmb{r} = \pmb{u}$ , which satisfies the condition  $-\Omega(\pmb{q}) \leq -\Omega(\pmb{r})$  (this is implied by Blondel et al. (2020, Prop. 4) and the fact that  $\Omega$  is strictly convex and symmetric). In this case,  $z^\top \pmb{q} = z_{y^*}$  is the score of the gold label and  $z^\top \pmb{r} = \frac{1}{|V|} \pmb{z} = \bar{z}$  is the average score.