SlowGuess/ACL_2020_OCR huggingface dataset:
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 2818-2826.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2Tensor for neural machine translation. CoRR, abs/1803.0741.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008.
Ronald Williams and Jing Peng. 1991. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3:241-268.
Fengshun Xiao, Yingting Wu, Hai Zhao, Rui Wang, and Shu Jiang. 2019. Dual skew divergence loss for neural machine translation. CoRR, abs/1908.08399.
A  $\alpha$ -Jensen to KL
For reference, we repeat eq. (9), the definition of the skew Jensen divergence for some strictly convex function  $G: \Omega \to \mathbb{R}$  and probability distributions  $p, q$ :
$$
\mathrm {J} _ {\alpha , G} (p \mid | q) := \frac {1}{\alpha (1 - \alpha)} \Big ((1 - \alpha) G (p) + \alpha G (q) - \nabla G ((1 - \alpha) p + \alpha q) \Big)
$$
We can rewrite the  $\alpha$ -Jensen divergence with convex generator function  $G$  in terms of the Bregman divergence
$$
\mathrm {J} _ {\alpha , G} (p \mid | q) = \frac {1}{\alpha (1 - \alpha)} \Big ((1 - \alpha) G (p) + \alpha G (q) - G ((1 - \alpha) p + \alpha q) \Big)
$$
$$
= \frac {1}{\alpha (1 - \alpha)} \Big ((1 - \alpha) G (p) + \alpha G (q) - G ((1 - \alpha) p + \alpha q)
$$
$$
\underbrace {- \alpha (1 - \alpha) \langle p - q , \nabla G ((1 - \alpha) p + \alpha q) \rangle - \alpha (1 - \alpha) \langle q - p , \nabla G ((1 - \alpha) p + \alpha q) \rangle} _ {= 0, \text {n o t e} p - q \text {i n f i r s t i n n e r p r o d u c t a n d} q - p \text {i n s e c o n d}})
$$
$$
- \underbrace {(1 - \alpha) \langle \alpha (p - q) , \nabla G ((1 - \alpha) p + \alpha q) \rangle} _ {\text {b r i n g} \alpha \text {i n s i d e t h e i n n e r p r o d u c t s i n c e} b \langle v , w \rangle = \langle b \cdot v , w \rangle}
$$
$$
\left. - \underbrace {\alpha \langle (1 - \alpha) (q - p) , \nabla G ((1 - \alpha) p + \alpha q) \rangle} _ {\text {l i k e w i s e , b r i n g} (1 - \alpha) \text {i n s i d e t h e i n n e r p r o d u c t}}\right)
$$
$$
- (1 - \alpha) \underbrace {\langle p - ((1 - \alpha) p + \alpha q) , \nabla G ((1 - \alpha) p + \alpha q) \rangle} _ {\text {d i s t r i b u t e} \alpha \text {a n d r e w r i t e}}
$$
$$
\left. - \alpha \underbrace {\langle q - ((1 - \alpha) p + \alpha q) , \nabla G ((1 - \alpha) p + \alpha q) \rangle} _ {\text {d i s t r i b u t e} (1 - \alpha) \text {a n d r e w r i t e}}\right)
$$
$$
= \frac {1}{\alpha (1 - \alpha)} \Big ((1 - \alpha) [ G (p) - G ((1 - \alpha) p + \alpha q) - \langle p - ((1 - \alpha) p + \alpha q), \nabla G ((1 - \alpha) p + \alpha q) \rangle ]
$$
$$
\left. \underbrace {+ \alpha [ G (q) - G ((1 - \alpha) p + \alpha q) - \langle q - ((1 - \alpha) p + \alpha q) , \nabla G ((1 - \alpha) p + \alpha q) \rangle ]}\right)
$$
regroup terms based on multiplier (either  $\alpha$  or  $1 - \alpha$ ) so we can rewrite equation as two Bregman divergences
$$
= \frac {1}{\alpha (1 - \alpha)} \Big ((1 - \alpha) D _ {G} (p, (1 - \alpha) p + \alpha q) + \alpha D _ {G} (q, (1 - \alpha) p + \alpha q) \Big)
$$
We look at the behavior of  $D_{\mathrm{J}_{\alpha ,G}}(p||q)$  as  $\alpha \rightarrow \{0,1\}$
$$
\begin{array}{l} \lim  _ {\alpha \rightarrow 0} \frac {1}{\alpha (1 - \alpha)} \left((1 - \alpha) D _ {G} (p, (1 - \alpha) p + \alpha q) + \alpha D _ {G} (q, (1 - \alpha) p + \alpha q)\right) \\ = \lim  _ {\alpha \rightarrow 0} \frac {1}{\alpha (1 - \alpha)} \Big ((1 - \alpha) \underbrace {D _ {G} (p , p)} _ {= 0} + \alpha D _ {G} (q, p) \Big) \\ = \lim  _ {\alpha \rightarrow 0} \frac {1}{(1 - \alpha)} D _ {G} (q, p) \\ = D _ {G} (q, p) \\ \end{array}
$$
6881
If we expand  $D_G(q,p)$  using our generator function  $G(p) = \sum_{i} p(i) \log p(i)$ , we get
$$
\begin{array}{l} D _ {G} (q, p) \\ = \sum_ {i} q (i) \log q (i) - \sum_ {i} p (i) \log p (i) - \langle q - p, \log (p) - 1 \rangle \\ = \sum_ {i} q (i) \log q (i) - \sum_ {i} p (i) \log p (i) + \sum_ {i} p (i) \log p (i) - \sum_ {i} q (i) \log p (i) \\ \underbrace {- \sum_ {i} p (i) + \sum_ {i} q (i)} \\ = 0 \text {s i n c e} q, p \text {a r e b o t h p r o b a b i l i t y d i s t r i b u t i o n s s u m m i n g t o} 1 \\ = \sum_ {i} q (i) \log q (i) - \sum_ {i} q (i) \log p (i) \\ = \operatorname {K L} (q | | p) \\ \end{array}
$$
Similarly, we can show  $\lim_{\alpha \to 1}J_{\alpha} = KL(p||q)$
B  $\alpha$ -Jensen to Jensen-Shannon
The proof that the  $\alpha$ -Jensen divergence is proportional to the Jensen-Shannon divergence is quite straightforward. If we evaluate  $\mathrm{J}_{\alpha}(p||q)$  at  $G = x\log x$  and  $\alpha = \frac{1}{2}$
$$
\begin{array}{l} \mathrm {J} _ {\alpha} (p \mid | q) = \frac {1}{\alpha (1 - \alpha)} \Big ((1 - \alpha) G (p) + \alpha G (q) - G ((1 - \alpha) p + \alpha q) \Big) \\ = 4 \cdot \left(\frac {1}{2} G (p) + \frac {1}{2} G (q) - G \left(\frac {1}{2} p + \frac {1}{2} q\right)\right) \\ = 4 \cdot \left(\frac {1}{2} p \log (p) + \frac {1}{2} q \log (q) - \frac {p + q}{2} \log \left(\frac {p + q}{2}\right)\right) \\ = 4 \cdot \left(\frac {1}{2} (p \log (p) - p \log (\frac {p + q}{2})) + \frac {1}{2} (q \log (q) - q \log (\frac {p + q}{2}))\right) \\ = 4 \cdot \left(\frac {1}{2} \mathrm {K L} (p \mid \mid \frac {p + q}{2}) + \frac {1}{2} \mathrm {K L} (p \mid \mid \frac {p + q}{2})\right) \\ = 4 \cdot \operatorname {J S} (p | | q) \\ \end{array}
$$
C Label Smoothing
For the case that  $\alpha \to 1$ ,  $p = u$ , and  $q = p_{\theta}$ , we have
$$
\begin{array}{l} \lim  _ {\alpha \rightarrow 1} \mathrm {J} _ {\alpha} (u | | p _ {\theta} (\cdot | \mathbf {x})) = \operatorname {K L} (u | | p _ {\theta}) \\ = \sum_ {y \in Y} u (y) \log \frac {u (y)}{p _ {\theta} (y \mid \mathbf {x})} \\ = \sum_ {y \in Y} u (y) \log u (y) - \sum_ {y \in Y} u (y) \log p _ {\theta} (y \mid \mathbf {x}) \\ = \log | Y | - \sum_ {y \in Y} u (y) \log p _ {\theta} (y \mid \mathbf {x}) \\ = - \sum_ {y \in Y} u (y) \log p _ {\theta} (y \mid \mathbf {x}) + N \\ \end{array}
$$
6882
When  $\mathrm{J}_1(u||p_\theta (\cdot |\mathbf{x}))$  is used as a regularizer for maximum likelihood training, we get the loss function
$$
\begin{array}{l} \mathcal {L} (\theta) \\ = \operatorname {K L} (\tilde {p} (\cdot | \mathbf {x}) | | p _ {\theta} (\cdot | \mathbf {x})) + \beta \cdot \operatorname {K L} (u (\cdot) | | p _ {\theta} (\cdot | \mathbf {x})) \\ = \underbrace {- \sum_ {y \in Y} \left(\tilde {p} (y \mid \mathbf {x}) + \beta \cdot u (y)\right) \log p _ {\theta} (y \mid \mathbf {x})} _ {\text {u n n o r m a l i z e d l a b e l - s m o o t h e d c r o s s - e n t r o p y l o s s}} + N \\ \end{array}
$$
where  $N$  is constant with respect to  $\theta$ .
D Classical Entropy Regularization
For the case that  $\alpha \to 0$ ,  $p = u$ , and  $q = p_{\theta}$ , we have
$$
\begin{array}{l} \lim  _ {\alpha \rightarrow 0} J _ {\alpha} (q | | p _ {\theta} (\cdot | \mathbf {x})) = K L (p _ {\theta} | | u) \\ = \sum_ {y \in Y} p _ {\theta} (y \mid \mathbf {x}) \log \frac {p _ {\theta} (y \mid \mathbf {x})}{u (y)} \\ = \sum_ {y \in Y} p _ {\theta} (y \mid \mathbf {x}) \log p _ {\theta} (y \mid \mathbf {x}) - \sum_ {y \in Y} p _ {\theta} (y \mid \mathbf {x}) \log u (y) \\ = - \mathrm {H} (p _ {\theta} (y \mid \mathbf {x})) - \log \frac {1}{| Y |} \sum_ {y \in Y} p _ {\theta} (y \mid \mathbf {x}) \\ = - \mathrm {H} (p _ {\theta} (y \mid \mathbf {x})) - \log \frac {1}{| Y |} \\ = - \mathrm {H} \left(p _ {\theta} (y \mid \mathbf {x})\right) - N \\ \end{array}
$$
When  $\mathrm{J}_0(u||p_\theta (\cdot |\mathbf{x}))$  is used as a regularizer for maximum likelihood training, we get the loss function
$$
\begin{array}{l} \mathcal {L} (\theta) \\ = \operatorname {K L} (\tilde {p} (\cdot | \mathbf {x}) | | p _ {\theta} (\cdot | \mathbf {x})) + \beta \cdot \operatorname {K L} (p _ {\theta} (\cdot | \mathbf {x} | | u (\cdot))) \\ = \underbrace {\operatorname {K L} \left(\tilde {p} (\cdot \mid \mathbf {x}) \mid | p _ {\theta} (\cdot \mid \mathbf {x})\right) - \beta \cdot H \left(p _ {\theta} (y \mid \mathbf {x})\right)} _ {\text {c o n f i d e n c e p e n a l t y c r o s s - e n t r o p y l o s s}} + \beta \cdot N \\ \end{array}
$$
E Bounds of  $\mathrm{J}_{\alpha}$
Upper bound of  $\mathrm{J}_{\alpha}$ :
First note that  $\mathrm{KL}(p\parallel q)$  is convex in  $q$  when  $\operatorname {supp}(p)\subseteq \operatorname {supp}(q)$ , which must be true since  $(1 - \alpha)u + \alpha p$  has support everywhere both  $p$  and  $u$  do for  $\alpha \in (0,1)$ . Therefore for  $\alpha \in (0,1)$
$$
\begin{array}{l} \operatorname {K L} (p \mid | (1 - \alpha) u + \alpha p) \leq (1 - \alpha) \operatorname {K L} (p \mid | u) + \alpha \operatorname {K L} (p \mid | p) \\ = (1 - \alpha) \mathrm {K L} (p \mid | u) \\ \end{array}
$$
$$
\operatorname {K L} (u \mid | (1 - \alpha) u + \alpha p) \leq \alpha \operatorname {K L} (u \mid | p)
$$
$$
\begin{array}{l} \mathrm {J} _ {\alpha} (u \mid | p) = \frac {\alpha}{\alpha (1 - \alpha)} \mathrm {K L} (p \mid | (1 - \alpha) u + \alpha p) + \frac {1 - \alpha}{\alpha (1 - \alpha)} \mathrm {K L} (u \mid | (1 - \alpha) u + \alpha p) \\ \leq \frac {\alpha (1 - \alpha)}{\alpha (1 - \alpha)} \mathrm {K L} (p | | u) + \frac {\alpha (1 - \alpha)}{\alpha (1 - \alpha)} \mathrm {K L} (u | | p) \\ = \operatorname {K L} (p \mid | u) + \operatorname {K L} (u \mid | p) \\ \end{array}
$$
Lower bound of  $\mathrm{J}_{\alpha}$ :
The bound from below is trivial given the definition of  $\mathrm{J}_{\alpha}$ , however, it can more easily be seen by expressing  $\mathrm{J}_{\alpha}$  as the sum of KL divergences as above:
$$
\mathrm {J} _ {\alpha} (u \parallel p) = \frac {\alpha}{\alpha (1 - \alpha)} \mathrm {K L} (p \parallel (1 - \alpha) u + \alpha p) + \frac {1 - \alpha}{\alpha (1 - \alpha)} \mathrm {K L} (u \parallel (1 - \alpha) u + \alpha p)
$$
Since  $\alpha > 0$  and necessarily  $\mathrm{KL}(\cdot ||\cdot) \geq 0$ , the lower bound  $0 \leq \mathrm{J}_{\alpha}(u||p)$  follows.
F No Sparse Solution for  $\mathbf{J}_1$
Proof. By definition, for any distribution  $p$  over a vocabulary  $Y$ :
$$
J _ {1} (u \mid p) = - \frac {1}{| Y |} \sum_ {y \in Y} \log p (y) + \log | Y | \tag {13}
$$
Thus, if  $p_{\theta}(y \mid \mathbf{x}) \to 0$  for some  $y \in Y$  and some  $\mathbf{x} \in \mathcal{X}$ , we have  $\mathrm{J}_1(u \parallel p) = \mathrm{KL}(u \parallel p_\theta) \to \infty$ . This means that label smoothing enforces  $p_{\theta}$  has support everywhere  $u > 0$ , i.e. over all words  $y \in Y$ . For any  $\alpha < 1$ ,  $\mathrm{J}_{\alpha}$  allows for sparse solutions since  $\lim_{x \to 0} x \log x = 0$ .
G Data Pre-Processing and Hyperparameter Settings
For training with convolutional architectures we set hyperparameters, e.g. dropout, learning rate, etc., following Gehring et al. (2017). On IWSLT'14 and MTTT tasks, we follow the recommended Transformer settings for IWSLT'14 in fairseq. $^{13}$  Hyperparameters for models trained on the WMT task are set following version 3 of the Tensor2Tensor toolkit (Vaswani et al., 2018). We use byte-pair encoding (BPE; Sennrich et al. 2016.) for all languages. Vocabulary sizes for WMT and IWSLT'14 are set from recommendations for the respective tasks in fairseq; for the MTTT tasks, vocabulary sizes are tuned on models with standard label smoothing regularization.
Similarly, the CNN/DailyMail data set is pre-processed and uses BPE following the same steps as (Lewis et al., 2019). Hyperparameters are the same as for their model fine-tuned on CNN/DailyMail. Details are available on the fairseq website. $^{14}$
<sup>13</sup>https://github.com/pytorch/fairseq/tree/master/examples/translation
$^{14}$ https://github.com/pytorch/fairseq/blob/master/examples/bart/README.cnn.md
6884
Figure 9: Model entropy vs. BLEU (validation set) on Multitarget Ted Talks Task Japanese to English (Ja-En) using a Transformer architecture; see Figure 3 for additional information.
Figure 10: Model entropy vs. BLEU (validation set) on IWSLT'14 German to English (De-En) using a convolutional architecture and generator function  $G(z) = ||z||_2^2$ ; see Figure 3 for additional information.
Figure 11: Model entropy vs. BLEU (validation set) on IWSLT'14 German to English (De-En) and Multitarget Ted Talks Task French to English (Fr-En) using Transformer and convolutional architectures; see Figure 3 for additional information.
6885
WMT'14 De-En (Convolutional)	MTTT Ja-En (Transformer)
α	β	\( \hat{H}(p_{\theta}) \)	BLEU	α	β	\( \hat{H}(p_{\theta}) \)	BLEU
No Regularization	-	0	0.15	33.2	-	0	0.19	13.8
Label Smoothing \( D_{J_1} \)(γ=0.1)	1	0.11	0.25	34.1 +0.9	1	0.11	0.27	15.2 +1.4
Label Smoothing \( D_{J_1} \)	1	0.35	0.42	34.6 +1.4	1	0.96	0.61	16.2 +2.4
Confidence Penalty \( D_{J_0} \)	0	0.60	0.79	34.7 +1.5	0	0.65	0.80	15.9 +2.1
GER \( D_{J_\alpha} \)	0.75	0.60	0.45	34.8 +1.6	0.42	1.7	0.76	15.9 +2.1
Table 5: Test BLEU for IWSLT'14 German-to-English using a convolutional architecture and for MTTT Japanese-to-English using a Transformer architecture; see Table 2 for additional information.
6886