SlowGuess/NAACL_2021_OCR huggingface dataset:
Training. We reimplemented GATEDATTN (Peters and Martins, 2019), an RNN model with separate encoders for lemma and morphological tags. We copied their hyperparameters, except that we used two layers for all encoders. We concatenated the high and low resource training data. In order to make sure the model paid attention to the low resource training data, we either oversampled it 100 times or used data hallucination (Anastasopoulos and Neubig, 2019) to generate synthetic examples. Hallucination worked well for some languages but not others, so we treated it as a hyperparameter.
Results. We compare to CMU-03 $^{6}$  (Anastasopoulos and Neubig, 2019), a two-encoder model with a sophisticated multi-stage training schedule. Despite our models' simpler training technique,
they performed nearly as well in terms of accuracy, while recording, to our knowledge, the best Levenshtein distance on this dataset.
Having shown the effectiveness of our technique on character-level tasks, we next turn to MT. To our knowledge, entmax loss has never been used for transformer-based MT; Correia et al. (2019) used entmax only for transformer attention.
Data. We made use of these language pairs:
- IWSLT 2017 German  $\leftrightarrow$  English (DE  $\leftrightarrow$  EN, Cettolo et al., 2017): 200k training examples.
- KFTT Japanese  $\leftrightarrow$  English (JA  $\leftrightarrow$  EN, Neubig, 2011): 330k training examples.
WMT 2016 Romanian  $\leftrightarrow$  English (RO  $\leftrightarrow$  EN, Bojar et al., 2016): 610k training examples.
WMT 2014 English  $\rightarrow$  German (wMT14, Bojar et al., 2014): 4.4 million training examples.
We used joint BPE (Sennrich et al., 2016) for all language pairs, with 25,000 merges for WMT14 and 32,000 merges for all other pairs.
Training. We trained transformers with the base dimension and layer settings (Vaswani et al., 2017). We optimized with Adam (Kingma and Ba, 2015) and used Noam scheduling with 20,000 warmup
7Although English and Japanese have different writing systems, we still found it beneficial to use joint BPE for  $\mathrm{JA}\leftrightarrow \mathrm{EN}$  because many subwords occur in both the English and Japanese training corpora. These include many named entities, which are often written with the native form alongside the transliteration.
<sup>6</sup>We specifically use the official task numbers from McCarthy et al. (2019), which are more complete than those reported in Anastasopoulos and Neubig (2019).
α	ε	DE→EN	EN→DE	JA→EN	EN→JA	RO→EN	EN→RO	WMT14
1	0	27.05 ± 0.05	23.36 ± 0.10	20.52 ± 0.13	26.94 ± 0.32	29.41 ± 0.20	22.84 ± 0.12	25.10 ± 0.18
>0	27.72 ± 0.11	24.24 ± 0.28	20.99 ± 0.12	27.28 ± 0.17	30.03 ± 0.05	23.15 ± 0.27	25.21 ± 0.04
1.5	0	28.12 ± 0.01	24.03 ± 0.06	21.23 ± 0.10	27.58 ± 0.34	30.27 ± 0.16	23.74 ± 0.08	25.46 ± 0.19
>0	28.11 ± 0.16	24.36 ± 0.12	21.34 ± 0.08	27.58 ± 0.16	30.37 ± 0.04	23.47 ± 0.04	25.45 ± 0.11
Table 3: MT results, averaged over three runs. For label smoothing, we select the best  $\epsilon$  on the development set. Note that wMT14 refers to WMT 2014 English  $\rightarrow$  German.
steps for WMT14 and 10,000 steps for the other pairs. The batch size was 8192 tokens.
Results. Table 3 reports our models' performance in terms of untokenized BLEU (Papineni et al., 2002), which we computed with SacreBLEU (Post, 2018). The results show a clear advantage for label smoothing and entmax loss, both separately and together: label-smoothed entmax loss is the best-performing configuration on 3 out of 7 language pairs, unsmoothed entmax loss performs best on another 3 out of 7, and they tie on the remaining one. Although label-smoothed cross-entropy is seen as an essential ingredient for transformer training, entmax loss models beat it even without label smoothing for every pair except EN→DE.
Model error. Stahlberg and Byrne (2019) showed that the bias in favor of short strings is so strong for softmax NMT models that the argmax sequence is usually the empty string. However, they did not consider the impact of sparsity or label smoothing. $^{8}$  We show in Table 4 how often the empty string is more probable than the beam search hypothesis. This is an upper bound for how often the empty string is the argmax because there can also be other hypotheses that are more probable than the empty string. The results show that  $\alpha$  and  $\epsilon$  both matter: sparsity substantially reduces the frequency with which the empty string is more probable than the beam search hypothesis, while label smoothing usually increases it. Outcomes vary widely with  $\alpha = 1.5$  and  $\epsilon = 0.1$ : WMT14 and DE↔EN models did not seriously suffer from the problem, EN→RO did, and the other three language pairs differed from one run to another. The optimal label smoothing value with cross-entropy is invariably  $\epsilon = 0.1$ , which encourages the cat got your tongue problem; on the other hand, entmax
loss does better with  $\epsilon = 0.01$  for every pair except  $\mathrm{RO}\rightarrow \mathrm{EN}$  in terms of BLEU.
Other inadequate strings. Even if a model rules out the empty string, it might assign nonzero probability to other short, inadequate strings. We investigated this with a depth-limited search inspired by Stahlberg and Byrne (2019)'s exact decoding technique. Unfortunately, the algorithm's exponential runtime made it unfeasible to perform the search for all language pairs, and in particular we found it too slow for the dense search space of cross entropy models, even after applying various optimizations. Therefore, we show results for EN  $\rightarrow$  RO entmax loss models in Table 5. These results show the same trend as on the empty string: short strings are usually pruned for entmax loss models with  $\epsilon = 0$  or  $\epsilon = 0.01$ , but are likely to have a higher score than the beam-decoded hypothesis with  $\epsilon = 0.1$ .
Label smoothing and sparsity. Peters et al. (2019) previously showed that RNN models trained with entmax loss become locally very sparse. Table 6 shows that this is true of transformers as well. Label smoothing encourages greater density, although for the densest language pair (WMT14) this only equates to an average support size of roughly 3300 out of a vocabulary of almost 30,000 word types. The relationship between density and overestimating the empty string is inconsistent with  $\epsilon = 0.1$ : WMT14 and DE  $\leftrightarrow$  EN models become much more dense but rarely overestimate the empty string (Table 4). The opposite occurs for RO  $\leftrightarrow$  EN: models with  $\epsilon = 0.1$  become only slightly more dense but are much more prone to model error. This suggests that corpus-specific factors influence both sparsity and how easily bad hypotheses can be pruned.
${}^{9}$  Specifically,when we pushed target sequences to the search's internal stack,we ordered them so that those ending in the end-of-sequence symbol would be popped first. We also discarded any sequence whose probability was lower than that of the beam-decoded hypothesis.
They trained with "transformer-base" settings, implying label smoothing, and did not compare to unsmoothed losses.
α	ε	DE→EN	EN→DE	JA→EN	EN→JA	RO→EN	EN→RO	WMT14
1	0	8.07 ± 1.21	12.97 ± 2.58	23.10 ± 1.01	14.38 ± 1.06	9.10 ± 0.82	4.32 ± 0.20	28.98 ± 1.44
0.01	9.66 ± 1.55	17.87 ± 1.63	22.96 ± 1.28	15.41 ± 1.62	12.19 ± 0.98	17.56 ± 5.55	31.34 ± 1.82
0.1	22.17 ± 1.79	30.25 ± 1.54	34.79 ± 1.12	31.19 ± 1.80	54.01 ± 14.54	47.24 ± 14.66	49.61 ± 1.44
1.5	0	0.50 ± 0.03	0.78 ± 0.71	0.03 ± 0.04	0.63 ± 0.66	0.18 ± 0.12	0.88 ± 0.46	3.51 ± 0.59
0.01	1.11 ± 0.70	6.27 ± 5.37	2.00 ± 2.59	1.57 ± 1.74	2.70 ± 2.26	0.92 ± 0.44	3.82 ± 1.33
0.1	0.96 ± 0.55	0.65 ± 0.30	15.67 ± 22.16	13.89 ± 19.53	35.13 ± 24.33	44.12 ± 2.39	3.46 ± 0.35
Table 4: Percentage of development set examples for which the model assigns higher probability to the empty string than to the beam-decoded hypothesis.
ε	l=1	l=2
0	1.25 ± 0.41	1.83 ± 0.55
0.01	2.58 ± 0.27	3.55 ± 0.27
0.1	55.85 ± 2.19	61.03 ± 2.13
Table 5: Percentage of development set examples for which EN  $\rightarrow$  RO entmax models assign higher probability to a hypothesis up to length  $\ell$  than to the beam-decoded hypothesis. Setting  $\ell = 0$  is equivalent to Table 4.
Calibration. This is the degree to which a model's confidence about its predictions (i.e. class probabilities) accurately measure how likely those predictions are to be correct. It has been shown (Müller et al., 2019; Kumar and Sarawagi, 2019) that label smoothing improves the calibration of seq2seq models. We computed the Expected Calibration Error (ECE; Naeini et al., 2015) $^{10}$  of our MT models and confirmed their findings. Our results, in Table 7, also show that sparse models are better calibrated than their dense counterparts. This shows that entmax models do not become overconfident even though probability mass is usually concentrated in a small set of possibilities. The good calibration of label smoothing may seem surprising in light of Table 4, which shows that label-smoothed models overestimate the probability of inadequate hypotheses. However, ECE depends only on the relationship between model accuracy and the score of the most likely label. This shows the tradeoff: larger  $\epsilon$  values limit overconfidence but make the tail heavier. Setting  $\alpha = 1.5$  with a moderate  $\epsilon$  value seems to be a sensible balance.
Label smoothing. Our work fits into a larger family of techniques that penalize model overconfidence. Pereyra et al. (2017) proposed the confidence penalty, which reverses the direction of the KL divergence in the smoothing expression. Meister et al. (2020b) then introduced a parameterized family of generalized smoothing techniques, different from Fenchel-Young Label Smoothing, which recovers vanilla label smoothing and the confidence penalty as special cases. In a different direction, Wang et al. (2020) improved inference calibration with a graduated label smoothing technique that uses larger smoothing weights for predictions that a baseline model is more confident of. Other works have smoothed over sequences instead of tokens (Norouzi et al., 2016; Elbayad et al., 2018; Lukasik et al., 2020), but this requires approximate techniques for deciding which sequences to smooth.
MAP decoding and the empty string. We showed that sparse distributions suffer less from the cat got your tongue problem than their dense counterparts. This makes sense in light of the finding that exact MAP decoding works for MI, where probabilities are very peaked even with softmax (Forster and Meister, 2020). For tasks like MT, this is not the case: Eikema and Aziz (2020) pointed out that the argmax receives so little mass that it is almost arbitrary, so seeking it with MAP decoding (which beam search approximates) itself causes many deficiencies of decoding. On the other hand, Meister et al. (2020a) showed that beam search has a helpful bias and introduced regularization penalties for MAP decoding that encode it explicitly. Entmax neither directly addresses the faults of MAP decoding nor compensates for the locality biases of beam search, instead shrinking the gap between beam search and exact decoding. It would be interesting, however, to experiment with these two approaches with entmax in place of softmax.
$^{10}ECE = \sum_{m = 1}^{M}\frac{|B_m|}{N} |\mathsf{acc}(B_m) - \mathsf{conf}(B_m)|$  partitions the model's  $N$  force-decoded predictions into  $M$  evenly-spaced bins and computes the difference between the accuracy  $(\mathsf{acc}(B_m))$  and the average probability of the most likely prediction  $(\mathsf{conf}(B_m))$  within that bin. We use  $M = 10$ .
ε	DE→EN	EN→DE	JA→EN	EN→JA	RO→EN	EN→RO	WMT14
0	0.09	0.08	0.12	0.08	0.11	0.07	0.25
0.01	0.17	0.14	0.19	0.15	0.21	0.18	0.48
0.1	8.74	5.67	6.13	7.62	1.98	0.84	11.82
Table 6: Average percentage of the target vocabulary with nonzero probability with forced decoding for entmax loss MT models. For cross-entropy models, this is always  $100\%$ .
α	ε	DE→EN	EN→DE	JA→EN	EN→JA	RO→EN	EN→RO	WMT14
1	0	0.146	0.149	0.186	0.167	0.166	0.188	0.055
0.01	0.147	0.131	0.175	0.162	0.160	0.176	0.049
0.1	0.078	0.077	0.116	0.095	0.102	0.125	0.016
1.5	0	0.123	0.110	0.147	0.147	0.133	0.152	0.036
0.01	0.113	0.090	0.145	0.141	0.132	0.151	0.033
0.1	0.049	0.039	0.099	0.098	0.102	0.123	0.021
Table 7: MT development set Expected Calibration Error with 10 bins. Lower values indicate better calibration.
We generalized label smoothing from cross-entropy to the wider class of Fenchel-Young losses. When combined with the entmax loss, we showed meaningful gains on character and word-level tasks, including a new state of the art on multilingual G2P. In addition, we showed that the ability of entmax to shrink the search space significantly alleviates the cat got your tongue problem in machine translation, while also improving model calibration.
This work was supported by the European Research Council (ERC StG DeepSPIN 758969), by the P2020 programs MAIA and Unbabel4EU (LISBOA-01-0247-FEDER-045909 and LISBOA-01-0247-FEDER-042671), and by the Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020. We thank Wilker Aziz, Vlad Niculae, and the anonymous reviewers, for their helpful discussion and feedback.
Antonios Anastasopoulos and Graham Neubig. 2019. Pushing the limits of low-resource morphological inflection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 984-996, Hong Kong, China. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR.
Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, and Joydeep Ghosh. 2005. Clustering with bregman divergences. Journal of machine learning research, 6(Oct):1705-1749.
Mathieu Blondel, André FT Martins, and Vlad Niculae. 2020. Learning with fenchel-young losses. Journal of Machine Learning Research, 21(35):1-69.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Néveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131-198, Berlin, Germany. Association for Computational Linguistics.
Jonathan Borwein and Adrian S Lewis. 2010. Convex analysis and nonlinear optimization: theory and examples. Springer Science & Business Media.
John S Bridle. 1990. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pages 227-236. Springer.
M Cettolo, M Federico, L Bentivogli, J Niehues, S Stüker, K Sudoh, K Yoshino, and C Federmann. 2017. Overview of the IWSLT 2017 evaluation campaign. In Proc. IWSLT.
Gonçalo M. Correia, Vlad Niculae, and André F. T. Martins. 2019. Adaptively sparse transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2174-2184, Hong Kong, China. Association for Computational Linguistics.
Bryan Eikema and Wilker Aziz. 2020. Is map decoding all you need? the inadequacy of the mode in neural machine translation. arXiv preprint arXiv:2005.10283.
Maha Elbayad, Laurent Besacier, and Jakob Verbeek. 2018. Token-level and sequence-level loss smoothing for RNN language models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2094-2103, Melbourne, Australia. Association for Computational Linguistics.
Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016. Morphological inflection generation using character sequence to sequence learning. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 634-643, San Diego, California. Association for Computational Linguistics.
Martina Forster and Clara Meister. 2020. SIGMORPHON 2020 task 0 system description: ETH Zürich team. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 106-110, Online. Association for Computational Linguistics.
Kyle Gorman, Lucas F.E. Ashby, Aaron Goyzueta, Arya D. McCarthy, Shijie Wu, and Daniel You. 2020. The sigmorphism 2020 shared task on multilingual grapheme-to-phoneme conversion. In Proc. SIGMORPHON.
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016. Improved neural machine translation with smt features. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 151-157.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. ICLR.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computational Linguistics.
Julia Kreutzer, Jasmijn Bastings, and Stefan Riezler. 2019. Joey NMT: A minimalist NMT toolkit for novices. In Proc. EMNLP-IJCNLP.
Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86.
Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv:1903.00802.
Michal Lukasik, Himanshu Jain, Aditya Menon, Seungyeon Kim, Srinadh Bhojanapalli, Felix Yu, and Sanjiv Kumar. 2020. Semantic label smoothing for sequence to sequence problems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4992-4998, Online. Association for Computational Linguistics.
Andre FT Martins and Ramón Fernandez Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proc. ICML.
Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019. The SIGMORPHON 2019 shared task: Morphological analysis in context and cross-lingual transfer for inflection. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 229–244, Florence, Italy. Association for Computational Linguistics.
Clara Meister, Ryan Cotterell, and Tim Vieira. 2020a. If beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2173-2185, Online. Association for Computational Linguistics.
Clara Meister, Elizabeth Salesky, and Ryan Cotterell. 2020b. Generalized entropy regularization or: There's nothing special about label smoothing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6870-6886, Online. Association for Computational Linguistics.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? In Advances in Neural Information Processing Systems, pages 4694-4703.
Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 212-223, Brussels, Belgium. Association for Computational Linguistics.
Mahdi Pakdaman Naeini, Gregory F Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, volume 2015, page 2901. NIH Public Access.
Graham Neubig. 2011. The Kyoto free translation task. http://www.phontron.com/kftt.
Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al. 2016. Reward augmented maximum likelihood for neural structured prediction. In Advances In Neural Information Processing Systems, pages 1723-1731.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In Proc. NeurIPS Autodiff Workshop.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548.
Ben Peters and André F. T. Martins. 2019. IT-IST at the SIGMORPHON 2019 shared task: Sparse two-headed models for inflection. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 50-56, Florence, Italy. Association for Computational Linguistics.
Ben Peters and André F. T. Martins. 2020. One-size-fits-all multilingual models. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 63-69, Online. Association for Computational Linguistics.
Ben Peters, Vlad Niculae, and André F. T. Martins. 2019. Sparse sequence-to-sequence models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1504-1519, Florence, Italy. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
D Raj Reddy et al. 1977. Speech understanding systems: A summary of results of the five-year research effort. Department of Computer Science. CarnegieMell University, Pittsburgh, PA, 17.