SlowGuess/NAACL_2022_OCR huggingface dataset:
Figure 8: BLEU scores with beam search (beam size of 4) for German-to-synthetic-English translation with different IBM-3 sampling temperatures.
pose, we trained an IBM-3 model (Brown et al., 1993) on the German-English training data after subword segmentation using MGIZA (Gao and Vogel, 2008). IBM-3 is a generative symbolic model that describes the translation process from one language into another with a generative story, and was popular for finding word alignments for statistical (phrase-based) machine translation (Koehn, 2009). The generative story consists of different steps such as distortion (word reordering), fertility  $(1:n$  word mappings), and lexical translation (word-to-word translation) that describe the translation process. The parameters of IBM-3 define probability distributions for each step. In this work we do not use IBM-3 for finding word alignments. Instead, for the original German sentences we sample synthetic English-like translations from the model with different sampling temperatures to control the ambiguity levels of the translation task. A low sampling temperature generates sentence pairs that still capture some of the characteristics of MT such as word reorderings, but the mapping is mostly deterministic (i.e. the same source token is almost always translated to the same target token). A high temperature corresponds to more randomness, i.e. more intrinsic uncertainty. Appendix B contains more details about sampling from IBM-3. We train NMT models using either softmax or SCONES on the synthetic corpora.
Fig. 8 shows that softmax and SCONES perform similarly using beam search: high IBM-3 sampling
Figure 9: Exact search for German-to-synthetic-English translation with different IBM-3 sampling temperatures. BLEU scores are shown relative to those of the beam search softmax output in Fig. 8 for the respective temperatures.
temperature translation tasks are less predictable, and thus lead to lower BLEU scores. The difference between both approaches becomes clear with exact search (Fig. 9). While the translations with the global highest probability for high IBM-3 sampling temperatures are heavily degraded for softmax and SCONES with  $\alpha = 1$ , the drop is much less dramatic for SCONES with  $\alpha = 0.2$  (solid purple curve). Setting  $\alpha$  to a low value enables the model to assign its highest probability to adequate translations, even when the translation task is highly uncertain.
Our approach draws insights from multi-label classification (MLC) (Tsoumakas and Katakis, 2007; Zhang and Zhou, 2006, 2014). One of the earliest approaches for MLC was to transform the problem into multiple binary classification problems while ignoring the correlations between labels (Boutell et al., 2004). More recent work has modeled MLC in the sequence-to-sequence framework with a decoder that generates the labels sequentially, thus preserving the inter-label correlations (Yang et al., 2018). Most prior work in MLC focuses on classification and is not directly applicable to MT. In contrast, our training strategy is tailored for sequence-to-sequence problems. Unlike prior work (Yang et al., 2018), SCONES allows us to perform MLC style training with any underlying NMT architecture by simply changing the loss function. By jointly training all label-specific binary classifiers, our strategy is able to account for label correlations.
Ma et al. (2018) used an MLC objective to improve machine translation. Unlike our approach, they attempted to predict all words in the target sentence with a bag-of-words loss function. We formulate the next word prediction at each time step as an MLC problem to handle intrinsic uncertainty, but our models are predicting ordered target sequences, not bags of words.
The speed-ups from SCONES can be partially attributed to avoiding the normalization of the output over the full vocabulary. The same idea motivated earlier work on self-normalized training (Gutmann and Hyvarinen, 2010; Mnih and Teh, 2012; Devlin et al., 2014; Goldberger and Melamud, 2018). As described in Sec. 2, unlike work on self-normalization, SCONES does not try to approximate a distribution over the full vocabulary. Rather, its output consists of multiple binary classifiers that do not share probability mass by design to be able to better represent intrinsic uncertainty.
Machine translation is a task with high intrinsic uncertainty: a source sentence can have multiple valid translations. We demonstrated that NMT models and specifically Transformers, can learn to model mutually non-exclusive target sentences from single-label training data using our SCONES loss function. Rather than learn a single distribution over all target sentences, SCONES learns multiple binary classifiers that indicate whether or not a target sentence is a valid translation of the source sentence. SCONES yields improved translation quality over conventional softmax-based models for six different translation directions, or (alternatively) speed-ups of up to  $3.9\mathrm{x}$  without any degradation in translation performance. We showed that SCONES can be tuned to mitigate the beam search curse and the problem of inadequate and empty modes in standard NMT. Our experiments on synthetic language translation suggest that, unlike softmax-trained models, SCONES models are able to assign their highest probability to adequate translations even when the underlying task is highly ambiguous.
The SCONES loss function is easy to implement. Adapting standard softmax-based sequence-to-sequence architectures such as Transformers requires only replacing the cross-entropy loss function with SCONES and the softmax with sigmoid activations. The remaining parts of the training
and inference pipelines can be kept unchanged. SCONES can be potentially useful in handling uncertainty for a variety of ambiguous NLP problems beyond translation, such as generation and dialog. We expect this work to encourage research on modeling techniques that can address ambiguity in much better ways compared to current models.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown. 2004. Learning multi-label scene classification. Pattern Recognition, 37(9):1757-1771.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: Composable transformations of Python+NumPy programs.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263-311.
Armen Der Kiureghian and Ove Ditlevsen. 2009. Aleatory or epistemic? Does it matter? Structural safety, 31(2):105-112.
Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1370-1380, Baltimore, Maryland. Association for Computational Linguistics.
Markus Dreyer and Daniel Marcu. 2012. HyTER: Meaning-equivalent semantics for translation evaluation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 162-171, Montreal, Canada. Association for Computational Linguistics.
Bryan Eikema and Wilker Aziz. 2020. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Proceedings of the 28th
International Conference on Computational Linguistics, pages 4506-4520, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Qin Gao and Stephan Vogel. 2008. Parallel implementations of word alignment tool. In *Software Engineering*, Testing, and Quality Assurance for Natural Language Processing, pages 49-57, Columbus, Ohio. Association for Computational Linguistics.
Jacob Goldberger and Oren Melamud. 2018. Self-normalization properties of language modeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 764-773, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Michael Gutmann and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700-1709, Seattle, Washington, USA. Association for Computational Linguistics.
Kevin Knight. 1999. A statistical mt tutorial workbook. In Prepared for the 1999 JHU Summer Workshop, pages 1-37.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv:1903.00802.
Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018. Bag-of-words as target for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 332-338, Melbourne, Australia. Association for Computational Linguistics.
Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, page 419-426, Madison, WI, USA. Omnipress.
Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 212-223, Brussels, Belgium. Association for Computational Linguistics.
Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3956-3965. PMLR.
Sebastian Padó, Daniel Cer, Michel Galley, Dan Jurafsky, and Christopher D Manning. 2009. Measuring machine translation quality as semantic equivalence: A metric based on entailment features. Machine Translation, 23(2-3):181-193.
Miika Pihlaja, Michael Gutmann, and Aapo Hyvärinen. 2010. A family of computationally efficient and simple estimators for unnormalized statistical models. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI'10, page 442-449, Arlington, Virginia, USA. AUAI Press.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Pavel Sountsov and Sunita Sarawagi. 2016. Length bias in encoder decoder models and a case for global conditioning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1516-1525, Austin, Texas. Association for Computational Linguistics.
Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3356-3362, Hong Kong, China. Association for Computational Linguistics.
Felix Stahlberg, Ilia Kulikov, and Shankar Kumar. 2022. Uncertainty determines the adequacy of the mode and the tractability of decoding in sequence-to-sequence models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826.
Grigorios Tsoumakas and Ioannis Katakis. 2007. Multi-label classification: An overview. Int J Data Warehousing and Mining, 2007;1-13.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. 2018. SGM: Sequence generation model for multi-label classification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3915-3926, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations.
Min-Ling Zhang and Zhi-Hua Zhou. 2006. Multilabel neural networks with applications to functional genomics and text categorization. IEEE Transactions on Knowledge and Data Engineering, 18(10):1338-1351.
Min-Ling Zhang and Zhi-Hua Zhou. 2014. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819-1837.
A Time complexity of exact search
The exact search algorithm of Stahlberg and Byrne (2019) we used in the paper is guaranteed to find the global best translation. Its runtime, however, varies greatly between language pairs and source sentences. Therefore, we limit the number of explored states per sentence by 1M to keep the decoding time under control. If the 1M threshold is reached, the optimality of the found translation is not guaranteed anymore. Fortunately, for most of our models and test sets, exact search was able to find and verify the global best translation earlier. Table 7 lists the runs for which a fraction of the sentences did not terminate before 1M steps. In these rare cases, we use the best translation found thus far by exact search as an approximation to the global best translation.
B Sampling from IBM-3
The parameters of the IBM-3 model (Brown et al., 1993) are composed of a set of fertility probabilities  $n(\cdot |\cdot), p_0, p_1$ , a set of translation probabilities  $t(\cdot|\cdot)$ , and a set of distortion probabilities  $d(\cdot|\cdot)$ . According to the IBM Model 3, the following generative
Languages	Run	#incomplete sent.
de-en	SCONES (α = 0.2)	1.45%
de-en	SCONES (α = 0.5)	0.90%
de-en	SCONES (α = 0.7)	0.05%
en-de	SCONES (α = 0.2)	4.01%
fi-en	SCONES (α = 0.2)	1.30%
en-fi	Softmax	0.10%
en-fi	SCONES (α = 0.2)	3.20%
lt-en	Softmax	0.20%
lt-en	SCONES (α = 0.2)	5.20%
en-lt	Softmax	0.10%
en-lt	SCONES (α = 0.2)	5.31%
synthetic-0.1	Softmax	1.05%
synthetic-0.1	SCONES (α = 0.2)	0.55%
synthetic-0.1	SCONES (α = 0.5)	1.10%
synthetic-0.1	SCONES (α = 1.0)	1.25%
synthetic-0.2	Softmax	1.00%
synthetic-0.2	SCONES (α = 0.2)	5.10%
synthetic-0.2	SCONES (α = 0.5)	7.65%
synthetic-0.2	SCONES (α = 1.0)	2.65%
synthetic-0.3	Softmax	0.10%
synthetic-0.3	SCONES (α = 0.2)	12.6%
synthetic-0.3	SCONES (α = 0.5)	17.3%
synthetic-0.3	SCONES (α = 1.0)	1.80%
synthetic-0.5	SCONES (α = 0.2)	25.0%
synthetic-0.5	SCONES (α = 0.5)	25.2%
synthetic-0.7	SCONES (α = 0.2)	25.9%
synthetic-0.7	SCONES (α = 0.5)	20.3%
Table 7: Fraction of sentences for which exact search did not terminate before 1M steps. For runs that are not listed here, exact search terminated within 1M steps for all sentences.
process produces the target language sentence  $\mathbf{y}$  from a source language sentence  $\mathbf{x}$  (Knight, 1999):
1. For each source word  $x_{i}$  indexed by  $i = 1,2,\ldots ,|\mathbf{x}|$ , choose the fertility  $\phi_{i}$  with probability  $n(\phi_i|x_i)$ .
2. Choose the number  $\phi_0$  of "spurious" target words to be generated from  $x_0 = \mathrm{NULL}$ , using probability  $p_1$  and the sum of fertilities from step 1.
3. Let  $m = \sum_{i=0}^{\left|\mathbf{x}\right|} \phi_i$ .
4. For each  $i = 0,1,2,\ldots ,|\mathbf{x}|$  and each  $k =$ $1,2,\dots ,\phi_{i}$  , choose a target word  $\tau_{ik}$  with probability  $t(\tau_{ik}|x_i)$
5. For each  $i = 1,2,\ldots ,|\mathbf{x}|$  and each  $k =$ $1,2,\dots ,\phi_{i}$  , choose a target position  $\pi_{ik}$  with probability  $d(\pi_{ik}|i,|\mathbf{x}|,m)$
6. For each  $k = 1,2,\ldots ,\phi_0$  , choose a position  $\pi_{0k}$  from the  $\phi_0 - k + 1$  remaining vacant positions in  $1,2,\dots ,m$  , for a total probability of  $\frac{1}{\phi_0!}$
7. Output the target sentence with words  $\tau_{ik}$  in positions  $\pi_{ik}$  ( $0 \leq i \leq |\mathbf{x}|$ ,  $1 \leq k \leq \phi_i$ ).
First, we estimate the IBM-3 model parameters using the MGIZA (Gao and Vogel, 2008) word alignment tool. Then, we sample English-like target sentences for the German source sentences following the generative story above. To control the level of uncertainty in the synthetic translation task we alter the entropies of the  $n(\cdot|\cdot)$ ,  $t(\cdot|\cdot)$ , and  $d(\cdot|\cdot)$  distributions by choosing different sampling temperatures  $\gamma \in \mathbb{R}^+$ . Instead of sampling directly from a categorical distribution  $P(\cdot)$  over categories  $\mathcal{C}$ , temperature sampling uses the following distribution:
$$
P _ {\gamma} (c) = \frac {e ^ {\log P (c) / \gamma}}{\sum_ {c ^ {\prime} \in \mathcal {C}} e ^ {\log P \left(c ^ {\prime}\right) / \gamma}} \tag {10}
$$
for each  $c \in \mathcal{C}$ . A low temperature amplifies large differences in probabilities, and thus leads to a lower entropy and less ambiguity.
C Implementation of SCONES in JAX
Fig. 10 provides an implementation of the SCONES loss function (Sec. 2) in JAX (Bradbury et al., 2018). We bound the inverse model probability (false_logprob) by  $e^{-30}$  in line 12
Figure 10: JAX implementation of the SCONES loss function.
for numerical stability. The JAX implementation generalizes the SCONES loss defined in the main paper in Eq. 6 with a label smoothing (Szegedy et al., 2016) factor  $\lambda \in [0,1]$  (1 in Fig. 10) such that the positive loss component  $\mathcal{L}_{+}(\cdot)$  becomes the following cross-entropy:
$$
\begin{array}{l} \mathcal {L} _ {+} (\mathbf {x}, \mathbf {y}, i) = - (1 - \lambda) \log P \left(z _ {\mathbf {x}, \mathbf {y} _ {<   i}} = 1 | \mathbf {x}, \mathbf {y} _ {<   i}\right) \\ - \lambda \log P \left(z _ {\mathbf {x}, \mathbf {y} _ {\leq i}} = 0 \mid \mathbf {x}, \mathbf {y} _ {<   i}\right). \tag {11} \\ \end{array}
$$
Similarly, the negative loss component  $\mathcal{L}_{-}(\cdot)$  with label smoothing can be written as:
$$
\begin{array}{l} \mathcal {L} _ {-} (\mathbf {x}, \mathbf {y}, i) = - \sum_ {w \in \mathcal {V} \backslash \{y _ {i} \}} ( \\ (1 - \lambda) \log P (z _ {\mathbf {x}, \mathbf {y} <   i \cdot w} = 0 | \mathbf {x}, \mathbf {y} <   i) \\ + \lambda \log P \left(z _ {\mathbf {x}, \mathbf {y} _ {<   i} \cdot w} = 1 | \mathbf {x}, \mathbf {y} _ {<   i})\right). \tag {12} \\ \end{array}
$$
The label smoothing extension is provided for the sake of completeness – we did not use label smoothing in any of the experiments in the main paper since it did not yield improvements in our setups.