SlowGuess/ACL_2020_OCR huggingface dataset:
Content tokens are more likely to suffer from miscalibration. From Figure 7 we find that the most relative changes of content tokens (i.e., "Noun", "Verb" and "Adj") are positive, while most of the relative changes of the content-free tokens (i.e., "Prep.", "Dete.", "Punc.", "Others") are negative. Among content tokens, the verbs ("Verb") face the over-estimation problem instead of the underestimation problem. Surprisingly, the adjectives ("Adj") suffer from under-estimation problem on large data (e.g., En-De and Zh-En).
5.5 Word Granularity
BPE segmentation is the preliminary step for current NMT systems, which may segment some
Figure 7: Effect of POS tags on miscalibration.
(a) Over-Estimation
Figure 8: Effect of word granularity on miscalibration.
words into sub-words. To explore the effect of word granularity on the miscalibration of NMT models, we divide the tokens after BPE segmentation into two categories: Sub-Words that are divided into word fragments by BPE (e.g., with “@@”), and Full Words that are not divided by BPE. Figure 8 depicts the results.
Sub-words suffer more from miscalibration, while full words are less likely to be miscalibrated. The relative changes of sub-words are all positive for both over- and under-estimation, while those of full words are all negative. Sennrich et al. (2016) showed that BPE addresses the open-vocabulary translation by encoding rare and unknown words as sequences of sub-word units. Our results confirm their claim: the behaviors of sub-words and full words correlate well with those of low- and high-frequency tokens respectively.
6 Revisiting Advances in Architecture and Regularization
Guo et al. (2017) have revealed that the miscalibration on classification tasks is closely related to lack of regularization and increased model size. In this section we check whether the conclusion holds on the inference of NMT models, which belong to a family of structured generation.
<sup>3</sup>If a token is a sub-word segmented by BPE, the token shares the syntactic role of the full word that it belongs to. <sup>4</sup>https://nlp.stanford.edu/software/tagger.shtml <sup>5</sup>http://www.phontron.com/kytea/
Label Smoothing	Dropout	Beam Size = 10	Beam Size = 100
BLEU	ECE	Over.	Under.	BLEU	ECE	Over.	Under.
×	×	23.03	25.49	58.3%	9.6%	22.90	26.46	59.4%	9.3%
✓	×	24.51	14.99	42.3%	17.3%	24.58	15.97	42.8%	16.9%
×	✓	27.52	20.75	52.3%	10.1%	26.93	22.57	53.6%	9.8%
✓	✓	27.65	14.26	39.7%	14.1%	27.68	14.75	40.1%	14.2%
GRADUATED	✓	27.76	5.07	29.1%	31.6%	28.07	5.23	29.5%	31.4%
Table 3: Results of label smoothing and dropout on the En-De task. "Over:" and "Under:" denote over-estimation and under-estimation, respectively.
(a) None
(b) Vanilla
Figure 9: Reliability diagrams of different label smoothing strategies: (a) no label smoothing; (b) vanilla label smoothing; (c) graduated label smoothing. The results are reported on the WMT14 En-De translation task.
(c) Graduated
One criticism of NMT inference is that the translation performance inversely decreases with the increase of search space (Tu et al., 2017). Quite recently, Kumar and Sarawagi (2019) claimed that this problem can be attributed to miscalibration. Accordingly, we also report results on large beam size and find that reducing miscalibration can improve the NMT performance in large beam size.
6.1 Regularization Techniques
We revisit two important regularization techniques that directly affect confidence estimation:
- Label Smoothing (Szegedy et al., 2016): distributing a certain percentage of confidence from the ground truth label to other labels uniformly in training.
- Dropout (Hinton et al., 2012): randomly omitting a certain percentage of the neural networks on each training case, which has been shown effective to prevent the over-fitting problem for large neural networks.
For comparison, we disable label smoothing or dropout to retrain the model on the whole training set. The results are shown in Table 3. We find that label smoothing improves the performance by
greatly reducing the over-estimation, at the cost of increasing the percentage of under-estimation error. Dropout alleviates the over-estimation problem, and does not aggravate under-estimation. Although label smoothing only marginally improves performance on top of dropout, it is essential for maintaining the translation performance in larger search space (i.e., Beam Size = 100).
As seen from Table 3, reducing ECE can only lead to marginal BLEU gains. We attribute this phenomenon to the fact that ECE is another metric to evaluate NMT models, which is potentially complementary to BLEU. Accordingly, ECE is not necessarily strictly negatively related to BLEU.
Graduated Label Smoothing Inspired by this finding, we propose a novel graduated label smoothing approach, in which the smoothing penalty for high-confidence predictions is bigger than that for low-confidence predictions. We firstly use the model trained by vanilla label smoothing to estimate the confidence of each token in the training set, then we set the smoothing penalty to 0.3 for tokens with confidence above 0.7, 0.0 for tokens with confidence below 0.3, and 0.1 for the remaining tokens.
As shown in Table 3, the graduated label smoothing can improve translation performance by alle
Enc.	Dec.	Para.	Beam Size = 10	Beam Size = 100
BLEU	ECE	Over.	Under.	BLEU	ECE	Over.	Under.
BASE	BASE	88M	27.65	14.26	39.7%	14.1%	27.68	14.75	40.1%	14.2%
DEEP	DEEP	220M	28.86	14.99	40.3%	14.1%	28.64	15.55	41.8%	14.0%
DEEP	BASE	145M	29.09	14.28	39.6%	14.1%	29.29	14.53	39.6%	14.2%
WIDE	WIDE	264M	28.66	16.09	42.3%	12.6%	28.42	17.22	43.2%	12.5%
WIDE	BASE	160M	28.97	14.83	39.7%	13.6%	29.09	15.06	39.8%	13.7%
Table 4: Effect of model size by enlarging encoder ("Enc.") and decoder ("Dec.") on the En-De dataset.
viating inference miscalibration, and the improvement is more significant in large beam size. Figure 9 shows the reliability diagrams of different label smoothing strategies. The graduated label smoothing can effectively calibrate the predictions with  $0.4 \leq$  confidence  $\leq 0.8$ , while is less effective for low- (i.e.,  $< 0.4$ ) and high-confidence (i.e.,  $> 0.8$ ) predictions. We believe that the design of more advanced techniques to solve this problem is a worthwhile future direction of research.
6.2 Increased Model Size
The model size of NMT models has increased significantly recently (Bahdanau et al., 2015; Vaswani et al., 2017; Wang et al., 2019). We evaluated the inference calibration of models with different sizes. We increase model size in the following two ways:
- Deeper model: both the encoder and the decoder are deepened to 24 layers;
- Wider model: the hidden size of the encoder and the decoder is widened to 1024.
The BLEU score and inference ECE of different models are shown in Table 4.
Increasing model size negatively affects inference calibration. We find that increasing both the encoder and the decoder increases the inference calibration error despite increasing the BLEU, confirming the finding of Guo et al. (2017) that increased model size is closely related to model miscalibration. This leads to a performance drop in a larger search space (i.e., Beam Size = 100).
Only enlarging the encoder improves translation quality while maintaining inference calibration. As the decoder is more directly related to the generation, it is more likely to result in miscalibration. In order to maintain the performance improvement and do not aggravate over-estimation, we propose to only increase the size of encoder and keep the decoder unchanged. Results in Table 4 indicate
that only enlarging the encoder can achieve better performance with fewer parameters compared to enlarging both the encoder and the decoder. In a larger search space (i.e., Beam Size = 100), models with high inference ECE will generate worse translations while models with low inference ECE can achieve improved translation performance.
Although NMT models are well-calibrated in training, we observe that they still suffer from miscalibration during inference because of the discrepancy between training and inference. Through a series of in-depth analyses, we report several interesting findings which may help to analyze, understand and improve NMT models. We revisit recent advances and find that label smoothing and dropout play key roles in calibrating modern NMT models. We further propose graduated label smoothing that can reduce the inference calibration error effectively. Finally, we find that increasing model size can negatively affect the calibration of NMT models and this can be alleviated by only enlarging the encoder. As well-calibrated confidence estimation is more likely to establish trustworthiness with users, we plan to apply our work to interactive machine translation scenarios in the future.
We thank all anonymous reviewers for their valuable comments and suggestions for this work. This work was supported by the National Key R&D Program of China (No. 2017YFB0202204), National Natural Science Foundation of China (No. 61925601, No. 61761166008, No. 61772302), Beijing Advanced Innovation Center for Language Resources (No. TYR17002), and the NExT++ project supported by the National Research Foundation, Prime Ministers Office, Singapore under its IRC@Singapore Funding Initiative.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Shu-Tao Xia. 2019. Adaptive Regularization of Labels. arXiv.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In NAACL.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In ICML.
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207.0580.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Volodymyr Kuleshov and Percy S Liang. 2015. Calibrated structured prediction. In NeurIPS.
Aviral Kumar and Sunita Sarawagi. 2019. Calibration of Encoder Decoder Models for Neural Machine Translation. In ICLR Debugging Machine Learning Models Workshop.
Aditya Krishna Menon, Xiaoqian J Jiang, Shankar Vembu, Charles Elkan, and Lucila Ohno-Machado. 2012. Predicting accurate probabilities with a ranking loss. In ICML, volume 2012, page 703. NIH Public Access.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. 2019. When Does Label Smoothing Help? In NeurIPS.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In AAAI.
Khanh Nguyen and Brendan O'Connor. 2015. Posterior calibration and exploratory analysis for natural language processing models. In EMNLP.
Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In ICML, pages 625-632.
Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In ICML.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In ACL.
John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61-74.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In CVPR.
Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In AAAI.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In ACL.
Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019. Improving back-translation with uncertainty-based confidence estimation. In EMNLP.
Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018. Beyond error propagation in neural machine translation: Characteristics of language also matter. In EMNLP.
Dong Yu, Jinyu Li, and Li Deng. 2011. Calibration of confidence measures in speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 19(8):2461-2473.
Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699. ACM.
Wenliang Zhong and James T Kwok. 2013. Accurate probability calibration for multiple classifiers. In Twenty-Third International Joint Conference on Artificial Intelligence.