SlowGuess/NAACL_2022_OCR huggingface dataset:
diversity during beam search; Diverse Beam (Vijayakumar et al., 2018) that improves the method of Li et al. (2016) by grouping the outputs; Hard-MOE (Shen et al., 2019) utilizing a mixture model, where different translations are obtained by controlling hidden states; Multinominal Sampling (Sun et al., 2020) that randomly selects words at each timestep to form diverse translations.
Table 1 shows the main results. Obviously, all basic models equipped with multi-candidate optimization achieve higher diversity while preserving semantic quality of translations, including conventional NMT model (Transformer) and diverse NMT models (Head Sampling and Tree2Code), showing universal effectiveness of the proposed framework. We further draw several conclusions:
1) The higher diversity among translations, the lower BLEU score they obtain, which is consistent with prior findings (Shen et al., 2019; Wu et al., 2020; Sun et al., 2020). The main reason is that the n-gram-based metric (BLEU) fails to accurately evaluate the quality of translations that syntactically differ from their references. The model-based metric (COMET) shows that our framework yields comparable translation quality compared with cor
responding basic models. More discussions about the correlation between these two automatic metrics and human evaluation are given in Section 4.6.
2) Soft constrained training exhibits better performance than hard constrained training on three basic models. The underlying reason is that soft constrained training can fully utilize candidate translations to optimize models.
3) The improvement of Transformer is smaller than that of Head Sampling and Tree2Code. We attribute this to the relatively less diversity of references generated by conventional NMT model, which limits the effects of our framework.
To investigate the effectiveness of different components in our framework, we further compare hard constrained training and soft constrained training with their several variants upon our best basic model Tree2Code on Chinese-English translation task, as concluded in Table 2:
1) Directly fine-tuning models on the whole multi-reference training set (Tree2Code+HCT without DR and QR) benefits translation quality while significantly harms its diversity, suggesting the importance of two rewards.
2) Using only the diversity reward  $(\mathrm{HCT}_d$  and
Model	DR	QR	BLEU (Top1)	BLEU	COMET	div-BLEU
Tree2Code	✘	✘	44.97	42.99	56.79	35.09
+HCT	✘	✘	45.14	43.10	57.05	33.80
✓	✘	44.50	42.16	56.40	37.23
✘	✓	44.72	43.38	56.98	34.88
✓	✓	44.31	42.52	56.88	37.72
+SCT	✓	✘	44.57	41.87	56.63	38.95
✘	✓	44.86	43.67	57.34	36.97
ED	✓	44.20	42.37	57.21	38.01
✓	CM	43.98	41.67	56.16	38.64
✓	✓	44.17	42.22	56.76	38.93
Table 2: Ablation study examined on the Chinese-English translation task. "HCT" and "SCT" individually represent hard constrained training and soft constrained training. "DR" and "QR" denote diversity and quality rewards, respectively. "ED": using edit distance as diversity reward; "CM": using COMET as quality reward.
$\mathrm{SCT}_d)$  significantly increases divBLEU while decreases BLEU and COMET. We speculate that candidate translations with high diversity but low quality lead to this phenomenon.
3) On the contrary, when we only consider the quality reward  $(\mathrm{HCT}_q$  and  $\mathrm{SCT}_q)$ , the results show high COMET but limited improvements on divBLEU. This is because candidate translations are semantically closer to references under current setting, which may harm the diversity of multi-reference pseudo corpus.
4) Jointly considering both diversity and quality (HCT and SCT) makes a better trade-off between translation diversity and quality, suggesting that both rewards are essential for diverse NMT.
5) When replacing BLEU with edit distance to define diversity reward, we observe the diversity drop of translations (SCT (ED) v.s. SCT). Meanwhile, changing reconstruction-BLEU to COMET also harms the translation quality (SCT (CM) v.s. SCT). All these confirm the advantages of our proposed two rewards.
6) We additionally report the BLEU score of Top1 candidate translation set (BLEU (Top1)). Interestingly, BLEU fluctuates more than BLEU (Top1) among different variants, which gives us a hint that the superiority of our frameworks lies in the translation on Top2-Top  $K$  variants. We will further explore this problem in Section 4.6.
Furthermore, we propose several hypotheses and experimental analyses for deeper insights to diverse NMT task, therefore explain why and how
(b)
Figure 2: Manual Evaluation Results. (a) Basic model equipped with our framework (Tree2Code+SCT) achieves comparable manual evaluation score with conventional NMT model (Transformer). (b) COMET has higher correlation with manual evaluation considering Kendall's Tau coefficient (Kendall, 1938).
our framework benefits the model performance. Specifically, we choose to analyze Transformer, Tree2Code and Tree2Code+SCT on Chinese-English translation task.
Hypothesis 1 Model-based metric COMET is more suitable for quality evaluation of diverse translations than n-gram-based metric BLEU.
Analysis Intuitively, the improvement of translation diversity may cause more mismatched n-grams between hypotheses and references, leading to a drop in n-gram-based metrics, i.e. BLEU. In order to make the evaluation more convincing, we conduct human evaluation on the translation results. Specifically, we randomly sample 300 source sentences from MT04-06 sets, and then use three models to generate diverse translations as human-evaluated cases. Next, three linguistic experts are asked to score (0-5) these translations according to the fluency and the accuracy. Each sentence is evaluated by two experts independently, and will be further reviewed by another expert if the disagreement of the former two experts exceeds 3. From Figure 2 (a), we can observe that Tree2Code+SCT gets higher manual evaluation scores than its basic model (Tree2Code), and yields comparable translation quality to the Transformer baseline. $^{8}$
Furthermore, we employ the Kendall's Tau coefficient  $\tau$  (Kendall, 1938) to quantify the correlation between automatic evaluation and manual evaluation, which is calculated over all the human-evaluated cases and defined as
$$
\tau = \frac {2}{m (m - 1)} (| C | - | D |). \tag {8}
$$
8 More details of manual evaluations are in Appendix A.
Figure 3: The confidence variances among Top  $K$  hypothesis translations at different training steps.
Here,  $m$  is the number of human-evaluated cases,  $|C|$  is the number of times a metric assigns a higher score to the "better" hypothesis and  $|D|$  is the number of times a metric assigns a higher score to the "worse" hypothesis. As illustrated in Figure 2 (b), we find that the  $\tau$  is 0.45 for BLEU and 0.58 for COMET, indicating the latter one is more suitable for evaluating translation quality of diverse NMT models than the former one.
Hypothesis 2 Multi-candidate optimization improve translation diversity by reducing the confidence variance among candidate translations.
Analysis We serve the predicted translation probability as the confidence of each candidate translation (Nguyen and O'Connor, 2015; Wang et al., 2019), and draw the confidence variance of Top  $K$  translations during training in Figure 3. When training on a single-reference corpus (Tree2Code), the confidence variance of Top  $K$  translations shows an upward trend as the pre-training step grows. Then, it will keep growing if we fine-tune the diverse NMT model with original training strategy, while starting to decline if using our training strategy. This proves that single-reference training encourages the model to fit the one-hot translation. On the contrary, multi-candidate optimization can reduce the confidence variance, and thus offer NMT model more opportunities to generate diverse translations.
Hypothesis 3 Multi-candidate optimization improves the quality of Top2-TopK translations.
Analysis We measure the quality of Top  $K$  candidate translations using COMET and manual evaluation, respectively. As shown in Figure 4, there exist large quality gaps between Top1 and the rest translations. However, after introducing our framework, the COMET and manual evaluation scores of Top2-Top  $K$  translations are improved. This shows that
Figure 4: The COMET scores of Top  $K$  hypothesis generated by Tree2Code and Tree2Code+SCT.
Figure 5: The COMET and divBLEU scores under different settings of  $K_{train} / K_{infer}$ .  $K_{train}$  represents the reference number during training, while  $K_{infer}$  indicates the candidate translation number during inference.
multi-candidate optimization can provide effective guidance for Top2-Top  $K$  candidate translations, thus improving overall quality.
Hypothesis 4 More references during training leads to better overall performance.
Analysis We explore different combinations of reference number for training and candidates translation number for inference on Tree2Code+SCT. as illustrated in Figure 5, each row  $(K_{train})$  and column  $(K_{infer})$  represents the number of generated references for training and the candidate translation number during inference, respectively. We have several interesting observations:
As for the quality, the COMET scores in upper left triangle ( $K_{train} \geq K_{infer}$ ) are higher than those in lower right triangle ( $K_{train} \leq K_{infer}$ ). This suggests that references for training should be more than candidates generated during inference for sufficient guidance. As for the diversity, it is obvious that the divBLEU scores in the upper right triangle are also higher than those in the lower left triangle. That is, divBLEU raises as  $K_{infer}$
<sup>9</sup>From another point of view, the existing optimization on mini-batch is a local fit to single-reference training data, while multi-candidate optimization narrows such quality gap by affecting the distribution of training data.
Src	依 巴拉 告诉 今日 新闻 电视台 说 ,「这 是一个 恐怖 夜晚」。
Ref	Ibarra told today 's news television station : " This is a horrible night . "
Transformer	Ibarra told today 's news television station , " This is a terrorist evening . "
Ibarra told today 's news television station , " This is a terrible evening . "
Ibarra told today 's news television station that " This is a terrorist evening . "
Tree2Code	Ibarra told today 's news television station that " This is a terrible night . "
According to a barra , today 's news television station said : " This is a terrible night . "
This is a terrible night , according to a news television station today .
Tree2Code+SCT	This is a terrible night according to Ibarra told today 's news TV station .
Ibarra told today 's news television station , " This is a terrible night . "
Speaking to news TV today , Ibarra said , " This is a terrible night . "
Figure 6: An example of NIST Chinese-English diverse translation.
and  $K_{train}$  grow. However, the improvements of diversity gradually become marginal.
From Figure 6, we can see that there are only some simple substitutions (highlighted in blue) in Transformer's results. Tree2Code generates more diverse translations, while containing more mis-translation and under-translation problems (highlighted in red). After applying our framework, Tree2Code+SCT generates better translations in terms of both diversity and quality.
In this paper, we first point out that the widely used single-reference training is not the preferred solution for diverse NMT. It causes discrepancy between training and inference, and prevents the model from learning one-to-many mapping relationships. Consequently, we propose a novel multi-candidate optimization framework which is model-irrelevant and can be compatible with any diverse NMT model. Empirical results suggest that: 1) Multi-candidate optimization is an universally effective manner on boosting the performance of diverse NMT; 2) Model-based metrics can better reflect the translation quality than its n-gram-based counterpart under diverse NMT context; 3) Multi-candidate optimization offers NMT abilities to reduce the confidence variance and improve the translation quality of candidate translations.
The project was supported by National Natural Science Foundation of China (No. 62036004, No. 61672440), Natural Science Foundation of Fujian Province of China (No. 2020J06001), Youth Innovation Fund of Xiamen (No. 3502Z20206059), National Key Research and Development Program of China (No. 2018YFB1403202), and Alibaba Group through Alibaba Innovative Research Program. We also thank the reviewers for their insightful comments.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. In ICLR 2017.
Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. 2020. Advaug: Robust adversarial augmentation for neural machine translation. In ACL 2020, pages 5961-5970.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018a. Understanding back-translation at scale. In EMNLP 2018, pages 489-500.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018b. Classical structured prediction losses for sequence to sequence learning. In NAACL-HLT 2018, pages 355-364.
Silin Gao, Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. Paraphrase augmented task-oriented dialog generation. In ACL 2020, pages 639-649.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In NeurIPS 2016, pages 820-828.
Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. 2018. Sequence-to-sequence data augmentation for dialogue language understanding. In COLING 2018, pages 1234-1245.
Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37:547-579.
Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81-93.
Huda Khayrallah, Brian Thompson, Matt Post, and Philipp Koehn. 2020. Simulated multiple reference training improves low-resource machine translation. In EMNLP 2020, pages 82-89.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In EMNLP 2004, pages 388-395.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL 2007, pages 177-180.
Marie-Anne Lachaux, Armand Joulin, and Guillaume Lample. 2020. Target conditioning for one-to-many generation. In Findings, EMNLP 2020, pages 2853-2862.
Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A simple, fast diverse decoding algorithm for neural generation. CoRR, abs/1611.08562.
Huan Lin, Liang Yao, Baosong Yang, Dayiheng Liu, Haibo Zhang, Weihua Luo, Degen Huang, and Jinsong Su. 2021. Towards user-driven neural machine translation. In ACL 2021, pages 4008-4018.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP 2015, pages 1412-1421.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a. Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. In ACL 2020, pages 4984-4997.
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. 2020b. Results of the WMT20 Metrics Shared Task. In WMT 2020, pages 688-725.
Paul Michel and Graham Neubig. 2018. Extreme adaptation for personalized neural machine translation. In ACL 2018, pages 312-318.
Khanh Nguyen and Brendan O'Connor. 2015. Posterior calibration and exploratory analysis for natural language processing models. In EMNLP 2015, pages 1587-1598.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL 2002, pages 311-318.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In ICLR 2016.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In EMNLP 2020, pages 2685-2702.
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. BLEURT: learning robust metrics for text generation. In ACL 2020, pages 7881-7892.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. In ACL 2015, pages 1715-1725.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL 2016, pages 86-96.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In ACL 2016.
Tianxiao Shen, Myle Ott, Michael Auli, and Marc'Aurelio Ranzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. In ICML 2019, pages 5719-5728.
Raphael Shu, Hideki Nakayama, and Kyunghyun Cho. 2019. Generating diverse translations with sentence codes. In ACL 2019, pages 1823-1827.
Aaron Smith, Christian Hardmeier, and Joerg Tiedemann. 2016. Climbing Mont BLEU: The Strange World of Reachable High-BLEU Translations. In EAMT 2016, pages 269-281.
Jinsong Su, Zhixing Tan, Deyi Xiong, Rongrong Ji, Xiaodong Shi, and Yang Liu. 2017. Lattice-based recurrent neural network encoders for neural machine translation. In AAAI 2017, volume 31.
Zewei Sun, Shujian Huang, Hao-Ran Wei, Xinyu Dai, and Jiajun Chen. 2020. Generating diverse translation by manipulating multi-head attention. In AAAI 2020, pages 8976-8983.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NeurIPS 2014, pages 3104-3112.
Zhixing Tan, Jinsong Su, Boli Wang, Yidong Chen, and Xiaodong Shi. 2018. Lattice-to-sequence attentional neural machine translation models. Neurocomputing, 284:138-147.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS 2017, pages 5998-6008.
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In AAAI 2018, pages 7371-7379.
Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek F. Won, and Lidia S. Chao. 2022. UniTE: Unified Translation Evaluation. In ACL 2022.
Yu Wan, Baosong Yang, Derek F Wong, Lidia S Chao, Haihua Du, and Ben CH Ao. 2020. Unsupervised neural dialect translation with commonality and diversity modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9130-9137.
Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In ACL 2020, pages 3544-3552.
Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019. Improving back-translation with uncertainty-based confidence estimation. In EMNLP-IJCNLP 2019, pages 791-802.
John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEU: training neural machine translation with semantic similarity. In ACL 2019, pages 4344-4355.
Lijun Wu, Li Zhao, Tao Qin, Jianhuang Lai, and TieYan Liu. 2017. Sequence prediction with unlabeled data by reward function learning. In *IJCAI* 2017, pages 3098-3104.
Xuanfu Wu, Yang Feng, and Chenze Shao. 2020. Generating diverse translation from model distribution with dropout. In EMNLP 2020, pages 1088-1097.
Renjie Zheng, Mingbo Ma, and Liang Huang. 2018. Multi-reference training with pseudo-references for neural translation and text generation. In EMNLP 2018, pages 3188-3197.
A Manual Evaluation Details
A.1 Score Definition
We define the quality in manual evaluation as follows: 1 - Totally incomprehensible. The content is confused and most of the target is left untranslated or unintelligible. 2 - Bad. Only a small part of target sentence can be understood, specific details are unintelligible, target is very poor in terms of readability or fluency. 3 - Neither good nor bad. Translation has notable fluency and readability issues, but it is understandable overall. 4 - Good. It is grammatically correct, but could be better in terms of style and readability. 5 - Very good. It equals quality of human translation. Only a few minor issues (like capitalization), that don't affect the readability of the target, are allowed.
A.2 Results of Top  $K$  Hypotheses
Figure 7: Manual scores of Top  $K$  hypotheses.
As illustrated in Figure 7, our framework (Tree2Code+SCT) leads to higher manual scores than basic model (Tree2Code) in terms of Top1-TopK hypotheses, which is consistent with the overall results in Figure 2 (a).