SlowGuess/NAACL_2022_OCR huggingface dataset:
Huan Lin $^{1}$  Baosong Yang $^{2}$  Liang Yao $^{2}$  Dayiheng Liu $^{2}$  Haibo Zhang $^{2}$  Jun Xie $^{2}$  Min Zhang $^{3}$  Jinsong Su $^{1,4*}$
$^{1}$ School of Informatics, Xiamen University
2Alibaba Group 3Soochow University 4Pengcheng Lab, Shenzhen
huanlin@stu.xmu.edu.cn
{yangbaosong.ybs, yaoliang.yl, liudayiheng.1dyh, zhanhui.zhb}@alibaba-inc.com
qingjing.xj@alibaba-inc.com minzhang@suda.edu.cn jssu@xmu.edu.cn
Diverse NMT aims at generating multiple diverse yet faithful translations given a source sentence. In this paper, we investigate a common shortcoming in existing diverse NMT studies: the model is usually trained with single reference, while expected to generate multiple candidate translations in inference. The discrepancy between training and inference enlarges the confidence variance and quality gap among candidate translations and thus hinders model performance. To deal with this defect, we propose a multi-candidate optimization framework for diverse NMT. Specifically, we define assessments to score the diversity and the quality of candidate translations during training, and optimize the diverse NMT model with two strategies based on reinforcement learning, namely hard constrained training and soft constrained training. We conduct experiments on NIST Chinese-English and WMT14 English-German translation tasks. The results illustrate that our framework is transparent to basic diverse NMT models, and universally makes better trade-off between diversity and quality. Our source code is available at https://github.com/DeepLearnXMU/MultiCanOptim.
Recently, neural machine translation (NMT) has achieved impressive progress in improving translation quality (Sutskever et al., 2014; Luong et al., 2015; Vaswani et al., 2017). Despite the remarkable success, NMT models still suffer from lacking translation diversity, which is essential due to the following reasons. First, similar to natural language, variability and expressiveness are the core features of translations. Second, only focusing on increasing translation accuracy during training will bias the NMT model to common phrases, exacerbating data sparsity (Khayrallah et al., 2020).
In conclusion, improving translation diversity is a promising direction in NMT community.
To achieve diverse NMT, several studies have explored various training or decoding strategies, including: 1) constraining decoding with diversity regularization (Li et al., 2016; Vijayakumar et al., 2018), 2) sampling from the mixture of models (Shen et al., 2019; Wu et al., 2020), and 3) conditioning decoding with diverse signals (Shu et al., 2019; Sun et al., 2020). However, all these approaches train models on single-reference corpus, while expecting them to generate multiple candidate translations during inference. We argue that such discrepancy between training and inference prevents the models from learning one-to-many relations efficiently. Firstly, since the predictions of NMT models are encouraged to fit the one-hot distribution of single-reference corpus, the model confidence of generating Top1 candidate translations will be much larger than that of the rest candidates, limiting translation diversity. Secondly, only one reference is used to get optimization signal at the training time, resulting in significant quality drops of Top2-Top  $K$  translations. One direct way addressing these issues is to train the models using multi-reference training data. Nevertheless, its construction is quite expensive and thus impractical.
To overcome the above issues, in this paper, we propose a novel multi-candidate optimization framework for diverse NMT. The basic idea is to guide an NMT model to learn diverse translation from its candidate translations based on reinforcement learning (RL). During training, the model generates multiple candidate translations, of which rewards are quantified according to their diversity and quality. Since directly optimizing model parameters with the above two rewards involves backpropagating through discrete decoding decision, we explore two specific methods to train the diverse NMT model: 1) Hard constrained training. We transform the rewards to discrete scalars, prevent
* Jinsong Su is the corresponding author. This work was done when Huan Lin was interning at DAMO Academy, Alibaba Group.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2622 - 2632
ing the model from learning those candidate translations with low rewards. 2) Soft constrained training. We introduce minimum risk training (MRT) to minimize the risks of obtaining diversity and quality rewards. Compared with previous works, our proposed framework reduces the confidence variance among candidate translations and improves the quality of Top2-Top  $K$  translations during inference, achieving better performance in terms of both diversity and quality. Overall, the major contributions of our work are three-fold:
- We point out and empirically verify that the discrepancy between training and inference in diverse NMT negatively impacts the translation diversity and quality.
- We propose a novel multi-candidate optimization framework based on RL, enabling an NMT model to learn one-to-many relations from its candidate translations. Our framework is transparent to model architecture, thereby can be employed individually or complemented to existing diverse NMT models.
- Extensive experimental results on NIST Chinese-English and WMT14 English-German datasets show that our framework can efficiently smooth the confidence distribution and raise the quality of Top2-Top  $K$  candidate translations, surpassing several commonly-used diverse NMT models.
Diverse NMT. Improving translation diversity has been a hot topic in NMT community in recent years, such as lattice-based NMT (Su et al., 2017; Tan et al., 2018) and personalized NMT (Michel and Neubig, 2018; Lin et al., 2021). Existing works for diverse NMT can be categorized into three major categories. The first category produces diverse translations by applying diversity regularization to decoding algorithm (Li et al., 2016; Vijayakumar et al., 2018). The second category improves translation diversity by sampling from a mixture of models. In this aspect, Shen et al. (2019) adopt conditional mixture models to control the generation of translations. Wu et al. (2020) derive a large number of models with Bayesian modeling, which are sampled to generate diverse translations. Unlike the former two categories, the third one attempts to condition the decoding procedure with diverse signals. Typically, Shu et al. (2019) use syntactic codes to condition translation process.
Further, Lachaux et al. (2020) replace the syntactic codes with latent domain variables derived from target sentences, which is more computationally efficient. Sun et al. (2020) sample the encoder-decoder attention heads of Transformer to affect source word selection. Despite their successes, an obstacle of these approaches lies in the discrepancy between training and inference, that is, learning diverse translations from a single-reference corpus. This enlarges the confidence and quality gaps among candidate translations, limiting the potential of diverse NMT models.
Multi-Candidate Optimization in Natural Language Generation. Since single-reference corpus is insufficient to model one-to-many relations in natural language generation (NLG), researchers have introduced multi-candidate optimization to NLG tasks such as image captioning and paraphrasing. Most of representative works among them generate pseudo training references and focus on improving diversity (Zheng et al., 2018; Hou et al., 2018; Gao et al., 2020). Conversely, in NMT community, previous studies on multicandidate optimization mainly aim at improving low-resource translation quality rather than diversity, which is similar to other data augmentation methods in NMT, such as back-translation and forward-translation (Sennrich et al., 2016; Edunov et al., 2018a; Cheng et al., 2020; Wan et al., 2020). For example, Khayrallah et al. (2020) improve the translation quality of low-resource language pairs by sampling paraphrases of the reference sentence. Different from these tasks, diverse NMT is more challenging since it requires the generation results to be accurate as well as diverse. For better balancing quality and diversity, we propose a novel multi-candidate optimization framework with RL.
Reinforcement Learning in NMT. Reinforcement learning (RL) has become an appealing path for advancement in NMT, as it firstly allows to optimize non-differentiable objectives, and secondly reduces exposure bias in auto-regressive sequence generators. To this end, various methods have been proposed. In Ranzato et al. (2016), Wu et al. (2017) and Edunov et al. (2018b), the authors employ the REINFORCE algorithm to optimize models with metric-based reward (i.e. senetnce-level BLEU). Different from them, He et al. (2016) propose to train two reverse NMT models through a dual-learning mechanism. And Bahdanau et al. (2017)
Figure 1: Our multi-candidate optimization framework: the diverse NMT model generates  $K$  candidate translations to receive individual diversity and quality rewards, which are then used for model optimization.
use actor-critic method that predicts the reward by a critic network. In this work, we follow Shen et al. (2016), Wieting et al. (2019) and Wang and Sennrich (2020) that adopt minimum risk training (MRT) to minimize the reward during training. To the best of our knowledge, our work is the first attempt employing RL to model one-to-many relations for diverse NMT.
3 Multi-Candidate Optimization Framework
As a significant extension of conventional NMT, given a source sentence, a diverse NMT model aims at producing a set of different candidate translations. Similar to conventional NMT, the most commonly-used training strategy of diverse NMT is to minimize the training objective based on maximum likelihood estimation (MLE):
$$
\mathcal {L} _ {m l e} (\theta) = - \sum_ {n = 1} ^ {N} \log P _ {\theta} (y ^ {(n)} | x ^ {(n)}), \tag {1}
$$
where  $(x^{(n)},y^{(n)})$  is the  $n$ -th instance in the training corpus of size  $N$ , and  $P_{\theta}(y^{(n)}|x^{(n)})$  denotes the translation model with parameters  $\theta$ . It can be said that the one-to-many relations are the basis of diverse NMT. However, as mentioned above, the model is unable to effectively learn such relations from a single-reference training corpus. Accordingly, the discrepancy between training and inference has become a bottleneck limiting the performance of diverse NMT models.
To deal with the above issue, we propose a multicandidate optimization framework based on reinforcement learning (RL). As shown in Figure 1, a diverse NMT model generates  $K$  candidate translations using its original method as additional references during training. Particularly, given a source sentence  $x^{(n)}$ , the model picks an action each time
it generates a candidate translation  $y_{k}^{(n)}$ . Diversity and quality rewards of  $y_{k}^{(n)}$  are observed once it is completed, which are then used to optimize model parameters. Please note that our framework is model-irrelevant and thus can be compatible with any diverse NMT model. Next, we will introduce the reward computation and training procedure in following subsections.
3.1 Reward Computation
Conventional RL in NMT usually takes sentence-level BLEU (Papineni et al., 2002) as reward. However, in diverse NMT, ideal translations should be semantically equal to their source sentences, as well as diverse from each other. To this end, we exploit two highly generic evaluation metrics to encourage the diversity and quality of candidate translations:
Diversity Reward. This reward measures the difference between each candidate translation  $\hat{y}_k^{(n)}$  and other translations, including the original reference  $y^{(n)}$  and the rest candidate translations  $\{\hat{y}_{k'}^{(n)}\}_{k' = 1,k'\neq k}^K$ . We can model the difference with arbitrary method such as Jaccard distance (Jaccard, 1901), edit distance (Levenshtein et al., 1966) or BLEU (Papineni et al., 2002). Here, we serve BLEU as the similarity assessment since it is less sensitive to sentence length. Formally, the diversity reward of  $\hat{y}_k^{(n)}$  is defined as follows:
$$
\mathrm {D R} \left(\hat {y} _ {k} ^ {(n)}\right) = 1 - \text {B L E U} _ {\mathrm {s}} \left(\hat {y} _ {k} ^ {(n)}, \left\{y ^ {(n)} \right\} \cup \left\{\hat {y} _ {k ^ {\prime}} ^ {(n)} \right\} _ {k ^ {\prime} = 1, k ^ {\prime} \neq k} ^ {K}\right), \tag {2}
$$
where  $\mathrm{BLEU}_{\mathrm{s}}(*)$  indicates sentence-level  $\mathrm{BLEU}^1$ .
Quality Reward. One common approach to evaluate the quality of each candidate translation is to compare it with the corresponding reference. However, such a method biases the model to candidate translations syntactically similar to original references, therefore harms translation diversity. In order to tackle this problem, it is better to use semantic evaluation metrics such as reconstruction-BLEU (He et al., 2016), COMET (Rei et al., 2020), BLEURT (Sellam et al., 2020), UniTE (Wan et al., 2022) and so on. Here, we choose reconstruction-BLEU, which uses a reverse NMT model trained on initial single-reference corpus to translate each candidate  $\hat{y}_k^{(n)}$  back to a source sentence  $\hat{x}_k^{(n)}$ , and
<sup>1</sup>We calculate sentence-level BLEU with SacreBLEU in https://github.com/mjpost/sacrebleu
then evaluates the BLEU score between  $\hat{x}_k^{(n)}$  and  $x^{(n)}$ :
$$
\mathrm {Q R} \left(\hat {y} _ {k} ^ {(n)}\right) = \mathrm {B L E U} _ {\mathrm {s}} \left(\hat {x} _ {k} ^ {(n)}, x ^ {(n)}\right). \tag {3}
$$
Compared with COMET and BLEURT that pretrained using out-of-domain data, the reconstruction model in our metric is trained on the same corpus as NMT model, thereby reducing the impact of domain inconsistency on evaluation accuracy.
The calculations of diversity and quality rewards involve undifferentiated discrete operations, leading to a challenge in the back-propagation at training time. To address this issue, we explore two approaches, separately termed as hard constrained training and soft constrained training.
Hard Constrained Training (HCT). An intuitive idea is utilizing translations only with high diversity and quality rewards. Along with this strategy, we pair  $x^{(n)}$  with its candidate translations  $\{\hat{y}_k\}_{k=1}^K$  and original reference  $y^{(n)}$  to form a new multi-reference training instance, then optimize the model with MLE objective:
$$
\begin{array}{l} \mathcal {L} _ {h c t} (\theta) = \sum_ {n = 1} ^ {N} \log P _ {\theta} \left(y ^ {(n)} \mid x ^ {(n)}\right) \tag {4} \\ + \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {K} \alpha_ {k} \cdot \log P _ {\theta} (\hat {y} _ {k} ^ {(n)} | x ^ {(n)}), \\ \end{array}
$$
$$
\alpha_ {k} = \left\{ \begin{array}{l l} 0 & \mathrm {D R} (\hat {y} _ {k} ^ {(n)}) <   \delta_ {d} \text {o r} \mathrm {Q R} (\hat {y} _ {k} ^ {(n)}) <   \delta_ {q} \\ 1 & \text {e l s e .} \end{array} \right.
$$
Here,  $\alpha_{k}$  is used to re-weight the training objective of candidate translations.  $\delta_{d}$  and  $\delta_{q}$  indicate corpus-level diversity and quality rewards of the initial model (i.e. a pre-trained diverse NMT model) on development set, respectively. Hard constrained training is easy to implement. However, the candidate translations are still far from utilization. First, all candidate translations are treated equally although they possess different diversity and quality. Second, some candidate translations are totally discarded although they may provide guidance for the model training.
Soft Constrained Training (SCT). To fully utilize all candidate translations, we employ MRT to directly optimize diversity and quality rewards.
We choose MRT since it does not require extra parameters compared with other RL techniques (He et al., 2016; Bahdanau et al., 2017). Similar to hard constrained training, a multi-reference training instance consists of each original source sentence and all its candidate translations. Specifically, we define the losses of diversity and quality rewards as  $1 - \mathrm{DR}(\hat{y}_k^{(n)})$  and  $1 - \mathrm{QR}(\hat{y}_k^{(n)})$ , respectively. Please refer to Equations 2 and 3 for definitions of  $\mathrm{DR}(*)$  and  $\mathrm{QR}(*)$ . We then apply these two losses to softly weight the posterior distribution  $P_{\theta}(\hat{y}_k^{(n)}|x^{(n)})$ . The goal is to minimize two risks:
$$
\mathcal {R} _ {d} (\theta) = \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {K} P _ {\theta} \left(\hat {y} _ {k} ^ {(n)} \mid x ^ {(n)}\right) \cdot \left(1 - \mathrm {D R} \left(\hat {y} _ {k} ^ {(n)}\right)\right), \tag {5}
$$
$$
\mathcal {R} _ {q} (\theta) = \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {K} P _ {\theta} \left(\hat {y} _ {k} ^ {(n)} \mid x ^ {(n)}\right) \cdot \left(1 - \mathrm {Q R} \left(\hat {y} _ {k} ^ {(n)}\right)\right). \tag {6}
$$
Completely different to MLE (Equation 1) that aims at reducing the discrepancy between a candidate translation and its single reference, MRT encourages the model to maximize rewards via generating more diverse and accurate translations. Following Wieting et al. (2019), we first pre-train the diverse NMT model with  $\mathcal{L}_{mle}(\theta)$ , and then fine-tune it with the combination of  $\mathcal{L}_{mle}(\theta)$ ,  $\mathcal{R}_d(\theta)$  and  $\mathcal{R}_q(\theta)$ :<sup>2</sup>
$$
\mathcal {L} _ {s c t} (\theta) = \mathcal {L} _ {m l e} (\theta) + \mathcal {R} _ {d} (\theta) + \mathcal {R} _ {q} (\theta). \quad (7)
$$
Consequently, soft constrained training possesses two advantages comparing with its hard counterpart: 1) It provides more guidance for model training by exploiting all candidate translations; 2) It directly incorporates the diversity and quality rewards into training objective, thereby distinguishing different effects of candidate translations.
In this section, we carry out several groups of experiments to investigate the effectiveness of our proposed framework on Chinese-English and English-German translation tasks.
2We have also tried weighted sum of three terms. Results shows no significant difference with the non-weighted version.
In order to make comparison with existing diverse NMT models (Li et al., 2016; Vijayakumar et al., 2018; Shen et al., 2019; Sun et al., 2020), we build multi-reference corpus and examine our framework on translation tasks commonly used in previous diverse NMT studies:
- NIST Chinese-to-English. This training set contains about 1.34M news sentence pairs.<sup>3</sup> We use MT03 as development set and MT04, MT05, MT06 as test sets, and report the average scores on test sets as final results.
WMT14 English-German. This training data consists of 4.5M sentence pairs<sup>4</sup>. We use the newstest 2013 as the development set, and the newstest 2014 as the test set.
For the above two datasets, We adopt Moses tokenizer (Koehn et al., 2007) to deal with English and German sentences, and segment the Chinese sentences with the Stanford Segmentor. Following common practices, we employ byte pair encoding (Sennrich et al., 2015) with 32K merge operations to segment words into subword units. We use a joint dictionary for English-German translation task while assigning individual vocabularies for Chinese-English translation task. In addition, we remove the examples in datasets where the length of source or target sentence exceeds 100 words.
We develop all diverse NMT models on Transformer-base (Vaswani et al., 2017) $^6$ . At the pre-training stage, we set the batch size as 32,768 tokens for NIST and 12,500 tokens for WMT14. Other configurations are identical to common settings in previous studies (Vaswani et al., 2017; Shu et al., 2019; Sun et al., 2020). During finetuning, we keep other settings consistent with the pre-training stage, but reduce the learning rate by a factor of 10. Using early-stopping strategy, we evaluate the model every 500 steps and stop training if the translation diversity or quality on development set does not raise for 10 consecutive evaluations. Considering computational efficiency, we set  $K$  as 3 in all our experiments as default.
We use the following three metrics to assess the quality and diversity of candidate translation sets.
- BLEU. Following previous studies (Shen et al., 2019; Wu et al., 2020; Sun et al., 2020), we use average BLEU of  $K$  candidate translation sets to evaluate translation quality.
COMET. It is based on pre-trained language model and has shown higher correlations with human judgements in a variety of metrics tasks (Mathur et al., 2020b). We adopt it since n-gram-based metrics may fail to robustly match paraphrases and capture distant dependencies, resulting in a diverse translation with high faithfulness and fluency but a low BLEU score (Smith et al., 2016; Mathur et al., 2020a). Similar to BLEU, we report the average COMET score of  $K$  candidate translation sets as default. Particularly, we normalize the results of COMET with sigmoid function.
divBLEU. We define divBLEU to measure the differences among  $K$  candidate translations on a test set of size  $S$ :
$$
1 - \mathrm {B L E U} _ {\mathrm {c}} (\{\hat {y} _ {k} ^ {(s)} \} _ {s = 1} ^ {S}, \{\hat {y} _ {k ^ {\prime}} ^ {(s)} \}),
$$
where  $1 \leq k \leq K, 1 \leq k' \leq K, k \neq k'$ . The second term denotes pairwise-BLEU (Shen et al., 2019; Wu et al., 2020; Sun et al., 2020) that compares each candidate translation set with each other.
All BLEU metrics used in this paper are case-sensitive. Concretely, corpus-level BLEU is calculated with Moses script<sup>7</sup>. To raise the reliability, we run all models three times with different random seeds and report the average results.
We apply our framework to the following models:
- Transformer (Vaswani et al., 2017) refers to the baseline. We pick its Top  $K$  hypotheses in beam search as the diverse translations.
- Tree2Code (Shu et al., 2019) generates diverse candidates with various syntactic codes.
- Head Sampling (Sun et al., 2020) generates different words by sampling attention heads.
We also display the reported results of several dominant diverse NMT models on the same datasets: Diverse Decoding (Li et al., 2016) employing diversity regularization terms to encourage translation
$^{7}$ https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl
3The training set is a combination of LDC2002E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08 and LDC2005T06.
4The preprocessed data can be found and downloaded from http://nlp.stanford.edu/projects/nmt/.
<sup>5</sup>https://nlp.stanford.edu/
Our codes are implemented upon https://github.com/facebookresearch/XLM/.
Model	Chinese-English	English-German
BLEU	COMET	divBLEU	BLEU	COMET	divBLEU
Existing Diverse NMT Systems
Diverse Decoding (Li et al., 2016)	43.18	-	19.76	25.27	-	21.43
Diverse Beam (Vijayakumar et al., 2018)	39.58	-	41.93	23.27	-	33.87
HardMOE (Shen et al., 2019)	38.54	-	39.30	23.22	-	31.97
Multinominal Sampling (Sun et al., 2020)	20.62	-	89.28	11.99	-	87.16
Head Sampling (Sun et al., 2020)	42.66	-	33.82	25.62	-	21.34
Our Implementations
Transformer (Vaswani et al., 2017)	44.67	57.83	13.89	26.29	55.03	19.06
+HCT	44.23	57.72	14.01	26.07	55.01	19.76
+SCT	43.78	57.41	\(14.98^‡\)	26.01	54.85	\(20.33^†\)
Tree2Code (Shu et al., 2019)	42.99	56.79	34.80	25.43	53.52	26.11
+HCT	42.52	56.88	\(37.57^‡\)	25.15	53.74	\(28.71^‡\)
+SCT	42.26	57.06	\(38.78^‡\)	25.40	54.19	\(29.98^‡\)
Head Sampling (Sun et al., 2020)	42.52	56.40	34.02	25.16	53.28	21.24
+HCT	42.46	56.58	\(36.38^‡\)	25.03	53.66	\(23.55^†\)
+SCT	42.02	57.03	\(37.56^‡\)	24.91	54.02	\(24.07^‡\)
Table 1: Main results on NIST Chinese-English (average scores of MT04, MT05 and MT06) and WMT14 English-German tasks. “HCT” and “SCT” individually indicate hard constrained training and soft constrained training. “BLEU” and “COMET” denote translation quality assessed by the n-gram-based and the model-based metrics, respectively. “divBLEU” indicates the diversity among candidates. We also calculate p-value with bootstrap sampling (Koehn, 2004) to estimate statistical significance.  $\ddagger/\dagger$ : significantly better than corresponding basic models ( $p < 0.01/0.05$ ). All results are derived from 3 independent runs.