SlowGuess/NAACL_2022_OCR huggingface dataset:
we train AT for 150K steps and train NAT for 300K steps with dropout 0.2. On WMT16 En  $\leftrightarrow$  Ro and IWSLT14 De  $\rightarrow$  En, we train AT for 18K steps and train NAT for 150K steps with dropout 0.3. We finetune NAT for 3K steps. The learning rate warms up to  $5 \cdot 10^{-4}$  within 10K steps in pretraining and warms up to  $2 \cdot 10^{-5}$  within 500 steps in RL finetuning, and then decays with the inverse square-root schedule. We average the last 5 checkpoints to obtain the final model. We use GeForce RTX 3090 GPU for the training and inference. We implement our models based on the open-source framework of fairseq (Ott et al., 2019).
Knowledge Distillation For baseline NAT models, we follow previous works on NAT to apply sequence-level knowledge distillation (Kim and Rush, 2016) to make the target more deterministic. Our method applies diverse distillation with  $k = 3$  by default, that is, we use SeedDiv to generate 3 reference sentences for each source sentence.
Beam Search Decoding For autoregressive models, we use beam search with beam width 5 for the inference. For NAT, the most straightforward way is to generate the sequence with the highest probability at each position. Furthermore, CTC-based models also support beam search decoding optionally combined with n-gram language models (Kasner et al., 2020). Following Gu and Kong (2020), we use beam width 20 combined with a 4-gram language model to search the target sentence, which can be implemented efficiently in  $\mathbf{C} + + ^2$ .
We compare the performance of DDRS and existing methods in Table 2. Compared with the competitive CTC baseline, DDRS achieves a strong improvement of more than 1.5 BLEU on average, demonstrating the effectiveness of diverse distillation and reference selection. Compared with exist-
$^{2}$ https://github.com/parlance/ctcdecode
Models	BLEU	Speedup
Transformer-base	27.51	1.0×
Transformer-big (teacher)	28.64	0.9×
R2L-Transformer-big (teacher)	27.96	0.9×
DDRS-base	27.99	14.7×
+beam20&lm	28.95	5.0×
DDRS-big	28.84	14.1×
+beam20&lm	29.82	4.8×
Table 3: Performance of DDR5 on the test set of WMT14 En-De with Transformer-big for distillation.
Models	En-Fr	De-En
Transformer	40.15	34.17
CTC	38.40	31.37
DDRS	39.91	33.12
DDRS +beam20&lm	40.59	34.74
ing methods, DDRS beats the state-of-the-art for one-pass NAT on all benchmarks and beats the autoregressive Transformer on most benchmarks with  $14.7 \times$  speedup over it. The performance of DDRS is further boosted by beam search and 4-gram language model, which even outperforms all iterative NAT models with only one-pass decoding. Notably, on WMT16 En  $\leftrightarrow$  Ro, our method improves state-of-the-art performance levels for NAT by over 1 BLEU. Compared with autoregressive models, our method outperforms the Transformer with knowledge distillation, and meanwhile maintains  $5.0 \times$  speedup over it.
We further explore the capability of DDRs with a larger model size and stronger teacher models. We use the big version of Transformer for distillation, and also add 3 right-to-left (R2L) teachers to enrich the references. We respectively use Transformer-base and Transformer-big as the NAT architecture and report the performance of DDRs in Table 3. Surprisingly, the performance of DDRs can be further greatly boosted by using a larger model size and stronger teachers. DDRs-big with beam search achieves 29.82 BLEU on WMT14 En-De, which is close to the state-of-the-art performance of autoregressive models on this competitive dataset and improves the state-of-the-art performance for NAT by over 1 BLEU with only one-pass decoding.
We also evaluate our approach on a large-scale dataset WMT14 En-Fr and a small-scale dataset IWSLT14 De-En. Table 4 shows that DDR5 still
Table 4: Performance of DDR5 on the test sets of WMT14 En-Fr and IWSLT14 De-En.
\( \mathcal{L}_{sum} \)	\( \mathcal{L}_{mid} \)	\( \mathcal{L}_{max} \)	λ	Reward	\( \mathrm{BLEU}_1 \)	\( \mathrm{BLEU}_2 \)
✓	24.61	25.97
✓	25.23	26.90
✓	25.31	26.88
✓	✓	0	25.41	26.99
✓	✓	✓	1/3	25.48	27.13
✓	✓	✓	2/3	25.59	27.18
✓	✓	1	25.45	27.09
✓	✓	✓	2/3	random	25.63	27.26
✓	✓	✓	2/3	average	25.79	27.51
✓	✓	✓	2/3	maximum	25.92	27.60
Table 5: Ablation study on WMT14 En-De with different combinations of techniques.  $\mathrm{BLEU}_1$  is the BLEU score on validation set.  $\mathrm{BLEU}_2$  is the BLEU score on test set. The validation performance of CTC baseline is 24.57 BLEU.  $\lambda$  is the length of the first training stage. random means the reward of a random reference, average means the average reward, and maximum means the maximum reward among all references.
achieves considerable improvements over the CTC baseline and DDRs with beam search can outperform the autoregressive Transformer.
In Table 5, we conduct an ablation study to analyze the effect of techniques used in DDRs. First, we separately use the loss functions defined in Equation 7, Equation 8 and Equation 9 to train the model. The summation loss  $\mathcal{L}_{sum}(\theta)$  has a similar performance to the CTC baseline, showing that simply using multiple references is not helpful for NAT due to the increase of data complexity. The other two losses  $\mathcal{L}_{mid}(\theta)$  and  $\mathcal{L}_{max}(\theta)$  achieve considerable improvements to the CTC baseline, demonstrating the effectiveness of reference selection.
Then we use different  $\lambda$  to verify the effect of the annealing schedule. With the annealing schedule, the loss is a combination of the three losses but performs better than each of them. Though the summation loss  $\mathcal{L}_{sum}(\theta)$  does not perform well when used separately, it can play the role of pretraining and improve the final performance. When  $\lambda$  is  $2/3$ , the annealing schedule performs the best and improves  $\mathcal{L}_{max}(\theta)$  by about 0.3 BLEU.
Finally, we verify the effect of the reward function during the fine-tuning. When choosing a random reference to calculate the reward, the fine-tuning barely brings improvement to the model. The average reward is better than the random reward, and the maximum reward provided by the selected reference performs the best.
Models	LCE	Lsum	Lmid	Lmax
AT	27.70	28.08	27.37	27.21
NAT	26.09	25.97	26.90	26.88
Table 6: The performance of AT and CTC-based NAT on the same diverse distillation dataset of WMT14 EnDe with different loss functions.  $\mathcal{L}_{CE}$  is the cross-entropy loss with sequence-level distillation.  $\mathcal{L}_{sum}$ ,  $\mathcal{L}_{mid}$ , and  $\mathcal{L}_{max}$  described in section 3.2.1 are losses for the diverse distillation dataset.
Methods	pwb ↓	rfb↑	BLEU
HardMoe	53.57	24.77	24.51
Dropout	69.71	26.23	25.35
SeedDiv	59.87	26.99	25.92
4.4 DDRs on Autoregressive Transformer
Though DDRs is proposed to alleviate the multimodality problem for NAT, it can also be applied to autoregressive models. In Table 6, we report the performance of the autoregressive Transformer when trained by the proposed DDRs losses. In contrast to NAT, AT prefers the summation loss  $\mathcal{L}_{sum}$ , and the other two losses based on reference selection even degrade the AT performance.
It is within our expectation that AT models do not benefit much from reference selection. NAT generates the whole sentence simultaneously without any prior knowledge about the reference sentence, so the reference may not fit the NAT output well, in which case DDRs is helpful by selecting an appropriate reference for the training. In comparison, AT models generally apply the teacher forcing algorithm (Williams and Zipser, 1989) for the training, which feeds the golden context to guide the generation of the reference sentence. With teacher forcing, AT models do not suffer much from the multi-modality problem and therefore do not need reference selection. Besides, as shown in Table 1, another disadvantage is that the training cost of DDRs is nearly  $k$  times as large, so we do not recommend applying DDRs on AT.
4.5 Effect of Diverse Distillation
In the diverse distillation part of DDRs, we apply SeedDiv to generate multiple references. There are also other diverse translation techniques that can
Table 7: Pairwise BLEU (pwb) and reference BLEU (rfb) scores of diverse translation techniques and their DDRS performance on WMT14 En-De validation set. pwb and rfb scores are measured on WMT14 En-De test set without compound split.
Models	WMT14	WMT16
EN-DE	DE-EN	EN-RO	RO-EN
CTC w/o RL	26.09	29.50	33.55	32.98
BLEU	26.48	30.02	33.64	33.31
METEOR	26.44	29.95	33.68	33.25
GLEU	26.58	29.96	33.59	33.34
BERTScore	26.51	30.20	33.69	33.42
BLEURT	26.66	30.05	33.71	33.35
Table 8: BLEU scores on WMT test sets when using different automatic metrics as reward to finetune CTC.
be used for diverse distillation. In this section, we evaluate the effect of diverse distillation techniques on the performance of DDRs. Besides SeedDiv, we also use HardMoe (Shen et al., 2019) and Concrete Dropout (Wu et al., 2020) to generate multiple references, and report their performance in Table 7. When applying other techniques for diverse distillation, the performance of DDRs significantly decreases. The performance degradation indicates the importance of high reference BLEU in diverse distillation, as the NAT student directly learns from the generated references.
4.6 Effect of Reward
There are many automatic metrics to evaluate the translation quality. To measure the effect of reward, we respectively use different automatic metrics as reward for RL, which include traditional metrics (BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), GLEU (Wu et al., 2016)) and pretraining-based metrics (BERTScore (Zhang* et al., 2020), BLEURT (Sellam et al., 2020)). We report the results in Table 8. Comparing the three traditional metrics, we can see that there is no significant difference in their performance. The two pretraining-based metrics only perform slightly better than traditional metrics. Considering the performance and computational cost, we use the traditional metric BLEU as the reward.
4.7 Number of References
In this section, we evaluate how the number of references affects the DDRs performance. We set the number of references  $k$  to different values and train the CTC model with reference selection. We report the performance of DDRs with different  $k$  in Table 9. The improvement brought by increasing  $k$  is considerable when  $k$  is small, but it soon becomes marginal. Therefore, it is reasonable to use a middle number of references like  $k = 3$  to balance the distillation cost and performance.
k	1	2	3	5	8
BLEU	25.0	25.6	25.9	26.0	26.1
Table 9: Performance of DDR5 with different number of references  $k$  on WMT14 En-De validation set.
Models	Distill	Train	Total	BLEU
CTC (64K)	5.5h	26.4h	31.9h	26.34
CTC (128K)	5.5h	52.5h	58.0h	26.59
DDRS (32K)	16.5h	19.3h	35.8h	27.60
Table 10: Performance and time cost of models with different batch sizes on WMT14 En-De. The time cost is measured on 8 GeForce RTX 3090 GPUs. The cost of Distill includes training teacher models and decoding source sentences.
4.8 Time Cost
The cost of preparing the training data is larger for DDRs since it requires training  $k$  teacher models and using each model to decode the training set. We argue that the cost is acceptable since the distillation cost is minor compared to the training cost of NAT, and we can reduce the training cost to make up for it. In Table 10, we report the performance and time cost of models with different batch sizes on the test set of WMT14 En-De. DDRs makes up for the larger distillation cost by using a smaller training batch, which has a similar cost to the CTC model of 64K batch and achieves superior performance compared to models of 128K batch.
Gu et al. (2018) proposes non-autoregressive translation to reduce the translation latency, which suffers from the multi-modality problem. A line of work introduces latent variables to model the nondeterminism in the translation process, where latent variables are based on fertilities (Gu et al., 2018), vector quantization (Kaiser et al., 2018; Roy et al., 2018; Bao et al., 2021) and variational inference (Ma et al., 2019; Shu et al., 2020). Another branch of work proposes training objectives that are less influenced by the multi-modality problem to train NAT models (Wang et al., 2019; Shao et al., 2019, 2020, 2021; Sun et al., 2019; Ghazvininejad et al., 2020; Shan et al., 2021; Du et al., 2021). Some researchers consider transferring the knowledge from autoregressive models to NAT (Li et al., 2019; Wei et al., 2019; Guo et al., 2020a; Zhou et al., 2020; Sun and Yang, 2020). Besides, some work propose iterative NAT models that refine the model outputs
with multi-pass iterative decoding (Lee et al., 2018; Gu et al., 2019; Ghazvininejad et al., 2019; Ran et al., 2020; Kasai et al., 2020). Our work is most related to CTC-based NAT models (Graves et al., 2006; Libovický and Helcl, 2018; Kasner et al., 2020; Sahara et al., 2020; Zheng et al., 2021; Gu and Kong, 2020), which apply the CTC loss to model latent alignments for NAT. In autoregressive models, translations different from the reference can be evaluated with reinforcement learning (Ranzato et al., 2015; Norouzi et al., 2016), probabilistic n-gram matching (Shao et al., 2018), or an evaluation module (Feng et al., 2020).
Our work is also related to the task of diverse machine translation. Li et al. (2016); Vijayakumar et al. (2018) adjust the beam search algorithm by introducing regularization terms to encourage generating diverse outputs. He et al. (2018); Shen et al. (2019) introduce latent variables with the mixture of experts method and use different latent variables to generate diverse translations. Sun et al. (2020) generates diverse translations by sampling different attention heads. Wu et al. (2020) train the translation model with concrete dropout and samples different models from a posterior distribution. Li et al. (2021) generate different translations for the input sentence by mixing it with different sentence pairs sampled from the training corpus. Nguyen et al. (2020) augment the training set by translating the source-side and target-side data with multiple translation models, but they do not evaluate the diversity of the augmented data.
In this paper, we propose diverse distillation with reference selection (DDRS) for NAT. Diverse distillation generates a dataset containing multiple references for each source sentence, and reference selection finds the best reference for the training. DDRS demonstrates its effectiveness on various benchmarks, setting new state-of-the-art performance levels for non-autoregressive translation.
We thank the anonymous reviewers for their insightful comments. This work was supported by National Key R&D Program of China (NO.2017YFE9132900).
Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, ICLR 2015; Conference date: 07-05-2015 Through 09-05-2015.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Yu Bao, Shujian Huang, Tong Xiao, Dongqi Wang, Xinyu Dai, and Jiajun Chen. 2021. Non-autoregressive translation by learning target categorical codes. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5749-5759, Online. Association for Computational Linguistics.
Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021. Order-agnostic cross entropy for non-autoregressive machine translation. In ICML.
Yang Feng, Wanying Xie, Shuhao Gu, Chenze Shao, Wen Zhang, Zhengxin Yang, and Dong Yu. 2020. Modeling fluency and faithfulness for diverse neural machine translation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 59-66. AAAI Press.
Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. 2017. Ensemble distillation for neural machine translation. arXiv preprint arXiv:1702.01802.
Xinwei Geng, Xiaocheng Feng, and Bing Qin. 2021. Learning to rewrite for non-autoregressive neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3297-3308, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Aligned cross entropy for non-autoregressive machine translation. In ICML.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6121.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, page 369-376, New York, NY, USA. Association for Computing Machinery.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Conference on Learning Representations.
Jiatao Gu and Xiang Kong. 2020. Fully non-autoregressive neural machine translation: Tricks of the trade.
Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2020a. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, 34:7839-7846.
Junliang Guo, Linli Xu, and Enhong Chen. 2020b. Jointly masked sequence-to-sequence model for non-autoregressive neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 376-385, Online. Association for Computational Linguistics.
Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. 2018. Sequence to sequence mixture model for diverse machine translation. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 583-592, Brussels, Belgium. Association for Computational Linguistics.
Chenyang Huang, Hao Zhou, Osmar R Zaiane, Lili Mou, and Lei Li. 2021. Non-autoregressive translation with layer-wise prediction and deep supervision. ArXiv, abs/2110.07515.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. 2018. Fast decoding in sequence models using discrete latent variables. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2390-2399. PMLR.
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020. Non-autoregressive machine translation with disentangled context transformer. In ICML.
Zdeněk Kasner, Jindřich Libovický, and Jindřich Helcl. 2020. Improving fluency of non-autoregressive machine translation.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173-1182, Brussels, Belgium. Association for Computational Linguistics.
Jicheng Li, Pengzhi Gao, Xuanfu Wu, Yang Feng, Zhongjun He, Hua Wu, and Haifeng Wang. 2021. Mixup decoding for diverse machine translation. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 312-320, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A simple, fast diverse decoding algorithm for neural generation.
Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Hint-based training for non-autoregressive machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5708-5713, Hong Kong, China. Association for Computational Linguistics.
Jindrich Libovicky and Jindrich Helcl. 2018. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3016-3021, Brussels, Belgium. Association for Computational Linguistics.
Ye Liu, Yao Wan, Jianguo Zhang, Wenting Zhao, and Philip Yu. 2021. Enriching non-autoregressive transformer with syntactic and semantic structures for neural machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1235-1244, Online. Association for Computational Linguistics.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. FlowSeq: Non-autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4282-4292, Hong Kong, China. Association for Computational Linguistics.