SlowGuess/ACL_2020_OCR huggingface dataset:
We adopt sequence-level knowledge distillation (Kim and Rush, 2016) on the training set of each task, which has been proved by previous NAT models that it can produce less noisy and more deterministic training data (Gu et al., 2017). As stated by Wang et al. (2019), the performance of the AT teacher will affect the final performance of the NAT student model. While AT teachers used in previous works have various performance, we utilize the teacher model which has similar performance with the one used in our main baseline CM-NAT (Ghazvininejad et al., 2019) to construct a fair comparison. In addition, we also provide the performance of our model trained by a weakened AT teacher (denoted as WT in Table 2) which has similar performance with the one used in (Wang et al., 2019) to compare with them.
5.1.5 Training and Inference
We train the model with 8/1 Nvidia 1080Ti GPUs on the WMT datasets and IWSLT14 dataset respectively, and we utilize the Adam optimizer while following the same settings used in the original Transformer. During inference, we generate multiple translation candidates by taking the top  $B$  length predictions into consideration, and select the translation with the highest probability as the final result. We set  $B = 3$  on WMT tasks and  $B = 4$  on IWSLT14 tasks. We also report the clock time of inference latency on a single Nvidia 1080Ti GPU in our experiments, where we set the batch size to 1 and calculate the average per sentence translation time on newstest2014 for the WMT14 En-Det task to keep consistence with previous works.
As for evaluation, we use BLEU scores (Papineni et al., 2002) as the evaluation metric, and
$^{1}$ https://wit3.fbk.eu/  
 $^{2}$ https://www.statmt.org/wmt16/  
translation-task  
 $^{3}$ https://www.statmt.org/wmt14/  
translation-task
Models	WMT14	WMT16	IWSLT14
En-De	De-En	En-Ro	Ro-En	De-En	Latency	Speedup
Transformer (Vaswani et al., 2017)	28.04*	32.69*	34.13*	34.46*	32.99*	607 ms	1.00×
Transformer (Weak Teacher)	27.40*	31.29*	/	/	/	-	-
NAT-FT (NPD 100) (Gu et al., 2017)	19.17	23.20	29.79	31.44	24.21†	257 ms	2.36×
Imitate-NAT (Wei et al., 2019)	24.15	27.28	31.45	31.81	/	/	/
NAT-Reg (NPD 9) (Wang et al., 2019)	24.61	28.90	/	/	28.04	40 ms	15.1×
FCL-NAT (NPD 9) (Guo et al., 2020)	25.75	29.50	/	/	29.91	38 ms	16.0×
NAT-DCRF (NPD 9) (Sun et al., 2019)	26.07	29.68	/	/	29.99	63 ms	9.63×
NAT-IR (k=5) (Lee et al., 2018)	20.26	23.86	28.86	29.72	/	/	/
NAT-IR (k=10)	21.61	25.48	29.32	30.19	23.94†	404† ms	1.50×
CM-NAT (k=4) (Ghazvininejad et al., 2019)	25.94	29.90	32.53	33.23	30.42*	62* ms	9.79×
CM-NAT (k=10)	27.03	30.53	33.08	33.31	31.71*	161* ms	3.77×
JM-NAT (k=4)	27.05	31.51	32.97	33.21	31.27	45 ms	13.5×
JM-NAT (k=10)	27.69	32.24	33.52	33.72	32.59	106 ms	5.73×
JM-NAT (WT) (k=4)	26.82	30.59	/	/	/	-	-
JM-NAT (WT) (k=10)	27.31	31.02	/	/	/	-	-
Table 2: The BLEU scores of our proposed JM-NAT and the baseline methods on the WMT14 En-De/De-En, WMT16 En-Ro/Ro-En and IWSLT14 De-En tasks. We report the best results for the baseline methods and also list the inference latency on clock as well as the speedup w.r.t autoregressive models. “†” indicates that the result is provided by (Wang et al., 2019), “*” indicates the results obtained by our implementation, “/” indicates the corresponding result is not reported in the original paper, and “–” indicates the same numbers as above. “Weak Teacher (WT)” indicates the NAT is trained with a weakened AT teacher through knowledge distillation. NPD stands for Noisy Parallel Decoding utilized in previous works. “k” represents the number of iterations while inference.
report the tokenized case-sensitive scores for the WMT datasets, as well as the tokenized case-insensitive scores for the IWSLT14 dataset. Our implementation is based on fairseq (Ott et al., 2019) and is available at https://github.com/lemmonation/jm-nat.
The main results are listed in Table 2. We denote our model as Jointly Masked NAT (JM-NAT), and show the results when the upper bound of iterations  $k$  is set to 4 and 10. As can be observed from Table 2, our model achieves comparable performance with its AT teacher on all datasets (only 0.5 BLEU score behind in average), while achieving  $5 +$  times speedup on the inference latency. Compared with the pure NAT models with  $O(1)$  time complexity, with similar inference latency by setting  $k = 4$ , our model outperforms all baselines with a consistent margin on different tasks. Compared with the models based on iterative refinement, JM-NAT also shows consistent superiority with the same time complexity. Our model outperforms CM-NAT (Ghazvininejad et al., 2019) with margins from 0.41 to 1.71 on different tasks, illustrating the boosted performance brought by the jointly masked model as well as the proposed loss functions. It is
worth noting that CM-NAT utilizes a much stronger AT teacher on the WMT14 En-De task (using the large configuration of Transformer and achieving 28.65 BLEU score). Our model, even with less iterations or a weaker AT teacher, still outperforms CM-NAT in most cases, and it is straightforward to further improve our performance with a stronger teacher.
5.3.1 Encoder Performance
As there does not exist a clear metric (such as the perplexity in language generation tasks) to evaluate the quality of the encoder in a sequence-to-sequence model, we adopt a naive version of the adversarial attack on text (Belinkov and Bisk, 2017) to the encoder input to test the robustness of the encoder. Specifically, during inference, we follow the same strategy used in Section 3 to add noise to the source sentence  $x$ . Given the noise ratio  $\alpha \in (0,1)$ , we randomly select  $\lfloor \alpha \cdot T_x \rfloor$  (where  $\lfloor \cdot \rfloor$  stands for the rounding function) source tokens and either drop or replace them with other tokens in the vocabulary. We increase  $\alpha$  from 0 to  $10\%$  and test the performance of each model on the validation set of the IWSLT14 De-En task, and show the results in Figure 3. We compare our model with baselines
NAT-FT	NAT-Reg	CM-NAT	JM-NAT
2.30	0.90	0.48	0.17
including NAT-FT and CM-NAT. According to the results, compared with CM-NAT, which is also an iterative decoding based method, our model shows more robust performance with regard to the noise on the encoder input, showing the efficacy of the proposed masking strategy and the better quality of our encoder.
5.3.2 Repetitive Words
As studied by Wang et al. (2019), the tendency of producing repetitive words in translation is a major drawback of NAT models. We propose to alleviate this problem by training the decoder with the consecutive masking strategy as well as the  $n$ -gram loss function. We compute the average number of consecutive repetitive tokens per sentence in the translation results on the validation set of the IWSLT14 De-En task. Results are shown in Table 3. Without introducing explicit regularizations (Wang et al., 2019), our method is still able to alleviate the problem of repetitive words. Compared with CM-NAT who also utilizes an iterative decoding method, the superiority of our method demonstrates the proposed consecutive masking strategy better solves the problem than random masking.
We conduct the ablation study on the validation set of the IWSLT14 De-En task to illustrate the contribution of different components in our model. Results are shown in Table 4. For the encoder, both encoder masking and the objective function  $L_{\mathrm{enc}}$  contribute to the final performance, and encoder masking provides the most prominent performance promotion. On the decoder side, both of the consecutive masking strategy and the  $n$ -gram loss function are indispensable to produce solid performance as they are tied together through the hyper-parameter  $n$ . In addition, all the proposed components are effective in alleviating the repetitive translations, and the  $n$ -gram loss function contributes the most.
Figure 3: The performance of considered NAT models when adding noise to the encoder input. The X-axis indicates the ratio of noise, and the Y-axis indicates the  $\Delta$ BLEU score compared with feeding the input without noise.
Table 3: The comparison on the average number of sentence repetitive tokens on the validation set of the IWSLT14 De-En task.
Model Variants	BLEU	ΔBLEU	Reps
JM-NAT	33.82	-	0.17
On the Encoder Side
w/o Lenc	33.32	-0.50	0.21
w/o Encoder Mask & Lenc	32.15	-1.67	0.23
On the Decoder Side
w/o Lgram	33.27	-0.55	0.30
w/o Consecutive Mask	32.97	-0.85	0.25
Table 4: The ablation study on different components of the proposed model conducted on the validation set of IWSTL14 De-En task. "Reps" indicates the average number of repetitive translations computed same as in Table 3.
We further conduct case studies to intuitively demonstrate the performance of different models and the generation process of our model. Results are listed in Table 5. As we discussed in Section 1, repetitive translations and missing translations are two stubborn problems of NAT models. In Table 5, both NAT-FT and CM-NAT tend to generate repetitive words (such as "eliminate diabetes diabetes" and "reduce cancer risk risk") as well as incomplete translations (both of them miss the word "eliminate" in the second clause), while our model achieves better results.
In this paper, we propose a jointly masked sequence-to-sequence model for non-autoregressive neural machine translation. We first empirically investigate the functionalities of the non-autoregressive translation model, and
Source:	wasäre , wenn sie die genetischen veränderungen machen konnten , um diabetes oder alzheimerr zu beseitigen oder das reduzieren des krebsrisikos oder schlaganfälle zu eliminieren ?
Target:	what if you could make the genetic changes to eliminate diabetes or alzheimer &apos;s or reduce the risk of cancer or eliminate stroke ?
NAT-FT:	what if you could make the genetic changes in order to eliminate diabetes diabetes or alzheimer disease or reduce reduce the cancer of cancer or strostroke ?
CM-NAT:	what if you could make the genetic changes to eliminate diabetes or alzheimer alzheimer &apos;s , or reduce cancer risk risk or strostro dents ?
JM-NAT:	what if you could make the genetic changes to eliminate diabetes or alzheimer &apos;s disease or the reduce cancer risk or eliminate stroke ?
Table 5: A case study on the translation results of different models on the IWSLT14 De-En task. We set  $k = 10$  for our model. The bold italics represent the repetitive words in the translation results.
improve the training of the encoder by masking its input and introducing a prediction based loss function. For the decoder, we propose to utilize consecutive masking and introduce an  $n$ -gram based loss function to alleviate the problem of repetitive translations. Our model outperforms all compared NAT baselines and achieves comparable performance with autoregressive models on five benchmark tasks with  $5 +$  times speed up on the inference latency.
In the future, we will extend the investigation on the functionalities of the encoder and decoder to other sequence-to-sequence tasks such as text summarization and text style transfer to explore more applications of our model.
This research was supported by the National Natural Science Foundation of China (No. 61673364, No. U1605251) and the Fundamental Research Funds for the Central Universities (WK2150110008). We would like to thank the Information Science Laboratory Center of USTC for the hardware and software services. We thank the anonymous reviewers as well as Zhirui Zhang and Tianyu He for helpful feedback on early versions of this work.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. arXiv preprint arXiv:1711.02173.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6114-6123.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2017. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281.
Jiatao Gu, Changhan Wang, and Jake Zhao. 2019. Levenshtein transformer. arXiv preprint arXiv:1905.11006.
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. 2019. Non-autoregressive neural machine translation with enhanced decoder input. In Proceedings of the AAAI Conference on Artificial Intelligence.
Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2020. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt,
William Lewis, Mu Li, et al. 2018. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567.
Tianyu He, Xu Tan, and Tao Qin. 2019. Hard but robust, easy but sensitive: How encoder and decoder perform in neural machine translation. arXiv preprint arXiv:1908.06259.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529.
Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, pages 177-180.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. arXiv preprint arXiv:1802.06901.
Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Hint-based training for non-autoregressive translation. arXiv preprint arXiv:1909.06708.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018. Bag-of-words as target for neural machine translation. arXiv preprint arXiv:1805.04871.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
Chenze Shao, Yang Feng, and Xilin Chen. 2018. Greedy search with probabilistic n-gram matching for neural machine translation. arXiv preprint arXiv:1809.03132.
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. 2019. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. arXiv preprint arXiv:1911.09320.
Xin Sheng, Linli Xu, Junliang Guo, Jingchang Liu, Ruoyu Zhao, and Yinlong Xu. 2020. Introvnmt: An introspective model for variational neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible sequence generation via insertion operations. arXiv preprint arXiv:1902.03249.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. 2019. Fast structured decoding for sequence models. In Advances in Neural Information Processing Systems, pages 3011-3020.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019. Non-autoregressive machine translation with auxiliary regularization. In Proceedings of the AAAI Conference on Artificial Intelligence.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019. Imitation learning for non-autoregressive neural machine translation. arXiv preprint arXiv:1906.02041.
Sean Welleck, Kianté Brantley, Hal Daumé III, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. arXiv preprint arXiv:1902.02192.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.