SlowGuess/NAACL_2021_OCR huggingface dataset:
Our best results are obtained with length parallel decoding, which employs a pretrained Transformer to rerank the multiple parallels generated candidates of different target lengths. Specifically, on a large scale WMT14 dataset, CNAT surpasses the NAT-DCRF by 0.71 BLEU score in DE-EN but
Table 3: Results of NAT models with parallel decoding on test set of WMT14. "N" means the number of candidates to be re-ranked.
Model	Iteration	WMT14
EN-DE	DE-EN	Speedup
IR-NAT	1	13.91	16.77	11.39×
2	16.95	20.39	8.77×
5	20.26	23.86	3.11×
10	21.61	25.48	2.01×
CMLM	4	26.08	30.11	/
10	26.92	30.86	/
CNAT	1	25.56	29.36	10.37×
CNAT (N=9)	1	26.60	30.75	5.59×
Table 4: Results of NAT models with iterative refinements on test set of WMT14. "Iteration" means the number of iteration refinements.
slightly under the NAT-DCRF around 0.20 BLEU in EN-DE, which shows that the CNAT is comparable to the state-of-the-art NAT model. Also, we can see that a larger "N" leads to better results ( $N = 100$  vs.  $N = 10$  of NAT-FT,  $N = 19$  vs.  $N = 9$  of NAT-DCRF, etc.); however, it always comes at the degradation of decoding efficiency.
We also compare our CNAT with the NAT models that employ an iterative decoding technique and list the results in Table 4. The iterative-based non-autoregressive Transformer captures the target language's dependencies by iterative generating based on the previous iteration output, which is an important exploration for a non-autoregressive generation. With the iteration number increasing, the performance improving, the decoding speed-up dropping, whatever the IR-NAT or CMLM. We can see that the CNAT achieves a better result than the CMLM with four iterations and IR-NAT with ten iterations, even close to the CMLM with ten iterations while keeping the benefits of a one-shot generation.
Translation Efficiency. As depicted in Figure 2, we validate the efficiency of CNAT. Put simply, the decoding speed is measured sentence-by-sentence, and the speed-up is computed by comparing it with the Transformer. Figure 2a and Figure 2b show the BLEU scores and decoding speed-up of NAT models. The former compares the pure NAT models. The latter compares NAT model inference with advanced decoding techniques (parallel reranking or iterative-based decoding) $^3$ .
We can see from Figure 2 that the point of
3Our results are conducted on a single GeForce GTX 1080-TI GPU. Please note that the result in Figure 2a and Figure 2b may be evaluated under different hardware settings, and it may not be fair to compare them directly.
(a) Pure NAT decoding.
(b) NAT with advanced decoding techniques.
Figure 2: BLEU and decoding speed-up of NAT models on WMT14 DE-EN test set. Each point represents the decoding method run with its corresponding setting in Table 2, Table 3 or Table 4.
Methods	Latent BLEU	Translation BLEU
CNAT w/ zref	100.00	59.12
CNAT w/ mref	39.72	31.59
CNAT	38.59	31.15
CNAT is located on the top-right of the baselines. The CNAT outperforms our baselines in BLEU if speed-up is held, and in speed-up if BLEU is held, indicating CNAT outperforms previous state-of-the-art NAT methods. Although iterative models like CMLM achieves competitive BLEU scores, they only maintain minor speed advantages over Transformer. In contrast, CNAT remarkably improves the inference speed while keeping a competitive performance.
Effectiveness of Categorical Modeling. We further conduct the experiments on the test set of IWSLT14 to analyze the effectiveness of our categorical modeling and its influence on translation quality. We regard the categorical predictor as a sequence-level generation task and list its BLEU score in Table 5.
As see, a better latent prediction can yield a better translation. With the  $z_{\mathrm{ref}}$  as the latent sequence, the model achieves surprisingly good performance on this task, showing the usefulness of the learned categorical codes. We also can see that the CNAT decoding with reference length only slightly (0.44 BLEU) better than it with predicted length, indicat
Table 5: Results on the test of IWSLT14 to analyze the effectiveness of categorical modeling. “w/  $z_{\mathrm{ref}}$ ” denote CNAT generate the tokens condition on the latent sequence which is quantized from the reference target. “w/  $m_{\mathrm{ref}}$ ” denote the CNAT generate the tokens condition on the reference length.
Line	K	Predictor	GateNet	BLEU
32	64	128	CRF	AR
1	✓	✓	✓	30.13
2	✓	✓	✓	31.87
3	✓	✓	✓	30.82
4	✓	✓	29.32
5	✓	✓	✓	28.23
6	✓	✓	24.00
7	✓	✓	25.43
8	24.25
Table 6: Ablation study on the dev set of IWSLT14. Note that we train all of the configurations with knowledge distillation. "AR" denotes an autoregressive Transformer predictor. The line 8 is our NAT baseline.
ing that the model is robust.
We further conduct the ablation study with different CNAT variant on dev set of IWSLT14.
Influence of  $K$ . We can see the CRF with the categorical number  $K = 64$  achieves the highest score (line 2). A smaller or larger  $K$  neither has a better result. The AR predictor may have a different tendency: with a larger  $K = 128$ , it achieves a better performance. However, a larger  $K$  may lead to a higher latency while inference, which is not the best for non-autoregressive decoding. In our experiments, the  $K = 64$  can achieve the high-performance and be smaller enough to keep the low-latency during inference.
CRF versus AR. Experiment results show that the CRF-based predictor is better than the AR predictor. We can see that the CRF-based predictor surpasses the Transformer predictor 3.5 BLEU (line 2 vs. line 5) with the GateNet; without the
H-score	C-score	V-measure
w/ POS tags	0.70	0.47	0.56
w/ Frequency	0.62	0.48	0.54
Table 7: Clustering evaluation metrics on the test set of IWSLT14 to analyze the learned codes.
GateNet, the gap enlarges to 5.3 BLEU (line 4 vs. line 6). It is consistent with our intuition that CRF is better than Transformer to model the dependencies among latent variables on machine translation when the number of categories is small.
GateNet. Without the GateNet, the CNAT with AR predictor degenerates a standard LT model with a smaller latent space. We can see its performance is even lower than the NAT-baselines (line 6 vs. line 8). Equipping with the GateNet and the schedule sampling, it outperforms the NAT baseline with a large margin (around 4.0 BLEU), showing that the GateNet mechanism plays an essential role in our proposed model.
4.3 Code Study
To analyze the learned category, we further compute its relation to two off-the-shelf categorical information: the part-of-speech (POS) tags and the frequency-based clustered classes. For the former, we intuitively assign the POS tag of a word to its sub-words and compute the POS tag frequency for the latent codes. For the latter, we roughly assign the category of a subword according to its frequency. It needs to mention that the number of frequency-based classes is the same as that of the POS tags.
Quantitative Results. We first compute the V-Measure (Rosenberg and Hirschberg, 2007) score between the latent categories to POS tags and subwords frequencies. The results are listed in Table 7.
Overall, the "w/ POS tags" achieves a higher V-Measure score, indicating that the latent codes are more related to the POS tags than sub-words frequencies. The homogeneity score (H-score) evaluates the purity of the category. We also can see that the former has a relatively higher H-score than the latter (0.70 vs. 0.62), which is consistent with our intuition.
Case Analysis. As shown in Figure 3, we also depict the POS tags distribution for the top 10 frequent latent variables on the test set of IWSLT14<sup>4</sup>.
Figure 3: The POS tags distribution for the top 10 frequent latent variables on the test set of IWSLT14. We list the top 3 frequent POS tags for each latent variable.
We can see a sharp distribution for each latent variable, showing that our learned fuzzy classes are meaningful.
Non-autoregressive Machine Translation. Gu et al. (2018) first develop a non-autoregressive Transformer (NAT) for machine translation, which produces the outputs in parallel, and the inference speed is thus significantly boosted. Due to the missing of dependencies among the target outputs, the translation quality is largely sacrificed.
A line of work proposes to mitigate such performance degradation by enhancing the decoder inputs. Lee et al. (2018) propose a method of iterative refinement based on the previous outputs. Guo et al. (2019) enhance decoder input by introducing the phrase table in statistical machine translation and embedding transformation. There are also some work focuses on improving the decoder inputs' supervision, including imitation learning from autoregressive models (Wei et al., 2019) or regularizing the hidden state with backward reconstruction error (Wang et al., 2019).
Another work proposes modeling the dependencies among target outputs, which is explicitly missed in the vanilla NAT models. Qian et al. (2020); Ghazvininejad et al. (2019) propose to model the target-side dependencies with a masked
4More details can be found in Appendix B.
language model, modeling the directed dependencies between the observed target and the unobserved words. Different from their work, we model the target-side dependencies in the latent space, which follows the latent variable Transformer fashion.
Latent Variable Transformer. More close to our work is the latent variable Transformer, which takes the latent variable as inputs to modeling the target-side information. Shu et al. (2019) combine continuous latent variables and deterministic inference procedure to find the target sequence that maximizes the lower bound to the log-probability. Ma et al. (2019) propose to use generative flows to the model complex prior distribution. Kaiser et al. (2018) propose to autoregressively decode a shorter latent sequence encoded from the target sentence, then simultaneously generate the sentence from the latent sequence. Bao et al. (2019) model the target position of decode input as a latent variable and introduce a heuristic search algorithm to guide the position learning. Akoury et al. (2019) first autoregressively predict a chunked parse tree and then simultaneously generate the target tokens from the predicted syntax.
We propose CNAT, which implicitly models the categorical codes of the target language, narrowing the performance gap between the non-autoregressive decoding and autoregressive decoding. Specifically, CNAT builds upon the latent Transformer and models the target-side categorical information with vector quantization and conditional random fields (CRF) model. We further employ a gated neural network to form the decoder inputs. Equipped with the scheduled sampling, CNAT works more robust. As a result, the CNAT achieves a significant improvement and moves closer to the performance of the Transformer on machine translation.
We would like to thank the anonymous reviewers for their insightful comments. Shujian Huang is the corresponding author. This work is supported by the National Science Foundation of China (61772261), National Key R&D Program of China (No. 2019QY1806), the Fundamental Research Funds for the Central Universities (No.
14380076), and the program B for Outstanding Ph.D. candidate of Nanjing University.
Nader Akoury, Kalpesh Krishna, and Mohit Iyyer. 2019. Syntactically supervised transformers for faster neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1269-1281, Florence, Italy. Association for Computational Linguistics.
Yu Bao, Hao Zhou, Jiangtao Feng, Mingxuan Wang, Shujian Huang, Jiajun Chen, and Lei Li. 2019. Non-autoregressive transformer by position learning. arXiv preprint arXiv:1911.10677.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1171-1179.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6121, Hong Kong, China. Association for Computational Linguistics.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. 2019. Non-autoregressive neural machine translation with enhanced decoder input. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 3723-3730. AAAI Press.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. 2018. Fast decoding in sequence models using discrete latent variables. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2395-2404. PMLR.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, pages 282-289. Morgan Kaufmann.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173-1182, Brussels, Belgium. Association for Computational Linguistics.
Zhuohan Li, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Hint-based training for non-autoregressive translation. In NeuralIPS (to appear).
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. FlowSeq: Nonautoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4282-4292, Hong Kong, China. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics* (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2020. Glancing transformer for non-autoregressive neural machine translation. arXiv preprint arXiv:2008.07905.
Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 410-420, Prague, Czech Republic. Association for Computational Linguistics.
Aurko Roy, Ashish Vaswani, Niki Parmar, and Arvind Neelakantan. 2018. Towards a better understanding of vector quantized autoencoders. arXiv.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. 2019. Latent-variable nonautoregressive neural machine translation with deterministic inference using a delta posterior. arXiv preprint arXiv:1908.07181.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhi-Hong Deng. 2019. Fast structured decoding for sequence models. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3011-3020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory, 13(2):260-269.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019. Non-autoregressive machine translation with auxiliary regularization. In AAAI.
A Non-Indo-European Translation
Dataset. We apply the CNAT to the non-Indo-European translation tasks on the LDC Chinese-English $^5$  (denote as LDC ZH-EN, 1.30M sentence pairs) and MT02 test set of NIST ZH-EN dataset. We use NLPIRICTCLAS $^6$  and Moses tokenizer for Chinese and English tokenization, respectively.
Model	BLEU
Transformer	28.05
NAT	12.31
CNAT	22.16
Results. We can see than in Table 8 that our model can enhance the performance of NAT with a large margin (22.16 vs. 12.31).
B Learned Latent Codes
Table 8: Results on the MT02 set of different models.
ID	Top 1	Top 2	Top 3
0	NOUN(41.06%)	ADJ(20.74%)	VERB(20.09%)
1	NOUN(88.73%)	ADJ(3.48%)	VERB(2.34%)
2	NOUN(58.06%)	ADJ(15.40%)	VERB(12.29%)
3	PUNCT(100.00%)	—	—
4	PUNCT(99.80%)	NOUN(0.14%)	PROPN(0.05%)
5	PRON(99.29%)	NOUN(0.42%)	PROPN(0.20%)
6	VERB(77.95%)	NOUN(9.93%)	ADJ(8.45%)
7	DET(99.33%)	PUNCT(0.47%)	PART(0.15%)
8	ADJ(56.24%)	ADV(15.06%)	NUM(11.87%)
9	NOUN(52.01%)	VERB(16.72%)	ADJ(15.75%)
10	CCONJ(99.71%)	VERB(0.11%)	NOUN(0.11%)
11	PART(71.00%)	ADP(27.70%)	SCONJ(0.99%)
12	PRON(60.71%)	SCONJ(30.15%)	DET(8.88%)
13	ADP(90.70%)	SCONJ(5.91%)	ADV(3.09%)
14	DET(96.81%)	NOUN(1.81%)	PROPN(1.13%)
15	NOUN(88.49%)	VERB(4.28%)	ADV(3.10%)
16	VERB(90.35%)	NOUN(4.82%)	AUX(4.40%)
17	ADV(67.37%)	PART(15.02%)	NOUN(8.31%)
18	PRON(68.46%)	DET(29.31%)	NOUN(1.65%)
19	ADV(61.05%)	SCONJ(36.01%)	ADP(2.31%)
20	VERB(93.70%)	NOUN(6.03%)	PROPN(0.23%)
21	PRON(98.59%)	NOUN(0.78%)	ADJ(0.39%)
22	ADP(93.45%)	ADV(2.40%)	SCONJ(1.95%)
23	AUX(85.14%)	VERB(14.51%)	NOUN(0.36%)
24	AUX(80.60%)	VERB(18.55%)	NOUN(0.60%)
25	VERB(44.78%)	AUX(40.84%)	PROPN(13.49%)
26	ADP(73.68%)	ADV(9.68%)	SCONJ(9.55%)
27	AUX(99.47%)	NOUN(0.53%)	—
28	AUX(76.11%)	VERB(10.26%)	PART(9.70%)
29	ADP(89.48%)	ADV(3.93%)	NOUN(3.93%)
30	VERB(43.34%)	ADP(34.41%)	AUX(14.47%)
31	DET(53.56%)	PRON(46.44%)	—
32	ADV(95.89%)	SCONJ(3.72%)	PROPN(0.23%)
33	ADP(52.12%)	SCONJ(27.84%)	ADV(8.74%)
34	DET(62.63%)	PRON(20.73%)	ADV(9.90%)
35	VERB(76.34%)	AUX(23.66%)	—
36	AUX(100.00%)	—	—
37	ADV(47.90%)	PRON(40.51%)	NOUN(11.59%)
38	NOUN(99.17%)	ADJ(0.73%)	ADV(0.10%)
39	NOUN(35.74%)	VERB(28.94%)	ADJ(28.21%)
40	NOUN(49.84%)	ADJ(27.23%)	ADV(16.19%)
41	PRON(92.22%)	NOUN(4.75%)	PROPN(1.08%)
42	PRON(92.73%)	DET(7.05%)	ADV(0.22%)
43	ADP(32.17%)	NOUN(23.05%)	ADJ(21.61%)
44	PUNCT(100.00%)	—	—
45	PRON(100.00%)	—	—
46	AUX(94.36%)	VERB(4.59%)	NOUN(1.05%)
47	ADV(67.33%)	ADJ(31.75%)	NOUN(0.79%)
48	ADP(91.70%)	SCONJ(8.16%)	ADJ(0.14%)
49	PUNCT(100.00%)	—	—
50	PART(70.91%)	AUX(25.04%)	ADP(2.85%)
51	CCONJ(99.52%)	ADP(0.48%)	—
52	ADP(69.34%)	SCONJ(15.89%)	ADV(14.13%)
53	PUNCT(100.00%)	—	—
54	NOUN(58.00%)	VERB(30.58%)	ADJ(6.15%)
55	VERB(71.57%)	AUX(28.04%)	NOUN(0.39%)
56	NUM(75.73%)	NOUN(20.33%)	PRON(2.49%)
57	DET(86.03%)	ADJ(6.77%)	PROPN(3.71%)
58	ADP(61.07%)	ADV(31.77%)	NOUN(5.37%)
59	CCONJ(90.75%)	NOUN(8.48%)	ADJ(0.51%)
60	ADP(78.74%)	SCONJ(19.93%)	ADV(1.00%)
61	VERB(59.11%)	NOUN(31.56%)	ADJ(9.33%)
Table 9: The distribution of pos tags for latent variables. For each latent variable, we list the top 3 frequent pos tags and their corresponding percentages.
${}^{5}$  LDC2002E18, LDC2003E14, LDC004T08, and LDC2005T06
http://ictclas.nlpir.org/