What Have We Achieved on Non-autoregressive Translation? (2024)

Abstract: Recent advances have made non-autoregressive (NAT) translation comparable to autoregressive methods (AT). However, their evaluation using BLEU has been shown to weakly correlate with human annotations. Limited research compares non-autoregressive translation and autoregressive translation comprehensively, leaving uncertainty about the true proximity of NAT to AT. To address this gap, we systematically evaluate four representative NAT methods across various dimensions, including human evaluation. Our empirical results demonstrate that despite narrowing the performance gap, state-of-theart NAT still underperforms AT under more reliable evaluation metrics. Furthermore, we discover that explicitly modeling dependencies is crucial for generating natural language and generalizing to out-of-distribution sequences.

heading: 1 Introduction
text: Non-autoregressive translation, where the model generates translations in parallel, demonstrates notable decoding speed advantages compared with traditional autoregressive translation (Vaswani et al., 2017) and large language models for translation (OpenAI, 2023). However, it suffers from performance degradation compared to autoregressive counterparts (Gu et al., 2018). The degradation stems from the independence assumption, which ignores the inter-token language dependency on the target side. Various methods are proposed to mitigate the performance gap (Ghazvininejad et al., 2019; Qian et al., 2021; Saharia et al., 2020; Du et al., 2021; Li et al., 2022; Huang et al., 2022b,c).
Although representative methods (Saharia et al., 2020; Li et al., 2022; Huang et al., 2022b,c) have reported comparable translation performance to AT, almost all NAT methods are evaluated under BELU scores (Papineni et al., 2002). Although
* Equal contribution. † Corresponding author.
BLEU has been long adopted, recent work (Freitag et al., 2022) argues that it is not a reasonable metric, considerably underperforming alternative metrics such as COMET (Rei et al., 2020) or large language model evaluation (Kocmi and Federmann, 2023). Limited work has been devoted to a systematic evaluation of advanced NAT against AT, leaving a significant gap in the research literature.
To address this gap, we conduct a comprehensive evaluation of representative NAT methods, aiming to reveal existing limitations and provide insights for future research. Our primary focus is on fully non-autoregressive methods which generate translations in a one-shot manner, achieving the most decoding efficiency advantage. We consider MgMO (Li et al., 2022) for advanced optimization, CTC (Saharia et al., 2020) for modeling latent alignment, and DAT (Huang et al., 2022b) for explicit target-side dependency modeling. CMLM (Ghazvininejad et al., 2019) is adopted as the representative iterative NAT method. All models are tested on representative benchmark datasets under a comprehensive evaluation , including rule-based metrics, model-based metrics and GPT4-based metrics (Kocmi and Federmann, 2023). Moreover, we conduct human evaluation under the MQM framework (Freitag et al., 2021) to gain further insights into the performance of NAT models that may be overshadowed by global automatic evaluations.
Automatic evaluation demonstrates varying degrees of advantage for AT over NAT models. In general, DAT achieves the most competitive performance, followed by MgMO and CTC. Under rule-based evaluation metrics such as BELU and chrf (Popović, 2015), DAT can achieve comparable or even superior performance compared to AT. However, this competitiveness diminishes when using model-based metrics such as COMET (Rei et al., 2020) or GPT4-based evaluation, under which AT significantly outperforms all NAT mod-
7585
els. Fine-grained human evaluation indicates that NAT models incorporating explicit dependency modeling (e.g., DAT and CMLM) achieve similar levels of translation fluency with AT, yet suffering various translation accuracy errors. Compared with AT, NAT tends to produce more grammar or punctuation errors. Models without explicit dependency modeling (MgMO and CTC) suffer the most mistranslation and omission errors. On the other hand, models with latent alignments (CTC and DAT) are more prone to spelling and addition errors.
Most of these errors are due to NAT’s inadequate dependency modeling. Specifically, DAT’s addition errors occur when it generates repeated translations, known as n-gram repetition. This can be easily overlooked by BLEU evaluation, which measures n-gram precision, explaining why DAT performs well in terms of BLEU but not COMET. The n-gram repetition mainly stems from the weak, though explicit, dependency modeling. DAT limits inter-token dependency within one step using a one-linear-layer attention module for decoding efficiency. In contrast, AT can depend on the entire generation history and encode it with powerful Transformer blocks. To validate our assumption, we train an asymmetric AT with a one-layer decoder and observe similar n-gram repetitions. Furthermore, adding an additional linear layer to the transition attention in DAT effectively reduces the repetition, corroborating our hypothesis.
Apart from translation quality, we compare AT with NAT from the perspective of generalization and robustness. Empirical findings demonstrate that explicit dependency modeling is crucial for generating human-like languages and generalizing to out-of-distribution samples, which NAT methods lack or are still weak at. On the other hand, weak dependency exhibits stronger robustness to input perturbations, as it is less affected by exposure bias (Bengio et al., 2015; Ranzato et al., 2016). Future research on NAT should focus on how to consolidate explicit language dependency while maintaining decoding efficiency. We release our resources at https://github.com/HJZnlp/NAT_ vs_AT.

heading: 2 Method
text: We begin with a brief introduction to autoregressive and non-autoregressive machine translation, before introducing four representative NAT methods.

heading: 2.1 Neural Machine Translation
text: The machine translation task can be formally defined as a sequence-to-sequence generation problem, where the model generates the target language sequence y = {y1, y2, ..., yT } from the target vocabulary V , given the source language sequence x = {x1, x2, ..., xS} based on the conditional probability pθ(y|x) (θ denotes the model parameters). Autoregressive Translation. Autoregressive neural machine translation factorizes the conditional probability to ∏T i=1 p(yi|y1, ..., yt−1,x), where the model is trained in a teacher-forcing way with cross-entropy (XE):
LAT = − log p(y|x) = − T∑
i=1
log pθ(yi|x, y<i).
(1) During inference, the model sequentially generates tokens based on previous predictions.
Non-autoregressive Translation. In contrast, non-autoregressive machine translation (Gu et al., 2018) ignores the dependency between target tokens and factorizes the probability as ∏T i=1 p(yi|x), where tokens at each time step are predicted independently. Vanilla NAT models are optimized with XE loss with target dependency ignored:
LNAT = − log p(y|x) = − T∑
i=1
log pθ(yi|x), (2)
with an additional loss for length prediction:
Llength = − log pθ(T |x). (3)
Challenges of NAT. The major difficulty of nonautoregressive translation lies in that the decoder side relies solely on the source-side information without any target inputs, e.g., history predictions in AT. Autoregressive models utilize previous token predictions to select the next token from the distribution over the whole vocabulary space:
pθ(yi|y<i,x) = softmax(WPTransformer(y<i,x), (4) where WP is the vocabulary projection weight. The inter-token dependency involves layers of Transformer blocks. In contrast, NAT models generate translations in a "one-shot" manner, ignoring or weakening the strong language dependency on the target side. As a result, vanilla NAT is not capable of properly modeling the highly multi-modal
distribution of target translations, i.e., a source sentence can have multiple valid translations. Various methods aim to alleviate the conditional independence assumption. In this work, we consider four representative methods: (1) alternative optimization with model architecture unchanged (Li et al., 2022); (2) introducing latent alignments based on an upsampled decoder prediction (Saharia et al., 2020); (3) building shallow but explicit target-side dependency (Huang et al., 2022b); and (4) iterative decoding (Ghazvininejad et al., 2019).

heading: 2.2 NAT with Advanced Optimization
text: Instead of exerting token-by-token cross-entropy supervision, Li et al. (2022) propose multigranularity optimization (MgMO) to collect multigranularity feedback on generations sampled from the models and gather them for backpropagation:
LMO = − K∑
k=1
qθ(h k|x)R(hk,yk), (5)
where K is the sample space size. qθ(hk|x) is defined as the normalized probability for each hypothesis hk:
qθ(h k|x;α) = p̂θ(h k|x)α∑ h′∈K(x) p̂θ(h ′|x)α , (6)
where K(x) denotes the sample space and α controls the distribution sharpness. R(h,y) is a reward function that encourages the generations to be similar with references under various granularity. MgMO requires no architecture modification and thus maintains decoding efficiency.

heading: 2.3 NAT with Latent Alignments
text: Saharia et al. (2020) introduce latent alignment models, e.g., Connectionist Temporal Classification (CTC) (Graves et al., 2006), to mitigate the target-side independence assumption. CTC utilizes a sequence of discrete latent alignment variables to monotonically align the non-autoregressive predictions of the model and target side tokens. The marginal probability over latent alignments a is derived as:
LLA = − log pθ(y|x) = − log ∑
a∈β(y) pθ(y|a,x)pθ(a|x), (7)
where β(y) is a function that returns all possible alignments for a sequence y. Then a =
{a1, . . . , aM} is predicted by the decoder output states H = {h1, . . . ,hM}, where ai ∈ V ∪ {“_”}. “_” is a special blank token to allow many-to-one and null alignment. For instance, for a target sequence “thank you”, valid alignments a include “_ thank thank you” and “thank _ you _”. The decoder state length is set as several times the source sequence length to allow long translations. The alignment probability pθ(a|x) is derived by:
pθ(a|x) = M∏
i=1
pθ(ai|x)
=
M∏
i=1
softmax(WPhi). (8)
Since ai ∈ V ∪ {“_”}, the posterior probability of y becomes:
pθ(y|a,x) = { 1 if a ∈ β(y) 0 otherwise.
(9)
MgMO and CTC avoid token-by-token CE supervision by introducing segment-level optimization or marginalizing latent alignments. However, they suffer independence assumption in generating tokens (Equation 2) or alignments (Equation 8). Consequently, both MgMO and CTC cannot inherently handle multi-modal problems and heavily rely on techniques such as knowledge distillation (Zhou et al., 2020a) to mitigate this limitation.

heading: 2.4 NAT with Explicit Dependency
text: Huang et al. (2022b) propose directed Acyclic Transformer (DAT) to construct explicit dependencies, by formalizing an alignment as a path in a direct acyclic graph. Similar to CTC, the decoder state length is upsampled to M and H = [h1, . . . ,hM ] denotes the decoder output hidden states, which are defined as the vertex states. The probability of path a is redefined as the position transition probability:
pθ(a|x) = ∏
i
pθ(ai+1|ai,x) = ∏
i
Eai,ai+1 ,
where E ∈ RM×M is the transition matrix normalized by rows. a = {a1, a2, . . . , aT } is a possible path represented by a sequence of vertex indexes of the vertex states H, i.e., ai ∈ {1, 2, 3, . . . ,M}. Specifically, the transition matrix is obtained by:
E = softmax( QKT√
d ), (10)
Q = HWQ, K = HWK,
where d is the hidden size, WQ and WK are learnable matrices. Conditioned on the vertex states in H and the selected path a, the posterior probability of y is computed as:
pθ(y|a,x) = T∏
i=1
Pθ(yi|ai,x)
= T∏
i=1
softmax(Wphai), (11)
where hai is the representation of the i-th vertex on the path a.
Different from previous NAT methods, DAT explicitly models token dependencies through vertex transitions. DAT first parallelly predicts a subset of all possible tokens for translating the source sentence and stores it as H, whose size is usually several times (e.g., 8) that of the source sequence. In contrast to Equation 4, the inter-token dependency is a one-step local transition for each vertex hi, to determine the next token from the rest of the set, i.e., {hi+1, . . . ,hM}:
pθ(y) = T∏
i=1
pθ(yai |yai−1), (12)
pθ(yai |yai−1) = softmax(WPhargmax(Eai−1,ai )), (13)
where yai is the predicted token of the i-th vertex on the path a*. The explicit though weak dependency modelled by one-layer linear weights WQ and WK alleviate the necessity of knowledge distillation, yet suffering n-gram repeating issues (discussed in Section 4.3).

heading: 2.5 NAT with Iterative Refinement
text: The iterative NAT model (Ghazvininejad et al., 2019) is typically trained with conditional masked language modeling (CMLM) to build inter-token dependencies:
LCMLM = − ∑
yt∈Y(y) logpθ(yt|Ω(y,Y(y)),x),
(14) where Y(y) is a randomly selected subset of target tokens and Ω denotes a function that masks a selected set of tokens in Y(y). During decoding, starting from a sequence of initiative tokens, e.g., “<unk>”, CMLM models iteratively refine translations from previous iterations to generate target language sequences.
*We omit conditional dependency on x for simplicity.

heading: 3 Experiment and Setup
text: Datasets and Models. We conduct experiments on WMT16 En⇒Ro and WMT21 De⇒En with 4 representative NAT methods apart from the vanilla NAT and AT. For knowledge distillation, We train an autoregressive model on the raw data as the teacher model to generate the distilled dataset. To achieve peak performance, we employ glancing training (Qian et al., 2021) for CTC and DAT training. Based on empirical results, we adopt standard beam search for all models during inference. Further details are provided in Appendix C.
Evaluation. For translation quality, we adopt four commonly used metrics, which include two rule-based metrics, i.e., BLEU score(Papineni et al., 2002) and chrf (Popović, 2015), and two modelbased metrics, i.e., COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020). Specifically, for COMET, we utilize the wmt22-comet-da model (Rei et al., 2022), and for BLEURT, the BLEURT20 model (Pu et al., 2021) is employed. Kocmi and Federmann (2023) propose a GPT-based metric, namely GEMBA, to evaluate translation quality, and demonstrate state-of-the-art correlation with human labels. We adopt GEMBA-GPT4-DA based on GPT-4 (OpenAI, 2023) as an advanced evaluation metric. For human evaluation, we follow (Freitag et al., 2021), an evaluation methodology based on the Multidimensional Quality Metrics (MQM) framework, which provides a hierarchical analysis of translation errors. Human evaluation details can be found in Appendix D.

heading: 4 Translation Quality

heading: 4.1 Automatic Evaluation
text: The automatic evaluation results on WMT16 En⇒Ro and WMT21 De⇒En are presented in Table 1. DAT obtains the most competitive performance compared with the AT counterpart across all automatic metrics, followed by MgMO and CTC. MgMO and CTC achieve stronger performance than the representative iterative method, CMLM, when considering COMET and GEMBA which have shown better correlation with human annotation (Rei et al., 2020; Kocmi and Federmann, 2023). Notably, MgMO obtains comparable performance with CTC, without modifying model architecture.
Reliance on Knowledge Distillation. In both translation directions, all fully non-autoregressive methods except DAT and CMLM suffer more from
training without distillation. Typically, the vanilla NAT models suffer a decrease of more than 7 BLEU points without KD. For strong NAT methods such as MgMO and CTC, on WMT21 De⇒En, the BLEU scores decrease by more than 2 and 3 points, respectively. On the contrary, the performance of DAT and CMLM is as similarly affected as the AT counterpart, due to explicit dependency modeling similar to AT. In the subsequent sections, we utilize knowledge distillation by default to analyze NAT models in the best-performing setting.
Evaluation Metrics. We consider a set of representative metrics to comprehensively compare NAT methods with AT. We perform significance tests on all pairs of NAT models and their AT counterparts across all metrics. Except for DAT, current NAT methods significantly underperform AT methods in various evaluation metrics including rule-based (BLEU and chrf), model-based (COMET and BLEURT), and GPT4-based metrics (GEMBA), particularly in the raw data setting. A notable observation is that DAT models are more competitive with AT models when evaluated using rule-based metrics, which assess the similarity between generated text and references. In contrast, AT models outperform DAT models significantly under model-based metrics or GPT4 evaluation (GEMBA). These metrics evaluate translation quality by measuring semantic similarity between two sentences based on parametric knowledge. To gain a deeper understanding of this phenomenon, we conduct human evaluation using a systematic and fine-grained framework, i.e., MQM (Freitag et al., 2021), to further compare NAT with AT.

heading: 4.2 Human Evaluation
text: The evaluation results, obtained by averaging the error counts from three translators, are presented in Table 2. We omit human evaluation on the vanilla NAT due to its poor performance under automatic evaluation. The performance ranking of human evaluation aligns with the automatic evaluation: AT performs the best, followed by DAT, MgMO, CTC, and CMLM. Models with explicit dependency modeling (AT, DAT and CMLM) generate more fluent translations than those without (MgMO and CTC). , with fewer fluency errors. Despite comparable fluency to AT, DAT exhibits low translation accuracy. All NAT methods, particularly CMLM, generate non-translations in certain cases.
A fine-grained error visualization is presented in Figure 1. The mistranslation error type at the major level has the highest proportion among all models, with the models lacking explicit dependency (MgMO and CTC) producing the most errors. AT performs generally better than NAT except for a considerable number of omission errors. In contrast, NAT models tend to generate translations with additional or duplicated content (addition),
particularly CTC and DAT which increase decoder length to model latent alignments. These two models also exhibit more spelling errors. Compared to AT, NAT models tend to produce more punctuation errors and grammar errors. Similar to AT, MgMO and CTC translations also frequently lack partial source content (omission).
We explore human annotations to understand typical patterns. Regarding omission errors, AT often exhibits incomplete generation at the sentence’s end. On the other hand, MgMO and CTC frequently omit content throughout the entire sentence, such as missing adjectives or verbs. The NAT’s grammar errors primarily stem from incorrect verb tense and singular/plural usage, resulting from its limited language dependency modeling. The case study indicates that, for CTC, the major addition errors are attributed to generating words with spelling errors, which are regarded as irrelevant content by annotators. For DAT, these addition errors stem from n-gram repetition, where the model generates a repeated segment from the previous context. For example, “By the beginning of November, there are seven races until the beginning of November.” To give an intuitive representation, we present several cases for the aforementioned error types in Appendix G. All these patterns can be attributed to inadequate language dependency modeling with limited or redundant decoding length.

heading: 4.3 Effects of Explicit Dependency
text: Repetition Ratio. We first examine token repetition ratio (Zhou et al., 2020a; Ghazvininejad et al., 2020; Du et al., 2021) in model translations, which is the ratio of generations with repeated tokens, e.g., “He is is a lawyer”. The results are shown in Table 3. We can observe that models without latent alignment modeling (NAT, MgMO and CMLM) suffer severe token repetition during generation.
N-gram Repetition. Besides consecutive unigram repetition, a more subtle phenomenon is nonadjacent n-gram repetition. Such a repetition can be overlooked under traditional metrics such as BLEU
score, which only calculates the n-gram precision of the generations. Consequently, translations that contain n-gram repetition may even achieve higher BLEU scores. This could explain why DAT performs better than AT under rule-based metrics but not under model-based or GPT4-based metrics. We collect the n-gram repetition count for each model, as shown in the left part of Figure 2. We can observe that DAT demonstrates a stronger tendency to generate repeating n-grams with higher counts across various n-gram granularity (2 to 10), which aligns with a substantial number of addition errors in human evaluation.
Enhancing Dependency Modeling. DAT utilizes one-linear-layer attention modules to model local vertex transitions. Such explicit dependency modeling can be limited when dealing with long sequence generation. For example, consider the sentence "By the beginning of November, there are seven races until the beginning of November." In this case, both the beginning and the end of the sentence are valid positions for the temporal prepositional phrase "the beginning of November" In DAT, at the vertex state corresponding to the token "races," only information from that current vertex state is used to determine the index of the next vertex state using one-linear-layer attention layers. In contrast, AT considers all previously generated tokens and utilizes Transformer decoder layers to determine the next token. Under weak dependency modeling in DAT, early generations can be ignored and repeated phrases can be falsely pointed to (e.g., “the beginning of November”). To validate this assumption, we train an asymmetrical AT model with a shallow decoder to simulate weak dependency modeling, and a deep encoder to guarantee model size. As shown in Figure 2 (right-side), AT11/1 (11-layer encoder and 1-layer decoder) also tends to generate repeated n-grams, and adding one decoder layer (AT-10/2) mitigates this issue. Nev-
ertheless, AT-11/1 performs better than DAT as it relies on the entire generation history rather than just considering the current token. To alleviate this issue without influencing decoding efficiency, we introduce an additional linear layer for both Q and K to strengthen the token transition modeling. This refined model is referred as DAT∗ (Appendix H). With 0.7% additional parameters compared to DAT alone, we observe that DAT∗ exhibits less n-gram repetition while maintaining decoding speed. We present a case study of how DAT∗ alleviates n-gram repetition in Appendix G. Nevertheless, DAT∗ is limited in a one-step local transition foundation and cannot fundamentally resolve n-gram repetition.

heading: 5 Generalization and Robustness
text: Length Generalization. Figure 3 illustrates that all models experience a decline in performance as the length of the source sequence increases, albeit at varying rates. AT surpasses all NAT methods in length generalization. Models incorporating explicit dependency (e.g., AT, DAT, and CMLM) exhibit slower degradation compared to others. Notably, CTC and CMLM experience severe performance drops on sequences longer than 60.
Cross-domain Generalization. Figure 4 illustrates the cross-domain performance averaged across 5 domains. Models with explicit dependency, such as AT and DAT, achieve high crossdomain performance. On the other hand, CTC and CMLM demonstrate substantial degradation in performance when tested on out-of-domain datasets This is due to CTC models generating spelling errors and CMLM models propagating errors from early steps. These issues are further exacerbated in cross-domain testsets that contain more terminologies, leading to subpar performance. The complete
results can be found in Appendix I.
Compositional Generalization. We measure compositional generalization on GoGnition (Li et al., 2021) which evaluates the ability to translate unseen phrases of simple and known semantic units. The results are shown in Table 4 †. Instancelevel CTER and Aggregate-level CTER denote the compound translation error rates of translating novel compounds. Despite the narrowing gap in in-domain and out-of-domain testsets, we observe a significant difference in compositional generalization between DAT and AT. This discrepancy is reflected in higher error rates, indicating a disparity in dependency modeling capabilities.
Robustness to Input Perturbations. Finally, we explore models’ robustness to different input perturbations, including random replacement, deletion and permutation (Details in Appendix C), with results shown in Figure 5. In contrast to previous findings that suggest explicit modeling provides advantages, models without explicit incorporation of modeling (e.g., MgMO and CTC) are less affected by input noises. This can be because explicit dependency generation may introduce exposure bias (Bengio et al., 2015; Ranzato et al., 2016), where errors occurring at early time steps (AT and DAT) or iterative steps (CMLM) can accumulate and propagate into future predictions, making them susceptible to input perturbations. For complete results, please refer to Appendix J.
To the best of our knowledge, this is the first comparison of NAT and AT in terms of generalization and robustness. In addition to the disparity
†We only evaluate AT and DAT as they do not rely on knowledge distillation.
in translation performance on benchmark datasets, inadequate language dependency modeling causes NAT methods to significantly lag behind AT. However, this weak dependency does provide an advantage in resisting input perturbations.

heading: 6 Related Work
text: We discuss several representative NAT methods in Section 2. A more detailed discussion on NAT advances is presented in Appendix A. Different from surveys (Xiao et al., 2023; Li et al., 2023) that conduct a comprehensive survey on recent NAT advances, we focus on comparing NAT with AT comprehensively. Our work is also related to previous work analyzing neural machine translation (Appendix B). Zhou et al. (2020a) find that knowledge distillation boosts NAT performance by reducing data complexity. Agrawal et al. (2022) discuss knowledge transfer in the context of multilingual NAT. Huang et al. (2022a) understand the learning process of NAT both theoretically and empirically. Differently, we focus on systematically comparing common NAT techniques with their AT counterparts in a systematic manner to showcase existing performance gaps for future research.

heading: 7 Conclusion
text: We compared representative NAT methods with AT under a comprehensive evaluation that encompasses a set of evaluation dimensions, including human evaluation. Our research aims to fill in the research gap of the real competitiveness of NAT to AT. Both automatic and human evaluations indicated that despite the narrowing gap, NAT methods underperform AT, with varying error patterns such as translation omission, spelling errors and n-gram repetitions. Our empirical results and analyses demonstrated that explicit dependency model-
ing is crucial for generating human-like languages, although strong dependence can suffer explore bias. Future research on NAT should focus on how to consolidate explicit language dependency while maintaining decoding efficiency.
Limitations
We systematically evaluate NAT and AT, highlighting performance gaps for future research. However, there are limitations: Firstly, we assess stateof-the-art NAT models using research-oriented datasets (WMT, OOD, CG), which mainly consist of English-centric text with a formal style and limited topic range. Secondly, each NAT model is annotated with only 100 samples. This may not cover all potential error types. Finally, we focus primarily on fully non-autoregressive methods due to their superior decoding efficiency. Our results are also limited to training-from-scratch methods; extending conclusions to large language models is left for future work.
Ethical Considerations
We honor the ACL Code of Ethics. No private data or non-public information is used in this work. For human annotation, we hired three annotators who have degrees in English Linguistics or Applied Linguistics. Before formal annotation, annotators were asked to annotate 100 samples randomly extracted from the dataset, and based on average annotation time we set a fair salary (i.e., 32 dollars per hour) for them. During their training annotation process, they were paid as well. The annotation does not involve any personally sensitive information. The annotation strictly follows the annotation guide of MQM (Freitag et al., 2021), with details presented in Appendix D. We adhere to the terms of companies offering commercial LLM APIs and express our gratitude to all global collaborators for their assistance in utilizing these APIs.

heading: Acknowledgement
text: We would like to thank all reviewers for their insightful comments and suggestions to help improve the paper. This work has been supported by the China Strategic Scientific and Technological Innovation Cooperation Project Grant No. 2022YFE0204900 (The Macao counterpart project Grant No. FDCT/0070/2022/AMJ). This work is also supported by a grant from Lan-bridge Information Technology Co., Ltd. We extend our gratitude
to colleagues at Lan-bridge for their professional annotations and evaluations of model translations. Special thanks to Xianchao Zhu and Jing Li for their invaluable assistance in organizing the annotation process.

heading: A Recent Advances in NAT
text: Various techniques have been proposed to address the performance limitations of NAT. Guo et al. (2020); Liu et al. (2020); Ding et al. (2021a); Qian et al. (2021) devise dedicated training curriculums to reduce the learning difficulty of NAT models, whereas Zhou et al. (2020b); Ding et al. (2021c,b) propose improved distillation training. Latent variable modeling has received significant attention in enhancing NAT performance (Libovický and Helcl, 2018; Kaiser et al., 2018; Ma et al., 2019; Saharia et al., 2020; Bao et al., 2022, 2021). Typically, Huang et al. (2022b) explicitly models target dependency as paths in a directed acyclic graph. Another line of research focuses on enhancing the cross-entropy loss or alternating to metric-based objectives (Sun et al., 2019; Shao et al., 2020; Ghazvininejad et al., 2020; Du et al., 2021; Shao et al., 2019; Li et al., 2022). In contrast to fully non-autoregressive methods mentioned earlier, another approach decomposes one-shot generation into multiple iterative non-autoregressive generations (Gu et al., 2019; Ghazvininejad et al., 2019; Huang et al., 2022c). Schmidt et al. (2022) align common NAT techniques and compare translation quality and speed implications under uniform environments. Despite claiming improved performance and comparability with autoregressive models (AT), these approaches are limited in their evaluation using rule-based metrics like BLEU score (Papineni et al., 2002), which demonstrates poor correlation with human preference (Rei et al., 2020; Freitag et al., 2022).

heading: B Analysis Research in NMT
text: Voita et al. (2021) interpret NMT’s learning process during training, and Ferrando et al. (2022); Yan et al. (2022) interpret and analyze model predictions during inference. Müller et al. (2020a) study NMT generalization ability to novel domains, whereas Li et al. (2021) demonstrate that NMT’s weak compositional generalization capability. Additional metrics proposed by Niu et al. (2020) quantify the effects of input perturbations. Hallucination, which refers to the generation of unrelated outputs by the model, has also been extensively studied (Xu et al., 2023; Guerreiro et al., 2023).

heading: C Experiment Setup
text: Datasets To evaluate general translation performance, we choose WMT16 En⇒Ro, a widely used
benchmark dataset for non-autoregressive translation. In addition, we select a large-scale benchmark dataset, i.e., WMT21 De⇒En, which consists of 101.35M parallel sentences and is further filtered to 88.66M. We apply BPE (Sennrich et al., 2016) on the concatenated training sets with 32,000 operations. Knowledge distillation is commonly used for training NAT models (Gu et al., 2018; Sun et al., 2019; Ghazvininejad et al., 2019, 2020). We train an autoregressive Transformer base model on the raw dataset as the teacher model and use it to generate the distilled dataset. To assess cross-domain translation, we employ the test sets from (Müller et al., 2020b), which encompass test instances from 5 domains: medical, IT, koran, law, and subtitles, and we evaluate the models (trained on WMT21 De⇔En) on these test sets. For compositional generalization, we utilize CoGnition (Li et al., 2021) with its original data configurations. Following the approach in (Edunov et al., 2018), we measure model robustness on the WMT21 De⇒En testset by introducing three types of input noise: (1) word deletion with a probability of 0.1; (2) word replacement with "<unk>" with a probability of 0.1; (3) word swapping within a range of 3 words with a probability of 0.1.
Model Settings We adhere to the bestperforming model configuration outlined in the corresponding papers (Vaswani et al., 2017; Gu et al., 2018; Saharia et al., 2020; Li et al., 2022; Huang et al., 2022b; Ghazvininejad et al., 2019). For all models, we utilize Transformer with a Transformer_Base configuration: both the encoder and decoder comprise 6 layers with 8 attention heads. The hidden dimension is set to 512, while the feedforward layer dimension is set to 2,048. The model is trained using Adam (Kingma and Ba, 2015) optimizer. We apply a weight decay of 0.01 and label smoothing of 0.1. The learning rate initially increases to 5 · 10−4 within the first 10K steps and subsequently decays exponentially. For glancing training, the glancing probability is gradually annealed from 0.5 to 0.1 in 200k steps. All results are based on models trained on the KD dataset unless otherwise stated. For inference, we present results obtained through beam search with a beam size of 5. In the case of iterative models such as CMLM, we set the number of iterative steps as 10. Following the official
DA-Transformer guidelines‡, we compared Viterbi Decoding with beam search and found that beam search slightly outperformed Viterbi in terms of BLEU and COMET metrics. Therefore, we chose beam search to present DAT’s best performance. The performance comparison, including BLEU, COMET and n-gram repetition is presented in the following table 5. Despite Viterbi Decoding’s reduction in n-gram repetition, this issue was still more pronounced in DAT compared to AT and other NAT methods, aligning with the conclusion in our paper. We utilized 4 NVIDIA V100 GPUs for our computations, dedicating two days for the CTC process and five days for DA. Other methods were executed within one day each.

heading: D Human Annotation
text: We follow Freitag et al. (2021), an evaluation methodology based on the Multidimensional Quality Metrics (MQM) framework, which provides a hierarchical analysis of translation errors. We adopt two common error hierarchy categories: Accuracy and Fluency. Accuracy covers fine-grained 4 error sub-types such as Addition, Omission, Mistranslation and Untranslated Text, whereas Fluency covers Punctuation, Spelling, Grammar, Register, Inconsistency and Character Encoding. Translations that are too badly garbled to permit error classification are classified as Non-translation. In addition to the error type, each error is also annotated with a severity label: minor and major. We follow the error weighting in Freitag et al. (2021) to compute the weighted error counts for each system. Annotation details are presented in Appendix E. We hire three expert translators to conduct side-by-side human evaluations on the 5 German-English translation models, i.e., AT, NAT, MgMO, CTC, DA and CMLM. We randomly sample 100 translations from the WMT21 De⇒En testset and ask translators to annotate translation errors for each instance
‡https://github.com/thu-coai/DA-Transformer/ tree/v1.0
following the MQM annotation guideline. We average the error counts from the 3 annotators as human evaluation results. For conducting human annotation, we hired three annotators who have degrees in English Linguistics or Applied Linguistics. Before formal annotation, annotators were asked to annotate 100 sampled translations from 5 systems, and based on average annotation time we set a fair salary (i.e., 32 dollars per hour) for them. During their training annotation process, they were paid as well.

heading: E MQM Annotation
text: We present the details of the error type description in Table 7, the error severity description in Table 8 and error weights in Table 6.

heading: F Human Evaluation Results
text: The annotation results (average from 3 translators) are presented in Table 9.

heading: G Case Study
text: We present a case study to showcase the n-gram repetition phenomenon in Table 10. We present several cases to showcase the spelling errors of CTC and DAT in Table 11. A case study of omission errors is shown in Table 12. A case study of grammar and punctuation errors is shown in Table 13. A case study of how DAT∗ alleviates n-gram repetition is presented in Table 14.

heading: H Model Details of DAT∗
text: To strengthen the inter-token dependency of DAT, we increase the depth of the transition model by en-
coding Q in Equation 10 with an additional linear layer:
Q∗ = ReLU(Q)W∗Q, (15)
where ReLU is the rectified linear unit activation function. The same applies to K. We refer to this model as DAT∗.

heading: I Cross-domain Performance
text: The complete cross-domain performance on 5 De⇒En out-of-domain testsets are presented in Table 15.
Compositional generalization in NMT refers to the model’s generality to translate compounds (e.g., phrases) of known semantic units (e.g., words). We test AT and DAT on the CoGnition dataset since they do not rely on knowledge distillation, and present the results in Table 4. As shown, DAT underperforms the AT counterpart in compositional generalization by a considerable margin, due to its weak dependency modeling. DAT∗

heading: J Robustness to Noisy Input
text: The translation performance on the WMT21 De⇒En testsets with different types of noises are shown in Table 16, where “None” denotes the performance on the original testset without noise.
Case 1
Source Sentence In Sachen Kindergarten- respektive Krippenplätzen hat sie bereits Kontakt mit einer örtlichen Einrichtung aufgenommen.
Reference Sentence Regarding kindergarten respectively nursery places she has already established contact with the local facilities.
Case 1
Source Sentence Bis Anfang November stehen sieben Rennen an.
Reference Sentence Until the beginning of November seven races are planned.
DAT Vertex Predictions <BOS> By The Seven There Seven races are been races By the As of early beginning of the beginning of of November , there there will there will be be been are been are seven seven event@@ seven races races have seven seven the races p@@ races are place have run are been planned in p@@ ending scheduled until the run scheduled p@@ ending up by beginning the early beginning beginning of early of November November beginning November . <EOS>

heading: Future Work
text: Future research on NAT should focus on how to consolidate explicit language dependency while maintaining decoding efficiency. Limitations
We systematically evaluate NAT and AT, highlighting performance gaps for future research. However, there are limitations: Firstly, we assess stateof-the-art NAT models using research-oriented datasets (WMT, OOD, CG), which mainly consist of English-centric text with a formal style and limited topic range. Secondly, each NAT model is annotated with only 100 samples. This may not cover all potential error types. Finally, we focus primarily on fully non-autoregressive methods due to their superior decoding efficiency