Improving Non-Autoregressive Translation Models Without Distillation
Authors: Xiao Shi Huang
Felipe Perez
Maksims Volkovs
Venue: ICLR 2022 Poster (2022)
Keywords: Natural Language Processing
Deep Learning
Non-autoregressive Machine Translation
Transformer
Distillation

Abstract: Transformer-based autoregressive (AR) machine translation models have achieved significant performance improvements, nearing human-level accuracy on some languages. The AR framework translates one token at a time which can be time consuming, especially for long sequences. To accelerate inference, recent work has been exploring non-autoregressive (NAR) approaches that translate blocks of tokens in parallel. Despite significant progress, leading NAR models still lag behind their AR counterparts, and only become competitive when trained with distillation. In this paper we investigate possible reasons behind this performance gap, namely, the indistinguishability of tokens, and mismatch between training and inference. We then propose the Conditional Masked Language Model with Correction (CMLMC) that addresses these problems. Empirically, we show that CMLMC achieves state-of-the-art NAR performance when trained on raw data without distillation and approaches AR performance on multiple datasets. Full code for this work will be released at the time of publication.

Published as a conference paper at ICLR 2022
IMPROVING NON-AUTOREGRESSIVE TRANSLATION MODELS WITHOUT DISTILLATION
Xiao Shi Huang, Felipe Pérez, Maksims Volkovs
Layer 6 Al
{gary, felipe, maks) ) 1ayeers.al ai
ABSTRACT
Transformer-based autoregressive (AR) machine translation models have achieved significant performance improvements, nearing human-level accuracy on some languages. The AR framework translates one token at time which can be time consuming, especially for long sequences. To accelerate inference, recent work has been exploring non- -oooreeeeeeresivvee (NAR) approaches that translate blocks of tokens in parallel. Despite significant progress, leading NAR models still lag behind their AR counterparts, and only become competitive when trained with distillation. this paper we investigate possible reasons behind this performance gap, namely, the indistinguishabiiity of tokens, and mismatch between training and inference. We then propose the Conditional Masked Language Model with Correction (CMLMC) that addresses these problems. Empirically, we show that CMLMC achieves state- of-the-an art NAR performance when trained raw data without distillation, and approaches AR performance on multiple datasets. Code for this work is available here: httts://githute com/ ayer6ai abs /CMLMC
INTRODUCTION
Neural machine translation (NMT) models based the Transformer architecture have achieved
leading performance (Vaswani et al., 2017 Barrault et al., 2019 Huang et al., 2020). Majority of the proposed approaches are based the autoregressive (AR) principle, where translation is done one token at a time conditioning on already translated tokens. AR inference scales linearly with the number of tokens and full forward pass through the decoder is required for each translated token. This can be prohibitively expensive for long sequences, particularly as leading models are becoming increasingly larger size. To mitigate this problem, recent works have explored the non-autoregressive (NAR) approach where subsets of tokens are translated in parallel (Gu et al. 2018) Ghazvininejad al. 2019 Kasai et al. 2020). NAR models achieve significantly faster inference speed that no longer depends on sequence length. However, despite considerable progress, leading NAR models still require sequence-level knowledge distillation (Kim & Rush 2016) to achieve competitive accuracy. In practice, a large AR Transformer model trained the raw data is used as the teacher for distillation (Ghazvininejad et al. 2019). This process is expensive, as every new language pair requires training new teacher. It is also non-standard, and raises questions to the necessity and the underlying problems solved by distillation (Zhou et al. 2020 Ding et al. 2021).
of the leading NAR approaches, the Conditional Masked Language Model (CMLM) (Ghazvininejad et al., 2019). CMLM achieved leading NAR performance on multiple NMT datasets especially when combined with semi-autoregrresive training (Ghazvininejad et al. 2020b)- but only when the model is trained on distilled data. Without distillation, CMLM performance drops significantly below AR benchmarks. The need for distillation indicates that CMLM alone is unable to fully leverage the information available in the raw training data (Ding et al., 202T). Here, we identify two shortcomings of CMLM that, when addressed, significantly improve NAR translation quality and narrow the gap between raw and distilled performance.
In this work we focus
First, input token representations in CMLM can become nearly indistinguishable, especially for adjacent positions. In AR models this problem is avoided by a combination of causal masked attention, sequential inference, and learned positional encodings (PEs). However, unmasked attention and simultaneous translation of token blocks in CMLM loses most of the information that distinguishes tokens. This problem is particularly severe during the first inference step, where the input is fully masked. The model thus only relies learned PEs to distinguish tokens, which is not sufficient. Poor token separation can cause significant translation errors, including the identified phenomenon of token repetition stemming from the related multi-modall problem (Zhou et al. 2020).

---

Published as a conference paper at ICLR 2022
Second, there is a misalignment between CMLM's training and inference procedures. During training CMLM is optimized with a masked loss analogous to language model training in popular models such BERT (Devlin et al., 2019). However, CMLM inference always starts with a fully masked sentence and translates all tokens simultaneously. Iterative refinement is then applied where subsets of low confidence tokens are masked and re-translated at each iteration. During training the model rarely sees a fully masked sentence, and is not trained to self-correct from the initial fully masked translation that can contain significant errors. The misalignment between the two procedures can cause disconnect, where optimization of the training loss does not transfer to improvements in translation quality.
In this work we propose the Conditional Masked Language Model with Correction (CMLMC). Our model builds on the CMLM architecture and addresses the aforementioned problems. We modify the decoder structure by exposing the positional encodings and incorporating causal attention layers to differentiate adjacent tokens. We also propose a novel correction loss that teaches the model how to correct translation mistakes made in early decoding iterations from the fully masked sentence. With these improvements, CMLMC achieves new state-of-the-art undistilled NAR results and approaches AR performance multiple NMT benchmarks.
2 RELATED WORK
Neural machine translation is sequence to sequence prediction problem where source sentence
X (zi,...., , )mm) in one language is transformed into target sentence Y (V)....: ynu) in
another language. In AR setting this problem is modelled as: arg maxy I[ P(y xx))) where
Yzı = (yi,..., D1-1) so every token is conditioned on all translated tokens before it (Cho et al. 2014).
Greedy inference approach is typically taken where tokens translated left-to-right sequentially. so a sequence of length n requires n forward passes through the model. Transformer- based AR models currently achieve leading accuracy most NMT benchmarks (Barrault et al. 2019), but as models become larger linear inference can become prohibitively expensive for long sequences.
The NAR approach is proposed to mitigate this problem. Early work NAT (Gu et al., 2018) models the problem as: arg maxy P(n I[ [[] P(y: where X) is the probability over sequence length. Since [ (X) is independent of Y, inference can be parallelized and all tokens (or their subsets) can be translated simultaneously. The conditional independence assumption can be overly restrictive and typically does not hold in natural language. To relax it, NAT adds positional attention and full self-attention between all tokens. However, the performance of this model is still considerably worse than the AR benchmarks, which authors attribute to the multi-modality problem (Gu et al.. 2018). NAT tackle this problem by applying knowledge distillation (Kim & Rush] 2016) on the training data, where a large AR model is used as a teacher. Distillation significantly improves performance, and has since been widely adopted to reduce the performance gap to AR benchmarks.
Recently, multiple NAR approaches have been proposed that build on the work of NAT (Gu et al., 2018). Flowseq (Ma et al. 2019) introduces latent variables and generative flow to decouple token dependencics in the output. CMLM (Ghazvininejad et al. (019), the focus of our work, extends masked language loss training (Devlin et al. 2019) to NMT to facilitate more robust handling of target language input initialization and iterative refinement. Several works build on CMLM to improve performance such as incorporating the energy of teacher optimization (Tu et al.) 2020 adding raw data prior (Ding et al. 1) and improving the encoder-decoder architecture Kasai et al. 2021). DisCo (Kasai et al. 2020) utilizes attention masking instead of token masking to improve both training and inference efficiency. ReorderNAT (Ran et al.) (22T)) uses an explicit module to re-order the input sequence. GLAT (Qian et al., (202)) compares intermediate predictions with ground truth to dynamically adjust the masking percentage according to translation difficulty. Recent work, including AXE (Ghazvininejad et al., 2000a)). Imputer (Saharia et al.) (2020), FullyNAT (Gu Kong 202T), and OAXÉ (Du et al. 2 achieve improved performance with the alignment loss (Libovicky & Helcl 2018) that has been found to be more suitable for one-step NAR tasks (Haviy et al. 2021). This important direction is orthogonal to our focus in this paper, and can be readily incorporated in future work. The closest to our work is the SMART model (Ghazviminejad et al.) 2020b) which also focuses on the training-inferenc, gap in NAR training; and SMART in Section 3.2, Finally, it is important to note that while significant progress has been made, the majority of work in NAR has focused maximizing the distilled performance. Despite its
discuss the key differences between our approach
2

---

Published as a conference paper at ICLR 2022
importance, NAR performance raw data remains significantly below leading AR baselines and is the focus on this work.
3 CMLMC: CONDITIONAL MASKED LANGUAGE MODEL WITH CORRECTION
In this section, we analyze the CMLM architecture and identify several weaknesses in its design, namely, lack of token distinguishabiltyy and training-inference mismatch. We then propose the CMLMC model, which addresses these problems leading to significant improvements in accuracy.
3.1 TOKEN DISTINGUISHABILITY
To generate accurate translations the model must be able to align the source and translated sentences. This is particularly important in Transformer-basee architectures where source sentence is typically encoded as memory, and decoder attends over memory during translation. To infer which part(s) of the memory attend for given translated token, the decoder needs know exactly where the token is in the translated sequence and its relative position to other translated tokens. Transformer attention doesn't 't preserve positional information so AR models this is achieved by a combination of causal masked attention, positional encodings (PEs), and sequential inference.
Causal masked attention matrix has a lower- triangular shape so each token only attends over tokens that come before it. PEs (fixed or learned) get added to the input token em- beddings at each position, and enable the model to infer the relative and absolute dis- tance between tokens (Yan et al. 2019). Fi- nally, sequential inference translates one ken at time conditioning on all previously translated tokens, the model can identify exactly which token is being translated at each step. Jointly these design choices lead to adequate separation between tokens and provide strong positional information.
of Pairs 4k
CMLM
Sentences 6k
CMLM
RevealPosition
RevealPosition
2k
-.88
0.0 Cosine Similarity
of Token Repetitions
Figure 1: IWSLT' 14 De-En ((a) Binned cosine similar- ity between all unique pairs of the first 200 well-trained PEs. Disrrbbbtnnt of token repetitions across test sentences after initial translation from fully masked sen- tence. For each repetition count, we find the number of test sentences where least one token consecutively repeats that many times.
In NAR models, parallel translation significantly complicates sentence alignment. In particular, CMLM replaces causal masked attention with a full attention over all tokens. Unmasked attention combined with paraliel inference over all tokens implies that the model has to rely solely on PEs for position information. learned PEs for different positions are very similar to each other, CMLM would not be able to distinguish them. Token indistinguishabiliy is especially problematic during the first inference step since CMLM inference always starts with a fully masked sentence. During this step, all tokens are replaced with the same mask embedding <M> summed into PEs. The model thus has to translate the entire sentence using only PEs to distinguish the input tokens.
Figure ((a) shows binned cosine similarity between unique pairs of trained PE encodings learned by the CMLM model on IWSLT' 14 De-En dataset. We see that the similarity distribution has a long tail on the positive side. Significant number of PE pairs have cosine similarity above 0.5, and over 200 pairs have similarity greater than 0.7. These results indicate that during inference with a fully masked sentence, some token pairs can be nearly indistinguishable for CMLM. This can lead to severe translation errors, the most direct of which is token repetition where indistinguishable tokens get translated to the same word. Figure [(b) shows the distribution of consecutive token repetitions across test sentences on the same dataset. We see that CMLM can produce over 5 repetitions in translated sentence, and hundreds of sentences have 3 or more repetitions. It is evident that CMLM has difficulty translating from fully masked sentence, and bad initial translation can be difficult to correct even with multiple steps of iterative refinement.
We address the problem of token distinguishability with dual strategy. First, each decoder block in CMLM is augmented with a causal masked attention layer, inserted between the unmasked self attention and encoder attention layers, leading the following block structure: FULL-AIT + MASKED-ATT > ENCODER-ATT + FFN. Re-introducing masked attention brings back the left-to-right sequential hierarchy and breaks symmetry between positions with similar PEs. We
3

---

Published as a conference paper at ICLR 2022
deliberately insert this layer before the encoder attention, as it aligns source and translated sentences, so properly separating imput tokens is particularly important there. We use the standard masked attention layer where the upper triangular portion of the SoftMax matrix is set to 0.
Second, we modify how token embeddings and PEs are combined. The majority of transformer- based models simply sum the two embeddings. However, this can be insufficient to propagate the positional information to the upper layers since the model has to balance the scales of both embeddings. This can again be particularly problematic during the fully masked inference step when learned mask embedding <M> is added to PEs. <M> is shared across all masked positions, and in training receives gradient updates that are typically larger in magnitude than individual PEs. Empirically we found that gradient magnitude for <M> can be up to 50x larger than the average gradient magnitude for first 100 most commonly updated tokens throughout training. This can in turn make the magnitude of <M> much larger than PE, making it difficult for the model to distinguish PEs at different positions once they are summed with <M>. To deal with this problem we instead propose to combine token embeddings and PEs with feed-forward layer (FFN). Formally, given an input token embedding Y/: and positional encoding pe; at position i, we first expand by concatenating them and then shrink back down with an FFN: vo- (FV pe:l). Here, [. is the concatenation operation and FFN: R2d + Rd where d is the original embedding dimension. FFN enables the model to appropriately adjust the embedding scales and deepasspp specific dimensions.
The joint effect of these architecture modifications is shown in Figure I We refer to this method as RevealPosition. From Figure I(a) we see that PE cosine similarity distribution is now mainly clustered around O, and no pair has similarity above 0.5. This is significant improvement from CMLM where hundreds of PE pairs have cosine similarity above 0.7. Furthermore, from Figure [Tbb also see that RevealPosition consistently reduces the number of sentences that have repetitions across all repetition counts. In particular, the most frequent one and two token repetitions are reduced by over 30% and 35% respectively. In the experiments section we further show that these relatively simple modifications lead to considerable improvements in BLEU of over point. However, despite the reduction in repetitions. RevealPosition can still make mistakes, so in the next section we proposed a new loss function that aims to teach the model how to correct them.
3.2 TRAINING/INFERENCE MISMATCH
is generally accepted that training and inference procedures should match as closely as possi- ble (Ranzato et al. 2015 Mihaylova & Martins 2019). When this is the case, improving training loss during optimization typically translates to better inference performance. In AR models, training and inference are well aligned, in both cases the model translates one token at a time conditioned on all previously translated tokens. The only major difference is that during training previously translated tokens are fixed to ground truth, but during inference they are set to model predictions.
This is not the case many NAR models, including CMLM. The training strategy CMLM follows masked language model approach similar to BERT (Devlin et al. (019)) pretraining: random subset of tokens from the target sentence Y get masked, effectively splitting Y into masked tokens Y Ymask and observed ground truth tokens Yobs: masked tokens replaced with the mask embedding <M>, and the sentence is passed through the decoder to get predictions for Ymask- The loss function then aims maximize the probability of masked tokens for reconstruction accuracy. Unlike the fixed 15% masking typically used in BERT, CMLM uses a more aggressive strategy, sampling the number of masked tokens uniformly between 1 and sentence length. During inference CMLM always starts with fully masked sentence and translates all tokens. Iterative refinement are then applied, where a subset of tokens with the lowest probability is re masked and re- re-translated at every iteration.
Comparing these training and inference procedures we can see a mismatch. During training CMLM not trained to correct its predictions. In particular, it is not trained to recognize the errors made in the crucial first inference step. As we discussed in the previous section, translations from this first step can have significant errors (repeated tokens etc.), so learning to correct these errors can be critical to model performance. To better align training and inference, we build error correction mechanism into our model by adding a correction loss term during training. This term focuses on correcting mistakes after inference with fully masked input. Formally, given source sentence X and a target sentence Y, we first split Y into Yohs and Y Ymask as in CMLM training. We then apply the decoder to the fully masked sentence Ye, where every token is replaced with <M>, obtaining
4

---

Published as a conference paper at ICLR 2022
Lynaek
Leovr
work
Transformer llicadder
CML Decoder
CMLMC Decoder
CMLMC Decocer
Ground auth
Model Prediction
Subatittion Token
prbekon
NLP
<M
Your
Ymaak
Yerod
Ymash
Figure 2: CMLMC loss example. Here, source German sentence X= [wir arbeiten an NLP] is translated to the target English sentence Y= [we work on NLP]. First, sampled mask Y mask masks the [on] token. Masked sentence passed through the CMLMC decoder predict the masked out token in the Lask loss. Then, fully masked sentence Yo is passed through the CMLMC decoder that translates all tokens simultaneously to Yø=[ [work work on on]. Sampled first token [work] from Ye is substituted for [we], and the resulting sentence [work work <M> NLP] passes through the CMLMC decoder to correct the [work] + [we] in the Leorr loss.
Yg = Decoder(X, Y, Here, Yz is identical to the output of the first inference step, where all tokens are predicted at once and can contain significant errors. With probability p we replace each token y E Yoss with a corresponding predicted token ğ E Yg. This splits Yobs into two sets: Ypred- where tokens are substituted with predictions from Yz, and the remaining portion Yobs Vpredd Our correction loss then aims predict the tokens in Ypred:
Leorr
log(P(y) Xpred, Yose's Yobe\Yyred Y Ymakk X))
(1)
BEY pored
To compute this loss we make forward pass through the decoder with a sentence that now contains mask Ymask- predictions ypred, and unmasked ground truth tokens Yohe Ypred. The model generates predictions for every substituted token in Y Ypred, and we maximize the probability of the corresponding ground truth tokens. This process approximates the correction procedure, where model re-translates a subset of unmasked tokens by conditioning on current translation. Early self-correction steps
primarily condition on tokens from Ye obtained during the initial pass. As shown in Ghazvininejad (ct al., 2019), these early steps are very important and lead to large improvements in BLEU. So the correction loss Leorr aims to gradually optimize the model for this process during training. In addition to Leorr, we keep the original mask prediction loss from CMLM:
Lmaak
log(P(y)\Yobs Ymask X))
VEY meaa
Note that Yprea is dropped from this loss and conditioning is done on ground truth tokens in Yoes- The final loss that we use in our model is a combination of correction and mask prediction losses:
LCMLMC
T.corr + T.mask
Figure 2 illustrates how the joint loss is computed for an example German to English translation. The mismatch between NAR training and inference procedures is also recognized by SMART (Gaazvinine approach. SMART applies the decoder during training to generate
jad et al. 2020b). Similarly predictions for a subset of tokens and then learns to self-correct these predictions while simultane- ously reconstructing masked tokens. Our approach however has two key differences. First, SMART generates token predictions from partially masked sentence [Ymask, ). While this does teach the model to self-correct, the decoder always sees some ground truth tokens Yous as input. This procedure thus does not address the key first inference step where tokens are predicted from the fully masked sentence with no ground truth input. We discussed that the most significant mistakes are made during this step (also empirically shown in CMLM) so learning to correct them is particularly important. We consequently argue that our approach to instead apply the decoder to fully masked sentence Yo for self-correction loss better addresses the training/inferenee gap. Second, SMART uses the same input with masked tokens Ymask and decoder predicted tokens Ypred for both mask reconstruction and self-correction tasks. This can potentially hamper training since to predict masked tokens decoder also conditions on Ypred which can contain significant errors especially early in optimization. To preserve clean signal in masked training we only condition on ground truth tokens Yots in Lyaas-- Empirically, we demonstrate that our approach leads to significant improvements in accuracy particularly on undistilled data.
5

---

Published as a conference paper at ICLR 2022
3.3 CMLMC TRAINING AND INFERENCE
In previous sections we introduced modifications to CMLM that improve token distinguishability and better align training with inference. Combined they form our CMLMC approach. Algorithm I outlines the training procedure for CMLMC. Given train- ing corpus S = {(,,, Y of source and target sentence pairs, for each pair (X,Y) ple mask Ymask uniformly from the interval [1, [Y]] We then compute correction and mask losses Leorr and Lmass as outlined in Section 3.2 and update model parameters using gradients. During infer- ence we follow the same iterative approach used in CMLM. First, fully masked sentence Yg is passed through the decoder that translates all to- kens once to get Y In subsequent iterations t > 1. a subset of tokens with the lowest likelihood in Y is masked and re- tr-rrsasllted by making pass
Algorithm 1: CMLMC Training
input: paired training data S = {(X,, Y.) learning rate token substitute probability initialize model parameters θ while not convergent do for (X, Y) E S do
first sam-
sample mask: - Ymask and Yous make forward pass:
Yz Decoder(X,, Ya)
substitute tokens:
Yous + Ypred and Yous (Pred
compute loss:
LCMLMc(X, (X,Y) Leorr + Limask
update model:
è = θ
1) 1 (CMENC(X,X))
end end return Q
through the decoder to get f-(t+1) The length of Yo is predicted by the special LENGTH token that is appended to the source sentence in the en- coder (Ghazvininejad et al. 2019). We repeat the inference procedure for top-k predicted lengths and select the translation with the highest average token probability, analogous to using beam search in AR models.
EXPERIMENTS
We evaluate our approach on multiple public NMT datasets: IWSLT' De-En/En-De, WMT' De- En/En-De, and WMT' 16 Ro-En/En-Ro. We use the same training//alidationn// sets as in previous work (Ghazvininejad et al., 2019) and report test set performance in BLEU for direct comparison. For each dataset compute performance on both raw and distilled settings, resulting in 12 dataset in total. Distillation is done using the Transformer small small//aee//ttteeeeeee /base/large AR model (Vaswani et al. 2017) for the IWSLT' 14/WMT' 16 16/WM'T' 14 datasets respectively, accounting for model capacity and dataset size. We compare CMLMC against leading NAR baselines including, Flowseq (Ma et al. 2019). CMLM and its variants, CMLM+SMART and CMLM+
(Ghazvininejad al. 2019 2020b
Ding et al. 2021), LevenshteinNAR (Gu et al. 2019), DisCo (Kasai et al. 2020), ENGINE (Tu et al. 2020), ReorderNAT (Ran et al. 2021), GLAT (Qian et al. 2021), as well as the models using the alignment losses. Descriptions for all baselines are in the related work section.
All experiments are done using the Fairseq library (Gehring al., 2017). To stay consistent with previous work, on the IWSLT' 14 dataset we use the Transformer, emall configuration 512-1024-4, while on the WMT datasets we use the Transformerkawe configuration 512-2048-8 for encoder and decoder in CMLMC. The numbers correspond to embedding dimension, FFN layer size, and number of attention heads respectively. Hapaer-aramettrs for each dataset are selected through grid search and are listed in Table B.1 Appendix. We apply our modifications to CMLM using the code released by the authors" [ In all experiments we use a linear FFN to combine the token and PE embeddings. For datasets where CMLM performance is not reported, we train the model ourselves using the original code, and choose the doyerr-paameee implement SMART to match the performance on the reported datasets since code is not publicly available, this implementation is then used to evaluate SMART on datasets not reported in the original paper. For CMLMC training we compute the correction loss Leorr by randomly substituting 30% (p = 0.3) of tokens in Yoes with predictions from Yo: the substitution percentage is chosen by grid search from 1, 0.2, 0.3, 0.5]. At inference time we follow the procedure outlined in Section 3.3 As in CMLM, we use 10 iterative refinement steps, but lower the beam search length to 3. Adam optimizer (Kingma& & Ba, 2015) with default settings is used for all experiments, and we train the models on the IBM servers with 160 POWER9 CPUs, 600GB RAM and 4 Tesla V100 GPUs (32G).
by applying the same grid search as in CMLMC. We also
httg::/githha.cmmmffccchttrerere Prrdicc

---

Published as a conference paper at ICLR 2022
Table 1: Results and ablation study on the WMT datasets. Indicates our training results, as the original papers did not report results on these datasets. For CMLM we report re-run results from (Kasai et 2021) as they are better than those reported in the original paper.
Model
Param 1M De-En
WMT14
En-De distill
IM Ro-Ea IM En-Ro distill
distill
raw
raw
distill
raw
AR Transformer Kasai al
2020
2021
65M
31,09
31.8
27.74
28.3
34,46
34,8
34.16
34.6
Flowsey Ma et
2019
258M
28.29
30.68
23.64
25.31
32,91
32.84
32.15
32.20
CMLM Gauzyimangan et
2019
67M
29.40
31.20
24.61
27.40
32.87
33.31
32.86
33.7
CMLM-SMAKI (Chazvinneedd
20200
67M
29.58
31.27
25.10"
27.65
32.86"
33.53"
32.71
33.85
OMIM-RawPrior (Ding et
2021
67M
27.8
33.7
LevenshtelnNAR
67M
27.73
33.02
DisCo Kasui et al. 2020
67M
31.31
25.64
27.34
32.25
33.25
33.22
ENGINE eluret
67M
ReorderNKT Ran df 2021
46M
31.13
26.49
31.99
31.70
GLAT (Qian or 2021
73M
31.02
26.55
33.84
32.87
CMLM-AXE (hazviningaad et al. 2020u
67M
24.90
27.90
20.40
23.53
31.42
31.54
30.47
30.75
Aligned
CMLM-OANE DDrreta
30.2
26.1
33.3
32.4
Imputer Saharia at
2120
67M
31.8
25.0
28.2
34.1
34.4
FullyNAT (Gu & Kong
8889
31.39
23.58
27.49
34.16
33.79
CMLM-RevPos
73M
30.02
30.84
25.53
27.70
33
33.92
33:41
34.23
CMLM-Corr
67M
31.31
33.98
34.08
33.75
CMLMC
73M
30.92
31.41
26.40
28.37
34,13
34,13
34.14
34.57
CMLMC456
63M
30.59
31.23
26.31
27.91
33.94
33.93
33.86
34.47
4.1
RESULTS
Results on the WMT datasets are shown in Table [ We see that CMLMC outperforms all NAR benchmarks using CE loss, in many cases by wide margin. On the distilled setting, with the exception of WMT De-En, CMLMC improves over CMLM by more than 0.8 BLEU points, and achieves new state-of the-th--rtt results for Cross Troos--Ettropybbase when compared to models using alignment loss, CMLMC only falls short on WMT 14 De-En when compared to Imputer (Saharia et al. 2020), but is still superior to all other baselines despite using the standard CE loss. Similarly, the raw setting CMLMC improves over CMLM by 1.5 to over BLEU points on all datasets. Improvements of this magnitude lead to CMLMC largely closing the gap to AR performance
NAR translation on all datasets. Even
undistilled data. On all datasets except En-De the difference between raw CMLMC and AR is less than 0.5 BLEU which to the best of our knowledge is the first time when such small performance difference is achieved without distillation. This implies that it possible to train NAR models without distillation and achieve comparable translation quality to AR counterparts. So we believe that with further research in this area, the performance difference between the two frameworks can be eliminated. Jointly, these results, and in particular the improvement over CMLM,, indicate that the identified problems do cause performance bottleneck. They also indicate that our proposed solutions are effective and can lead to significant improvements, while being conceptually simple and easy to implement. Results on the IWSLT' 14 datasets are shown Table A. in the Appendix. CMLMC also shows considerable improvements of over 0.7 BLEU distilled datasets and over 1.8 BLEU on raw datasets compared to the leading baselines.
Analysis of iterative refinement is shown in Table 3(a) For both CMLM and CMLMC compute BLEU accuracy at different iterations. We also compute total inference time to translate the entire IWSLT' 14 De-En test set and fraction of token repetitions. Fraction of repetitions is computed by counting all consecutively repeating tokens and dividing by the total number of translated tokens. This number can thus be interpreted the repetition rate that we expect see from each model, for example at 5% we expect to see 50 repetitions in every 1000 translated tokens. From Table 3(a] we see that the repetition rate consistently lower CMLMC. This can partially explain the over points gain in BLEU after the initial translation from fully masked sentences (iteration 1). This gain is preserved throughout the refinement iterations, and after the last iteration CMLMC has repetition rate that is almost 50% lower than CMLM. The architectural changes to distinguish tokens together with correction training, enable CMLMC to start with fewer mistakes and more effectively correct them during iterative refinement. We also see that additional masked attention layers in the decoder blocks add an overhead during inference forward passes. However, after the full 10 iterations this overhead is around 20% which we do not believe to be significant for a model of this size. Contrasting CMLMC with SMART, we note that CMLMC outperforms SMART on all datasets, with an average BLEU gain of 0.54 in the distilled datasets and 1.15 in raw, further supporting our arguments in section 3.2
7

---

Published as a conference paper at ICLR 2022
Auto- Auto-Regressive
CMLM
CMLMC
CMLM
CMLMC
Iter
BLEU
Time
Repetition
BLEU
Time
Repetition
Berr
de
26.52
15.0s
7.42%
29.77
16.9s
5.47%
28.61
18.55
4.05%
32.20
21.2s
2.47%
fr
30.56
29.66
1.58%
33.79
35.2s
0.78%
31.22
38.55
1.18%
33.94
46.7s
0.53%
31.80
52.26
0.83%
34.21
63.5s
0.35%
Figure 3: 3(a) IWSLT' 14 De-En BLEUs, inference times, and token repetitions for different number of correction iterations. Inference time measures the wall time from when model and test data are loaded until the last sentence has been translated. [ Translation language probability visualization for a multi-target En-De/Es/Fr corpus. Each models is trained to translate all three languages.
4.2 ABLATION STUDY
Ablation results the WMT datasets are shown in Table 1 IWSLT" ablations are shown in Table A in the Appendix. Here, MMeMss incorporates the architectural changes in the decoder to better distinguish tokens (Section 3.1 but is trained with the original CMLM loss. On the other hand, CMLM+Corr keeps the original CMLM architecture but incorporates the correction loss during training (Section 3.2). We observe that both CMLM+RevPos and CMLM+Cor improve performance over CMLM on all datasets in both raw and distilled settings. The improvements are significant and range from 0.2 to over 2 points gain in BLEU. CMLM+Corr generally leads to larger improvement, demonstrating the importance of training the model to correct mistakes. Additional ablation study on the effect of Masked Attention layers and concatenation of PEs are shown in Table indicating the necessity of both in the RevealPosition mechanism. The gains over CMLM are more pronounced on raw setting than distilled. We hypothesize that a smaller impact on distilled setting is due to the simplification of the underlying training data structure. Knowledge distillation compresses multiple modes in the raw data into a single mode learned by the AR model (Zhou et al.,
2020). NAR models trained with this data are less likely make translation mistakes related to multi-modality so correction is not as important in this case. We still see however, that even on distilled data CMLM+Corr produces significant improvements of up to 1.3 BLEU points. Finally, combining both approaches in CMLMC gives additional improvement, and in all cases CMLMC outperforms both CMLM+RevPos and CMLM+Corr. This indicates that distinguishing tokens and error correction are complementary and jointly lead to better translations.
Architectural modifications to the decoder outlined in Section 3.1 introduce additional parameters through weights in masked attention layers and input FFN. From [ we see that CMLMC has around 9% more parameters than CMLM. To remove the performance effect of extra parameters, we also train smaller model CMLMC458 where input embedding dimension is reduced from 512 to 456; this model has 63M parameters which is 6% smaller than CMLM. As seen in Table the reduction in parameters does impact the performance, however, CMLMC156 still outperforms nearly all baselines especially on raw data. This demonstrates that most performance gains in CMLMC come from modifications and not from additional parameters.
4.3 MULTI-MODALITY
We discussed that knowledge distillation is a time-consuming and non-standard step. yet all leading NAR models rely on it since their raw scores significantly underperform the distilled counterparts. Previous work (Zhou et al..(022) argues that distillation primarily helps to resolve the multi-modality problem in the raw dataset, which NAR models cannot handle directly. Following that work, we use the aligned sentences from the Europarl corpus to create a multi-target English to German, Spanish and French (En-De/Es/Fr) corpus. For every En source sentence, we have three target sentences in De, Es, and Fr, effectively creating a tri-modal data. For each architecture we train a single model on raw data to translate in all three languages by sharing the encoder/decoder layers but learning token representations for each language. During inference the model is applied to En test sentences, and we estimate the probability of translation belonging to each of the three target languages by computing relative count of translated tokens from each language. Figure 3(b) visualizes the results where probabilities close are shown at the extremes of the corresponding languages.
8

---

Published as a conference paper at ICLR 2022
Table 2: Qualitative examples of test sentence translation from CMLM and CMLMC on the IWSLT 14 De-En dataset. For both models we show translations after the first self-correction iteration (from fully masked sentence) and after the last iteration.
Source:
also, werde ich sie ihaen zeigen. das ist einer, zwei, drei, vier, fonf.
CMLM iter
so, i'm going to show you.
one one, two, three three, four, five.
CMLM iter 10
i'm going to show you. this
two, three, four, five.
CMLMC iter 1
i'm going to show you you. this's 's one, two, three, four, live.
CMLMC iter 10
so, i'm going to show you. that's 's one, two, three, four, five.
Source:
das entspricht der leistung von hundert atonkraftwerkenn, weil das geht besonders schnell überall.
CMLM iter
that's equivalent of a hundred nuclear nuclear power power power power. because it's all going very quickly. that's the equivalent of hundred of nuclear power power power, because it's going on very quickly.
CMLM iter 10
CMLMC iter I
that's the performance hundred nuclear nuclear power power, because it's going fast. that's the performance of hundred of nuclear power plants, because it's going very fast.
CMLMC iter 10
We see that at iteration 1 CMLM outputs a highly mixed distribution where translations have tokens from multiple languages, which clearly shows the multi-modality problem. After 10 iterations of self-correction CMLM is able to separate the languages and mostly produces translations where the majority of tokens are from one target language. This demonstrates that iterative refinement is an important component of NAR translation that helps the models settle into one mode. We also see that CMLMC has much better language separation at iteration than CMLM. So our proposed modifications improve the sentence. By comparing the graphs from CMLM and CMLMC at iteration 10 vs autoregressive baseline, we observe that they all exhibit similar degree of separation. However, as we have seen from other experiments their BLEU scores are typically quite different. This suggests that the multi-modality problem might not be the only issue NAR models face when trained on raw datasets. We believe that an additional investigation is necessary here and leave it for future work.
modality problem even when translation is done from fully masked
4.4 QUALITATIVE ANALYSIS
IWSLT' 14 De-En translation examples are shown in Table We focus on the common NAR problem of token repetition and show two example seniences that are representative of the repetition errors that CMLM typically makes. We also show the final translation after 10 iterative refinements and compare against the CMLMC translation. The first sentence has sequence of numbers "one, two, three, four, five". Embeddings for number tokens tend to be close together in the learned space, and CMLM has difficulty removing these repetitions. After the first iteration "one" and "three" tokens repeat, and while CMLM is able to correct these, it can't 't fully remove repetitions and the final translation still has repeating "two" token. CMLMC on the other hand has no difficulty with the number tokens and translates them correctly after the first iteration.
The second sentence shows long repetition that is difficult correct. Here, "power" is repeated four times by CMLM after the first iteration. After 10 iterations CMLM is able to correct the "nuclear" repetition and remove one "power" token but the other three remain. Since each refinement iteration re-translates subset of tokens with the lowest probability, longer repetitions are increasingly harder correct as all repeating tokens need to eventually end up in the lowest probability set. We see that CMLMC also repeats "power", but only once, allowing the correction mechanism to come into play and fix the error. These examples demonstrate that improved token distinguishability can lead to initial translations with fewer repetition mistakes. Our model is then the able to more effectively correct them, resulting the better end translations.
5 CONCLUSION
We introduce the Conditional Masked Language Model with Correction (CMLMC), an NAR trans- lation model that addresses the design shortcomings in leading NAR approaches. Through a dual strategy of revealing positional information and adding error correction mechanism, we significantly improve NAR translation performance on both raw and distilled datasets. In particular, when trained on raw data, CMLMC approaches the performances of leading AR models which to the best of our knowledge is the first such result in NAR. Future work involves expanding our approach to other NAR models and further investigation into the relationship between multi-modalits and distillation.

---

Published as a conference paper at ICLR 2022
REFERENCES
Loic Barrault, Ondiej Bojar, Marta R Costa-Jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, al. Findings of the 2019 conference on machine translation. In WMT, 2019.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
Liang Ding. Longyue Wang, Xuebo Liu, Derek Wong, Dacheng Tao, and Zhaopeng Tu. Under- standing and improving lexical choice in non-autoregressive translation. In ICLR, 2021.
Cunxiao Du, Zhaopeng Tu, and Jing Jiang. Order-agnostic cross entropy for non-autoregressive machine translation. In Arxiv, 2021.
Jonas Gehring. Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning. In ICML, 2017.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP, 2019.
Marjan Ghazvininejad, V. Karpukhin, Luke Zettlemoyer, and Omer Levy. Aligned cross entropy for non-autoregressive machine translation. In /CML, 2020a.
Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoyer. Semi-autoregressive training improves mask-predict decoding. arXiv preprint 1xXiv:2001.0878, 2020b.
Jiatao Gu and Xiang Kong. Fully non-autoregressive neural machine translation: Tricks of the trade. In ACL, 2021.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. In ICLR, 2018.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In NeurIPS, 2019.
Adi Haviv, Lior Vassertail, and Omer Levy. Can latent alignments improve autoregressive machine translation? In NAACL, 2021.
Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In ICML, 2020.
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Non-autorrerressive machine translation with disentangled context transformer. In PMLR, 2020.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah Smith. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. In ICLR, 2021.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In EMNLP, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie- Van Liu. Hint-based training for non-autoregressive machine translation. In EMNLP-IJCNLP, 2019.
Jindfich Libovický and Jindfich Helel. End-to-end non-autoreereessvve neural machine translation with connectionist temporal classification. In EMNLP, 2018.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance the adaptive learning rate and beyond. In ICLR, 2020.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. FlowSeq: Non- autoregressive conditional sequence generation with generative flow. In EMNIP-IJCNIPP 2019.
10

---

Published as a conference paper at ICLR 2022
Tsvetomila Mihaylova and André F. T. Martins. Scheduled sampling for transformers. In ACL, 2019.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. Glancing transformer for non-autoregressive neural machine translation. In ACL, 2021.
Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. Guiding non-autoreeressive neural machine translation decoding with reordering information. AAAI, 2021.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autoregressive machine translation with latent alignments. In EMNLP. 2020.
Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. ENGINE: Energy-based inference networks for autrerooorrre machine translation. ACL, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Yiren Wang. Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. Non-autoregressive machine translation with auxiliary regularization. AAAI, 2019.
Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. Tener: Adapting transformer encoder for named entity recognition. arXiv preprint 4xX1v:111.04474.20 2019.
Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non- autoregressive machine translation. In ICLR, 2020.

---

Published as a conference paper at ICLR 2022
A CMLMC PERFORMANCE ON IWSLT' 14 EN-DE DATASET
Table A.1: Results and ablation study on the IWSLT 14 De-En and En datasets. Indicates CMLM and SMART models that were trained by us, as (Ghavvininejad et al. 2019) and (Ghazvininejad et al. 2020b) does not report results on these datasets.
Model
Param
IWSLT
#
De-En
En-De
raw
distill
raw
distill
AR Transformer Liu et al.
2020
38M
34.66
35,30
28.56
29.26
NAR-Reg (Wang et al 2019)
46M
28.04
NAR-Hint (Li al 2019
46M
28.80
Flowseq (Ma et al.
2019)
73M
24.75
27.55
CMLM (Ghazvmmeeja
2019
38M
31.80"
33.42"
25.60*
27.59*
CMLM+SMARI Ghazzvmnejja al. 2020b
38M
30.74
33.48*
24.55*
27.74*
ENGINE (Tu ct al.
2020
67M
33.17
Mas.ee-ts
44M
33.08
33.70
26.03
27.93
CMLM+ConcatPE
39M
33.23
33.90
26.12
27.90
CMLM+ RevPos
46M
33.50
33.96
26.38
28.13
CMLM+
38M
33.90
34.53
26.92
28.39
CMLMC
46M
34.28
34.78
27.55
28.51
CMLMC456
38M
33.83
34.44
27.35
28.47
B CMLMC HYPERPARAMETERS
Table B.1: CMLMC hyper-parameters.
Parameters
IWSLT'14
WMT'14
16
learning rate
0.0005
0.0007
0.0005
warmup
30k
40k
15k
dropout
0.3
0.2
0.3
updates
1755
150k
120k
epochs
300
250
200
GPU
1xTesla V100
4xTTsll V100
4xTesla V100
tokens/GPU
8192
8192
8192
12


Reviews:
[{"review_id": "oQUPDo5OaTp", "reviewer": "Reviewer_dfXP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* The paper is very clearly written and easy to follow. \n* The issues behind performance gap between NAR and AR models have been clearly explained and methods to circumvent those issues have been well motivated and justified.\n* The resulting model CMLMC achieves state of the art scores for semi-autoregressive machine translation, and more importantly achieves better performance on non-distilled datasets. \n\n**Questions/Concerns**\n* *More Ablation Experiments*: In case it is easy enough to execute, it will be informative to see results for `SMART + RevPos`  in Table 1 as well. \n\n\n* *Comparison with SMART (I)* : In Section 3.2, the authors discuss the similarities and key differences with a similar approach to bridge train-test mismatch for NAR models - SMART (Ghazvininejad et al. 2020). The authors say -  \"First, SMART generates token predictions from a partially masked sentence $\\{Y_{mask}, Y_{obs}\\}$. While this does teach the model to self-correct, the decoder always sees some ground truth tokens $Y_{obs}$ as input.\" This is slightly wrong, as in SMART, the masking length can be set to sequence length which would then result in predictions from a fully masked sequence (1st inference step). CMLMC's approach can be considered as a re-weighted form of SMART, where all weight is given to all masked sequence (which the authors demonstrate performs better empirically). if the authors agree with this, it would be great to clarify this in the paper. \n\n\n* *Comparison with SMART (II)*: Second key difference, as the authors mention, lies in the introduction of an additional correction loss instead of modifying the existing CE loss function. While the arguments presented by the authors make intuitive sense, it will be nice to perform some empirical assessment of this claim. i.e. comparing CMLMC with re-weighted SMART.\n\n\n* Question: Did the authors try adjusting relative weights of correction loss and CE loss?\n ", "summary_of_the_paper": "Authors describe two important issues behind performance gap between semi-autoregressive CMLM (conditional masked language models) and autoregressive models in Machine Translation - (i) indistinguishability of tokens, and (ii) training and inference mismatch. They propose Conditional Masked Language Model with Correction (CMLMC) to address these issues through intermediate causal attention layers, and FFN layers for combining token and positional encodings for (i) and adding a correction loss for the first inference step for (ii). Authors show that CMLMC outperforms existing NAR methods on many benchmarks when trained on distilled and non-distilled datasets. ", "summary_of_the_review": "The paper is well written, presents well motivated improvements for improving NAR models. The empirical evidence is convincing, and proper ablation experiments have been presented. I have provided some suggestions for experiments and technical corrections to the authors. I believe this paper presents interesting findings which will be relevant to the machine translation community. I recommend acceptance for the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "dXeqH0t4_0", "reviewer": "Reviewer_6uRn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths:\n1. Training a NAR model without the help of the autoregressive model is appealing and important.\n2. The performance improvements are impressive, which remarkably narrows the performance gap to the autoregressive model without the knowledge distillation.\n\n#### Weaknesses:\n1. The novelty is somewhat limited. The indistinguishability of decoder inputs and the mismatch problems have been discussed in previous works[1][2], the authors do not present a clear discussion among them.\n2. To my understanding, the correction loss works very similarly to SMART[2]. It is unclear to me the key differences between them and why the former works better than the latter, though the author has discussed it on Page 5.\n3.  Although this paper focuses on the performances trained with the raw datasets, several comparisons may not be fair for distilled datasets. Some results of baselines in Table 2 use the Transformer-base as a teacher in WMT14 instead of Transformer-large.\n\n#### Questions:\n1. Is the distinguishability of the decoder inputs matter to an iterative-based model? Figure 1 only analyzes the first inference step but misses the 4- or 10-iterations results.  I know this issue is important to fully non-autoregressive models and discussed by Wang et al. 2019[1]. Need to say, their method can be incorporated directly into your setting.\n2. I notice that the CMLMC model achieves significant improvements in the first inference step than the CMLM model (Figure 3, it is also very curious to report a Table in the Figure environment). What is the result of WMT14? One-step results can be directly compared with most NAR models, which will provide a stronger baseline for subsequent work.\n\n#### Typos:\n1. Which \"GLAT (Haviv et al., 2021a)\"?\n\n#### References\n1. Wang et al. AAAI-2019, Non-Autoregressive Machine Translation with Auxiliary Regularization\n2. Ghazvininejad et al. 2020. Semi-Autoregressive Training Improves Mask-Predict Decoding.\n3. Qian et al. ACL-2021. Glancing Transformer for Non-autoregressive Neural Machine Translation.", "summary_of_the_paper": "This paper argues that the indistinguishability of tokens and the mismatch between training and inference leads to inferior translation quality in the previous state-of-art non-autoregressive~(NAR) model --- CMLM. \nTo address the above issues, the authors propose the CMLMC model and achieve better performances without the help of the autoregressive model. \nSpecifically, CMLMC first re-introduces a causal masked attention layer to the CMLM model and uses a feed-forward layer to combine the tokens and positional encodings to enhance the distinguishability of the decoder inputs. \nThen, the CMLMC model introduces a correction loss that predicts the correct tokens based on the predicted tokens,  avoiding the mismatch between the training and inference stages.\nExperiments results on several translation benchmarks show the CMLMC model achieves impressive performance improvements.", "summary_of_the_review": "Overall, although the paper presented impressive results, it missed some meaningful discussions, such as the importance of indistinguishability and why its correction loss works better than SMART. In addition, it needs to polish the draft.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cYSIb1nAed", "reviewer": "Reviewer_BqHZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the paper is mainly about tackling a well-scoped problem (token repetition) using an existing architecture (CMLM), which is well-known in the domain of non-autoregressive MT. The modifications by the authors all make sense, arguably there can be a difference in terms of how each factor contributed to the overall performance improvement, and the authors do show ablated results with a leave-one-out approach. The performance gained by the newly introduced factors looks effective, and they don't seem to overcomplicate the method. The analysis by the authors shows that the proposed ideas help mitigate the multi-modality problem, although the problem is not fully eliminated.\n\nSince the token repetition problem is not fully removed, but reduced to around half the occurrence rate, could the authors confirm that there are still repeated tokens in the translation results?\n\nOverall, I think the effort to remove distillation using AR models is a good direction in general. Eventually, non-autoregressive models will become useful by removing this constraint. ", "summary_of_the_paper": "This paper investigates a few different ways to mitigate token repetition issues in non-autoregressive machine translation (MT) systems, which is also known as the multi-modality problem. The authors highlight that this is mainly due to the mismatch between training and inference in non-autoregressive (NAR) models. In addition to the mismatch, the indistinguishability of tokens makes the problem worse. The authors explain that autoregressive (AR) models suffer less from the same issues because of a combination of multiple factors, such as causal attention, auto-regressive modelling, and introducing positional embedding (the last point is more specific for the Transformer architecture). The authors introduce an alternative path for combining the positional embedding with token embedding using a shallow neural network (RevealPosition), introduce intermediate causal attention between self-attention and cross-attention, and finally a correction loss to deal with repeated tokens.", "summary_of_the_review": "This paper shows interesting observation points to understand why the multi-modality problem occurs in non-autoregressive MT systems, and proposes a few ways to mitigate the issue. The modifications seem effective and improve both raw and distilled performances. More impressively, the gains are bigger in the raw setting. \n\nI also find that it is nice to remove the restriction of non-autoregressive models that rely heavily on autoregressive models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LC5XZJ6EAbS", "reviewer": "Reviewer_ssed", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The proposed method is very simple, and can be applied to many iterative NAR models (and perhaps more, e.g., when we use BERT-like models for generation in general).\n- The experiments show the strong performance compared to prior NAR models, especially when distillation is not applied. \n- Removing the necessity of distillation will have a practical impact for NAR models because distillation is, as authors point out, time and resource consuming. \n\n**Weaknesses**\n- While it is correctly acknowledged in Sec. 4.1, the gap between raw and distillation setting for WMT14 EN-DE is very substantial. This might be because of the diverging and relatively flexible word order of German, which makes the multimodality problem more severe. This suggests that the proposed method has a clear limitation in addressing the multimodality. \n- (Question) Since the modified objective involves two separate decoder passes for every batch (for Lmask and Lcorr), I suspect that translates to an increase in training time. How does this increase compare to the training time increase you would get from knowledge distillation? If the proposed method's increase is much smaller than applying knowledge distillation, it would mean that the method can save training overhead compared to approaches that require knowledge distillation. \n\n**Updates**\nMy question above is fully addressed in the response. I would strongly recommend that the authors add the discussion on increased training time to the final version, as one major benefit of removing KD is training time savings. I increased the score from 6 to 8. ", "summary_of_the_paper": "This paper presents two techniques that improve iterative non-autoregressive machine translation (NAR). The first technique modifies the conditional masked language modeling objective from prior work in a way that makes training more similar to iterative inference. The other improvement is the way positional information and tokens are encoded. This is designed to encourage the model to distinguish between target positions and avoid repetitions. ", "summary_of_the_review": "Overall, this paper presents s simple, effective approach to improve NAR machine translation. Their claim is supported by their experiments. I still have some concern described above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a modification of the training objective of non-autoregressive MT which claims most of the improvements that other approaches obtain only through knowledge distillation (KD) from an autoregressive teacher. 

The strategy has been largely appreciated as simple and the results suggest that it's rather effective. One of us was not ready to accept certain aspects of the comparisons in the paper, and challenged the paper also from a speed of generation point of view. While I see that NAT-MT is very much concerned with speed, removing the dependency on an autoregressive teacher is an important step in the NAT-MT agenda (as KD has various drawbacks, e.g., it corrupts the statistics of the training data), and, in my view, disentangling the two desiderata (e.g., faster models, and no KD) is okay at this stage. I hope the authors will not take this recommendation as a reason to ignore the comments in that review, rather, that they take it as an opportunity to address those comments as well as possible (for example, by positioning the work more carefully wrt speed and the possibly negative impact of KD).

On style: Table 1 should fit within the margins of the paper, please fix it. Also, avoid boldfacing model names and avoid vertical bars (please check this nice guide on making [tables look nice](https://people.inf.ethz.ch/markusp/teaching/guides/guide-tables.pdf)).
Decision: Accept (Poster)