Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents two techniques that improve iterative non-autoregressive machine translation (NAR). The first technique modifies the conditional masked language modeling objective from prior work in a way that makes training more similar to iterative inference. The other improvement is the way positional information and tokens are encoded. This is designed to encourage the model to distinguish between target positions and avoid repetitions. 
main_review: **Strengths**
- The proposed method is very simple, and can be applied to many iterative NAR models (and perhaps more, e.g., when we use BERT-like models for generation in general).
- The experiments show the strong performance compared to prior NAR models, especially when distillation is not applied. 
- Removing the necessity of distillation will have a practical impact for NAR models because distillation is, as authors point out, time and resource consuming. 

**Weaknesses**
- While it is correctly acknowledged in Sec. 4.1, the gap between raw and distillation setting for WMT14 EN-DE is very substantial. This might be because of the diverging and relatively flexible word order of German, which makes the multimodality problem more severe. This suggests that the proposed method has a clear limitation in addressing the multimodality. 
- (Question) Since the modified objective involves two separate decoder passes for every batch (for Lmask and Lcorr), I suspect that translates to an increase in training time. How does this increase compare to the training time increase you would get from knowledge distillation? If the proposed method's increase is much smaller than applying knowledge distillation, it would mean that the method can save training overhead compared to approaches that require knowledge distillation. 

**Updates**
My question above is fully addressed in the response. I would strongly recommend that the authors add the discussion on increased training time to the final version, as one major benefit of removing KD is training time savings. I increased the score from 6 to 8. 
summary_of_the_review: Overall, this paper presents s simple, effective approach to improve NAR machine translation. Their claim is supported by their experiments. I still have some concern described above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors describe two important issues behind performance gap between semi-autoregressive CMLM (conditional masked language models) and autoregressive models in Machine Translation - (i) indistinguishability of tokens, and (ii) training and inference mismatch. They propose Conditional Masked Language Model with Correction (CMLMC) to address these issues through intermediate causal attention layers, and FFN layers for combining token and positional encodings for (i) and adding a correction loss for the first inference step for (ii). Authors show that CMLMC outperforms existing NAR methods on many benchmarks when trained on distilled and non-distilled datasets. 
main_review: **Strengths**

* The paper is very clearly written and easy to follow. 
* The issues behind performance gap between NAR and AR models have been clearly explained and methods to circumvent those issues have been well motivated and justified.
* The resulting model CMLMC achieves state of the art scores for semi-autoregressive machine translation, and more importantly achieves better performance on non-distilled datasets. 

**Questions/Concerns**
* *More Ablation Experiments*: In case it is easy enough to execute, it will be informative to see results for `SMART + RevPos`  in Table 1 as well. 


* *Comparison with SMART (I)* : In Section 3.2, the authors discuss the similarities and key differences with a similar approach to bridge train-test mismatch for NAR models - SMART (Ghazvininejad et al. 2020). The authors say -  "First, SMART generates token predictions from a partially masked sentence $\{Y_{mask}, Y_{obs}\}$. While this does teach the model to self-correct, the decoder always sees some ground truth tokens $Y_{obs}$ as input." This is slightly wrong, as in SMART, the masking length can be set to sequence length which would then result in predictions from a fully masked sequence (1st inference step). CMLMC's approach can be considered as a re-weighted form of SMART, where all weight is given to all masked sequence (which the authors demonstrate performs better empirically). if the authors agree with this, it would be great to clarify this in the paper. 


* *Comparison with SMART (II)*: Second key difference, as the authors mention, lies in the introduction of an additional correction loss instead of modifying the existing CE loss function. While the arguments presented by the authors make intuitive sense, it will be nice to perform some empirical assessment of this claim. i.e. comparing CMLMC with re-weighted SMART.


* Question: Did the authors try adjusting relative weights of correction loss and CE loss?
 
summary_of_the_review: The paper is well written, presents well motivated improvements for improving NAR models. The empirical evidence is convincing, and proper ablation experiments have been presented. I have provided some suggestions for experiments and technical corrections to the authors. I believe this paper presents interesting findings which will be relevant to the machine translation community. I recommend acceptance for the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates a few different ways to mitigate token repetition issues in non-autoregressive machine translation (MT) systems, which is also known as the multi-modality problem. The authors highlight that this is mainly due to the mismatch between training and inference in non-autoregressive (NAR) models. In addition to the mismatch, the indistinguishability of tokens makes the problem worse. The authors explain that autoregressive (AR) models suffer less from the same issues because of a combination of multiple factors, such as causal attention, auto-regressive modelling, and introducing positional embedding (the last point is more specific for the Transformer architecture). The authors introduce an alternative path for combining the positional embedding with token embedding using a shallow neural network (RevealPosition), introduce intermediate causal attention between self-attention and cross-attention, and finally a correction loss to deal with repeated tokens.
main_review: In general, the paper is mainly about tackling a well-scoped problem (token repetition) using an existing architecture (CMLM), which is well-known in the domain of non-autoregressive MT. The modifications by the authors all make sense, arguably there can be a difference in terms of how each factor contributed to the overall performance improvement, and the authors do show ablated results with a leave-one-out approach. The performance gained by the newly introduced factors looks effective, and they don't seem to overcomplicate the method. The analysis by the authors shows that the proposed ideas help mitigate the multi-modality problem, although the problem is not fully eliminated.

Since the token repetition problem is not fully removed, but reduced to around half the occurrence rate, could the authors confirm that there are still repeated tokens in the translation results?

Overall, I think the effort to remove distillation using AR models is a good direction in general. Eventually, non-autoregressive models will become useful by removing this constraint. 
summary_of_the_review: This paper shows interesting observation points to understand why the multi-modality problem occurs in non-autoregressive MT systems, and proposes a few ways to mitigate the issue. The modifications seem effective and improve both raw and distilled performances. More impressively, the gains are bigger in the raw setting. 

I also find that it is nice to remove the restriction of non-autoregressive models that rely heavily on autoregressive models.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper argues that the indistinguishability of tokens and the mismatch between training and inference leads to inferior translation quality in the previous state-of-art non-autoregressive~(NAR) model --- CMLM. 
To address the above issues, the authors propose the CMLMC model and achieve better performances without the help of the autoregressive model. 
Specifically, CMLMC first re-introduces a causal masked attention layer to the CMLM model and uses a feed-forward layer to combine the tokens and positional encodings to enhance the distinguishability of the decoder inputs. 
Then, the CMLMC model introduces a correction loss that predicts the correct tokens based on the predicted tokens,  avoiding the mismatch between the training and inference stages.
Experiments results on several translation benchmarks show the CMLMC model achieves impressive performance improvements.
main_review: #### Strengths:
1. Training a NAR model without the help of the autoregressive model is appealing and important.
2. The performance improvements are impressive, which remarkably narrows the performance gap to the autoregressive model without the knowledge distillation.

#### Weaknesses:
1. The novelty is somewhat limited. The indistinguishability of decoder inputs and the mismatch problems have been discussed in previous works[1][2], the authors do not present a clear discussion among them.
2. To my understanding, the correction loss works very similarly to SMART[2]. It is unclear to me the key differences between them and why the former works better than the latter, though the author has discussed it on Page 5.
3.  Although this paper focuses on the performances trained with the raw datasets, several comparisons may not be fair for distilled datasets. Some results of baselines in Table 2 use the Transformer-base as a teacher in WMT14 instead of Transformer-large.

#### Questions:
1. Is the distinguishability of the decoder inputs matter to an iterative-based model? Figure 1 only analyzes the first inference step but misses the 4- or 10-iterations results.  I know this issue is important to fully non-autoregressive models and discussed by Wang et al. 2019[1]. Need to say, their method can be incorporated directly into your setting.
2. I notice that the CMLMC model achieves significant improvements in the first inference step than the CMLM model (Figure 3, it is also very curious to report a Table in the Figure environment). What is the result of WMT14? One-step results can be directly compared with most NAR models, which will provide a stronger baseline for subsequent work.

#### Typos:
1. Which "GLAT (Haviv et al., 2021a)"?

#### References
1. Wang et al. AAAI-2019, Non-Autoregressive Machine Translation with Auxiliary Regularization
2. Ghazvininejad et al. 2020. Semi-Autoregressive Training Improves Mask-Predict Decoding.
3. Qian et al. ACL-2021. Glancing Transformer for Non-autoregressive Neural Machine Translation.
summary_of_the_review: Overall, although the paper presented impressive results, it missed some meaningful discussions, such as the importance of indistinguishability and why its correction loss works better than SMART. In addition, it needs to polish the draft.
review_title: null

Meta-review: This paper proposes a modification of the training objective of non-autoregressive MT which claims most of the improvements that other approaches obtain only through knowledge distillation (KD) from an autoregressive teacher. 

The strategy has been largely appreciated as simple and the results suggest that it's rather effective. One of us was not ready to accept certain aspects of the comparisons in the paper, and challenged the paper also from a speed of generation point of view. While I see that NAT-MT is very much concerned with speed, removing the dependency on an autoregressive teacher is an important step in the NAT-MT agenda (as KD has various drawbacks, e.g., it corrupts the statistics of the training data), and, in my view, disentangling the two desiderata (e.g., faster models, and no KD) is okay at this stage. I hope the authors will not take this recommendation as a reason to ignore the comments in that review, rather, that they take it as an opportunity to address those comments as well as possible (for example, by positioning the work more carefully wrt speed and the possibly negative impact of KD).

On style: Table 1 should fit within the margins of the paper, please fix it. Also, avoid boldfacing model names and avoid vertical bars (please check this nice guide on making [tables look nice](https://people.inf.ethz.ch/markusp/teaching/guides/guide-tables.pdf)).
Decision: Accept (Poster)