Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This article proposed a distill multiple steps method to decrease the number of required steps to speed up iterative non-autoregressive transformer.
strength_and_weaknesses: Strengths:

1. The training of the distilled student model obtained improved training efficiency with a certain translation quality. 

2. The proposed approach gained improvement on the CMLM on the WMT14 EN-De and WMT16 Ro-EN benchmarks.

Weaknesses:

1. For many comparison methods in Table 1, the reported results appeared to be lower than the original paper (e.g., the CMLM model).

2. If the training time of the teacher model is also calculated, the efficiency of the final student model may not necessarily be improved.

3. As shown in Figure 3, the BLEU scores of the proposed method were inferior to those of the baseline CMLM and CMLMC models.

4. There were too many hyper-parameters (e.g., n, lambda, and p) in the proposed DiMS.
clarity,_quality,_novelty_and_reproducibility: The contribution of this study is small in the machine translation community.
summary_of_the_review: Please see review in the previous sections.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use progressive distillation for distilling non-autoregressive models, and update the teacher based on the exponential moving average of the student. The experiments assess the idea on the machine translation task and exhibit some improvement when comparing with single-step results.
strength_and_weaknesses: Strength: reducing the number of iterations for the non-autoregressive approach is a critical topic for the field, and the use of progressive distillation is a reasonable choice.
Weakness: the final results are worse than without applying the progressive distillation, and therefore making the contribution limited.
clarity,_quality,_novelty_and_reproducibility: The uploaded file seems like an image-converted pdf, making it difficult to read.
summary_of_the_review: While the idea of using progressive distillation for reducing the inference iterations of non-autoregressive models seems appealing, the resulting quality is not convincing and therefore the work provides limited contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Iterative NAT improves translation performance with multiple decoding steps, at the cost of sacrificing decoding speed. In response to this problem, this paper proposes to leverage distillation technique to imitate the behaviors of multiple steps with a single decoding step. To this end, the authors maintain two NAT models, namely teacher and student. 

The student model learns to predict the multiple-step output of the teacher with two additional training objectives for the imitation learning: 1) a KL-divergence loss of the probability distributions of target tokens between the teacher and the student; 2) an Euclidean distance between the last layer representations of the two models.

As the training processes, the performance of student model is improved (for both single- and multiple- decoding steps), which can further enhance the teacher model. Accordingly, the teacher model is updated by the student via a slow-exponential-moving average.

Experimental results show that the proposed approach consistently improves performance over CMLMC (Huang et al., 2022) across language pairs on both distilled and raw data.
strength_and_weaknesses: Pros:
1. An promising direction to improve fully NAT, which can maintain the advantage of decoding speed;

2. An interesting way to leverage the performance of multiple-step decoding via distillation.

Cons:

1. The approach is relatively complicated. It introduces 3 additional hyper-parameters (decoding steps of teacher model, interpolation weights of KL and Euclidean losses, and ), which potentially threat the robustness of the approach. Are these hyper-parameters the same across models (e.g. CMLMC and Imputer) and datasets (e.g. En-De and En-Ro).

2. The methods to distill multiple steps are practical but not innovative enough. The KL and Euclidean losses are straightforward, and widely-used in imitation learning. 

3. The distillation pipeline still relies on an external teacher model (e.g. the pertained NAT model in this work), which prevents the model from learning from scratch (e.g. DAT (Huang et al., ICML 2022)).


Suggested Revisions:
1. The ultimate goal of NAT is  to become an independent translation model that achieves comparable with or even better performance than the AT models. Now the majority of NAT models are base setting (i.e., 6 layers of 512 dimensions) trained on medium-scale data (e.g., 4.5M WMT14 En-De), which still lags behind AT models (e.g., big or deep models trained on 20M WMT17 En-De data). It would be appreciated if the authors can validate the effectiveness of NAT models on big settings and/or larger datasets.
clarity,_quality,_novelty_and_reproducibility: This paper is generally well written and the experiments that have been conducted make sense and are useful for evaluating the proposed methods. 
summary_of_the_review: This paper focuses on improving the performance of single-step NAT (on raw data), which is a promising and important direction to make NAT be independent model. The proposed Distill Multiple Steps (DiMS) method is effective across datasets and model architectures.

However, the method introduces several hyper-parameters, which require a grid-search for individual dataset (e.g. Section 4.5). This method also relies on an pretrained teacher model, which prevents NAT from learning from scratch.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to use progressive distillation for distilling non-autoregressive models. While the reviewers admits the it is a promising direction to improve NAT, all the reviewers point out the quality of the experiments is not satisfying. Both in terms of the replication of the baselines and the setup for the proposed model. The paper needs to be further improved on these matters before it reaches the acceptance bar of ICLR.
Decision: Reject