Revisiting the Iterative Non-Autoregressive Transformer
Authors: Yisheng Xiao, Pei Guo, Zechen Sun, Juntao Li, Kai Song, Min Zhang


Reviewer paper summaries: The paper brings forward an innovative approach to enhance the performance and efficiency of iterative Non-Autoregressive (NAR) models. This paper proposes the systematic inspection of relevant factors and the proposal of two new metrics - Decline Risk of Refinements (DRR) and Ratio of Over-Refinements (ROR), catered to gauge the stability and reliability of the refinement process. To further boost the efficiency of NAR models, an autoregressive (AR) model-trained ARScorer is proposed as a token-level scorer. Experiment results show the final models incorporating competitive strategies and ARScorer have displayed superior outcomes on various datasets, even with reduced decoding steps, and notably outdo their AR counterparts.
Summary of strengths: - The paper presents a comprehensive study and analysis of the iterative non-autoregressive models (NAR).
- The paper proposes new metrics (DRR and ROR) to assess the stability and reliability of the model's refinement process.
- The paper demonstrates improved performance compared to autoregressive models.
Summary of weaknesses: 1. The difference between the ARScorer and the best choice in terms of BLEU score in Table 1 and Table 5 indicates that there may be room for optimizing the ARScorer approach to boost language translation quality.
2. The proposed metrics DRR and ROR do not take into account the numerical difference between the two decoding steps when comparing them. For example, using BLEU difference between results instead of only transforming better or worse to 1 or 0. This could potentially skew the evaluations.
3. The paper does not directly study or evaluate the effect of ARScorer on decoding efficiency.
Comments, suggestions and typos: 1. Could you explain the 2.0 BLEU score difference between ARScorer and the superior choice, and is there potential for further improvements in ARScorer?
2. Why weren't also numerical differences considered in the proposed DRR and ROR metrics?
3. Would it be possible to incorporate other scoring methods (instead of ARScorer) to improve model efficiency and competitiveness? As ARScorer needs to deploy an external module while decoding.
Overall assessment scores: 3