Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper's main topic is the actual usability of the current status of non-auto-regressive translation (NAT) models.

Although previous papers have reported that the NAT models can achieve the same performance level with auto-regressive translation models while the decoding speed is much faster, like two to five times, this paper points out that it deeply relies on the batch size and computation environment.
This is a proper investigation for the community since some researchers might believe that NAT is always faster than standard auto-regressive models and became an excellent alternative to them.

The ideas of inducing skip-AT and skip-MT are really unique and somewhat innovative (since, I guess, no other researchers hardly think to employ such skip-decoding architecture).
Basically, this paper has several new findings that should be shared in the community for developing better technologies.




The following are the questions/concerns of this paper.

1,

"IR-NAT heavily relies on small batch size and GPU, which is rarely mentioned in prior studies."
I think this is an excellent investigation. However, this paper does not tell readers why this observation happens.
Please explain why the current NAT models are not suitable to work on CPUs and large batches.


2,

The intention of the statement, "which indicates that the balanced distribution of deterministic tokens is necessary" is unclear. Please elaborate on what the authors try to tell by this statement.


3,

The proposed method consists of many new components.
The authors provided the results of an ablation study in Table 5.
This is a really nice analysis.
However, the performance differences in −FT, −RPR, and −MixDistill are somewhat marginal. 
Actually, we can easily observe such 0.3-0.4 BLEU score difference by just changing random seeds for Transformer models.
Are there any statistically significant differences among them? Or any reasonable evidence that supports the difference?


4,

This is just a comment, and appreciate having the author's words.

There is an opinion that fully tuned implementation for standard auto-regressive models outperforms both decoding speed and accuracy. See the following presentation slide on WNGT-2020 (such as P33):
https://kheafield.com/papers/edinburgh/wngt20overview_slides.pdf

What would happen for the proposed method if we compared them on such a highly-tuned implementation?
review_title: Interesting findings

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a hybrid-regressive machine translation (HRT) approach—combining autoregressive (AT) and non-autoregressive (NAT) translation paradigms: it first uses an AT model to generate a “gappy” sketch (every other token in a sentence), and then applies a NAT model to fill in the gaps with a single pass. As a result the AT part latency is roughly reduced by half compared to a full AT baseline. The AT and NAT models share a majority part of the parameters and can be trained jointly with a carefully designed curriculum learning procedure. Experiments on several MT benchmarks show that the proposed approach achieves speedup over the full AT baseline with comparable translation quality.

The idea of combining AT and NAT is interesting, and the paper is very clearly written. The experiments and analysis are solid and well-designed. I vote for acceptance.

Pros:
- An interesting idea combining AT and NAT which can inspire future research.
- Experiments and analysis support most of the claims.
- The proposed curriculum training and mixed distillation can be useful in future research.

Cons:
- It is a bit sad that the translation quality drops a lot when k > 2. This limits and maximum speedup that can be achieved. 
- Some of the claims on the HRT’s performance compared to the AT baseline seems a bit misleading (see details below).

Details:
- From Table 3 it seems that AT models also benefit from mixed distillation. Based on the numbers here I’m assuming the AT models in Table 2 are trained w/o mixed distillation. Early on the paper claims that HRT can outperform AT in terms of translation quality, but this doesn’t seem to be the case if one compares HRT against AT both trained with mixed distillation. Can the authors clarify? If this is true, please tone down the claims on this point in the abstract, intro, and Section 5.
- Can the authors comment on the training cost of HRT in comparison to AT in terms of, e.g., number of epochs, wall-clock time?
- Does the AT baselines in Table 2 use the same relative positional encodings as HRT? If not, can the authors comment on how it may affect their performance? 
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: Non-autoregressive decoder (NAT) greatly improves translation efficiency, but often relies on iterative refinement (IR) to retain the translation performance. Unfortunately, this strategy makes the decoding efficiency of IR-based NAT much more sensitive to batch size and computing device: with larger batch size and CPU, IR-based NAT runs even slower than the standard auto-regressive decoder (AT). 

This paper targets at alleviating this IR bottleneck by proposing hybrid-regressive translation (HRT), which combines AT and NAT in two stages. AT in the first stage aims at offering an initial coarse target hypothesis, implemented by predicting target words at every k position, to serve as a better target context for NAT such that NAT can produce high-quality translation with only one iteration in the second stage. The authors introduce several tricks to optimize HRT, including joint training, mixed distillation and curriculum learning etc. Experiments on two WMT translation tasks (four translation directions) demonstrate the effectiveness of HRT, which consistently accelerates decoding by >50% compared to the AT baseline and yields comparable or even better translation quality.

Overall, this IR issue is under-studied in the literature, which deserves more attention. But I am not fully convinced by the current experiments, especially the comparisons.

My major concerns:

= Firstly, only comparing with MP10 is not enough. HRT consists of two parts. The first part AT acts very similar to the semi-autoregressive model (Wang et al, EMNLP 2018), which should be treated as a baseline. The second part, NAT or MP used in experiments, involves different variants when different iterations are used. For example, Figure 2 in Mask-Predict paper (Ghazvininejad et al, EMNLP 2019) discovers the trade-off between speedup and translation quality. Could you please show the Pareto frontier to prove that HRT reaches a better trade-off compared to MP? Besides, one thing should also be noticed: the speedup yielded by HRT seems less charming compared to recent NAT models, such as Levenshetain Transformer 3x-4x (Gu et al, NeurIPS 2019), and JM-NAT (k = 10) 5.73x (Guo et al, ACL 2020) where both models achieve similar translation quality to the AR baseline.

= If I understand correctly, the advantage of HRT in handling different batch sizes and computing devices mainly comes from its usage of one-pass NAT. In Figure 3, we observe that HRT1-1 produces clearly larger speedups as batch size increases from the CPU graphs. However, we observe a reversed trend with MP1 in Figure 1. Could you give readers some explanation? 

= Although this paper mainly focuses on improving NAT, efforts on accelerating decoding is not limited to NAT. Relevant studies, which should be at least discussed in related work, include model quantization [1] and simplified decoder [2, 3], to name a few, are all missed in the paper. In particular, some models like AAN [2] and Deep encoder+Shallow decoder[3] can already produce similar speedups (might be smaller) with comparable translation performance. These models do not rely on knowledge distillation and those complex optimization tricks used for HRT. Could you please explain the advantage of HRT over these models?

= The claim “Chunk is superior to Rand, which indicates that the balanced distribution of deterministic tokens is necessary” (above Section 4) is strong. From Figure 2, what I see is that Chunk performs slightly better than Random. It’s hard to conclude that the balanced distribution is *necessary*.

My minor concerns:

= How did you handle the positional encoding in AT of HRT? Did you consider the interval k?

= The proposed optimization tricks are somehow orthogonal to HRT itself. What if you apply them to the baselines, like MP and semi-autoregressive model?

= How many runs did you perform when reporting the speed numbers in Figure 3?

= I cann’t find the number *34.08* (Table 3) in Table 2 for En->Ro translation. Is there something wrong?

= The citation format should be adjusted accordingly. The authors always use the format of “author (year)”.

= It would be better to see more results on the translation of distant language pairs, such as WMT English-Chinese. 

[1] Bhandare et al.  Efficient 8-bit quantization of transformer neural machine language translation model. 2019

[2] Zhang et al. Accelerating neural transformer via an average attention network. ACL 2018

[3] Kasai et al. Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation. 2020

review_title: the studied problem is novel, but the comparison is not fully convincing.

Meta-review: This paper proposes a new method to combine non-autoregressive (NAT) and autoregressive (AT) NMT. Compared with the original iterative refinement for non-autoregressive NMT, their method first generates a translation candidate using AT and then fill in the gap using NAT.

All of the reviewers think the idea is interesting and this research topic is not well-studied. However, the empirical part did not convince all the reviewers. The revised version and response is good; however, it still does not solve some major concerns of reviewers.

Decision: Reject