Hybrid-Regressive Neural Machine Translation
Authors: Qiang Wang
Heng Yu
Shaohui Kuang
Weihua Luo
Venue:  (2021)
Keywords: neural machine translation
non-autoregressive translation

Abstract: Although the non-autoregressive translation model based on iterative refinement has achieved comparable performance to the autoregressive counterparts with faster decoding, we empirically found that such aggressive iterations make the acceleration rely heavily on small batch size (e.g., 1) and computing device (e.g., GPU).
By designing synthetic experiments, we highlight that iteration times can be significantly reduced when providing a good (partial) target context. 
Inspired by this, we propose a two-stage translation prototype -- Hybrid-Regressive Translation (HRT). HRT first jumpily generates a discontinuous sequence by autoregression (e.g., make a prediction every k tokens, k>1). Then, with the help of the partially deterministic target context, HRT fills all the previously skipped tokens with one iteration in a non-autoregressive way.
The experimental results on WMT'16 En-Ro and WMT'14 En-De show that our model outperforms the state-of-the-art non-autoregressive models with multiple iterations, even autoregressive models. Moreover, compared with autoregressive models, HRT can be steadily accelerated 1.5 times regardless of batch size and device.

Under review as a conference paper at ICLR 2021
HYBRID-REGRESSIVE
NEURAL MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Although the non-autoregressive translation model based on iterative refinement has achieved comparable performance to the autoregressive counterparts with faster decoding, we empirically found that such aggressive iterations make the acceleration rely heavily on small batch size (e.g., 1) and computing device (e.g.. GPU). By designing synthetic experiments, we highlight that iteration times can be significantly reduced when providing a good (partial) target context. Inspired by this, we propose a two-stage translation prototype - Hybrid- Regressive Trans- lation (HRT). HRT first jumpily generates discontinuous sequence by autore- gression (e.g., make a prediction every & tokens, & >)) Then, with the help the partially deterministic target context, HRT fills all the previously skipped to- kens with one iteration in a aooeerstreee way. The experimental results on WMT 16 En<+Ro and WMT' 14 En show that our model outperforms the state-of-the-arr non-autoregresssive models with multiple iterations, and the origi- nal autoregressive models. Moreover, compared with autoregressive models, HRT can be steadily accelerated 1.5 times regardless of batch size and device.
INTRODUCTION
Although autoregressive translation (AT) has become the de facto standard for Neural Machine Translation (Bahdanau al. 2015), its nature of generating target sentences sequentially (e.g.. from left to right) makes it challenging to respond quickly in production environment. One straightfor- ward solution is the non-autoregressive translation (NAT) (Gu al. 2017), which predicts the entire target sequence in one shot. However, such one-pass NAT models lack dependencies between target words and still struggles to produce smooth translations, despite many efforts developed (Ma al.,
2019
Guo et al. 2019a| Wang et al. 2019b Shao et al.
2019 Sun et al. 2019).
Recent studies show that extending one-pass NAT to multi-pass NAT, so-called iterative refinement (IR-NAT), is expected to break the performance bottleneck (Leee et al.] 2018 Ghazvininejad et al. 2019 Gu et al. 2019 Guo al.) 2020; Kasai et al. 2020a). Unlike one-
MP(=1)
MP
Relative Speedup Ratio
15
Faster
10
5
pass NAT, which outputs the prediction imme- diately, IR-NAT takes the translation hypothe- sis from the previous iteration as a reference and regularly polishes the new translation un- til achieving the predefined iteration count / or no changes appear in the translation. Compared with AT. IR-NAT with I=10 runs 2-5 times faster with a considerable translation accuracy, as reported by Guo et al. (2020).
0
Stower
0
10
20
30
Decoding Batch Size
Figure 1: Relative speedup ratio (a) compared MP with AT on GPU (solid) and CPU (dashed). The value of denotes running faster (positive) or slower (negative) |a times than AT.
However, we highlight that the fast decoding of IR-NAT heavily relies on small batch size and GPU, which is rarely mentioned in prior studies Without loss of generality, we take Mask-Predict (MP)
Unfortunattly, such a decoding setting is not common in practice. NMT systems deployed on GPUs tend to use larger batches to increase translation throughput, while the batch size of is used more frequently in offline systems running CPUs. e.g., smartphones.

---

Under review as a conference paper at ICLR 2021
(Ghavvinieejad al. 2019) example, typical IR-NAT paradigm based the conditional masked language model. Figure illustrates that when the batch exceeds 8, MP(1=10) is already running slower than AT, and the situation is worse CPU. Further analysis shows that the increase in batch size leads to the efficiency degradation of parallel computing in NAT models
To tackle this problem, first design synthetic experiment understand the relationship between target context and iteration times. We mask some proportion tokens on the translation generated by a pretrained AT and take the decoder input of the pretrained MP. Then we surprisingly found that even masking 70% AT hypothesis, and the remaining target context can help MP(I=1) to compete with the standard MP(I=10) (Figure 2). This result confirms that decoding with multiple iterations in NAT is unnecessary when providing a good (partial) reference hypothesis.
Inspired by this, we propose a two-stage translation prototype-HHbrid- -eebrisseeeeeseveeeeeeeeee Translation (HRT). After encoding, HRT first uses an autoregressive decoder (called Skip-AT) to produce a dis- continuous translation hypothesis. Concretely, at decoding step i, the SKip-AT decoder immediately predicts the (i + k--th token Vi+A without generating V/+1., ..... Vi++ where k is hyperparam- eter and > Then, a non-autoregressive decoder like MP (called Skip-MP) predicts previously skipped tokens with one iteration according to the deterministic context provided by Skip-AT. Since both Skip-AT and Skip-MP share the same model parameters, HRT does not increase parameters significantly. To train HRT effectively and efficiently, further propose joint training guided by curriculum learning and mixed distillation. Experimental results on WMT En and En show that HRT is far superior existing IR-NAT's and achieves comparable even better accuracy than the original AT with a consistent 50% decoding speedup on varying batch sizes and devices (GPU, CPU).
BACKGROUND
Given source sentence æ {21, 22...., RM and target sentence y {u1 , .NN there are several ways to model P(y x):
Autoregressive translation (AT) is the dominant approach in NMT, which decomposes P(y(æ) by chain rules:
NN II
p(yla)
P(ye ax, vee)
1=1
where V<r denotes the generated prefix translation before time step t. However, the existence of Vett requires the model must wait for yt-1 to be produced before predicting ys, which hinders the possibility of parallel computation along with time step.
Non-autoregressive translation (NAT) is first proposed by Gu et al. (2017). allowing the model to generate all target tokens simultaneously. NAT replaces yea with target-independent input and rewrites Eq. as:
N p(y|))= = P()))) II P(y (x
(2)
In Gu al. (2017), they monotonically copy the source embedding as according to a fertility
model. Subsequently, the researchers developed more advanced methods to enhance z, such as adversarial source embedding (Guo et al., 2019a), reordered source sentence (Ran et al,, 2019). latent variables (Ma et al. 2019 Shu al. 2019) etc, but there still is a huge performance gap between AT and NAT.
Iterative refinement based non-autoregressive translation (IR-NAT) extends the traditional one- pass NAT by introducing the multi-pass decoding mechanism (Lee et al. 2018 Ghazvininejad al.)
2019
Gu et al. 2019 Guo et al. 2020 Kasai et al. 2020a). IR-NAT applies a conversion function
2 Early experiment shows that when the batch size increases from to 32, the latency of AT is reduced by 22 times, while MP(1=10) only reduces by four times. Latency measured by the average time of translating constant test set. See Appendix (or details.
'Thanks to the proposed training algorithm, single HRT model can support both hybrid-regressing decod- ing and autoregressive decoding at inference. Here, the AT model refers to the autoregressive teacher model that generates the distillation data.
2

---

Under review as a conference paper at ICLR 2021
F on the deterministic hypothesis of previous iteration y' as the alternative to Common imple-
mentations of F include identity (Lee et al. 2018), random masking (Ghazvininejad et al. 2019) or
random deletion (Gu et al. 2019) etc. Thus, can predict by:
N
P - II P(emece (1) F(yr))
(3)
where N' is the number of refined tokens in F(y)). m(t) is the real position of t-th refined token in yr. In this way, the generation process of IR-NAT is simple: first, the NAT model produces an inaccurate translation as the initial hypothesis, and then iteratively refines it until converge or reaching the maximum number of iterations.
Mask-Predict (MP) is a typical instance of IR-NAT, trained by a conditional masked language model objective like BERT (Devlin al. 2019). In this work, we use MP as the representation of IR-NAT due to its excellent performance and simplification. In MP, F randomly masks some tokens over the sequence training but selects those predicted tokens with low confidences inference.
3 IS ITERATIVE REFINEMENT ALL YOU NEED?
As mentioned earlier, IR-NAT with multiple iterations slows down severely in some cases. It is natural think of reducing iterations to alleviate it. This section starts from synthetic experiments on WMT' 16 Ro and WMT' 14 En to verify the assumption that a sufficiently good decoder input can help reduce iterations. Here we construct the "good" decoder input from the translation hypothesis produced by an AT model.
Models We use the official MP models released by Ghazvininejad et al. (2019) Since the authors did not publish their AT baselines, we use the same data to retrain AT models with the standard Transformer- configuration (Vaswani al. (017) and obtain comparable performance with theirs (see Appendix B for more details).
Decoding AT models decode with beam sizes of 5 on both tasks. Then, we replace a certain percentage of AT translation tokens with <mask> and use as input to the MP model (see below for replacement strategy). Unlike the standard MP model that uses large beam size (e.g.. 5) and iterates several times (e.g., 10), the MP model used here only iterates once with beam size We substitute all input <mask>> with MP's predictions to obtain the final translation. We report case- sensitive tokenized BLEU score by multi-bleu.perl.
Mask Strategy We tested strategies to mask AT translations: Head, Tail, Random and Chunk. Given the masking rate Pmask and the translation length N, the number of masked to- kens is mmxxt==ax(1, [N Then Head/Tail always masks the first/last Nmask tokens, while Random masks the translation randomly. Chunk is slightly different from the above strate- gies. It first divides the target sentence into C' chunks, where C' = Ceil(N/k) and k is the chunk size. Then in each chunk, retain the first token, but mask other k-1 tokens. Thus, the actual masking rate in Chunk is 1-1/k instead of Pmask- To exclude randomness, we ran Random three times with different seeds and report the average results.
3.1 RESULTS
The experimental results are illustrated in Figure 2 where we see that:
balanced bidirectional context is critical. Compared with Tail 1 and Head, it obvious that Rand and Chunk both have better performance. We attribute it to the benefit of the bidirectional context in Rand and Chunk (Devlin et al., 2019), because Tail and Head can only provide unidirectional context (i.e., prefix suffix). In addition, compare Chunk with Random, we find that Chunk is moderately but consistently superior to Random, even if more tokens are masked. For instance, the WMT En-De task, when the chunk size is 4 (the masking rate is 75%), the BLEU score of Chunk is 27.03, which is +0.3 BLEU higher than that of Random with the masking rate of

---

Under review as a conference paper at ICLR 2021
head
0
tail
random
chunk
head
tail
random
chunk
1=10
ha I=10
BLEU
BLEU
be5.Tel
20
b=5, Im]
bc 1=1.11
hell
26
0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
masking rale Pmnakk
masking rale Proaak
Figure 2: Comparison of four masking strategies (Head, Tail, Random, Chunk) } in synthetic experiments on WMT En-Ro (Left) and En (Right) test sets. For Chunk. we the chunk size from {2, 3, 4). Dashed lines represent Mask-Predict's scores reported by Ghazvininejad et al. (2019) stands for "beam size" while / stands for "the number of iterations".
Table Examples training samples. <s2> special 
 for k=2. <m> short for <mask>>
Mode
Input
Target
AT

, y1, 1/2 y3, V4 U1. <m>>, <m>>, U1: <m>>
y1, U2, V3. 14: </S>
MP
PAD, U2, V3: PAD, </$>>
AK (k=2)
<,2> 1/2 34
y2 34 

Skip-MP (k=2)
<m>>, U2, <m> V4: <m>, 

y1 PAD, Vs PAD, 
 PAD
70%. Because the difference between Chunk and Random lies only in the distribution of <mask>, this experiment indicates that making <mask> uniformly on sequence is better than random S
Small beams and one iteration are sufficient. Compared with the standard MP with the beam size of 5 and 10 iterations, it is interesting to find that even if only 30%-40% of the AT translations are exposed, our MP using greedy search and one iteration can achieve quite comparable performance.
4 BYBRID-REGRESSIV TRANSLATION
A limitation in the above synthetic experiment is that the MP decoder input comes from an AT hypothesis, which impossible in practice [ To solve this problem as well as inspired by the Chunk strategy's ' success, we propose two-stage translation paradigm called Translation (HRT). Briefly speaking. HRT can autoregressively generate a discontinuous sequence with chunk size l: (stage I), and non-autoregressiivll fill the skipped tokens (stage II) in one model. Thus, the standard AT can regarded as the special case HRT when k=l without stage II.
Regriid-eereesivve
4.1 ARCHITECTURE
Overview. Our HRT consists of three parts: encoder, Skip-AT decoder (for stage D,, and Skip-MP decoder (for stage II). All components adopt the Transformer architecture (Vaswani et al,, 2017): : the encoder contains self-attention sublayers and feedforward sublayers, and additional cross-attention sublayers are added to the decoder. The two decoders have the same network structure and share model parameters, leading the same parameter size compared to the standard MP. The only differ- ence between the two decoders lies in the masking mode in the self-attention sublayer. The Skip-AT decoder masks future tokens to guarantee strict left-to-right generation, while the Skip-MP decoder eliminates this limitation to leverage the bi-directional context.
Simplified relative position representation. Another difference from the standard MP architecture that our decoder self-attention equips with relative position representation (RPR) (Shaw et al.
Chhnk guarantees that each unmasked token (except the first last in the sequence) deterministic tokens within the window size k. However, in extreme cases, when all <mask>> happen to concentrate on the left/right side of the sequence, Kandom will degrade into Head/Iaail Wee can directly return AT predictions translation results without going through MP.
4

---

Under review as a conference paper at ICLR 2021
2018 to enable the model to capture the positional relationship between words easily the decoder self attentitn with RPR is calculated by:
Precisely,
O Softmax Q,(K* + R
V
(4)
du
where R, is the relative position embedding Note that Eq. 4 only injects the relative positional representation in Key (K" + R2) without involving Value V. We found that this simplification has no negative impact on performance but significantly saves memory footprint.
No target length predictor. Most previous NAT methods need to jointly train the translation model with an independent translation length predictor. However, such a length predictor is unnecessary for us because the translation length by-product of Skip-AT, e.g., N Nat, where Nat the sequence length produced by Skip-AT Another bonus is that we can avoid carefully tuning the weighting coefficient between the length loss and the token prediction loss.
4.2 TRAINING
Training HRT models is not trivial because single HRT model needs to learn to generate sequenees by both autoregression and non-autoreerresionn This section will introduce three details of training HRT models, including chunk-aware training samples, curriculum learning, and mixed distillation. We describe the entire training algorithm in Appendix C
Chunk-aware training samples. As listed in Table I the training samples for Skip-AT and Skip- MP are different from the standard AT and MP. Compared with AT, Skip-AT shrinks the sequence length from N to N/k. It should be noted that, although the sequence feeding to Skip-AT AT is shortened, the input position still follows the original sequence rather than the surface position. For example, in Table the position of Skip-AT input (<s2>, y2. 41) is (0, 2, 4), instead of (0, 1, 2). Moreover, MP has the opportunity to mask any token over the target sequence without considering the position. However, the masking pattern in Skip-MP is deterministic, i.e., masking all non-first tokens in each chunk. Therefore, we can say that the training sample for Skip-AT and Skip-MP is in chunk-aware manner.
Curriculum learning. Unfortunately, direct joint training of Skip-AT and Skip-MP is problematic because the chunk-aware training samples cannot make full use of all the tokens in the sequence. For example, in Table I1 the target tokens y1 and y/ have no chance to be learned as the decoder input of either Skip-AT or Skip-MP. However, there is no such problem in AT and MP. Therefore, we propose to gradually transition from joint training {AT, MP} to (Skip-AT, Skip-MP) through curriculum learning (Bengio et al. 2009). In other words, the model is trained from chunk size to chunk size k (k>1). 1). More concretely, given a batch of original sentence pairs B = (x) [|== and let the proportion of chunk size k in B be Pk, start with pæ 0 and construct the training samples of AT and MP for all pairs. And then we gradually increase pk to introduce more learning signals for Skip-AT and Skip-MP until px=l. In implement, we schedule pk by:
Pk
(5)
where t and T are the current and total training steps, respectively. is a hyperparameter and we use >=1 to increase pæ in linear manner for all experiments.
Mixed Distillation. NAT models generally use the distillation data generated by AT models due to more smoothing data distribution (Zhou et al. 2020). However, making full use of distillation data may miss the diversity in raw data. To combine the best of both worlds, we propose simple and effective approach- - Mixed Distillation (MixDistill). During training. MixDistill randomly samples target sentence from the raw version y with probability Praw or its distillation version y² with probability I-Praw. where Praw is a hyperparameter 10 By learning from raw target sentences, we empirically found that HRT is less prone to over-fit in some simple tasks (e.g., WMT' 16 En En-Roo).
We keep the sinusoidal absolute position embedding unchanged.
$ RR = Emhhd(clip(i, 1)...., clip(i, N)), where clip(i, j) = max(w, min(w, j j-))), vo is the window size.
9 "More precisely, Nnat here the maximum target length rather than the realistic length because multiple
</>> may he predicted the last tokens.
Training with full raw data or full distillation data can be regarded as the special case of MixDistill when Pl =O.
5

---

Under review as a conference paper at ICLR 2021
The BLEU scores of our proposed HRT and the baseline methods four tasks. Unless otherwise stated, the used beam size "?" denotes dynamic iterations. "20L" stands for using a 20-layer encoder. "NPD" is short for Noisy Parallel Decoding. All HRT models only iterate once by non-autoregression.
Table 2:
IM
WMT'14
System
Iterations
En-Ro
Ro-En
En-De
De-En
Transformer
34.25
34.40
27.45
31.86
Transformer-201.
28.79
33.02
Reeaaive
Existing NATS
Iterative Retinement (Lee et al. 2018
29.66
30.30
21.54
25.43
Mask-Predict (Ghazviningaad al. 2019
1
27.32
28.20
18.05
21.83
Mask-Predict (Ghazvinnejad
2019
10
33.08
33.31
27.03
30.53
Lev DevTransformer (Gü ct 2019
?
27.27
JM-NAT (Guo et al. 2020
10
33,52
33,72
27.69
32,24
Non-Ierative
SAT (k=2) Wang et
2018a
26.90
FCL-NAT (NPD'9) Guo al.
2019b
25.75
29.50
HIRT =b, Bmpppppp =mp=1)
34.11
34.28
27.85
31.80
Our models
HRT (ba==5, Dmy=l)
34.36
34.55
27.98
31.93
HRT (b..=5, Dug=5)
1
34.53
34.80
28.10
32,07
HRT-20C (ba =l)=
28.79
"32.86"
HIRT-20L (bar5, =mp=1)
1
28.90
33.06
HRT-20L (b, 55)
1
28.99
33.08
4.3 INFERENCE
After encoding, the Skip-AT decoder starts from <sk> to autoregressively generate a discontinuous target sequence Yut (21, 22, 2m with chunk size until meeting 
. Then we construct the input of Skip-MP decoder Emp by appending k - <mask> before every The final translation generated by replacing all <mask> with the predicted tokens by Skip-MP decoder with one iteration. If there are multiple 
 existing, we truncate to the first 
, Note that the beam size but in Skip-AT can be different from the beam size mmp in Skip-MP as long as st. b 2 bmp. bat Dmp, then only feed the Skip-MP with the top hmp Skip-AT hypothesis. Finally, choose the hypothesis with the highest score:
m Σ (2 220)
score(ū)
(Dic+t |2, Tmp)
Skip-AT score
Skip-MP score
where --=lixk- In Appendix D we summarized the comparison with the existing three methods from the aspects of decoding step and calculation cost, including AT, MP, and cemi-autoregressive translation (SAT) (Wang et al. 201 20). Besides, thanks to the joint training of chunk size and simultaneously, the HRT model can also behave like standard AT model by forcing decoding by chunk size one (denoted as Ca=1). In this way, we can only use the Skip-AT decoder to generate the entire sequence without the help of Skip-MP. Thus, C,=11 can be regarded the performance upper bound when the decoding chunk size is k (denoted as Ca=k).
EXPERIMENTS
Datasets. We conducted experiments on four widely used tasks: WMT' English+-Romanian (En+>Ro, 610k) and WMT' 14 English++German (En++De, 4.5M). We replicated the data
processing as Ghazvininejad et al. (201)) for fair comparisons.
AT teachers for distillation. Since Ghazvininejad et al. (2019) only release the distillation data of En++Ro, not Ent+De, we retrained the AT' teacher models of En to produce the distillation the teacher, but we the deep PreNorm Transformer-Baa Base with a 20-layer encoder, which is faster to train and infer with comparable performance (Wang et al. 2019a).
data. Specifically, (Ghavviineeadd et al. (2019) use Transformer-Large
Models and hyperparameters. noted otherwise, the chunk size k=2 and \=1. Pr5 for En and Pra for
We ran all experiments on TITAN X (Pascal) GPUs. Unless

---

Under review as a conference paper at ICLR 2021
En++Ro according validation sets. The windows size of RPR is 16 HRT models finetuned on pretrained AT models for the same training steps 12 Other training byperparameters are the same as Vaswani et al. (2017) Wang et al. (2019a) (using deep-encoder). Please refer to them for more details.
Translation quality. Table reports the BLEU scores on four tasks. First of all, we can see that IR-NAT models significantly outperform those one-pass NAT models (e.g., SAT, FCL-NAT, FlowSeq). However, our small beam model (b===bm==1) can defeat the existing multiple-iteration models. Furthermore, when the beam sizes increase to 5, HRT equipped with standard 6-layer encoder achieves +1.0 BLEU point improvement in En compared to the previous best results (Guo et al. 2020). Even on harder Ent+De tasks, also outperform them with 0.4 BLEU score. We can easily trade-off between performance and speed by using = but bmp=1. Not only NAT models, but also we are surprised to find that HRT can even surpass the AT models trained from scratch. We attribute it to two reasons: (1) HRT is fine-tuned on a well-trained AT model, making training casier; (2) Mixing up AT and NAT has a better regularization effect than training alone. Besides, in line with Guo et al. (2020), which demonstrate that the encoder is critical for NAT, we can obtain further improvement of about +0.8 BLEU when using a deeper encoder.
On distant language pair. To verify our method can apply to distance language pairs, we conducted a new experiment on the Chinese-English (Zh-En) translation task. We use the same datasets as Wang et al. 8b): the training data is selected from LDC Corpus and con- tains 1.8M sentence pairs; NIPS MT06 as the valida- tion set and MT04, MT05, and MT08 as test sets. We train all models for 50k steps, and other training setting the same as WMT En-De. The experimental results are listed in Table 3 We can see that HRT is superior to the original AT model and MP model again, which indicates that HRT is effective in both close and distant language pairs.
Figure 3: BLEU scores on the Chinese-Englise task.
Model
MT04
MT05
MT08
AT
43.86
52.91
33,94
MP(I=10)
42.47
52.16
33.09
HRT1-1
43.96
53.16
33.99
HRT5-1
44.28
53.44
34.63
HRT5-5
44.31
53.77
34.74
Translation speed. Unlike the previous works that only run the model on GPU with batch size of I, we systematically test the decoding speed using varying batch sizes and devices on WMT' 14 En-+De test set (see Figure 4.. By default, the beam size is 5. can be seen that although HRT is slower than MP10 when running on GPU with a batch size of 1, MP10 dramatically slows down as the batch size increases. In contrast, HRT5- consistently more than 50% faster than AT without changing with the environment. These results show that our HRT can be an effective and efficient substitute for AT and IR-NAT.
6 ANALYSIS
Effect of Mixed Distillation. In Table we compared different data strategies, including raw data (Raw), sequence-level knowledge distillation (Dist.), and mixed distillation (Mix Dist.). Overall, Mix Dist. is superior to other methods across the board, which indicates that training with raw data and distillation data is complementary. distillation data is lower than the raw data on En->Ro task, which is against the previous results. As interpreted by Zhou et al. (2020), we suspect that when the translation model is strong enough, training by distillation data completely may make the learning too easy and lead to over-fitting.
addition, we also find that the performance of the
Effect of chunk size. We tested the chunk size k from {2. 3, 4}, and the results are listed in Table Obviously, we can see that: (1) A large has more significant acceleration on GPU because fewer autoregressive steps are required; (2) As increases, the performance of regresi-eeeesssive decoding
"'For autoregressive baselines, adding RPR in the Transformer decoder did not bring obvious improve- over the vanilla Transformer. For example, WMT' En- Transformer=27.45 and Trans- Pormer+
"iinee HRT needs to train Skip-AT and Skip-MP jointly (please see Algorithm in Appendix C the wall- clock time is about two times longer than AT in the same training epochs. One more thing to note is that the officially released MP models trained for 300k steps from scratch.
7

---

Under review as a conference paper at ICLR 2021
Batch=1
Butch=8
Batch=16
Butch=32
GPU
1:34
10E15-1
1:38
10015-9
MIN
2:45
3014
11.00
MPN
MPN
MP10
5.13
SIP10
1.30
MP10
MPP0
CPU
HKTI-11
2.18
HRTT-1
2,00
HSL22-
5.95
4.6
FIRTS-1
11ETS
1-74
11075-1
HRTT5-
1.09
5
HRTS
1:23
11R75-5
Figure 4: Comparison of decoding speed w.r.t batch size and computing device on WMT 14 En task. The x-axis is the relative speed compared to the corresponding autoregressive models (dashed lines at x=1). GPU:TITAN X (Pascal), CPU:Intel(R) Xeon(R) E3-2680 v3 @ 2.50GHz. MP(#) denotes the MP model with # iterations. HRT ( denotes HRT with bar= 1 and =22 Note that the BLEU score of MP4 is 25.94, which is significantly lower than that of HRT
(e.g., the BLEU score of HRTI- 1-1 is 27.85).
Table 3: Performance against different data strategies. Cx=1 represents decoding the HRT model in an autoregressive manner.
Table 4: The effect of training by different chunk sizes. Latency is tested in batch size
of 16 using Ca=k and ba:=bmp=1.
Lang.
Ca
Raw
Dist.
Mix Dist.
Chunk (k)
BLEU
Latency (sec.)
k
33.92
33.41
34.53
C5= k
Ca=1 1
GPU
CPU
En-=Ro
34.29
33.41
34.27
34.11
33,86
20.0
70.5
k
26.37
28.00
28.10
31.15
33.78
13.0
54.6
En->De
27.60
28.42
28.51
28.22
34.12
12.2
53.9
drops sharply (e.g., k=4 is 6 BLEU points lower than k=2.), but k has little effect on autoregressive modes. It indicates that the training difficulty of Skip-AT increases as k gets bigger. We think that skip-generation may require more fancy model architecture or training method, which is left for our future work.
Effect of decoding mode. We tested the well-trained HRT model with two decoding modes: au- (C==k). Concretely, We divided the test set of WMT' 14 En-+e into several groups according to the source sentence's 's length and then compared the two decoding modes in terms of translation speed and accuracy in each group (see Figure First of all, we can see that regardless of the source sentence length, the running speed of Ca=k is consistently faster than C,=1 on both GPU and CPU, thanks the shorter autoregressive length. This advantage is more evident on CPU: When the source length is less than 10, Ca=k runs 1.6 times faster than C2=1, while the speedup ratio increases to 2.0 when the source length > 50. As for accuracy, C.=kk has closed performance to Ca=1 when the length is between 10 and 30, but shorter or longer sen- tences will hurt the performance. This result indicates that we dynamically adjust the decoding chunk size C. according to the source sentence's 's length, the HRT model can be expected to improve the performance further at the expense of a certain speed.
toregressive (Ca=l) and
Ablation study. We also did an ablation study on WMT' 16 En-RRo test set. As shown in Table we can see that all introduced techniques help to improve performance. In particular, using mixed distillation prevents the HRT model from over-fitiee werr-ftting and leads to +1.1 BLEU points improvement compared to the standard distillation (-MixDistill). In addition, the other three methods, including training the HRT model from pretrained AT model (FT), using relative positional representation on decoder (RPR), and using curriculum learning (CL), can bring about 0.3-0.4 BLEU improve- ments each. It should be noted that removing curriculum learning makes the trained HRT model fail to decode by C,=1, whose BLEU score is only 5.18. Since the BLEU score decreases slightly (0.3-0.4 except -MixDistill) when each component is excluded independently, it is difficult to say
8

---

Under review as a conference paper at ICLR 2021
3.90
Da Sae Bp
1.000
0,900
2,00
1,000
200
Figure 5: Translation speed (Left: GPU, Middle: CPU) and BLEU score (Right) against source sentence's 's length for different decoding modes in HRT (k=2): Ca=1 denotes decoding by autore- gression, while Cy=k denotes regriseregressive decoding. Speed is measured at batch size=32, bar=5, bmp=
that the difference of BLEU is not caused by random fluctuation. To verify it, we try to exclude them all from the standard HRT (-ALL). Interestingly, the obtained model drops by 1.96 BLEU points, which is very close to the cumulative BLEU loss (2.13) of excluding each component separately. It indicates that these newly introduced components are complementary. In addition, we also test these methods in MP training. Please see Appendix E for details.
Table 5: Ablation study WMT 16 En >Ro test set.
System
AT
HRT
-FT
RPR
-ixtiililll
-CL(g=110)
-ALL 32.57
BLEU
34.25
34.53
34.14
34.22
33.41
34.22
7 RELATED WORK
Iterative refinement. Lee et al. (2018) first extend NAT from the conventional one-pass manner to the multi-pass manner. They add additional decoder to learn to recover from collapsed target sequence to gold one. Mask-Predich (Ghazvininejad et al. 2019) simplifies the two-decoder structure by introducing the conditional masked language model objective. During each iteration, Mask-Predict retains partial inputted target tokens according to the prediction confidence, while LevTransformer (Gu et al.) 2019) uses multiple diseriminators determine the edited tokens.
Combination of AT and NAT. The idea of incorporating AT in the NAT model is not new (Kaiser jt al. 2018: Ran et al. 2019; Akoury et al., 2019). The main difference from existing methods lies in the content of AT output, such as latent variables (Kaiser et al.) 2018). reordered source tokens
RRan et al. 2019). syntactic labels (Akoury et al. 2019) etc. contrast, our approach uses the deterministic target tokens, which has been proven effective in Ghhavvininejja et al. (2019)
Decoding acceleration. In addition to transforming the decoding paradigm from autoregressive to non-autoregressive, there are many works explore how achieve faster decoding from other as- pects. Zhang et al.) (2018c propose to optimize the beam search progress by recombining or prun- ing the translation hypothesis. Considering the network architecture, Zhang et al. (2018a) use light AAN instead of the standard self-attention module; Xiao et al. (2019) share the self-attention weight matrix across decoder layers; Kasai et al. (2020b) suggest using deep-encoder and network to keep high BLEU score and low delay. Moreover, some common model compression techniques, such as distillation (Kim & Rush 2016) and quantization (Bhandare et al. [019 Lin al.) 2020), have also helpful for acceleration. However, the above methods mainly focus the traditional autoregressive translation, which is orthogonal to our work.
8 CONCLUSION
We have pointed out that NAT, especially IR-NAT, cannot efficiently accelerate decoding when using a large batch running CPUs. Through well-designed synthetic experiment, we highlighted that given a good decoder input, the number of iterations in IR-NAT could be dramatically reduced. Inspired by this, proposed a two-stage translation paradigm HRT to combine AT and NAT's advantages. The experimental results show that HRT owning equivalent or even higher accuracy and 50% acceleration ratio varying batches and computing devices is good substitute for AT.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Nader Akoury, Kalpesh Krishna, and Mohit lyyer. Syntactically supervised transformers for faster neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 1269-1281, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning. ICML '09, pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 15534// URL 1553374.155333
Aishwarya Bhandare, Vamsi Sripathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Saletore. Efficient 8-bit quantization of transformer neural machine language translation model. arXiv preprint 0rXiv:/906.000 00532, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1111866/1//N19-1223. URL https:
www. wwwwwwwww....c http:/wwww...g/.nnegoeeeeeeeee htth//oohooool/ll 1423
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 6111-6120, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1166 URL
https: www www.awcwwb.bc org http///ahhhooool-eoeeaea
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoreegessive
neural machine translation. arXiv preprint arXiv:1711.02228 2017.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In Advances in Neural Information Processing Systems, pp. 11 11179-1118) 2019.
Junliang Guo, Xu Tan, Di He. Tao Qin, Linli Xu, and Tie- Yan Liu. Non-autoregressive neural machine translation with enhanced decoder input. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3723-3730, 2019a.
Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie- Yan Liu. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. arXiv preprint 19Xiv:1911.087 2019b.
Junliang Guo, Linli Xu, and Enhong Chen. Jointly masked sequence-to-sequenee model for non- autoregressive neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 376-385, 2020.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Con- ference on Machine Learning, pp. 2390-2399, 2018.
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Parallel machine translation with disentangled context transformer. In ICMI. 2020: 37th International Conference on Machine Learning, 2020a.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. Deep encoder, shal- low decoder: Reevaluating the speed- tradcoff machine translation. arXiv preprint arXiv:2006. 10369, 2020b.
10

---

Under review as a conference paper at ICLR 2021
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327,
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neu- ral sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1173-1182, Brussels, Belgium, October-November 2018. doi: 10.18653/1//D1-1J 1149. URL UtLhttts:://// /www. www.eb.wwwwbrrr..o aathooooyyDD18-1153
Ye Lin, Yanyang Li, Tengbo Liu, Tong Xiao, Tongran Liu, and Jingbo Zhu. Towards fully 8-bit integer inference for the transformer model. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pp. 3759-3765 3765, 2020.
Xuezhe Ma. Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. Flowseq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.
2019.
Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. Guiding non-auttoregreesivv neural machine translation decoding with reordering information. arXiv preprint arXiv: 2019.
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, Xilin Chen, and Jie Zhou. Retrieving sequential information for non-autoregreesive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3013-3024, 2019.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp- 464-468, New Orleans, Louisiana, June 2018, Association for Computational Linguistics. doi: 2074
10118653/1/NN18-220 URL htt.https// wwww. ac www..wbb..... www.:ew......thooooooooooooooooo www.ttooooooyyooo
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable
autoregressive neural machine translation with deterministic inference using a delta posterior. arXiv preprint arXiv: 0tt:::990000770 071801 2019.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. Fast structured decoding for sequence models. In Advances in Neural Information Processing Systems, pp. 301 1- 3020, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 6000-6010, 2017.
Chunqi Wang, Ji Zhang, and Haiqing Chen. Semi-autoregressive neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 479-488, 2018a
Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu. Multi-layer represen- tation fusion for neural machine translation. Proceedings of the 27th International Conference Computational Linguistics, pp. 3015-3026, 2018b.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1810-1822, Florence, Italy, July 2019a. htthttt:: wttp::/ww.bbbbbrg///eeee http://www.........raaaaaooooooa 1176
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Non-autoregressivv machine translation with auxiliary regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5377-5384, 2019b.
Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu. Sharing attention weights for fast transformer. In Proceedings of the Srenty-Eightte International Joint Conference on Artificial Intelligence, pp. 5292- 22988, 2019.
11

---

Under review as a conference paper at ICLR 2021
Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via average atten- tion network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1789-1798, 2018a.
Wen Zhang, Liang Huang, Yang Feng, Lei Shen, and Qun Liu. Speeding up neural machine trans- lation decoding by cube pruning. Proceedings of EMNLP 2018, 2018b.
Zhisong Zhang, Rui Wang, Masao Utiyama, Eiichiro Sumita, and Hai Zhao. Exploring recombina- tion for efficient decoding of neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4785-4790, 2018c.
Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non- autoregressive machine translation. In ICLR 2020: Eighth International Conference on Learning Representations, 2020.
12

---

Under review as a conference paper at ICLR 2021
A
SPEED DEGRADATION ANALYSIS OF NAT MODEL
Table 6: Time-consuming of decoding newstesf2014 by different batch sizes on Titian X GPU.
Model
BH=1
BH=8
BH=16
BH=32
AT
962s
151s
78s
43s
MP (/=10)
464s
125s
119s
129s
For the NAT model, we assume that the computation cost of each iteration is proportional to the size of decoder input tensor (BH BM, L, H), where BH is the batch size, BM is the beam size, L is the predicted target length, and H is the network dimension. In this way, the total cost of / iterations (generally, I < L) is Cnat I x O(BH BM H and simplify Cnat to I x O(BH x L). Likely, the computational cost of AT model is about Ca. L O(BH 1)
H). For convenience, we omit BM and
O(BHxL) Then, can denote the speedup ratio O are the keys to Thus, fewer iterations (small I) and faster parallel computation (large OBBRL IR-NAT.
Cat 4 O(BHx1) Ceat
O( especially in larger batches. As shown in Table O the direct evidence is that when decoding the test set of the WMT 14 De task, the time spent by AT decreases with the increase of batch size, while MP(/=10) cannot benefit from this. We test on Titian X GPU and report the average of runs. BM=5. Note that CPU has similar results. Specifically, we can see that when BH increases from I to 32, AT's latency reduces by 962/43 (about 22) times, while MP (I=10) only reduces by 464/129 (about 4) times. It means that ((B(x1I becomes lower as BH increases. Until O(BNx1) < J. NAT will start to be slower O(BHXL)
However, in practice, find it difficult increase O(BHx1)
O(BHLL) than AT.
L.
B AT TRANSFORMER IN SYNTHETIC EXPERIMENTS
Table 7: Performance of autoregressive models in the synthetic experiment.
AT Transformer
En-Ro
En-De
Vaswani et al. (2017)
27.3
Ghazvininejad et al. (2019)
34.28
27.74
Our implementation
34.25
27.45
In the synthetic experiment, trained all AT models with the standard Transformer-Base configu- ration: layer=6, dim=512, ffn=2048, head=8. The difference from Ghazvininejad et al. (2019) is that they trained the AT models for 300k steps, but we updated 50k/100k steps on En RRo and En- respectively. Although fewer updates, as shown in Table 7 our AT models have comparable perfor- mance with theirs.
C TRAINING ALGORITHM
Algorithm [ describes the process of training the HRT model. The HRT model is pre-initialized by a pre-trained AT model (Line 1). During training, the training batch B, randomly select a raw target sentence Y, or its distilled version Y' (Line 4-6). Then according to Eq. 5 into two parts: B and Ba=k, where / B = Ps: (Line 7-8). Next, we construct four kinds of training samples based corresponding batches: B Bat Bmd and 8mp, Finally, collect all training samples together and accumulate their gradients to update the model parameters, which results in the batch size being twice that of standard training.
divide B
13

---

Under review as a conference paper at ICLR 2021
Algorithm Training Algorithm for
Translation
Input: Training data D including distillation targets, pretrained AT model Mat. chunk size k. mixed distillation rate Praw
Output: Regressseeee Translation model Mart
1: Mart < Mat
finetune on pre-trained AT
2: for in 1, ................................ do
3:
X [1,,... ;n)), Y (y1,... yn Y (3/ Vn) < fetch a batch from D
4: for in 1.2..... do
5:
B. = (X; ) + sampling Y ~ (Y, ) with P(Y.) = Praw b mixed distillation
6:
end for
7:
Pk < get the chunk aware proportion by Eq.
> curriculum learning
8:
B B==1 B: [nxpa [nxpp
split batch
9:
Bat B f construct (Skip-AT, Skip-MP) training samples based on B Ba=k
10:
B B _ + construct (AT, MP)
training samples based on B
Optimize Mart using B Bat
c "
b joint training
12:
end for
Table 8: Compare hybrid-regressive translation (HRT) to autoregressive translation (AT), itera- tive refinement based non-autoregressive translation (IR-NAT), and semi-autoregrrssivve translation (SAT). Q(i) denotes the computation cost in autoregressive mode when producing the i-th token
(e.g., the prefix length is 1). @i( denotes the computation cost in non-autoregressive mode when producing tokens by one shot with a beam size of T=4 10, & generally
Method
Steps
Computing Cost
AT
L
Σ Q(i)
IR-NAT
I
IxQo=s(L)
SAT
L/k
L/k x (Q0== +
HRT
L/k+1
f/Ah Q(i k) (L)
D COMPUTATION COMPLEXITY
In Table we summarized the comparison with autoregressive translation (AT), iterative refinement based -uoreeroreeee translation (IR-NAT) and semi-autoregressive translation (SAT) (Wang et al. 2018a) (a)).)
AT. Although both HRT and AT contain a slow autoregressive generation process, HRT's length is k times shorter than AT. Considering that the computational complexity of self-attention is quadratic with its length, HRT can save more time in autoregressive decoding.
IR-NAT. Since Skip-AT provides high-quality target context, HRT does not need to use large beam size and multiple iterations like IR-NAT. The experimental results also show that our light NAT can make up for the increased cost in Skip-AT and can achieve stable acceleration regardless of the decoding batch size and rumning device.
SAT. SAT generates segments locally by non-autoregression, but it is still autoregressive between segments. We claim that the SAT reduces the decoding steps by k, but each token's calculation remains unchanged. In other words, in the time step i, there are - tokens used for self-attention. By contrast, only i/k tokens are involved our Skip-AT.
E APPLY THE OPTIMIZATION METHODS TO MASK-PREDICT
We conducted experiments on the WMT En-De task verify whether the optimization methods used in HRT training are complementary to MP, including fine-tuning from the pre-trained autoregressive
While the decoder self-attention module considers the previous tokens, we omit it here for the sake of clarity.
14

---

Under review as a conference paper at ICLR 2021
Table 9: Apply the optimization methods used in HRT training MP. BLEU scores are evaluated on the WMT 14 En->De test set.
Method
Steps
BLEU
MP (official)
300k
27.03
HRT (bat=5, bmp=1)
100k
27.98
MP FT + RPR MD
100k
27.02
MP + FT + RPR + MD
160k
27.32
MP FT + RPR MD
300k
27.63
model (FT), relative positional representation (RPR), and mixed distillation (MD). The joint training of Skip-AT and Skip-MP through curriculum learning is not involved, because it is incompatible with MP training. Table 9 shows that the optimization methods used in HRT training are complementary to MP training. With the help of FT+RPR+MD, the same BLEU score as the officially released model with 300k steps. What's more, when we train more steps, MP is improved by +0.61 BLEU points compared with the official model, but still falls behind our HRT model.
MP model with 100k steps
achieve almost
15


Reviews:
[{"review_id": "PtNkyC-ql4y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Non-autoregressive decoder (NAT) greatly improves translation efficiency, but often relies on iterative refinement (IR) to retain the translation performance. Unfortunately, this strategy makes the decoding efficiency of IR-based NAT much more sensitive to batch size and computing device: with larger batch size and CPU, IR-based NAT runs even slower than the standard auto-regressive decoder (AT). \n\nThis paper targets at alleviating this IR bottleneck by proposing hybrid-regressive translation (HRT), which combines AT and NAT in two stages. AT in the first stage aims at offering an initial coarse target hypothesis, implemented by predicting target words at every k position, to serve as a better target context for NAT such that NAT can produce high-quality translation with only one iteration in the second stage. The authors introduce several tricks to optimize HRT, including joint training, mixed distillation and curriculum learning etc. Experiments on two WMT translation tasks (four translation directions) demonstrate the effectiveness of HRT, which consistently accelerates decoding by >50% compared to the AT baseline and yields comparable or even better translation quality.\n\nOverall, this IR issue is under-studied in the literature, which deserves more attention. But I am not fully convinced by the current experiments, especially the comparisons.\n\nMy major concerns:\n\n= Firstly, only comparing with MP10 is not enough. HRT consists of two parts. The first part AT acts very similar to the semi-autoregressive model (Wang et al, EMNLP 2018), which should be treated as a baseline. The second part, NAT or MP used in experiments, involves different variants when different iterations are used. For example, Figure 2 in Mask-Predict paper (Ghazvininejad et al, EMNLP 2019) discovers the trade-off between speedup and translation quality. Could you please show the Pareto frontier to prove that HRT reaches a better trade-off compared to MP? Besides, one thing should also be noticed: the speedup yielded by HRT seems less charming compared to recent NAT models, such as Levenshetain Transformer 3x-4x (Gu et al, NeurIPS 2019), and JM-NAT (k = 10) 5.73x (Guo et al, ACL 2020) where both models achieve similar translation quality to the AR baseline.\n\n= If I understand correctly, the advantage of HRT in handling different batch sizes and computing devices mainly comes from its usage of one-pass NAT. In Figure 3, we observe that HRT1-1 produces clearly larger speedups as batch size increases from the CPU graphs. However, we observe a reversed trend with MP1 in Figure 1. Could you give readers some explanation? \n\n= Although this paper mainly focuses on improving NAT, efforts on accelerating decoding is not limited to NAT. Relevant studies, which should be at least discussed in related work, include model quantization [1] and simplified decoder [2, 3], to name a few, are all missed in the paper. In particular, some models like AAN [2] and Deep encoder+Shallow decoder[3] can already produce similar speedups (might be smaller) with comparable translation performance. These models do not rely on knowledge distillation and those complex optimization tricks used for HRT. Could you please explain the advantage of HRT over these models?\n\n= The claim “Chunk is superior to Rand, which indicates that the balanced distribution of deterministic tokens is necessary” (above Section 4) is strong. From Figure 2, what I see is that Chunk performs slightly better than Random. It’s hard to conclude that the balanced distribution is *necessary*.\n\nMy minor concerns:\n\n= How did you handle the positional encoding in AT of HRT? Did you consider the interval k?\n\n= The proposed optimization tricks are somehow orthogonal to HRT itself. What if you apply them to the baselines, like MP and semi-autoregressive model?\n\n= How many runs did you perform when reporting the speed numbers in Figure 3?\n\n= I cann’t find the number *34.08* (Table 3) in Table 2 for En->Ro translation. Is there something wrong?\n\n= The citation format should be adjusted accordingly. The authors always use the format of “author (year)”.\n\n= It would be better to see more results on the translation of distant language pairs, such as WMT English-Chinese. \n\n[1] Bhandare et al.  Efficient 8-bit quantization of transformer neural machine language translation model. 2019\n\n[2] Zhang et al. Accelerating neural transformer via an average attention network. ACL 2018\n\n[3] Kasai et al. Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation. 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OY8bdgfZyOV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a hybrid-regressive machine translation (HRT) approach—combining autoregressive (AT) and non-autoregressive (NAT) translation paradigms: it first uses an AT model to generate a “gappy” sketch (every other token in a sentence), and then applies a NAT model to fill in the gaps with a single pass. As a result the AT part latency is roughly reduced by half compared to a full AT baseline. The AT and NAT models share a majority part of the parameters and can be trained jointly with a carefully designed curriculum learning procedure. Experiments on several MT benchmarks show that the proposed approach achieves speedup over the full AT baseline with comparable translation quality.\n\nThe idea of combining AT and NAT is interesting, and the paper is very clearly written. The experiments and analysis are solid and well-designed. I vote for acceptance.\n\nPros:\n- An interesting idea combining AT and NAT which can inspire future research.\n- Experiments and analysis support most of the claims.\n- The proposed curriculum training and mixed distillation can be useful in future research.\n\nCons:\n- It is a bit sad that the translation quality drops a lot when k > 2. This limits and maximum speedup that can be achieved. \n- Some of the claims on the HRT’s performance compared to the AT baseline seems a bit misleading (see details below).\n\nDetails:\n- From Table 3 it seems that AT models also benefit from mixed distillation. Based on the numbers here I’m assuming the AT models in Table 2 are trained w/o mixed distillation. Early on the paper claims that HRT can outperform AT in terms of translation quality, but this doesn’t seem to be the case if one compares HRT against AT both trained with mixed distillation. Can the authors clarify? If this is true, please tone down the claims on this point in the abstract, intro, and Section 5.\n- Can the authors comment on the training cost of HRT in comparison to AT in terms of, e.g., number of epochs, wall-clock time?\n- Does the AT baselines in Table 2 use the same relative positional encodings as HRT? If not, can the authors comment on how it may affect their performance? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6h4lSPNYlri", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper's main topic is the actual usability of the current status of non-auto-regressive translation (NAT) models.\n\nAlthough previous papers have reported that the NAT models can achieve the same performance level with auto-regressive translation models while the decoding speed is much faster, like two to five times, this paper points out that it deeply relies on the batch size and computation environment.\nThis is a proper investigation for the community since some researchers might believe that NAT is always faster than standard auto-regressive models and became an excellent alternative to them.\n\nThe ideas of inducing skip-AT and skip-MT are really unique and somewhat innovative (since, I guess, no other researchers hardly think to employ such skip-decoding architecture).\nBasically, this paper has several new findings that should be shared in the community for developing better technologies.\n\n\n\n\nThe following are the questions/concerns of this paper.\n\n1,\n\n\"IR-NAT heavily relies on small batch size and GPU, which is rarely mentioned in prior studies.\"\nI think this is an excellent investigation. However, this paper does not tell readers why this observation happens.\nPlease explain why the current NAT models are not suitable to work on CPUs and large batches.\n\n\n2,\n\nThe intention of the statement, \"which indicates that the balanced distribution of deterministic tokens is necessary\" is unclear. Please elaborate on what the authors try to tell by this statement.\n\n\n3,\n\nThe proposed method consists of many new components.\nThe authors provided the results of an ablation study in Table 5.\nThis is a really nice analysis.\nHowever, the performance differences in −FT, −RPR, and −MixDistill are somewhat marginal. \nActually, we can easily observe such 0.3-0.4 BLEU score difference by just changing random seeds for Transformer models.\nAre there any statistically significant differences among them? Or any reasonable evidence that supports the difference?\n\n\n4,\n\nThis is just a comment, and appreciate having the author's words.\n\nThere is an opinion that fully tuned implementation for standard auto-regressive models outperforms both decoding speed and accuracy. See the following presentation slide on WNGT-2020 (such as P33):\nhttps://kheafield.com/papers/edinburgh/wngt20overview_slides.pdf\n\nWhat would happen for the proposed method if we compared them on such a highly-tuned implementation?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method to combine non-autoregressive (NAT) and autoregressive (AT) NMT. Compared with the original iterative refinement for non-autoregressive NMT, their method first generates a translation candidate using AT and then fill in the gap using NAT.

All of the reviewers think the idea is interesting and this research topic is not well-studied. However, the empirical part did not convince all the reviewers. The revised version and response is good; however, it still does not solve some major concerns of reviewers.

Decision: Reject