Hybrid-Regressive Paradigm for Accurate and Speed-Robust Neural Machine Translation
Year: 2023

Abstract:
This study provides empirical evidence that non-autoregressive translation (NAT) is less robust in decoding batch size and hardware settings than autoregressive translation (AT). To address this issue, we demonstrate that incorporating a small number of AT predictions can significantly reduce the performance gap between AT and NAT through synthetic experiments. In line with this, we propose hybridregressive translation (HRT), a two-stage translation prototype that combines the strengths of AT and NAT. Specifically, HRT initially generates discontinuous sequences using autoregression (e.g., making predictions for every k tokens, k > 1), and then fills in all previously skipped tokens simultaneously in a nonautoregressive manner. Experimental results on five translation tasks show that HRT achieves comparable translation quality to AT while providing at least 1.5x faster inference, irrespective of batch size and device. Moreover, HRT successfully retains the desirable characteristics of AT in the deep-encoder-shallow-decoder architecture, enabling further speed improvements without sacrificing BLEU scores.1

heading: 1 Introduction
text: Autoregressive translation (AT) such as Transformer has been the de facto standard for Neural Machine Translation (NMT) (Vaswani et al., 2017). However, AT predicts only one target word at a time, resulting in slow inference speed. To overcome this limitation, non-autoregressive translation (NAT) attempts to generate the entire target sequence in one step by assuming conditional independence among target tokens (Gu et al., 2018). While NAT offers efficiency, it often suffers from significant degradation in translation quality. Achieving a better trade-off between inference speed and translation quality remains an active area
∗Corresponding author. 1https://github.com/wangqiangneu/hrt
of research for NAT (Wang et al., 2018a; Ran et al., 2020; Qian et al., 2021; Huang et al., 2022b,a).
One of the most successful approaches to this issue is the iterative refinement mechanism (IRNAT) proposed by Lee et al. (2018), which has been widely adopted by several leading systems (Ghazvininejad et al., 2019; Kasai et al., 2020a; Guo et al., 2020; Saharia et al., 2020; Geng et al., 2021; Huang et al., 2022b). Specifically, IR-NAT, also known as multi-shot NAT, takes the translation hypothesis from the previous iteration as a reference to refine the new translation until it reaches the predefined iteration count I or no translation changes. Although a larger I can improve translation accuracy, it may also lead to a speedup degradation (Kasai et al., 2020b).
In this work, we build upon the findings of Kasai et al. (2020b) and examine the robustness of IR-NAT compared to AT. Our comprehensive examinations confirm that the inference speed of IRNAT is consistently less robust than that of AT when involving various decoding batch sizes and computing hardware. For example, when using a GPU, the ten-iteration non-autoregressive model has 1.7/1.2/0.7/0.4 times the inference speed of the AT model for decoding batch sizes of 1/8/16/32, respectively. However, when switching to CPU, the relative speed ratio drops to 0.8/0.4/0.3/0.3 times. Previous studies have highlighted the complementary nature of AT and NAT in terms of both translation quality (AT being superior) and inference speed (NAT being superior) (Wang et al., 2018a; Ran et al., 2020). Our findings, however, suggest that there is also complementary robustness in inference speed (AT being superior).
Taking a further step, we investigate how much target context (i.e., the number of target tokens) is sufficient for one-shot NAT to rival multi-shot NAT through synthetic experiments. Our findings suggest that given a well-trained CMLM model, even if 70% of AT translations are masked, the
5931
remaining target context can help the CMLM1 with greedy search compete with the standard CMLM10 with beam search (see Figure 2). This could enable us to build the desired target context more cheaply, replacing expensive multiple iterations. To our best knowledge, this is the first study of the masking rate issue in the inference phase of NAT.
Based on the observations from these experiments, we have proposed a novel two-stage translation prototype called hybrid-regressive translation (HRT). This method combines the advantages of autoregressive translation (AT) and nonautoregressive translation (NAT) by first using an autoregressive decoder to generate a discontinuous target sequence with an interval of k (k > 1), and then filling the remaining slots in a lightweight non-autoregressive manner. We have also created a multi-task learning framework, enhanced by curriculum learning, for effective and efficient training without adding any model parameters. Results on WMT En↔Ro, En↔De, and NIST Zh→En show that HRT outperforms prior work combining AT and NAT, and is competitive with state-of-theart IR-NAT models. Specifically, HRT achieved a BLEU score of 28.27 on the WMT En→De task, and is 1.5x faster than AT regardless of batch size and device. Additionally, HRT equipped with a deep-encoder-shallow-decoder architecture achieved up to 4x/3x acceleration on GPU/CPU, respectively, without sacrificing BLEU.

heading: 2 Background
text: Given a source sentence x = {x1, x2, . . . , xM} and a target sentence y = {y1, y2, . . . , yN}, there are several ways to model P (y|x):
Autoregressive Translation (AT) AT is the predominant technique in NMT, decomposing P (y|x) using the chain rule: P (y|x) =∏N
t=1 P (yt|x, y<t), where y<t denotes the prefix translation generated before time step t. Nevertheless, autoregressive models must wait for yt−1 to be generated before predicting yt, thus hindering parallelism over the target sequence.
Non-Autoregressive Translation (NAT) NAT has been proposed to generate target tokens simultaneously (Gu et al., 2018). This approach replaces the traditional autoregressive formulation of y<t with a target-independent input z, resulting in the following formulation: P (y|x) = P (N |x)×∏Nt=1 P (yt|x, z). Various approaches have been proposed for modeling z, such as using source embedding (Gu et al., 2018; Guo et al., 2019), reordering the source sentence (Ran et al., 2019), or using a latent variable (Ma et al., 2019; Shu et al., 2019).
Iterative Refinement based Non-Autoregressive Translation (IR-NAT) IR-NAT extends the traditional one-shot NAT by introducing an iterative refinement mechanism (Lee et al., 2018). We choose CMLM (Ghazvininejad et al., 2019) as the representative of IR-NAT due to its excellent performance and simplification. During training, CMLM randomly masks a fraction of tokens on y as the alternative to z, and is trained as a conditional masked language model (Devlin et al., 2019). Denoting ym/yr as the masked/residual tokens of y, we have: P (y|x) = ∏|y
m| t=1 P (y m t |x,yr). At infer-
ence, CMLM deterministically masks tokens from the hypothesis in the previous iteration ŷ(i−1) according to prediction confidences. This process is iterated until ŷ(i−1)=ŷ(i) or i reaches the maximum iteration count.

heading: 3 Acceleration Robustness Problem
text: In this section, we comprehensively analyze the inference acceleration robustness problem in IRNAT. Without loss of generality, we take CMLM as the agency of IR-NAT.2
Problem Description The inference overhead of the autoregressive translation model mainly con-
2From the perspective of inference speed, we note that most one-shot NAT models are closed to CMLM1. Especially, existing one-shot NAT models with CTC loss, such as GLAT and Fully-NAT, are theoretically slower than CMLM1 because they require a longer target sequence for inference.
centrates on the decoder side(Hu et al., 2020). Suppose that the decoder’s computational cost is proportional to the size of its input tensor (B,N,H), where B is the batch size, N is the target sequence length, and H is the network dimension. We omit H for convenience due to its invariance in NAT and AT. Thus, the total cost of AT model is about Cat ∝ N × O(B × 1) 3. Likely, the cost of Iiteration NAT is Cnat ∝ I × O(B × N). Given a fixed test set, We can use TD(·) to represent the translation time on computing device D. This allows us to calculate the relative speedup ratio α between I-iteration NAT and AT as:
α = TD(Cat) TD(Cnat) ∝ N I × E(B,D), (1)
where E(B,D)= TD(O(B×1))TD(O(B×N)) ≤ 1, denotes the parallel computation efficiency over sequence under batch size B and device D. When fixing N and I , α is completely determined by E(B,D). We note that most previous NAT studies only report the inference speed with D=GPU and B=1, without considering cases where B or D change.
Setup We systematically investigate the inference speed of CMLM 4 and AT under varying environments, including batch size B ∈ {1, 8, 16, 32}, device D ∈ {GPU,CPU}5, and the number of iterations I ∈ {1, 4, 10}, using a beam size of 5. We test inference speed on the widely used WMT En→De newstest2014 test set and report the average results over five runs (see Appendix A for details).
Results We plot the curve of relative speedup ratio (α) in Figure 1 and observe that:
i. α decreases as decoding batch size increases regardless of the number of iterations, as noted by Kasai et al. (2020b).
ii. α on CPU generally performs worse than GPU, except when using one iteration.
iii. The benefit of non-autoregressive decoding is more prone to disappear for larger numbers of iterations (I).
3Though the decoder self-attention module considers the previous i tokens, we omit it here for the sake of clarity.
4We use the officially released CMLM models from https: //github.com/facebookresearch/Mask-Predict
5We use 2080Ti GPUs and Intel Xeon(R) E5-2683 v4 CPU, unless otherwise stated.
For instance, when decoding a single sentence on the GPU, the inference speed of the ten-iteration non-autoregressive model is 170% that of the autoregressive model. However, when switching to batches of 32 on CPU, the IR-NAT model only reaches 30% of the AT model’s inference speed. These results demonstrate that AT and NAT possess different strengths, and combining the advantages of both models could be an effective way to achieve robust acceleration.

heading: 4 Synthetic Experiments
text: According to Equation 1, reducing the iteration count I helps to increase α. Recalling the refinement process of IR-NAT, we hypothesize that the essence of multiple iterations is to provide the decoder with a good enough target context (deterministic target tokens). This raises the question of how many target tokens need to be provided to make one-shot NAT competitive with IR-NAT? To answer it, we conduct synthetic experiments on WMT En→Ro and En→De to control the size of the target context by masking the partial translations generated by a pre-trained AT model. We then use a pre-trained CMLM model to predict these masks and observe the BLEU score curves under different masking rates.
Models We use the official CMLM models. Since the authors did not release the AT baselines, we used the same data to retrain AT models with the standard Transformer-Base configuration (Vaswani et al., 2017) and obtain comparable performance with the official ones (see Appendix B for details).
Decoding AT models decode with beam sizes of 5 on both tasks. Then we replace a certain percentage of AT tokens with [MASK] and feed them to CMLM. The used CMLM model only iterates once with beam size 1. We substitute all [MASK]s with CMLM’s predictions to obtain the final translation. We report case-sensitive tokenized BLEU scores by multi-bleu.perl.
Mask Strategies We tested four strategies to mask AT results: HEAD, TAIL, RANDOM, and CHUNK. Given the masking rate pmask and the translation length N , the number of masked tokens is Nmask=max(1, ⌊N×pmask⌋). Then HEAD/TAIL always masks the first/last Nmask tokens, while RANDOM masks the translation randomly. CHUNK is slightly different from the above strategies. It first divides the target sentence into C chunks, where
C = Ceil(N/k) and k is the chunk size. Then in each chunk, we retain the first token but mask other k−1 tokens. Thus, the actual masking rate in CHUNK is 1− 1/k instead of pmask. We ran RANDOM three times with different seeds to exclude randomness and report the average results.
Results The experimental results in Figure 2 demonstrate that CHUNK is moderately and consistently superior to RANDOM, and both strategies significantly outperform HEAD and TAIL. We attribute this success to the use of (1) bidirectional context (Devlin et al., 2019) (vs. HEAD and TAIL), and (2) the uniform distribution of deterministic tokens (vs. RANDOM)6. Furthermore, when using the CHUNK strategy, we find that exposing 30% AT tokens as the input of the decoder is enough to make our CMLM1(beam=1) competitive with the official CMLM10(beam=5), which emphasizes the importance of a good partial target context.

heading: 5 Hybrid-Regressive Translation
text: We propose a novel two-stage translation paradigm, Hybrid-Regressive Translation (HRT), which imitates the CHUNK process. In HRT, a discontinuous sequence with a chunk size of k is autoregressively generated in stage I, followed by nonautoregressive filling of the skipped tokens in stage II.

heading: 5.1 Architecture
text: Overview HRT consists of three components: encoder, Skip-AT decoder (for stage I), and SkipCMLM decoder (for stage II). All components adopt the Transformer architecture (Vaswani et al., 2017). The two decoders have the same network structure, and we share them to make the parameter size of HRT the same as the vanilla Transformer. The only difference between the two decoders lies in the masking pattern in self-attention: The Skip-AT decoder masks future tokens to guarantee strict left-to-right generation like the autoregressive Transformer. In contrast, the Skip-CMLM decoder eliminates it to leverage the bi-directional context like CMLM (Ghazvininejad et al., 2019).
No Target Length Predictor Thanks to Skip-AT, we can obtain the translation length as a by-product: Nnat=k ×Nat, where Nat is the sequence length produced by Skip-AT. Our approach has two major advantages over most NAT models, which jointly train both the translation length predictor and the translation model. Firstly, there is no need to carefully adjust the weighting coefficient between the sentence-level length prediction loss and the wordlevel target token prediction loss. Secondly, the length predicted by Skip-AT is more accurate due to its access to the already-generated sequence information.
6CHUNK ensures that each masked token (except the last k-1 ones in the sequence) meets two deterministic tokens within the window size of k. However, RANDOM may degrade into HEAD/TAIL in extreme cases.

heading: 5.2 Training
text: Next, we elaborate on how to train the HRT model efficiently and effectively. Please refer to Appendix C for the entire training algorithm.
Multi-Task Framework We learn HRT by jointly training four tasks: two primary tasks (TASK-SKIP-AT, TASK-SKIP-CMLM) and two auxiliary tasks (TASK-AT, TASK-CMLM). All tasks use cross-entropy as the training objective. Figure 3 illustrates the differences in training samples among these tasks. Notably, TASK-SKIP-AT shrinks the sequence length from N to N/k, while preserving the token positions from the original sequence. For example, in Figure 3 (c), the position of TASK-SKIP-AT input ([B2], y2, y4) is (0, 2, 4). Auxiliary tasks are necessary to leverage all tokens in the sequence, as the two primary tasks are limited by the fixed k. For example, in Figure 3 (c) and (d), y1 and y3 cannot be learned as the decoder input of either TASK-SKIP-AT or TASK-SKIP-CMLM.
Curriculum Learning To ensure the model is not overly biased towards auxiliary tasks, we propose gradually transferring the training tasks from auxiliary tasks to primary tasks through curriculum learning (Bengio et al., 2009). We start with a batch of original sentence pairs B, and let the proportion of primary tasks in B be pk=0. We construct the training samples of TASK-AT and TASK-CMLM for all pairs, then gradually increase pk to introduce more learning signals for TASK-SKIP-AT and TASK-SKIP-CMLM until pk=1. We schedule pk by pk = (t/T )
λ, where t and T are the current and total training steps, and λ is a hyperparameter set to 1 for linear increase.

heading: 5.3 Inference
text: As illustrated in Figure 4, HRT adopts a two-stage generation strategy. In the first stage, the Skip-AT decoder autoregressively generates a discontinuous target sequence ŷat = (z1, z2, . . . , zm) with chunk size k, starting from [BOSk] and ending with [EOS]. Then, the input of Skip-CMLM decoder ynat is constructed by appending k − 1 [MASK]s before every zi. The final translation is generated by replacing all [MASK]s with the predicted tokens after one iteration of the Skip-CMLM decoder. If multiple [EOS]s exist, we truncate to the first [EOS]. The beam sizes bat and bnat can be different from each other, as long as bat ≥ bnat. In our implementation, we use standard beam search in Skip-AT (bat >1) and greedy search in Skip-CMLM (bnat=1). Table 3 provides more details on the beam size setting in HRT. The translation hypothesis with the highest score S(ŷ) is chosen by summing the Skip-AT
score and the Skip-CMLM score:
m∑
i=1 logP (zi|x, z<i) ︸ ︷︷ ︸
Skip-AT score
+ m−1∑
i=0
k−1∑
j=1 logP (ŷi×k+j |x,ynat) ︸ ︷︷ ︸
Skip-CMLM score
(2)
where zi=ŷi×k.

heading: 5.4 Discussion
text: The basic idea of HRT is to apply Autoregressive Transformation (AT) and Non-Autoregressive Transformation (NAT) in sequence. This concept has been investigated before (Kaiser et al., 2018), Ran et al. (2019), and Akoury et al. (2019). The main differences between these methods lie in the content of the AT output, such as latent variable (Kaiser et al., 2018), reordered source tokens (Ran et al., 2019), and syntactic labels (Akoury et al., 2019). In contrast, our approach uses the deterministic target token following Ghazvininejad et al. (2019).
HRT is related to chunk-wise decoding, another line incorporating AT and NAT. Table 1 shows the differences between HRT and prior studies, including SAT (Wang et al., 2018a), RecoverSAT (Ran et al., 2020), and LAT (Kong et al., 2020). SAT and LAT follow the generation order of from-leftto-right, behaving similarly to HEAD as described in Section 4. In contrast, RecoverSAT and HRT generate discontinuous target contexts, which have been shown to perform better than HEAD according to our synthetic experiments. However, RecoverSAT cannot accurately generate the discontinuous context "a,c,e" via non-autoregression, resulting in error propagation for generating "b,d,f". However, HRT produces "a,c,e" through accurate autoregression. Additionally, although HRT requires more decoding steps, its non-autoregressive process is inexpensive due to greedy searches. In contrast, other methods require larger beams to explore translations of different lengths.

heading: 6 Experimental Results
text: Setup We conduct experiments on five tasks, including WMT’16 English↔Romanian (En↔Ro, 610k), WMT’14 English↔German (En↔De, 4.5M) and long-distance language pair NIST Chinese-English (Zh→En, 1.8M). For fair comparisons, we replicate the same data processing as Ghazvininejad et al. (2019) in four WMT tasks and follow the setup of Wang et al. (2018b) for Zh→En. Like previous work, we train
Method Generation SAT a, b→ c, d→ e, f
RecoverSAT a, c, e→ b, d, f LAT {a→ b→ c, d→ e→ f}
HRT through sequence-level knowledge distillation (Kim and Rush, 2016). Specifically, we use the standard Transformer-Base as teacher models for En↔Ro and Zh→En, while we use the deep PreNorm Transformer-Base with a 20-layer encoder for harder En↔De. We run all experiments on four 2080Ti GPUs. Unless noted otherwise, we use the chunk size k=2. We fine-tune HRT models on pre-trained AT models and take 100k/300k/100k training steps for En↔Ro/En↔De/Zh→En, respectively. Other training hyperparameters are the same as Vaswani et al. (2017) or Wang et al. (2019) (deep-encoder). We report both case-sensitive tokenized BLEU scores and SacreBLEU 7. We also report COMET as suggested by Helcl et al. (2022).
Beam Size on HRT We first verify the influence of two beam sizes of HRT (bat and bnat) on the BLEU score and relative speedup ratio by testing three different setups. The results are listed in Table 3. Consistent with our observations in synthetic experiments, using bnat=1 only slightly reduces BLEU but significantly improves decoding efficiency. Considering the trade-off between translation quality and speed, and for a fair comparison with other baselines (most prior related work uses beam size of 5), we use bat=5 and bnat=1 unless
7Signature: BLEU+case.mixed+lang.sourcetarget+numrefs.1+smooth.exp+tok.13a+version.1.5.1
otherwise stated.
Main Results We compare the performance of HRT with existing systems in different translation paradigms on four WMT tasks, as shown in Table 2. HRT with distillation data consistently outperforms that of raw data and most existing NAT, IR-NAT, and Semi-NAT models, obtaining a BLEU score of 28.27 on the widely used En→De task. Compared to the re-implemented typical semi-autoregressive model (SAT) and one-shot non-autoregressive model (GLAT-CTC), HRT obtains an improvement of approximately 1.7 BLEU points, with a more significant margin in COMET score. Moreover, HRT20−6 can improve by 0.7 BLEU and 0.02 COMET when using a deeper encoder. Interestingly, the evaluation results of BLEU and COMET are inconsistent, as observed
by Helcl et al. (2022). For instance, HRT20−6 has higher BLEU score than AT20−6 on En→De, but its COMET score is still lower. Furthermore, the experimental results on the Zh→En task, as reported in Table 4, demonstrate that the effectiveness of HRT is agnostic to language pairs, as it is close or superior to the original AT and CMLM model. We attribute this to two reasons: (1) HRT is fine-tuned on a well-trained AT model; (2) Multi-task learning on autoregressive and non-autoregressive tasks has better regularization than training alone.

heading: 7 Analysis
text: Impact of Chunk Size We tested chunk size k on the En→De task, as shown in Table 5. We observed that larger values of k had a more significant speedup on the GPU, as fewer autoregressive steps were required. However, as k increased, the performance of HRT dropped sharply; for example, k=4 was about 1.24 BLEU points lower than k=2 on the test set. This suggests that the training difficulty of Skip-AT increases as k becomes larger. Further investigation into more sophisticated training algorithms to address this is left for our future work.
Which decoding stage is more important? To understand the importance of the two decoding stages of HRT, we exchange the intermediate results of two HRT models (A and B). Specifically, we use the Skip-AT decoder of A to generate its discontinuous target sequence, which is then forced decoded by B’s Skip-AT decoder to obtain corresponding encoding representations and autoregressive model scores. Finally, B’s Skip-CMLM decoder generates the complete translation result based on these. We can reverse the order of A and B as well. We use two models (HRT and HRT20−6) with a large performance gap as A and B, respectively. As shown in Table 6, we find that using the result of stage I of the strong model brings a greater improvement (+0.61 BLEU) than that of stage II (+0.15 BLEU). This result supports our hypothesis that a good partial target context is essential.
Deep-encoder-shallow-decoder Architecture Kasai et al. (2020b) showed AT with deepencoder-shallow-decoder architecture can speed up translation without sacrificing accuracy, while CMLM fails. To validate whether HRT can inherit this, we compared HRT and AT with a 12-layer encoder and 1-layer decoder (HRT12−1 and AT12−1), using the same distillation data. As Table 7 shows, both AT12−1 and HRT12−1 benefit from the layer allocation, achieving comparable BLEU scores and double the decoding speed of the vanilla models. Specifically, HRT12−1 achieved an average acceleration of 4.2x/3.1x over the AT baselines. This suggests HRT12−1’s success was due to Skip-AT rather than Skip-CMLM. However, its COMET scores are lower than 6-6 architecture.
Further research into decoder depth and COMET correlation will be conducted.
Ablation Study In Table 8, we conduct an ablation study on the En→De task to investigate the contribution of fine-tuning from pre-trained AT (FT), training steps (TS), and curriculum learning (CL). We test two settings about CL: Fixing pk=1 is equivalent to removing auxiliary tasks; Fixing pk=0.5 assigns the same probability to the primary and auxiliary tasks. The results show that all components contribute to the performance, but CL and TS are the most critical, with a reduction of 0.74 and 0.45 BLEU points, respectively. Excluding all components from the vanilla HRT (-ALL) leads to a total reduction of 1.68 BLEU points.
Case study Table 9 presents a translation example from En→De validation set. Comparing CMLM5 and HRT, both having the same masking rate (50%), two main distinctions can be observed: (1) The distribution of masked tokens in CMLM is more discontinuous than in HRT (as indicated by the blue marks); (2) The decoder input of HRT contains more accurate target tokens than CMLM, due to the Skip-AT decoder (as indicated by the wavy marks). These differences make our model more effective in producing high-quality translations than CMLM, and suggest that our model can generate appropriate discontinuous sequences.

heading: 8 Conclusion
text: We noted that IR-NAT has robustness issues with inference acceleration. Inspired by our findings in synthetic experiments, we proposed HRT to take advantage of the strengths of both AT and NAT. Our experiments demonstrated that our approach surpasses existing semi-autoregressive and IR-NAT methods, providing competitive performance and consistent speedup, making it a viable alternative to autoregressive translation.

heading: 9 Limitations
text: The main limitation of HRT is that its upper bound on the inference speedup is lower than that of single-iteration NAT under the same network architecture. As demonstrated in Appendix A, the average speedup of single-iteration NAT (i.g., CMLM1) is 4.7x/3.2x on GPU/CPU, respectively, while that of HRT is 1.7x/1.6x. To achieve higher acceleration, HRT needs to employ the deep-encodershallow-decoder architecture. Increasing the chunk size is a simple way to reduce the autoregressive cost, yet it results in severe BLEU degradation (see Table 5). Further research should be conducted to maintain high translation performance with fewer autoregressive prompts.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their helpful comments. We also thank Shuqin Pan for the writing suggestions.

heading: A Detailed inference speed
text: In Table 10, we list the exact decoding time and relative speedup ratio of different models under varying environments on the En→De test set. When changing the batch size from 1 to 32, the decoding time of AT reduces 20.4x/4.6x on GPU/CPU, respectively, while that of CMLM10 only reduces 3.7/0.8x. In contrast, HRT inherits the good character of AT and achieves an 18.7x/3.8x speedup. On the other hand, HRT has more robust acceleration than multi-shot NAT, such as CMLM4, CMLM10. When using the deep-encoder-shallow-decoder architecture, HRT12−1 performance approaches the one-shot NAT (CMLM1) on both GPU and CPU. Besides, the overall results of HRT-20L are similar to those of HRT because the translation time is mainly consumed in the decoder. We also report the change of inference speed along with chunk size k.

heading: B AT Transformers in synthetic experiments
text: We trained all AT models in the synthetic experiment with the standard Transformer-Base configuration: layer=6, dim=512, ffn=2048, head=8. The difference from Ghazvininejad et al. (2019) is that they trained the AT models for 300k steps, but we updated 50k/100k steps on En→Ro and En→De, respectively. Although fewer updates, as shown in Table 11, our AT models have comparable performance with theirs.

heading: C Training algorithm
text: Algorithm 1 describes the training process of HRT. The HRT model is pre-initialized by a pretrained AT model (Line 1). Then according to the schedule strategy pk = ( t T )λ , we can divide the training batch B into two parts: Bp for primary tasks and Ba for auxiliary tasks, where |Bp|/|B| = pk (Line 4-5). Next, we construct four kinds of training samples based on corresponding batches: Batp (TASK-SKIP-AT), B at a (TASK-AT), Bnatp (TASK-SKIP-CMLM) and B nat a (TASK-CMLM). Finally, we collect all training samples together and accumulate their gradients to update the model parameters, which results in the batch size being twice that of standard training.
Algorithm 1 Training Algorithm for Hybrid-Regressive Translation Input: Training data D, pretrained AT model Mat, chunk size k, schedule coefficient λ Output: Hybrid-Regressive Translation model Mhrt
1: Mhrt ← Mat ▷ fine-tune on pre-trained AT 2: for t in 1, 2, . . . , T do 3: B = ⟨xi,yi⟩|ni=1 ▷ fetch a batch B from D 4: pk ← ( tT )λ ▷ curriculum learning 5: Bp,Ba ← B [ : ⌊n× pk⌋ ] ,B [ ⌊n× pk⌋ : ] ▷ split batch for different tasks 6: Batp ,B nat p ← construct training samples of primary tasks based on Bp 7: Bata ,B nat a ← construct training samples of auxiliary tasks based on Ba 8: Optimize Mhrt using Batp ∪Bata ∪Bnatp ∪Bnata ▷ joint training 9: end for
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Footnote 5 in section 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 6
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 6
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Further investigation into more sophisticated training algorithms to address this is left for our future work. Which decoding stage is more important? To understand the importance of the two decoding stages of HRT, we exchange the intermediate results of two HRT models (A and B). Specifically, we use the Skip-AT decoder of A to generate its discontinuous target sequence, which is then forced decoded by B’s Skip-AT decoder to obtain corresponding encoding representations and autoregressive model scores. Finally, B’s Skip-CMLM decoder generates the complete translation result based on these. We can reverse the order of A and B as well. We use two models (HRT and HRT20−6) with a large performance gap as A and B, respectively. As shown in Table 6, we find that using the result of stage I of the strong model brings a greater improvement (+0. 61 BLEU) than that of stage II (+0. 15 BLEU). This result supports our hypothesis that a good partial target context is essential. Deep-encoder-shallow-decoder Architecture Kasai et al. (2020b) showed AT with deepencoder-shallow-decoder architecture can speed up translation without sacrificing accuracy, while CMLM fails. To validate whether HRT can inherit this, we compared HRT and AT with a 12-layer encoder and 1-layer decoder (HRT12−1 and AT12−1), using the same distillation data. As Table 7 shows, both AT12−1 and HRT12−1 benefit from the layer allocation, achieving comparable BLEU scores and double the decoding speed of the vanilla models. Specifically, HRT12−1 achieved an average acceleration of 4. 2x/3. 1x over the AT baselines. This suggests HRT12−1’s success was due to Skip-AT rather than Skip-CMLM. However, its COMET scores are lower than 6-6 architecture. Further research into decoder depth and COMET correlation will be conducted. Ablation Study In Table 8, we conduct an ablation study on the En→De task to investigate the contribution of fine-tuning from pre-trained AT (FT), training steps (TS), and curriculum learning (CL). We test two settings about CL: Fixing pk=1 is equivalent to removing auxiliary tasks; Fixing pk=0. 5 assigns the same probability to the primary and auxiliary tasks. The results show that all components contribute to the performance, but CL and TS are the most critical, with a reduction of 0. 74 and 0. 45 BLEU points, respectively. Excluding all components from the vanilla HRT (-ALL) leads to a total reduction of 1. 68 BLEU points. Case study Table 9 presents a translation example from En→De validation set. Comparing CMLM5 and HRT, both having the same masking rate (50%), two main distinctions can be observed: (1) The distribution of masked tokens in CMLM is more discontinuous than in HRT (as indicated by the blue marks); (2) The decoder input of HRT contains more accurate target tokens than CMLM, due to the Skip-AT decoder (as indicated by the wavy marks). These differences make our model more effective in producing high-quality translations than CMLM, and suggest that our model can generate appropriate discontinuous sequences