SlowGuess/ACL_2020_OCR huggingface dataset:
Qiu Ran\*, Yankai Lin\*, Peng Li\*, Jie Zhou
Pattern Recognition Center, WeChat AI, Tencent Inc., China
{soulcaptran, yankailin, patrickpli, withtomzhou}@tencent.com
Non-autoregressive neural machine translation (NAT) predicts the entire target sequence simultaneously and significantly accelerates inference process. However, NAT discards the dependency information in a sentence, and thus inevitably suffers from the multi-modality problem: the target tokens may be provided by different possible translations, often causing token repetitions or missing. To alleviate this problem, we propose a novel semi-autoregressive model RecoverSAT in this work, which generates a translation as a sequence of segments. The segments are generated simultaneously while each segment is predicted token-by-token. By dynamically determining segment length and deleting repetitive segments, RecoverSAT is capable of recovering from repetitive and missing token errors. Experimental results on three widely-used benchmark datasets show that our proposed model achieves more than  $4\times$  speedup while maintaining comparable performance compared with the corresponding autoregressive model.
Although neural machine translation (NMT) has achieved state-of-the-art performance in recent years (Cho et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017), most NMT models still suffer from the slow decoding speed problem due to their autoregressive property: the generation of a target token depends on all the previously generated target tokens, making the decoding process intrinsically nonparallelizable.
Recently, non-autoregressive neural machine translation (NAT) models (Gu et al., 2018; Li et al., 2019; Wang et al., 2019; Guo et al., 2019a; Wei et al., 2019) have been investigated to mitigate the
Src.	es gibt heute viele Farmer mit diesen Ansatz
Feasible	there are lots of farmers doing this today
Trans.	there are a lot of farmers doing this today
Trans. 1	there are lots of farmers doing this today
Trans. 2	there are a lot farmers doing this today
Table 1: A multi-modality problem example: NAT models generate each target token independently such that they may correspond to different feasible translations, which usually manifests as repetitive (Trans. 1) or missing (Trans. 2) tokens.
slow decoding speed problem by generating all target tokens independently in parallel, speeding up the decoding process significantly. Unfortunately, these models suffer from the multi-modality problem (Gu et al., 2018), resulting in inferior translation quality compared with autoregressive NMT. To be specific, a source sentence may have multiple feasible translations, and each target token may be generated with respect to different feasible translations since NAT models discard the dependency among target tokens. This generally manifests as repetitive or missing tokens in the translations. Table 1 shows an example. The German phrase "viele Farmer" can be translated as either "lots of farmers" or "a lot of farmers". In the first translation (Trans. 1), "lots of" are translated w.r.t. "lots of farmers" while "of farmers" are translated w.r.t. "a lot of farmers" such that two "of" are generated. Similarly, "of" is missing in the second translation (Trans. 2). Intuitively, the multi-modality problem has a significant negative effect on the translation quality of NAT.
Intensive efforts have been devoted to alleviate the above problem, which can be roughly divided into two lines. The first line of work leverages the iterative decoding framework to break the independence assumption, which first generates an initial translation and then refines the translation
${}^{ \dagger  }$  indicates corresponding author
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3059-3069 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: An overview of our RecoverSAT model. RecoverSAT generates a translation as a sequence of segments. The segments are generated simultaneously while each segment is generated token-by-token conditioned on both the source tokens and the translation history of all segments (e.g., the token "are" in the first segment is predicted based on all the tokens colored green). Repetitive segments (e.g., the third segment "lots of") are detected and deleted automatically.
iteratively by taking both the source sentence and the translation of last iteration as input (Lee et al., 2018; Ghazvininejad et al., 2019). Nevertheless, it requires to refine the translations for multiple times in order to achieve better translation quality, which hurts decoding speed significantly. The other line of work tries to improve the vanilla NAT model to better capture target-side dependency by leveraging extra autoregressive layers in the decoder (Shao et al., 2019a; Wang et al., 2018), introducing latent variables and/or more powerful probabilistic frameworks to model more complex distributions (Kaiser et al., 2018; Akoury et al., 2019; Shu et al., 2019; Ma et al., 2019), guiding the training process with an autoregressive model (Li et al., 2019; Wei et al., 2019), etc. However, these models cannot alter a target token once it has been generated, which means these models are not able to recover from an error caused by the multi-modality problem.
To alleviate the multi-modality problem while maintaining a reasonable decoding speedup, we propose a novel semi-autoregressive model named RecoverSAT in this work. RecoverSAT features in three aspects: (1) To improve decoding speed, we assume that a translation can be divided into several segments which can be generated simultaneously. (2) To better capture target-side dependency, the tokens inside a segment is autoregressively generated conditioned not only on the previously generated tokens in this segment but also on those in other segments. On one hand, we observe that repetitive tokens are more likely to occur within a short context. Therefore, autoregressively generating a segment is beneficial for reducing repetitive tokens. On the other hand, by conditioning on previously
generated tokens in other segments, the model is capable of guessing what feasible translation candidates have been chosen by each segment and adapts accordingly, e.g., recovering from missing token errors. As a result, our model captures more target-side dependency such that the multi-modality problem can be alleviated naturally. (3) To make the model capable of recovering from repetitive token errors, we introduce a segment deletion mechanism into our model. Informally speaking, our model will mark a segment to be deleted once it finds the content has been translated in other segments.
We conduct experiments on three benchmark datasets for machine translation to evaluate the proposed method. The experimental results show that RecoverSAT is able to decode over  $4\times$  faster than the autoregressive counterpart while maintaining comparable performance. The source code of this work is released on https://github.com/ ranqiu92/RecoverSAT.
2.1 Autoregressive Neural Machine Translation
Autoregressive neural machine translation (AT) generates the translation token-by-token conditioned on translation history. Denoting a source sentence as  $\mathbf{x} = \{x_{i}\}_{i = 1}^{T^{\prime}}$  and a target sentence as  $\mathbf{y} = \{y_j\}_{j = 1}^T$  , AT models the joint probability as:
$$
P (\mathbf {y} | \mathbf {x}) = \prod_ {t = 1} ^ {T} P \left(y _ {t} \mid \mathbf {y} _ {<   t}, \mathbf {x}\right). \tag {1}
$$
where  $\mathbf{y}_{< t}$  denotes the generated tokens before  $y_{t}$ .
During decoding, the translation history dependency makes the AT model predict each token after all previous tokens have been generated, which makes the decoding process time-consuming.
2.2 Non-Autoregressive Neural Machine Translation
Non-autoregressive neural machine translation (NAT) (Gu et al., 2018) aims to accelerate the decoding process, which discards the dependency of translation history and models  $P(\mathbf{y}|\mathbf{x})$  as a product of the conditionally independent probability of each token:
$$
P (\mathbf {y} | \mathbf {x}) = \prod_ {t = 1} ^ {T} P (y _ {t} | \mathbf {x}). \tag {2}
$$
The conditional independence enables the NAT models to generate all target tokens in parallel.
However, independently predicting all target tokens is challenging as natural language often exhibits strong correlation across context. Since the model knows little information about surrounding target tokens, it may consider different possible translations when predicting different target tokens. The problem is known as the multi-modality problem (Gu et al., 2018) and significantly degrades the performance of NAT models.
RecoverSAT extends the original Transformer (Vaswani et al., 2017) to enable the decoder to perform generation autoregressively in local and non-autoregressively in global. An overview of the architecture of our RecoverSAT model is shown in Figure 1. As illustrated in the figure, RecoverSAT simultaneously predicts all segments "there are EOS", "lots of farmers EOS", "a lot DEL" and "doing this today EOS". And at each time step, it generates a token for each incomplete segment. The special token DEL denotes the segment should be deleted and EOS denotes the end of a segment. Combining all the segments, we obtain the final translation "there are lots of farmers doing this today".
Formally, assuming a translation  $\mathbf{y}$  is generated as  $K$  segments  $\mathbf{S}^1,\mathbf{S}^2,\dots ,\mathbf{S}^K$  , where  $\mathbf{S}^i$  is a subsequence of the translation'. For description simplicity, we assume that all the segments have the
same length. RecoverSAT predicts a token for each segment conditioned on all previously generated tokens at each generation step, which can be formulated as:
$$
P (\mathbf {y} | \mathbf {x}) = \prod_ {t = 1} ^ {L} \prod_ {i = 1} ^ {K} P (\mathbf {S} _ {t} ^ {i} | \mathbf {S} _ {<   t} ^ {1} \dots \mathbf {S} _ {<   t} ^ {K}; \mathbf {x}), \quad (3)
$$
where  $\mathbf{S}_t^i$  denotes the  $t$ -th token in the  $i$ -th segment,  $\mathbf{S}_{< t}^{i} = \{\mathbf{S}_{1}^{i},\dots ,\mathbf{S}_{t - 1}^{i}\}$  denotes the translation history in the  $i$ -th segment, and  $L$  is segment length.
Here, two natural problems arise for the decoding process:
- How to determine the length of a segment?
- How to decide a segment should be deleted?
We address the two problems in a uniform way in this work. Suppose the original token vocabulary is  $V$ , we extend it with two extra tokens EOS and DEL. Then for the segment  $\mathbf{S}^i$ , the most probable token  $\hat{\mathbf{S}}_t^i$  at time step  $t$ :
$$
\hat {\mathbf {S}} _ {t} ^ {i} = \underset {\mathbf {S} _ {t} ^ {i} \in V \cup \{\mathrm {E O S}, \mathrm {D E L} \}} {\arg \max } P \left(\mathbf {S} _ {t} ^ {i} \mid \mathbf {S} _ {<   t} ^ {1} \dots \mathbf {S} _ {<   t} ^ {K}; \mathbf {x}\right) \tag {4}
$$
has three possibilities:
(1)  $\hat{\mathbf{S}}_t^i\in V$  : the segment  $\mathbf{S}^i$  is incomplete and the decoding process for it should continue;
(2)  $\hat{\mathbf{S}}_t^i = \mathrm{EOS}$ : the segment  $\mathbf{S}^i$  is complete and the decoding process for it should terminate;
(3)  $\hat{\mathbf{S}}_t^i = \mathrm{DEL}$ : the segment  $\mathbf{S}^i$  is repetitive and should be deleted. Accordingly, the decoding process for it should terminate.
The entire decoding process terminates when all the segments meet EOS/DEL or reach the maximum token number. It should be noticed that we do not explicitly delete a segment when DEL is encountered but do it via post-processing. In other words, the model is trained to ignore the segment to be deleted implicitly.
3.2 Learning to Recover from Errors
As there is little target-side information available in the early stage of the decoding process, the errors caused by the multi-modality problem is inevitable. In this work, instead of reducing such errors directly, we propose two training mechanisms to teach our RecoverSAT model to recover
dynamically according to the sentence length. In other words, we can predict the target sentence length to determine the segment number during inference. In this case, our model can also decode in constant time.
Note that, by fixing segment length (token number of each segment) instead, the segment number  $K$  can be changed
from errors: (1) Dynamic Termination Mechanism: learning to determine segment length according to target-side context; (2) Segment Deletion Mechanism: learning to delete repetitive segments.
3.2.1 Dynamic Termination Mechanism
As shown in Section 3.1, instead of pre-specifying the lengths of segments, we let the model determine the lengths by emitting the EOS token. This strategy helps our model recover from multi-modality related errors in two ways:
1. The choice of the first few tokens is more flexible. Taking Figure 1 as an example, if the decoder decides the first token of the second segment is "of" instead of "lots" (i.e., "lots" is not generated in the second segment), it only needs to generate "lots" before "EOS" in the first segment in order to recover from missing token errors. In contrast, if the decoder decides the first token is "are", it can avoid repetitive token error by not generating "are" in the first segment;
2. As shown in Eq. 3, a token is generated conditioned on all the previously generated tokens in all the segments. Therefore, the decoder has richer target-side information to detect and recover from such errors.
However, it is non-trivial to train the model to learn such behaviour while maintaining a reasonable speedup. On one hand, as the decoding time of our RecoverSAT model is proportional to the maximum length of the segments, we should divide the target sentences of training instances into equal-length segments to encourage the model to generate segments with identical length. On the other hand, the model should be exposed to the multi-modality related errors to enhance its ability of recovering from such errors, which suggests that the target sentences of training instances should be divided randomly to simulate these errors.
To alleviate the problem, we propose a mixed annealing dividing strategy. To be specific, we randomly decide whether to divide a target sentence equally or randomly at each training step and gradually anneal to the equally-dividing method at the end of training. Formally, given the target sentence  $\mathbf{y}$  and the segment number  $K$ , we define the segment dividing index set  $\mathbf{r}$  as follows:
$$
s \sim \operatorname {B e r n o u l l i} (p), \tag {5}
$$
$$
\mathbf {r} = \left\{ \begin{array}{l l} \operatorname {E Q U A L} (T, K - 1) & s = 0 \\ \operatorname {R A N D} (T, K - 1) & s = 1 \end{array} , \right. \tag {6}
$$
where  $\operatorname{Bernoulli}(p)$  is the Bernoulli distribution with parameter  $p$ ,  $\mathrm{EQUAL}(n,m) = \{\lceil \frac{n}{m + 1} \rceil, \lceil \frac{2n}{m + 1} \rceil, \dots, \lceil \frac{mn}{m + 1} \rceil\}$ ,  $\mathrm{RAND}(n,m)$  sampling  $m$  non-duplicate indices from  $[1,n]$ . A larger value of  $p$  leads to better error recovering ability while a smaller one encourages the model to generate segments with similar lengths (in other words, better speedup). To balance the two aspects, we gradually anneal  $p$  from 1 to 0 in the training process, which achieves better performance (Section 4.5).
3.2.2 Segment Deletion Mechanism
Although the dynamic termination mechanism makes the model capable of recovering from missing token errors and reducing repetitive tokens, the model still can not recover from errors where token repetition errors have already occurred. We find the major errors of our model occur when generating the first token of each segment since it cannot see any history and future. In this situation, two repetitive segments will be generated. To alleviate this problem, we propose a segment-wise deletion strategy, which uses a special token DEL to indicate a segment is repetitive and should be deleted<sup>2</sup>.
A straightforward way to train the model to learn to delete a segment is to inject pseudo repetitive segments into the training data. The following is an example:
Target Sentence	there are lots of farmers doing this today
+ Pseudo Repetitive Segment	there are lots of farmers lots of DEL doing this today
Given the target sentence "there are lots of farmers doing this today", we first divide it into 3 segments "there are", "lots of farmers" and "doing this today". Then we copy the first two tokens of the second segment and append the special token DEL to the end to construct a pseudo repetitive segment "lots of DEL". Finally, we insert the repetitive segment to the right of the chosen segment, resulting in 4 segments. Formally, given the expected segment number  $K$  and the target sentence  $\mathbf{y}$ , we first divide  $\mathbf{y}$  into  $K - 1$  segments  $\mathbf{S}^1, \mathbf{S}^2, \dots, \mathbf{S}^{K - 1}$  and then build a pseudo repetitive segment  $\mathbf{S}_{rep}^i$  by copying the first  $m$  tokens of a randomly chosen segment  $\mathbf{S}^i$  and appending DEL to the end,  $m$  is uniformly
2It is more flexible to employ token-wise deletion strategy which could handle more complex cases. We will explore this in future.
sampled from  $[1,|\mathbf{S}^i |]$ . Finally,  $\mathbf{S}_{rep}^i$  is inserted at the right side of  $\mathbf{S}^i$ . The final  $K$  segments are  $\mathbf{S}^1,\mathbf{S}^2,\dots ,\mathbf{S}^i,\mathbf{S}_{rep}^i,\mathbf{S}^{i + 1},\dots ,\mathbf{S}^{K - 1}$
However, injecting such pseudo repetitive segments to all training instances will mislead the model that generating then deleting a repetitive segment is a must-to-have behaviour, which is not desired. Therefore, we inject pseudo repetitive segment into a training instance with probability  $q$  in this work.
We conduct experiments on three widely-used machine translation datasets: IWSLT16 En-De (196k pairs), WMT14 En-De (4.5M pairs) and WMT16 En-Ro (610k pairs). For fair comparison, we use the preprocessed datasets in Lee et al. (2018), of which sentences are tokenized and segmented into subwords using byte-pair encoding (BPE) (Sennrich et al., 2016) to restrict the vocabulary size. We use a shared vocabulary of 40k subwords for both source and target languages. For the WMT14 En-De dataset, we use newstest-2013 and newstest-2014 as validation and test sets respectively. For the WMT16 En-Ro dataset, we employ newsdev2016 and newstest-2016 as validation and test sets respectively. For the IWSLT16 En-De dataset, we use test2013 as the validation set.
For model hyperparameters, we follow most of the settings in (Gu et al., 2018; Lee et al., 2018; Wei et al., 2019). For the IWSLT16 En-De dataset, we use a small Transformer model ( $d_{model} = 278$ ,  $d_{hidden} = 507$ ,  $n_{layer} = 5$ ,  $n_{head} = 2$ ,  $p_{dropout} = 0.1$ ). For the WMT14 En-De and WMT16 En-Ro datasets, we use a larger Transformer model ( $d_{model} = 512$ ,  $d_{hidden} = 512$ ,  $n_{layer} = 6$ ,  $n_{head} = 8$ ,  $p_{dropout} = 0.1$ ). We linearly anneal the learning rate from  $3 \times 10^{-4}$  to  $10^{-5}$  as in Lee et al. (2018) for the IWSLT16 En-De dataset, while employing the warm-up learning rate schedule (Vaswani et al., 2017) with  $t_{warmup} = 4000$  for the WMT14 En-De and WMT16 En-Ro datasets. We also use label smoothing of value  $\epsilon_{ls} = 0.15$  for all datasets. We utilize the sequence-level distillation (Kim and Rush, 2016), which replaces the target sentences in the training dataset with sentences generated by an autoregressive model, and set the beam size of the technique to 4. We use the
encoder of the corresponding autoregressive model to initialize the encoder of RecoverSAT, and share the parameters of source and target token embedding layers and the pre-softmax linear layer. We measure the speedup of model inference in each task on a single NVIDIA P40 GPU with the batch size 1.
We use the Transformer (Vaswani et al., 2017) as our AT baseline and fifteen latest strong NAT models as NAT baselines, including: (1) fertility-based model: NAT-FT (Gu et al., 2018); (2) iterative decoding based models: NAT-IR (Lee et al., 2018) and CMLM (Ghazvininejad et al., 2019); (3) models learning from AT teachers: imitate-NAT (Wei et al., 2019), NART (Li et al., 2019) and FCL-NAT (Guo et al., 2019b); (4) latent variable framework based models: LV NAR (Shu et al., 2019) and FlowSeq (Ma et al., 2019); (5) regularization framework based model: NAT-REG (Wang et al., 2019); (6) models introducing extra target-side dependencies: SAT (Wang et al., 2018), SynST (Akoury et al., 2019), NAT-FS (Shao et al., 2019a), PNAT (Bao et al., 2019), NART-DCRF (Sun et al., 2019) and ReorderNAT (Ran et al., 2019).
The performance of our RecoverSAT model and the baselines is shown in Table 2. Due to the space limitation, we only show the results corresponding to the settings of the best BLEU scores for the baselines<sup>3</sup>. From Table 2, we can observe that:
(1) Our RecoverSAT model achieves comparable performance with the AT baseline (Transformer) while keeping significant speedup. When  $K = 2$ , the BLEU score gap is moderate (from 0.06 to 0.4, even better than Transformer on the WMT16 En→Ro and Ro→En tasks) and the speedup is about  $2 \times$ . When  $K = 10$ , the BLEU scores drop less than  $5\%$  relatively, and the speedup is considerably good (over  $4 \times$ ).
(2) Our RecoverSAT model outperforms all the strong NAT baselines except CMLM (on the WMT16 En  $\rightarrow$  Ro and Ro  $\rightarrow$  En tasks). However, the performance gap is negligible (0.16 and 0.12 respectively), and CMLM is a multi-step NAT method which is significantly slower than our model.
<sup>3</sup>A thorough comparison under other settings can be found in Appendix B.
Model	Iterative Decoding	WMT14 En-De	WMT16 En-Ro	IWSLT16 En-De
En→	De→	Speedup	En→	Ro→	Speedup	En→	Speedup
Transformer	27.17	31.95	1.00×	32.86	32.60	1.00×	31.18	1.00×
NAT-FT+NPD (n = 100)	19.17	23.20	-	29.79	31.44	-	28.16	2.36×
SynST	20.74	25.50	4.86×	-	-	-	23.82	3.78×
NAT-IR (iter = 10)	✓	21.61	25.48	2.01×	29.32	30.19	2.15×	27.11	1.55×
NAT-FS	22.27	27.25	3.75×	30.57	30.83	3.70×	27.78	3.38×
imitate-NAT+LPD (n = 7)	24.15	27.28	-	31.45	31.81	-	30.68	9.70×
PNAT+LPD (n = 9)	24.48	29.16	-	-	-	-	-	-
NAT-REG+LPD (n = 9)	24.61	28.90	-	-	-	-	27.02	-
LV NAR	25.10	-	6.8×	-	-	-	-	-
NART+LPD (n = 9)	25.20	29.52	17.8×	-	-	-	-	-
FlowSeq+NPD (n = 30)	25.31	30.68	<1.5×	32.20	32.84	-	-	-
FCL-NAT+NPD (n = 9)	25.75	29.50	16.0×	-	-	-	-	-
ReorderNAT	26.51	31.13	-	31.70	31.99	-	30.26	5.96×
NART-DCRF+LPD (n = 19)	26.80	30.04	4.39×	-	-	-	-	-
SAT (K = 2)	26.90	-	1.51×	-	-	-	-	-
CMLM (iter = 10)	✓	27.03	30.53	<1.5×	33.08	33.31	-	-	-
RecoverSAT (K = 2)	27.11	31.67	2.16×	32.92	33.19	2.02×	30.78	2.06×
RecoverSAT (K = 5)	26.91	31.22	3.17×	32.81	32.80	3.16×	30.55	3.28×
RecoverSAT (K = 10)	26.32	30.46	4.31×	32.59	32.29	4.31×	29.90	4.68×
Table 2: Performance (BLEU) of Transformer, the NAT/semi-autoregressive models and RecoverSAT on three widely-used machine translation benchmark datasets. NPD denotes the noisy parallel decoding technique (Gu et al., 2018) and LPD denotes the length parallel decoding technique (Wei et al., 2019).  $n$  denotes the sample size of NPD or LPD. iter denotes the refinement number of the iterative decoding method.
(3) As  $K$  grows, the BLEU scores drop moderately and the speedup grows significantly, indicating that our RecoverSAT model has a good generalizability. For example, the BLEU scores drop less than 0.45 when  $K$  grows from 2 to 5, and drop no more than 0.90 except on the WMT14 De→En task when  $K$  further grows to 10. Meanwhile, the speedup for  $K = 10$  is larger than  $4\times$ , which is considerably good.
(4) There are only 7 baselines (SynST, imitate-NAT+LPD, LV NAR, NART+LPD, FCL-NAT+NPD, ReorderNAT and NART-DCRF+LPD) achieving better speedup than our RecoverSAT model when  $K = 10$ . However, only ReorderNAT and NART-DCRF+LPD achieve comparable BLEU scores with our model. The improvements of both ReorderNAT and NART-DCRF are complementary to our method. It is an interesting future work to join these works together.
4.5 Effect of Dynamic Termination Mechanism