SlowGuess/ACL_2020_OCR huggingface dataset:
Junliang Guo Linli Xu* Enhong Chen
Anhui Province Key Laboratory of Big Data Analysis and Application, School of Computer Science and Technology, University of Science and Technology of China
guojunll@mail.ustc.edu.cn {linlixu,cheneh}@ustc.edu.cn
The masked language model has received remarkable attention due to its effectiveness on various natural language processing tasks. However, few works have adopted this technique in the sequence-to-sequence models. In this work, we introduce a jointly masked sequence-to-sequence model and explore its application on non-autoregressive neural machine translation (NAT). Specifically, we first empirically study the functionalities of the encoder and the decoder in NAT models, and find that the encoder takes a more important role than the decoder regarding the translation quality. Therefore, we propose to train the encoder more rigorously by masking the encoder input while training. As for the decoder, we propose to train it based on the consecutive masking of the decoder input with an  $n$ -gram loss function to alleviate the problem of translating duplicate words. The two types of masks are applied to the model jointly at the training stage. We conduct experiments on five benchmark machine translation tasks, and our model can achieve 27.69/32.24 BLEU scores on WMT14 English-German/German-English tasks with  $5+$  times speed up compared with an autoregressive model.
The encoder-decoder based sequence-to-sequence framework (Sutskever et al., 2014; Bahdanau et al., 2014) has achieved great success on the task of Neural Machine Translation (NMT) (Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Hassan et al., 2018; Sheng et al., 2020). In this framework, the encoder takes the source sentence as input and extracts its hidden representation, based on which the decoder generates the target sentence word by word and from left to right, i.e.,
in an autoregressive manner, which is a natural bottleneck for the inference speed due to the sequential conditional dependence.
As the performance of NMT models have been substantially promoted, the translation efficiency is becoming a new research hotspot. Non-autoregressive neural machine translation (NAT) models are proposed to reduce the translation latency while inference, by removing the conditional dependence between target tokens and predicting all tokens in parallel (Gu et al., 2017). As the context dependency cannot be utilized while decoding, the inference speedup of NAT models comes at the cost of the degradation in performance. As studied by previous works (Guo et al., 2019; Wang et al., 2019), the inferior accuracy of NAT models mainly occurs from two aspects: 1) the source-side information is not adequately encoded which results in incomplete translation; 2) the decoder cannot handle the task well which leads to repeated translations and poor performance on long sentences.
To tackle these problems and promote the performance of NAT models, in this paper, we empirically conduct a thorough study on the functionalities of the encoder and decoder in NAT models, and conclude that the encoder has a more direct influence on the final translation performance, and is harder to train than the decoder. Therefore, we propose a jointly masked sequence-to-sequence model which is inspired by the idea of masked language modeling (Devlin et al., 2018). Specifically, for the encoder, we follow the masking strategy of BERT (Devlin et al., 2018) and randomly mask a number of tokens of the source sentence. This strategy trains the encoder more rigorously by forcing it to encode the complete information with residual input. For the decoder, we mask the consecutive fragment of the target sentence to make the decoder concentrate more on predicting adjacent tokens, and propose an  $n$ -gram based loss function to learn
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 376-385 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
the consecutive tokens as a whole objective. In this way, we can alleviate the problem of repeated translations of NAT models. During inference, we adopt a mask-and-predict (Ghazvininejad et al., 2019) strategy to iteratively generate the translation result, which masks and predicts a subset of the current translation candidates in each iteration.
We verify the effectiveness of our model on five benchmark translation tasks including WMT14 English  $\leftrightarrow$  German, WMT16 English  $\leftrightarrow$  Romanian and IWSLT14 German  $\rightarrow$  English. Our model outperforms all the NAT models in comparison, and can achieve comparative performance with its autoregressive counterpart while enhanced with 5+ times speedup on inference (27.69/32.24 BLEU scores and 5.73 times speedup on the WMT14 En-De/De-En tasks with an autoregressive teacher of 28.04/32.69 BLEU scores).
- While previous works only concentrate on manipulating the decoder, we illustrate and emphasize the importance of the encoder in NAT models and propose the encoder masking strategy to improve its training.
- We propose the consecutive masking strategy of the decoder input and the  $n$ -gram loss function to alleviate the problem of repetitive translations of NAT models.
- We integrate the two parts above in the jointly masked sequence-to-sequence model which shows strong performance on benchmark machine translation datasets.
2.1 Non-Autoregressive Machine Translation
Neural machine translation (NMT) models have achieved great success in recent years. Traditional NMT models are based on the sequence-to-sequence framework (Bahdanau et al., 2014; Sutskever et al., 2014), taking the source sentence as input and generating the target sentence in an autoregressive manner. Specifically, given the source sentence  $x = (x_{1},x_{2},\dots,x_{T_{x}})$ , the target sentence  $y = (y_{1},y_{2},\dots,y_{T_{y}})$  is generated as:
$$
P (y | x) = \prod_ {t = 1} ^ {T _ {y}} P \left(y _ {t} \mid y _ {<   t}, x; \theta_ {\mathrm {e n c}}, \theta_ {\mathrm {d e c}}\right), \tag {1}
$$
where  $y_{< t}$  indicates the generated target tokens before timestep  $t$ , and  $\theta_{\mathrm{enc}}$  and  $\theta_{\mathrm{dec}}$  denote the pa
rameters of the encoder and decoder respectively. For a target sentence with length  $n$ , autoregressive models have to take  $O(n)$  iterations to generate it during inference. To break the sequential conditional dependency and make the generation process parallelizable, non-autoregressive machine translation (NAT) models are proposed to generate all target tokens independently (Gu et al., 2017) and reduce the time complexity from  $O(n)$  to  $O(k)$  where  $k$  is a constant number:
$$
P (y | x) = P \left(T _ {y} | x\right) \cdot \prod_ {t = 1} ^ {T _ {y}} P \left(y _ {t} | x; \theta_ {\mathrm {e n c}}, \theta_ {\mathrm {d e c}}\right), \tag {2}
$$
where  $P(T_{y}|x)$  is the explicit length prediction process for NAT models. Although the inference speed of NAT is significantly boosted, the translation accuracy is sacrificed due to the lack of context information at the target side. Therefore, lots of works have been conducted to promote the performance of NAT models. Specifically, Gu et al. (2017) takes a copy of the encoder input  $x$  as the decoder input and trains a fertility predictor to guide the copy procedure. Lee et al. (2018) and Ghazvininejad et al. (2019) generate the target sentence by iteratively refining the current translation. Other works enhance the performance of NAT models by utilizing auxiliary information, such as extra loss functions (Wang et al., 2019; Li et al., 2019; Sun et al., 2019; Wei et al., 2019; Shao et al., 2019), SMT components (Guo et al., 2019) and fine-tuning from an AT model (Guo et al., 2020). Recently, some works (Stern et al., 2019; Welleck et al., 2019; Gu et al., 2019) propose to change the generation order from the traditional left-to-right manner to a tree-based manner, resulting in a time complexity of  $O(\log n)$ . In this paper, we focus on the NAT model with  $O(k)$  generation complexity.
2.2 Masked Language Model
The masked language model proposed by BERT (Devlin et al., 2018) has become the essential component of the state-of-the-art pre-training methods (Song et al., 2019; Dong et al., 2019; Liu et al., 2019; Joshi et al., 2019; Lample and Conneau, 2019) in natural language understanding tasks. The standard paradigm of masked language modeling is to substitute a subset of tokens in the input sentence by a special symbol [MASK], and predict the missing tokens by the residual ones. We denote the residual tokens as  $x^{r}$  and the masked target tokens as  $x^{m}$ .
Δ Layers	+5	+10	+15
Δ Enc BLEU	+0.71	+1.05	+1.26
Δ Dec BLEU	+0.12	+0.18	+0.20
Table 1: The comparison of gains in BLEU score on the test set of the IWSLT14 German-English task when adding more layers to the encoder and decoder respectively of the NAT model.
As BERT is designed for language understanding tasks which can be handled with a single Transformer encoder, it is non-trivial to extend the paradigm into NMT tasks, where a sequence-to-sequence framework is utilized. To address that, XLM (Lample and Conneau, 2019) concatenates the source sentence and the target sentence as the encoder input to let the model learn the cross-lingual information, but still using a single Transformer encoder. MASS (Song et al., 2019) presents a sequence-to-sequence pre-training framework, which takes  $x^{r}$  as the encoder input and takes  $x^{m}$  as the decoder input as well as the target, still yielding a monolingual pre-training framework. In this paper, we propose a jointly masked language modeling method to handle the cross-lingual challenge in a unified sequence-to-sequence framework, based on which the translation accuracy of AT models and the inference speedup of NAT models can both be preserved.
3 Preliminary Study
To explore the functionalities of the encoder and decoder in NAT models, we conduct a thorough empirical study. We mainly follow the settings in (He et al., 2019). We train a basic NAT model proposed by Gu et al. (2017), except that we remove the fertility predictor and keep the decoder input as a hard copy of the source sentence in a similar way with (Guo et al., 2019; Wang et al., 2019). We conduct the following experiments on the IWSLT14 German to English dataset and train the model with the same number of training steps for each setting.
We study the importance of the encoder and decoder from three aspects. Firstly, we vary the number of encoder and decoder layers respectively to see which will bring more performance gain. Specifically, on a basic model with a 5-layer encoder and a 5-layer decoder, we increase the number of layers to the encoder and decoder separately. Results are illustrated in Table 1, from which we
(a) Convergence Speed
(b) Performance with Noisy Input
Figure 1: (a) The convergence speed of the encoder and the decoder. (b) The performance when adding noise to the encoder input, encoder output and decoder input in the inference stage of a basic NAT model.
can conclude that adding the layers of the encoder can bring more performance gain than the decoder.
Secondly, we compare the convergence speed of the encoder and decoder by initializing the NAT model with a pretrained decoder/encoder and fix it during training, while randomly initialize a trainable encoder/decoder. The convergence speed is illustrated by the BLEU score along with the training steps, as shown in Figure 1(a). From the results, we can observe that the decoder converges faster than the encoder. In conclusion, we find that the encoder is dealing with a more sophisticated task than the decoder, and the encoder is not adequately trained in the initial NAT model.
Thirdly, we further conduct an investigation on the encoder input, encoder output and decoder input to evaluate their importance in the inference stage. During inference, we add random noise to the three types of inputs respectively, by randomly replacing the embeddings of some tokens with random noise. This experiment is conducted on a basic 5-layer encoder and decoder NAT model, and the
results are illustrated in Figure 1(b). Obviously, the encoder input and encoder output both largely influence the translation quality, which implies that the encoder plays an important role in the inference of NAT models, while the decoder input is the least important due to its conditional independence in nature. In a word, the performance of NAT models rely more on the encoder rather than the decoder.
While most existing NAT works only focus on refining the decoder to obtain better performance, we have explored and shown the significance of the encoder in the previous section. Therefore, we propose to improve the translation performance by further manipulating the encoder, and we will introduce the proposed framework to tackle the problems discussed above in this section. We start with the problem definition.
Problem Definition Given a pair of source and target sentence  $(x,y)\in (\mathcal{X},\mathcal{Y})$  from the parallel training dataset  $\mathcal{X}$  and  $\mathcal{Y}$ , the negative log-likelihood objective function of an NMT model can be written as:
$$
L _ {\mathrm {n l l}} (x, y; \theta_ {\mathrm {e n c}}, \theta_ {\mathrm {d e c}}) = - \log P (y | x; \theta_ {\mathrm {e n c}}, \theta_ {\mathrm {d e c}}), \tag {3}
$$
where the conditional probability can be either Equation (1) or Equation (2) for AT or NAT models, and  $\theta_{\mathrm{enc}},\theta_{\mathrm{dec}}$  represent the parameters of the encoder and decoder respectively.
4.1 Encoder Masking
As studied in Section 3, the encoder needs to handle a harder task than the decoder but is not adequately trained in previous works. To maximize the functionality of the encoder, we propose to train it with masked language modeling.
The general masking strategy is as follows. Given a source sentence  $x = (x_{1}, x_{2}, \dots, x_{T_{x}})$ , we randomly sample a subset from  $x$ , denoted as  $x^{m}$  with  $T_{x}^{m}$  tokens, and substitute them with other tokens in position. Specifically, we follow the similar substitution strategy as BERT (Devlin et al., 2018): we randomly select  $10\%$  of the tokens in  $x$  of which  $80\%$  are substituted with a special symbol [MASK],  $10\%$  are substituted with a random token in the vocabulary, and  $10\%$  are kept unchanged. And we denote the substituted result of the source sentence as  $x^{r}$ . Then the loss function on the encoder of predicting the missing source tokens can
be written as:
$$
L _ {\mathrm {e n c}} \left(x ^ {m} \mid x ^ {r}\right) = - \sum_ {t = 1} ^ {T _ {x} ^ {m}} \log P \left(x _ {t} ^ {m} \mid x ^ {r}\right). \tag {4}
$$
4.2 Decoder Masking
For the decoder, as it is shown that the repetitive translations mainly result from the non-autoregressive nature of NAT, we alleviate this problem by applying a consecutive masking strategy and proposing a tailored  $n$ -gram based loss function. During training, given a target sentence  $y = (y_{1},y_{2},\dots,y_{T_{y}})$ , we randomly select multiple sets of consecutive tokens and mask them in a similar strategy as masking the encoder. Each set contains  $n$  consecutive tokens, and we denote the masked target set as  $y^{m}$  and the substituted result as  $y^{r}$ , and their corresponding lengths as  $T_{y}^{m}$  and  $T_{y}$ . Note that in the decoder, the total number of masking tokens is uniformly sampled from 1 to  $T_{y}$  instead of being computed with a fixed ratio. We provide an illustration of our framework in Figure 2, where  $n$  is set to 2. The loss function of predicting the masked target tokens can be written as:
$$
L _ {\mathrm {n l l}} \left(y ^ {m} \mid x ^ {r}, y ^ {r}\right) = - \sum_ {t = 1} ^ {T _ {y} ^ {m}} \log P \left(y _ {t} ^ {m} \mid x ^ {r}, y ^ {r}\right). \tag {5}
$$
We propose an  $n$ -gram based loss function, which has been applied to NMT models recently (Ma et al., 2018; Shao et al., 2018, 2019), to enhance the sentence-level information and alleviate the problem of repetitive translations of NAT models. The loss function is tied with the consecutive masking where  $n$  equals to the number of the consecutive masked tokens in each set. Specifically, given an  $n$ -gram  $g = (g_{1}, \dots, g_{n})$ , its occurrence count in the target sentence  $y$  can be written as:  $C_{y}(g) = \sum_{t=0}^{T_{y}-n} \prod_{i=1}^{n} 1\{g_{i} = y_{t+i}\}$ . As for the count in the masked sequence  $y^{m}$ , we introduce the probabilistic variant of the  $n$ -gram count to make the objective differentiable (Shao et al., 2018) by representing each token with the prediction probability:
$$
\tilde {C} _ {y ^ {m}} (g) = \sum_ {t = 0} ^ {T _ {y} ^ {m} - n} \prod_ {i = 1} ^ {n} 1 \left\{g _ {i} = y _ {t + i} ^ {m} \right\} \cdot p \left(y _ {t + i} ^ {m} | x\right). \tag {6}
$$
Considering all possible  $n$ -grams in  $y$ , the proposed
Figure 2: An illustration of the propose jointly masked sequence-to-sequence framework. “-” indicates that the token at this position is substituted by other tokens following the masking strategy.  $M$  and  $N$  indicate the number of layers of encoder and decoder respectively.
$n$ -gram based loss function can be written as:
$$
L _ {\text {g r a m}} \left(y, y ^ {m} \mid y ^ {r}, x ^ {r}\right) = \tag {7}
$$
$$
K - \sum_ {g} \min (C _ {y} (g), \tilde {C} _ {y ^ {m}} (g)),
$$
where  $\min(C_y(g), \tilde{C}_{y^m}(g))$  represents the matching count between  $y$  and  $y^m$  w.r.t the  $n$ -gram  $g$ , and  $K$  is the upper bound of the total matching count which equals to the number of sets of consecutive masked tokens. The  $n$ -gram loss function will encourage the model to treat the consecutive masked tokens as a whole objective to match the sequential fragments in the target sentence, thus reducing the occurrence of repetitive translations.
4.3 Jointly Masked Model
Based on the proposed framework, the objective function of our model contains three parts: the traditional negative log-likelihood loss function to predict the missing target tokens  $L_{\mathrm{nll}}(\cdot)$ , the prediction loss function on the encoder side  $L_{\mathrm{enc}}(\cdot)$ , and the  $n$ -gram loss function  $L_{\mathrm{gram}}(\cdot)$ . By integrating the three loss functions, given a training pair  $(x,y)$ , the complete objective function of our model is:
$$
\begin{array}{l} \min  _ {\Theta} L (x, y) = L _ {\mathrm {n l l}} \left(y ^ {m} \mid x ^ {r}, y ^ {r}; \theta_ {\mathrm {e n c}}, \theta_ {\mathrm {d e c}}\right) \\ + \alpha_ {1} L _ {\mathrm {e n c}} \left(x ^ {m} \mid x ^ {r}; \theta_ {\mathrm {e n c}}\right) \tag {8} \\ + \alpha_ {2} L _ {\mathrm {g r a m}} (y, y ^ {m} | y ^ {r}, x ^ {r}; \theta_ {\mathrm {e n c}}, \theta_ {\mathrm {d e c}}), \\ \end{array}
$$
where  $\Theta = (\theta_{\mathrm{enc}},\theta_{\mathrm{dec}})$ ,  $\alpha_{1}$  and  $\alpha_{2}$  are the hyperparameters that control the weights of different loss functions.
In the proposed training framework, the importance of the encoder has been emphasized by masking the encoder input and introducing  $L_{\mathrm{enc}}(\cdot)$ . The encoder is encouraged to produce better representations of other tokens in order to predict the missing tokens. On the decoder side, the consecutive masking strategy augmented with the  $n$ -gram based
loss function can help the model better capture the sentence-level information and alleviate the problem of repetitive translations.
4.4 Decoding Algorithm
For inference, we propose to iteratively refine the translation result in a mask-and-predict manner mainly following the strategy proposed in (Ghazvininejad et al., 2019), and details are introduced below.
During inference, the first step for NAT models is to determine the length of the target translation. We follow (Ghazvininejad et al., 2019) and introduce an additional prediction process to estimate the length by the source sentence. Specifically, we add a special token to the encoder and predict the target length with the output hidden vector of this token. The negative log-likelihood loss function of this token is then added to the word prediction loss in Equation (8) as the final loss. In experiments, we also consider selecting the translation with highest probability over multiple translation candidates with different target lengths to obtain better results.
Thereafter, based on the mask-and-predict paradigm, we design our decoding algorithm as follows. Given the target length  $T_{y}$ , we initiate the target sentence with [MASK] at all positions, and take it as the decoder input followed by conducting translation. Next, for each iteration, we apply consecutive masking to the translation candidates as we have done in the training stage. Specifically, we select several tokens with the lowest probabilities from the current translation candidates, and mask these tokens as well as their adjacent ones. The number of tokens to mask at each iteration follows a linear decay function utilized in (Ghazvininejad et al., 2019). As for the stop condition, the final translation is taken either when a pre-defined number of iterations is reached, or the translation candidates do not change between two iterations.
We evaluate our method on five widely used benchmark tasks: IWSLT14 German  $\rightarrow$  English translation (IWSLT14 De-En)<sup>1</sup>, WMT16 English  $\leftrightarrow$  Romanian translation (WMT16 En-Ro/Ro-En)<sup>2</sup>, and WMT14 English  $\leftrightarrow$  German translation (WMT14 En-De/De-En)<sup>3</sup>. We strictly follow the dataset configurations of previous works. For the IWSLT14 De-En task, we train the model on its training set with 157k training samples, and evaluate on its test set. For the WMT14 En-De/De-En task, we train the model on the training set with 4.5M training samples, where newstest2013 and newstest2014 are used as the validation and test set respectively. As for the WMT16 En-Ro task which has 610k training pairs, we utilize newsdev2016 and newstest2016 as the validation and test set. For each dataset, we tokenize the sentences by Moses (Koehn et al., 2007) and segment each word into subwords using Byte-Pair Encoding (BPE) (Sennrich et al., 2015), resulting in a 32k vocabulary shared by source and target languages.
5.1.2 Model Settings
We strictly follow the previous works to set the configurations of models. Our model is based on the Transformer (Vaswani et al., 2017) architecture, with multi-head positional attention proposed in (Gu et al., 2017). We utilize the small Transformer  $(d_{\mathrm{model}} = d_{\mathrm{hidden}} = 256, n_{\mathrm{head}} = 4)$  with 5-layer encoder and decoder for the IWSLT14 De-En task, and the base Transformer  $(d_{\mathrm{model}} = d_{\mathrm{hidden}} = 512, n_{\mathrm{layer}} = 6, n_{\mathrm{head}} = 8)$  for the WMT14 and WMT16 tasks. We set  $n = 2$  for all tasks, i.e., we consider two-gram matchings when calculating  $L_{\mathrm{gram}}$ . The hyper-parameters  $\alpha_{1}$  and  $\alpha_{2}$  are both set to 0.01 for all tasks.
5.1.3 Baselines
We consider seven recent works as our baselines, including five NAT works: NAT with fertility (NATFT) (Gu et al., 2017), NAT with Imitation Learning (Imitate-NAT) (Wei et al., 2019), NAT with Regularizations (NAT-Reg) (Wang et al., 2019),
NAT with Curriculum Learning (FCL-NAT) (Guo et al., 2020), NAT with Dynamic Conditional Random Field (NAT-DCRF) (Sun et al., 2019); and two iterative decoding based works: NAT with Iterative Refinement (NAT-IR) (Lee et al., 2018) and Conditional Masked NAT (CM-NAT) (Ghazvininejad et al., 2019). The first five models are purely non-autoregressive, whose time complexities during inference are all  $O(1)$ . The other two models are based on iteratively refining the translation results by  $k$  iterations, where  $k$  is a constant number, yielding  $O(k)$  complexity. In the experiments, we also compare with them in terms of the inference latency on clock.
5.1.4 Sequence-Level Knowledge Distillation