SlowGuess/NAACL_2021_OCR huggingface dataset:
Xi Ai
College of Computer Science Chongqing University
barid.x.ai@gmail.com
Bin Fang
fb@cqu.edu.cn
Translation quality can be improved by global information from the required target sentence because the decoder can understand both past and future information. However, the model needs additional cost to produce and consider such global information. In this work, to inject global information but also save cost, we present an efficient method to sample and consider a semantic draft as global information from semantic space for decoding with almost free of cost. Unlike other successful adaptations, we do not have to perform an EM-like process that repeatedly samples a possible semantic from the semantic space. Empirical experiments show that the presented method can achieve competitive performance in common language pairs with a clear advantage in inference efficiency. We will open all our source code on GitHub.
Successful NMT (Neural Machine Translation) (Vaswani et al., 2017; Bahdanau et al., 2015; Johnson et al., 2017; Ng et al., 2019) can translate sentences through left to right or through right to left. However, there is one critical limitation in this diagram. That is, the decoder can only have access to directional information (left-to-right or right-to-left) when processing auto-regressive (Graves, 2013).
To alleviate this pain, there have been three successful lines. 1) Generative NMT: (Zheng et al., 2020; Shah and Barber, 2018; Su et al., 2018; Zhang et al., 2016; Eikema and Aziz, 2019) adapt VAE (variational auto-encoder) (Altieri and Duvenaud, 2015; Kingma and Ba, 2015; Bowman et al., 2016) for NMT that is trained in generative model settings, modeling the semantics of the source and target sentences in latent space. 2) Deliberation: since the problem is caused by the one-pass process of decoding in the auto-regression process, (Xia et al., 2017) present a framework to predict a guess
target sentence in the first-pass and jointly considers the encoding and the guess target sentence in the second-pass. 3) Soft-prototype: (Wang et al., 2019) present a framework to generate a prototype on the encoder side and then the decoder can jointly use the encoding and the prototype. Although empirical results show the previous methods can successfully inject global information into the decoder, these methods either introduce computational complexity to the encoder-decoder architecture or employ an EM-like process in inferring, thus requiring even more than  $100\%$  additional time to produce and consider global information in inferring.
In this work, we present an efficient method to sample and consider a semantic draft as global information for decoding with almost free of cost, following the line of generative NMT. Concretely, we sample the semantic draft from semantic space that is a Gaussian inference model with learnable parameters. In the classic utilization of the semantic space, e.g., generative NMT, inferring needs to work with the EM-like process that could degrade the inference efficiency significantly. To mitigate the degradation but still use the semantic space, we train the encoder of NMT in multilingual settings and simultaneously train a cross-lingual generator to obtain an approximation of the target-sentence semantic, hence modeling the required semantic space from the approximation and the source-sentence semantic. In inferring, based on the source-sentence semantic and an approximation made by the cross-lingual generator, the semantic draft can be sampled from the semantic space in a one-shot style. Once the semantic draft has been sampled, we aggregate the semantic draft and the encoding so that the variational decoder can simply decompose the aggregation.
We train the model in generative settings with additional loss of KL-divergence that is used to optimize the semantic space, similar to generative NMT training (Zheng et al., 2020; Shah and Barber,
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3931-3941 June 6-11, 2021. ©2021 Association for Computational Linguistics
2018; Su et al., 2018; Zhang et al., 2016; Eikema and Aziz, 2019) and VAE training (Altieri and Duvenaud, 2015; Kingma and Ba, 2015; Bowman et al., 2016). Our work can build upon Transformer (Vaswani et al., 2017), LSTM/GRU (Hochreiter and Schmidhuber, 1997; Chung et al., 2014) and Convolutional sequence (Gehring et al., 2017). In this work, we use Transformer as an example to present our idea, evaluating our method on common translation tasks and 5 more comprehensive experiments.
Our empirical study shows that, compared to previously successful methods, our method can achieve competitive performance and has a clear advantage in inference efficiency. Since we do not change the architecture of the NMT model, our model is compatible with common techniques in NMT.
Notation  $x$  and  $y$  denotes word embeddings in the source language  $L_{1}$  and the target language  $L_{2}$ , respectively.  $X = (x_{0}, x_{2}, \ldots, x_{n}) \in R^{N \times d}$  and  $Y = (y_{0}, y_{2}, \ldots, y_{m}) \in R^{M \times d}$  are the sentences sampled from corpora in  $L_{1}$  and  $L_{2}$  respectively, where  $N$  and  $M$  are the sequence length and  $d$  is the word embedding dimension.  $X$  and  $Y$  are parallel sentences that are used in our supervised training. The translation task  $X \rightarrow Y$  is denoted as  $Y = Dec(Enc(X))$ , where  $Dec$  and  $Enc$  jointly construct an encoder-decoder model.  $s$  and  $t$  represent the source-sentence semantic for  $X$  and the target-sentence semantic for  $Y$  in translation, respectively.  $z$  is a latent variable to represent a semantic draft, sampled from the semantic space.
NMT (Vaswani et al., 2017; Bahdanau et al., 2015; Johnson et al., 2017; Ng et al., 2019) utilizes seq2seq learning (Sutskever et al., 2014) and autoregressive (Graves, 2013) to facilitate training and inferring. Concretely, the current translation  $y_{j}$  at time-step  $j$  is conditional on  $Enc(X)$  and  $y_{<j}$ , where  $y_{<j}$  is the previous translation before  $j$ . The intrinsic problem is caused because the translation  $y_{j}$  can only consider  $y < j$  without considering  $y > j$ . Intuitively, a semantic draft or global information including  $y < j$  and  $y > j$  can benefit the translation  $y_{j}$  because the translation can be consistent with neighboring information.
Some impressive methods have been proposed to produce and consider a draft providing global information for better translation quality. 1) Gener
ative NMT (including variational NMT) (Shah and Barber, 2018; Zheng et al., 2020; Su et al., 2018; Zhang et al., 2016; Eikema and Aziz, 2019) study latent and continuous space of semantic (Bowman et al., 2016) for NMT, which can sample  $z$ . These methods inject  $z$  into NMT to provide global information for better translation. Meanwhile, the encoder is encouraged to consider  $z$ . In this manner, generative NMT models the joint probability  $P_{nmt}(X,Y,z) = p(z)p(X|z)p_{nmt}(Y|X,z)$  in training. For inferring, the model utilizes the EM-like process to maximize a lower bound on  $\log(p(X,Y))$  by repeatedly guessing or predicting possible  $Y$  and resampling  $z$ . However, compared to NMT without  $z$ , generative NMT costs over  $100\%$  additional time in inferring typically. 2) Sharing the same idea of the reconsideration of the current translation, Deliberation (Xia et al., 2017) is proposed to deliberate the complete output of the first-pass decoding as the attention context of the second-pass decoding. With the Deliberation, the final translation is based on the understanding of a possible translation in the target language. Although Deliberation is employed without the EM-like process, which is more efficient than generative NMT in inferring, the doubled pass increases the time of auto-regressive in decoding that costs  $80\%$  additional time in inferring. 3) (Wang et al., 2019) further consider the inference efficiency and the storage cost, proposing Soft-prototype framework to use a prototype. The prototype is an approximation of the target sentence  $Y' = (y_0',\dots,y_i')$ , produced by a probability generator  $R$  that accepts any  $x$  to generate a probability  $p(y')$  over the target vocabulary to search  $y'$ .
These successful methods, although using different settings and frameworks, share the same idea to inject a draft of the required target sentence and introduce global information to the decoder. Therefore, the decoder can understand the target globally. Concretely, such an idea can be formulated into a framework as:
$$
Y = \operatorname {D e c} (\operatorname {E n c} (X), \text {d r a f t}) \tag {1}
$$
However, these successful methods either introduce computational complexity to NMT (Wang et al., 2019; Xia et al., 2017) or employ the EM-like process, showing significant degradation in inference efficiency, e.g., GNMT(Shah and Barber, 2018) needs  $110\%$  additional inferring time. Intuitively, a high-quality draft should include two main aspects:
1) a good draft should include a global semantic for the target sentence; 2) a draft should not degrade inference efficiency significantly.
3 NMT with Semantic Draft
In this section, we present our framework and method. We then discuss how to train the model in generative settings and how to tackle optimization challenges in practice.
Inspired from previously successful models, we employ the general framework  $Y = \text{Dec}(\text{Enc}(X), \text{draft})$  for our model, presenting the high-level architecture in Figure 1. Concretely, draft is instantiated to  $z$  that the general framework is modified to  $Y = \text{Dec}(\text{Enc}(X), z)$ . Since  $z$  is sampled from the semantic space, our decoder is a variational decoder (Altieri and Duvenaud, 2015; Kingma and Ba, 2015; Bowman et al., 2016).
3.1.1 Generative Semantic Draft
To obtain  $z$ , we leverage a similar generative process of GNMT (Shah and Barber, 2018), sampling  $z$  from the semantic space that is a Gaussian inference model trained by  $s$  and  $t$  or approximations of  $s$  and  $t$  at the very least. Typically,  $s$  and  $t$  are obtained by modeling the semantics of  $X$  and  $Y$  with the same parameters.
Semantic for Source Sentence  $s \in R^d$  is computed by averaging a set of vector representation. Specifically, we first process  $X$  to the NMT encoder before averaging, obtaining  $Enc(X)$ . Then, we compute  $s = \frac{1}{N} \sum_{k=0}^{n} Enc(X)_k$ .
Semantic for Target Sentence We encourage the model to learn an approximation of  $t$  instead of the "ground-truth target semantic". We assume  $G(s) \approx t$ , where  $G$  is a two-layer cross-lingual generator. In other words, we compute a dummy target-sentence semantic  $G(s)$  based on  $s$ . We will discuss this assumption in §4 Multilingual Encoder and Cross-lingual Generator and how to train the cross-lingual generator  $G$  in §3.2 Encoder and Generator Tweaking.
Semantic Space Typically, a Gaussian inference model is used for the semantic space, representing a variational distribution  $q_{z}(z|s,t)$  for sampling (Shah and Barber, 2018; Zhang et al., 2016; Zheng et al., 2020). It serves as an approximate posterior. Instead of  $q_{z}(z|s,t)$ , in our model, we use
$q_{z}(z|s,G(s))$  for our required semantic space because  $G(s)$  is encouraged to learn an approximation of  $t$ . Specifically, we concatenate  $s$  and  $G(s)$  to compute the mean and variance of the diagonal Gaussian as:
$$
\begin{array}{l} S = [ s, G (s) ] \\ q _ {z} (z | s, G (s)) = \mathcal {N} \left(W ^ {\mu} S, \operatorname {d i a g} \left(\exp \left(W ^ {\sigma} S\right)\right)\right) \tag {2} \\ \end{array}
$$
3.1.2 Decoding with Draft
As aforementioned,  $z$  is sampled from the semantic space  $q_{z}(z|s,G(s))$ . We then aggregate  $z$  and  $Enc(X)$ , processing the aggregation to the decoder for decoding. In other words, we add generative context to the encoding for the encoder-decoder attention in the decoder. Therefore, the decoder is a variational decoder that is conditional on  $z$  and  $X$ .
NMT Training To train the parameters of both NMT and the semantic space in generative settings, we follow the successful training strategy in previous works (Bowman et al., 2016; Zhang et al., 2016), using SGVB (stochastic gradient variational Bayes) (Kingma and Welling, 2014; Rezende et al., 2014) to perform approximate maximum likelihood estimation:
$$
\begin{array}{l} \mathcal {L} (Y | X) = \mathbb {E} _ {q _ {z} (z | s, G (s))} [ \log p _ {n m t} (Y | X, z) ] - \\ \lambda D _ {K L} q _ {z} (z | s, G (s)) \| p (z) \tag {3} \\ \end{array}
$$
where  $\lambda$  weighs the KL divergence term and  $p(z) = \mathcal{N}(0,I)$ .
Encoder and Generator Tweaking Intuitively, the semantic space should consider the shared semantics between  $s$  and  $t$ . Ideally,  $s$  and  $t$  should be obtained from a shared model by processing  $X$  and  $Y$ , which is discussed in generative NMT (Shah and Barber, 2018; Zheng et al., 2020; Eikema and Aziz, 2019). In spired by this idea, we use the same NMT encoder to compute  $Enc(Y)$ , obtaining the "ground-truth target semantic"  $t = \frac{1}{M} \sum_{k=0}^{m} Enc(Y)_k \in R^d$ . As aforementioned, we do not directly use  $t$  for our semantic space, which is different from generative NMT. Instead, we only use  $t$  to enforce and regularize  $G(s)$  in training. Concretely, we train the cross-lingual generator  $G$  to restore  $t$  from  $s$  so that  $G(s) \approx t$ .
Figure 1: High-level view of NMT with a semantic draft. Note that the "dotted line" is only used in training.  $s$  and  $t$  represent the sentence semantics. The semantic draft  $z$  is sampled from the semantic space that is a parameterized space to model the Gaussian inference distribution  $q_{z}(z|s,G(s))$ , where  $G$  is our cross-lingual generator.  $\sigma$  linearly increases over the course of training so that the model learns to predict without  $t$ .  $\cos$  denotes the similarity between  $G(s)$  and  $t$  that we encourage  $G(s)\approx t$ . The variational decoder decomposes the sum of a draft and the encoding.
4.1 Inferring with Almost Free Draft
Costly Draft In traditionally generative NMT, based on a random target sentence, the inference mode or the process of translation generating makes an initial guess  $z_{init}$  from the semantic space or the variational distribution  $q_{z}(z|s,t_{random})$ , where  $s$  is computed by  $X$  and  $t_{random}$  is obtained from a random  $Y_{random}$ . Then, it can generate a possible translation  $Y'$  and its semantic  $t'$ . To obtain a good translation, based on the last translation, the inference mode can re-sample a better semantic from the semantic space and regenerate a new translation to maximize a lower bound on  $\log(p(X,Y))$  in the EM-like process. Readers can also refer to Algorithm 1 in GNMT (Shah and Barber, 2018) for more details.
Almost Free Draft Unlike traditionally generative NMT, we do not need to make an initial guess and also do not employ the EM-like process to sample  $z$  for inferring, which improves the inference efficiency. In our model,  $G(s)$ , which is the dummy target semantic, plays a prominent role that aims to approximate  $t$  instead of making an initial guess. Therefore, we do not have to make an initial guess, and we can also eliminate the whole EM-like process because  $z$  is not randomly sampled, which results in a one-shot sampling. Since  $G$  is a simple generator, sampling  $z$  from  $q_{z}(z|s,G(s))$  does not hurt the inference efficiency significantly and is almost free of cost.
4.2 Multilingual Encoder and Cross-lingual Generator
Approximation of  $t$  In Encoder and Generator Tweaking operation, we jointly train the encoder
and the cross-lingual generator  $G$  to make  $G(s)$  and  $t$  as similar as possible. Since we input parallel sentences to the encoder, the encoder is encouraged to search multilingual properties. Specifically, we notice that  $s \approx t$  potentially<sup>1</sup>, which is studied and reported in previous works of multilingual BERT empirically (Devlin et al., 2019; Karthikeyan et al., 2020; Wu and Dredze, 2019). Meanwhile, Soft-prototype (Wang et al., 2019) and multilingual NMT (Wu et al., 2016; Johnson et al., 2017) also explore this aspect in NMT scenario. We further introduce the cross-lingual generator  $G$  to tweak/finetune the property, observing the significant benefits of regularizing. Most importantly, with the cross-lingual generator  $G$ , the model can greedily gain a dummy  $t$  by  $G(s)$  so that the semantic draft can be sampled in a one-shot generative style without the EM-like process.
Potential of  $s$  and  $G(s)$  Besides, we are aware that only injecting  $s$  or  $G(s)$  without processing to the semantic space may also provide global information or the shared semantic for decoding because  $s \approx t$  and  $G(s) \approx t$  potentially. We will present an ablation study in one of our comprehensive experiments §6.5 Necessity of Semantic Space and Multilingual Encoder to show the significance of  $G$ , the semantic space and their combination.
Semantic in Encoder and Decoder On the other hand, compared to generative NMT, which employs an auxiliary network to help the semantic space by feeding parallel sentences, our method simply processes the parallel sentences to the NMT
There is a difference between  $s$  or  $t$  and the output of multilingual BERT. Specifically,  $s$  and  $t$  are sentence representations, whereas multilingual BERT outputs a sequence of the word representation.
encoder that is equivalent to the auxiliary network in generative NMT. In this way, there is no need to pass  $z$  to the encoder to model a joint probability  $P_{nmt}(X,Y,z) = p(z)p(X|z)p_{nmt}(Y|X,z)$ . Specifically, as discussed in VAE (Altieri and Duvenaud, 2015; Kingma and Ba, 2015; Bowman et al., 2016; Zhang et al., 2016), if  $z$  involves in the process of encoding,  $z$  can guide and regularize the encoder to consider the shared semantic. Therefore, generative NMT models the joint probability in training, encouraged to consider  $z$  in both the encoder and the decoder. However, in our model, we let the multilingual encoder consider the implicitly shared semantic itself, and we inject  $z$  into the decoder that is encouraged to consider the shared semantic.
Figure 2: Comparison between our model and previous models. The "dotted line" indicates the flow of global information.  $Z$  denotes Gaussian semantic space. Net denotes an auxiliary network.  $G$  is a generator.  $1st$  denotes first-pass decoding. soft denotes soft-prototype.
In Figure 2, we compare our framework with previous successful models: GNMT (Shah and Barber, 2018), Deliberation (Xia et al., 2017) and Softprototype (Wang et al., 2019). We observe some significant differences from the perspective of our design:
- vs GNMT 1) The semantic space is built upon the multilingual encoder and the cross-lingual generator in our model; 2) the semantic/global information is only used in the decoder.
- vs Deliberation The global information comes from semantic space instead of the first-pass decoding.
- vs Soft-prototype The global information is sampled from the semantic space instead of target prototypes.
- Additionally, we notice an optimization solution for the EM-like process. (Eikema and Aziz, 2019) study an approximating method to maximize the lower bound on  $\log(p(X,Y))$  by employing an auxiliary distribution with only using source  $s$ , which boosts the inference efficiency with a single call (without the EM-like process) to an argmax solver. Compared to their work, our model has three major differences: 1) our model depends on both  $s$  and  $G(s)$ ; 2) an auxiliary distribution is not necessary in our model; 3) we focus on the process of draft generating.
4.4 Optimization Challenges
Collapse of  $\mathbf{D}_{\mathbf{KL}}$  (Bowman et al., 2016) report the collapse of  $D_{KL}$  term in the objective function Eq.3. Following the instructions of (Bowman et al., 2016; Shah and Barber, 2018), we apply two common strategies:  $I)\lambda$  linearly increases from 0 to 1 over the initial 50k steps during training; 2) we randomly drop a constant of  $30\%$  words when encoding  $X$ .
Warm-up of Generator Training is somewhat tricky when using the cross-lingual generator  $G$ . We apply a weight  $\sigma \in [0,1]$  for  $G(s)$  and a weight  $1 - \sigma$  for  $t$ , as presented in Figure 1.  $\sigma$  linearly increases from 0 to 1 over 50k steps after  $\lambda = 1$ . By this strategy, the semantic space is encouraged to rely on  $t$  in warm-up. Significantly, it avoids that  $\cos(G(s),t)$  is close to 0 at the beginning of training. After warm-up, i.e.,  $G(s) \approx t$ , we use  $G(s)$  for the rest of training.
5 Experiment Settings
To be comparable, we train our model on language pairs  $\{French, German\} \leftrightarrow English$  and a relative low-resource language pair Romanian  $\leftrightarrow$  English which are commonly used in previous work (Shah and Barber, 2018; Vaswani et al., 2017; Bahdanau et al., 2015; Zheng et al., 2020). Concretely, we download parallel corpora  $\{French, German,\} \leftrightarrow English$  from WMT  $2014^{2}$  (Bojar et al., 2014). For Romanian  $\leftrightarrow$
$^{2}$ http://www.statmt.org/wmt14/translation-task.html
English, we retrieve parallel corpora from WMT 2016  $^{3}$  (Bojar et al., 2016). The preprocess is simple in our case that we only remove sentences with over 50-word length in our training datasets. Following standard evaluation, the model is evaluated on newstest2014 for {French, German}  $\leftrightarrow$  English and newstest2016 for Romanian  $\leftrightarrow$  English. Case-sensitive BLEU score is computed by multi-BLEU.perl $^{4}$  to report the performance. We also employ beam search with beam size 4 and length penalty 0.6.
5.2 Model Settings
We implement presented model on Tensorflow 2.0 (Abadi et al., 2016). To be comparable with other models and baselines, the NMT settings are identical to big-Transformer (Vaswani et al., 2017). Specifically, we set model dimension, word embedding, head, encoder layer, decoder layer and FFN filter to 1024, 1024, 16, 6, 6 and 4096. Adam optimizer (Kingma and Ba, 2015) is employed with parameters  $\beta_{1} = 0.9, \beta_{2} = 0.98$  and  $\epsilon = 10^{-9}$ . We use a dynamic learning rate over the course of NMT training (Smith, 2017; Vaswani et al., 2017) $^{5}$ . The dropout rate is set to  $rate = 0.1$ , and label smoothing is used with gamma  $= 0.1$  (Mezzini, 2018). Parallel corpora for one translation task (e.g., Romanian  $\leftrightarrow$  English) are concatenated to train BPE (Sennrich et al., 2016b) with a balance strategy (Lample and Conneau, 2019) that forms a shared vocabulary with 40,000 subtokens. For data feeding efficiency, each minibatch of similar-length sentences are padded to the same length and may have a different number of elements in each mini-batch, such that batch_size  $\times$  padded_length  $<= 3000$ .
5.3 Reimplementation and Reconfiguration
To be fair, we reimplement some models on our machine with the same mini-batch size. We compare the reimplemented results to the reported results on the same test set to ensure the difference less than  $5\%$  (or 1.5) in BLEU. Then, we can confirm the reimplementation and reconfiguration.
6 Performance
6.1 Translation Task