SlowGuess/ACL_2020_OCR huggingface dataset:
Xuanli He
{xuanli.hl,gholamreza.haffari}@monash.edu
Gholamreza Haffari
affari}  $@$  monash.edu
Mohammad Norouzi
mnorouzi@google.com
This paper introduces Dynamic Programming Encoding (DPE), a new segmentation algorithm for tokenizing sentences into subword units. We view the subword segmentation of output sentences as a latent variable that should be marginalized out for learning and inference. A mixed character-subword transformer is proposed, which enables exact log marginal likelihood estimation and exact MAP inference to find target segmentations with maximum posterior probability. DPE uses a lightweight mixed character-subword transformer as a means of pre-processing parallel data to segment output sentences using dynamic programming. Empirical results on machine translation suggest that DPE is effective for segmenting output sentences and can be combined with BPE dropout for stochastic segmentation of source sentences. DPE achieves an average improvement of 0.9 BLEU over BPE (Sennrich et al., 2016) and an average improvement of 0.55 BLEU over BPE dropout (Provilkov et al., 2019) on several WMT datasets including English  $\leftrightarrow$  (German, Romanian, Estonian, Finnish, Hungarian).
The segmentation of rare words into subword units (Sennrich et al., 2016; Wu et al., 2016) has become a critical component of neural machine translation (Vaswani et al., 2017) and natural language understanding (Devlin et al., 2019). Subword units enable open vocabulary text processing with a negligible pre-processing cost and help maintain a desirable balance between the vocabulary size and decoding speed. Since subword vocabularies are built in an unsupervised manner (Sennrich et al., 2016; Wu et al., 2016), they are easily applicable to any language.
Given a fixed vocabulary of subword units, rare words can be segmented into a sequence of
subword units in different ways. For instance, "un+conscious" and "uncon+scious" are both suitable segmentations for the word "unconscious". This paper studies the impact of subword segmentation on neural machine translation, given a fixed subword vocabulary, and presents a new algorithm called Dynamic Programming Encoding (DPE).
We identify three families of subword segmentation algorithms in neural machine translation:
1. Greedy algorithms: Wu et al. (2016) segment words by recursively selecting the longest subword prefix. Sennrich et al. (2016) recursively combine adjacent word fragments that co-occur most frequently, starting from characters.
2. Stochastic algorithms (Kudo, 2018; Provilkov et al., 2019) draw multiple segmentations for source and target sequences resorting to randomization to improve robustness and generalization of translation models.
3. Dynamic programming algorithms, studied here, enable exact marginalization of subword segmentations for certain sequence models.
We view the subword segmentation of output sentences in machine translation as a latent variable that should be marginalized out to obtain the probability of the output sentence given the input. On the other hand, the segmentation of source sentences can be thought of as input features and can be randomized as a form of data augmentation to improve translation robustness and generalization. Unlike previous work, we recommend using two distinct segmentation algorithms for tokenizing source and target sentences: stochastic segmentation for source and dynamic programming for target sentences.
We present a new family of mixed character-subword transformers, for which simple dynamic programming algorithms exist for exact marginalization and MAP inference of subword segmenta
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3042-3051 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tions. The time complexity of the dynamic programming algorithms is  $O(TV)$ , where  $T$  is the length of the target sentence in characters, and  $V$  is the size of the subword vocabulary. By comparison, even computing the conditional probabilities of subword units in an autoregressive model requires  $O(TV)$  to estimate the normalizing constant of the categorical distributions. Thus, our dynamic programming algorithm does not incur additional asymptotic costs. We use a lightweight mixed character-subword transformer as a means to pre-process translation datasets to segment output sentences using DPE for MAP inference.
The performance of a standard subword transformer (Vaswani et al., 2017) trained on WMT datasets tokenized using DPE is compared against Byte Pair Encoding (BPE) (Sennrich et al., 2016) and BPE dropout (Provilkov et al., 2019). Empirical results on English  $\leftrightarrow$  (German, Romanian, Estonian, Finnish, Hungarian) suggest that stochastic subword segmentation is effective for tokenizing source sentences, whereas deterministic DPE is superior for segmenting target sentences. DPE achieves an average improvement of 0.9 BLEU over greedy BPE (Sennrich et al., 2016) and an average improvement of 0.55 BLEU over stochastic BPE dropout (Provilkov et al., 2019) $^{1}$ .
Neural networks have revolutionized machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Cho et al., 2014). Early neural machine translation (NMT) systems used words as the atomic element of sentences. They used vocabularies with tens of thousands words, resulting in prohibitive training and inference complexity. While learning can be sped up using sampling techniques (Jean et al., 2015), word based NMT models have a difficult time handling rare words, especially in morphologically rich languages such as Romanian, Estonian, and Finnish. The size of the word vocabulary should increase dramatically to capture the compositionality of morphemes in such languages.
More recently, many NMT models have been developed based on characters and a combination of characters and words (Ling et al., 2015; Luong and Manning, 2016; Vylomova et al., 2017; Lee et al., 2017; Cherry et al., 2018). Fully character based models (Lee et al., 2017; Cherry et al., 2018) demonstrate a significant improvement over word
based models on morphologically rich languages. Nevertheless, owing to the lack of morphological information, deeper models are often required to obtain a good translation quality. Moreover, elongated sequences brought by a character representation drastically increases the inference latency.
In order to maintain a good balance between the vocabulary size and decoding speed, subword units are introduced in NMT (Sennrich et al., 2016; Wu et al., 2016). These segmentation approaches are data-driven and unsupervised. Therefore, with a negligible pre-processing overhead, subword models can be applied to any NLP task (Vaswani et al., 2017; Devlin et al., 2019). Meanwhile, since subword vocabularies are generated based on word frequencies, only the rare words are split into subword units and common words remain intact.
Previous work (Chan et al., 2016; Kudo, 2018) has explored the idea of using stochastic subword segmentation with multiple subword candidates to approximate the log marginal likelihood. Kudo (2018) observed marginal gains in translation quality at the cost of introducing additional hyperparameters and complex sampling procedures. We utilize BPE dropout (Provilkov et al., 2019), a simple stochastic segmentation algorithm for tokenizing source sentences.
Dynamic programming has been used to marginalize out latent segmentations for speech recognition (Wang et al., 2017), showing a consistent improvement over greedy segmentation methods. In addition, dynamic programming has been successfully applied to learning sequence models by optimizing edit distance (Sabour et al., 2018) and aligning source and target sequences (Chan et al., 2020; Sahara et al., 2020). We show the effectiveness of dynamic programming for segmenting output sentences in NMT using a mixed character-transformer in a pre-processing step.
3 Latent Subword Segmentation
Let  $\pmb{x}$  denote a source sentence and  $\pmb{y} = (y_{1},\dots ,y_{T})$  denote a target sentence comprising  $T$  characters. The goal of machine translation is to learn a conditional distribution  $p(\pmb{y} \mid \pmb{x})$  from a large corpus of source-target sentences. State-of-the-art neural machine translation systems make use of a dictionary of subword units to tokenize the target sentences in a more succinct way as a sequence of  $M \leq T$  subword units. Given a subword vocabulary, there are multiple ways to segment a
<sup>1</sup>code and corpora: https://github.com/xlhex/dpe
rare word into a sequence of subwords (see Figure 1). The common practice in neural machine translation considers subword segmentation as a pre-process and uses greedy algorithms to segment each word across a translation corpus in a consistent way. This paper aims to find optimal subword segmentations for the task of machine translation.
Let  $z = (z_{1},..,z_{M + 1})$  denote a sequence of character indices  $0 = z_{1} < z_{2} < \ldots < z_{M} < z_{M + 1} = T$  in an ascending order, defining the boundary of  $M$  subword segments  $\{\pmb{y}_{z_i,z_{i + 1}}\}_{i = 1}^M$ . Let  $\pmb{y}_{a,b}\equiv [y_{a + 1},\dots ,y_b]$  denote a subword that spans the segment between  $(a + 1)^{\mathrm{th}}$  and  $b^{\mathrm{th}}$  characters, including the boundary characters. For example, given a subword dictionary  $\{\textsf{'c', 'a', 't', 'at', 'ca'}\}$ , the word 'cat' may be segmented using  $z = (0,1,3)$  as ('c', 'at'), or using  $z = (0,2,3)$  as ('ca', 't'), or using  $z = (0,1,2,3)$  as ('c', 'a', 't'). The segmentation  $z = (0,3)$  is not valid since 'cat' does not appear in the subword vocabulary.
Autoregressive language models create a categorical distribution over the subword vocabulary at every subword position and represent the log-probability of a subword sequence using chain rule,
$$
\begin{array}{l} \log p (\boldsymbol {y}, z) = \\ \sum_ {i = 1} ^ {| z |} \log p \left(\boldsymbol {y} _ {z _ {i}, z _ {i + 1}} \mid \boldsymbol {y} _ {z _ {1}, z _ {2}}, \dots , \boldsymbol {y} _ {z _ {i - 1}, z _ {i}}\right). \tag {1} \\ \end{array}
$$
Note that we suppress the dependence of  $p$  on  $\mathbf{x}$  to reduce notational clutter. Most neural machine translation approaches assume that  $\mathbf{z}$  is a deterministic function of  $\mathbf{y}$  and implicitly assume that  $\log p(\mathbf{y},\mathbf{z})\approx \log p(\mathbf{y})$
We consider a subword segmentation  $\mathbf{z}$  as a latent variable and let each value of  $\mathbf{z} \in \mathcal{Z}_y$ , in the set of segmentations compatible with  $\mathbf{y}$ , contribute its share to  $p(\mathbf{y})$  according to  $p(\mathbf{y}) = \sum_{\mathbf{z}} p(\mathbf{y}, \mathbf{z})$ ,
$$
\begin{array}{l} \log p (\boldsymbol {y}) = \\ \log \sum_ {\boldsymbol {z} \in \mathcal {Z} _ {y}} \exp \sum_ {i = 1} ^ {| z |} \log p \left(\boldsymbol {y} _ {z _ {i}, z _ {i + 1}} \mid \dots , \boldsymbol {y} _ {z _ {i - 1}, z _ {i}}\right). \tag {2} \\ \end{array}
$$
Note that each particular subword segmentation  $z \in \mathcal{Z}_y$  provides a lower bound on the log marginal likelihood  $\log p(\pmb{y}) \geq \log p(\pmb{y}, \pmb{z})$ . Hence, optimizing (1) for a greedily selected segmentation can be justified as a lower bound on (2). That said, optimizing (2) directly is more desirable. Unfortunately, exact marginalization over all segmentations is computationally prohibitive in a combinatorially large space  $\mathcal{Z}_y$ , especially because the
Figure 1: An illustration of different ways of segmenting 'unconscious' into subword units.
probability of each subword depends on the segmentation of its conditioning context. In the next section, we discuss a sequence model in which the segmentation of the conditioning context does not influence the probability of the next subword. We describe an efficient Dynamic Programming algorithm to exactly marginalize out all possible subword segmentations in this model.
4 A Mixed Character-Subword Transformer
We propose a mixed character-subword transformer architecture, which enables one to marginalize out latent subword segmentations exactly using dynamic programming (see Figure 2). Our key insight is to let the transformer architecture process the inputs and the conditioning context based on characters to remain oblivious to the specific choice of subword segmentation in the conditioning context and enable exact marginalization. That said, the output of the transformer is based on subword units and at every position it creates a categorical distribution over the subword vocabulary. More precisely, when generating a subword  $y_{z_i,z_{i + 1}}$ , the model processes the conditioning context  $(y_{z_1},\ldots ,y_{z_i})$  based solely on characters using,
$$
\log p (\boldsymbol {y}, \boldsymbol {z}) = \sum_ {i = 1} ^ {| z |} \log p \left(\boldsymbol {y} _ {z _ {i}, z _ {i + 1}} \mid y _ {z _ {1}}, \dots , y _ {z _ {i}}\right), \tag {3}
$$
where the dependence of  $p$  on  $\pmb{x}$  is suppressed to reduce notational clutter.
Given a fixed subword vocabulary denoted  $V$ , at every character position  $t$  within  $\pmb{y}$ , the mixed character-subword model induces a distribution over the next subword  $w \in V$  based on,
$$
p (w \mid y _ {1},.., y _ {t}) = \frac {\exp (f (y _ {1} , . . . , y _ {t}) ^ {\top} e (w))}{\sum_ {w ^ {\prime} \in V} \exp (f (y _ {1} , . . . , y _ {t}) ^ {\top} e (w ^ {\prime}))}
$$
where  $f(\cdot)$  processes the conditioning context using a Transformer, and  $e(\cdot)$  represents the weights of the softmax layer.
Algorithm 1 Dynamic Programming (DP) for Exact Marginalization
Input:  $y$  is a sequence of  $T$  characters,  $V$  is a subword vocabulary,  $m$  is the maximum subword length. Output:  $\log p(y)$  marginalizing out different subword segmentations.
1:  $\alpha_0\gets 0$
2: for  $k = 1$  to  $T$  do
3:  $\alpha_{k} \gets \log \sum_{j=k-m}^{k-1} \mathbb{1}[\pmb{y}_{j,k} \in V] \exp\left(\alpha_{j} + \log P_{\theta}(\pmb{y}_{j,k} | y_{1},.., y_{j})\right)$
4: end for
5: return  $\alpha_{T}$  the marginal probability  $\log p(\pmb {y}) = \log \sum_{\pmb {z}\in \mathcal{Z}_y}p(\pmb {y},\pmb {z})$
Figure 2: An illustration of the mixed character-subword Transformer. The input is a list of characters, whereas the output is a sequence of subwords.
As depicted in in Figure 2, the mixed character-subword Transformer consumes characters as input generates subwords as output. This figure only shows the decoder architecture, since as the encoder that processes  $\pmb{x}$  is a standard subword Transformer. Once a subword  $w$  is emitted at time step  $t$ , the characters of the subword  $w$  are fed into the decoder for time steps  $t + 1$  to  $t + |w|$ , and the next subword is generated at time step  $t + |w|$ , conditioned on all of the previously generated characters.
4.1 Optimization
The training objective for our latent segmentation translation model is  $\sum_{(\boldsymbol{x},\boldsymbol{y})\in \mathcal{D}}\log P_{\theta}(\boldsymbol{y}|\boldsymbol{x})$  where  $\mathcal{D}$  is the training corpus consisting of parallel bilingual sentence pairs. Maximizing the training objective requires marginalization and the computation of the gradient of the log marginal likelihood.
Exact Marginalization. Under our model, the probability of a subword only depends on the character-based encoding of the conditioning context and not its segmentation, as in (3). This means
that we can compute the log marginal likelihood for a single example  $\pmb{y}$ , exactly, using the Dynamic Programming algorithm shown in Algorithm 1. The core of the algorithm is line 3, where the probability of the prefix string  $y_{0,k}$  is computed by summing terms corresponding to different segmentations. Each term consists of the product of the probability of a subword  $y_{j,k}$  times the probability of its conditioning context  $(y_1,\dots ,y_j)$ . The running time of the algorithm is  $\mathcal{O}(mT)$ , where  $T$  is the length of the string, and  $m$  is the size of the longest subword unit in the vocabulary.
Gradient Computation. We use automatic differentiation in PyTorch to backpropagate through the dynamic program in Algorithm 1 and compute its gradient. Compared to a standard Transformer decoder, our mixed character-subword Transformer is 8x slower with a larger memory footprint, due to computation involved in the DP algorithm and large sequence length in characters. To address these issues, we reduce the number of transformer layers from 6 to 4, and accumulate 16 consecutive gradients before one update.
4.2 Segmenting Target Sentences
Once the mixed character-subword transformer is trained, it is used to segment the target side of a bilingual corpus. We randomize the subword segmentation of source sentences using BPE dropout (Provilkov et al., 2019). Conditional on the source sentence, we use Algorithm 2, called Dynamic Programming Encoding (DPE) to find a segmentation of the target sentence with highest posterior probability. This algorithm is similar to the marginalization algorithm, where we use a max operation instead of log-sum-exp. The mixed character-subword transformer is used only for tokenization, and a standard subword transformer is trained on the segmented sentences. For inference using beam search, the mixed character-subword transformer is not needed.
Algorithm 2 Dynamic Programming Encoding (DPE) for Subword Segmentation
Input:  $y$  is a sequence of  $T$  characters,  $V$  is a subword vocabulary,  $m$  is the maximum subword length. Output: Segmentation  $z$  with highest posterior probability.
$$
\beta_ {k} \leftarrow \max  _ {\left\{j \in [ k - m, k - 1 ] \mid \mathbf {y} _ {j, k} \in V \right\}} \beta_ {j} + \log P _ {\theta} (\mathbf {y} _ {j, k} | y _ {1},.., y _ {j})
$$
$$
b _ {k} \leftarrow \operatorname {a r g m a x} _ {\left\{j \in [ k - m, k - 1 ] \mid \mathbf {y} _ {j, k} \in V \right\}} \beta_ {j} + \log P _ {\theta} (\mathbf {y} _ {j, k} | y _ {1},.., y _ {j})
$$
$$
z \leftarrow \operatorname {b a c k t r a c e} \left(b _ {1},.., b _ {T}\right)
$$
$\triangleright$  backtrace the best segmentation using  $\pmb{b}$
Number of sentences	Vocab size
train	dev	test
En-Hu	WMT09	0.6M	2,051	2,525	32K
En-De	WMT14	4.2M	3000	3003	32K
En-Fi	WMT15	1.7M	1,500	1,370	32K
En-Ro	WMT16	0.6M	1,999	1,999	32K
En-Et	WMT18	1.9M	2,000	2,000	32K
Table 1: Statistics of the corpora.
Dataset We use WMT09 for En-Hu, WMT14 for En-De, WMT15 for En-Fi, WMT16 for En-Ro and WMT18 for En-Et. We utilize Moses toolkit² to pre-process all corpora, and preserve the true case of the text. Unlike Lee et al. (2018), we retain diacritics for En-Ro to retain the morphological richness. We use all of the sentence pairs where the length of either side is less than 80 tokens for training. Byte pair encoding (BPE) (Sennrich et al., 2016) is applied to all language pairs to construct a subword vocabulary and provide a baseline segmentation algorithm. The statistics of all corpora is summarized in Table 1.
Training with BPE Dropout. We apply BPE dropout (Provilkov et al., 2019) to each mini-batch. For each complete word, during the BPE merge operation, we randomly drop a particular merge with a probability of 0.05. This value worked the best in our experiments. A word can be split into different segmentations at the training stage, which helps improve the BPE baseline.
DPE Segmentation. DPE can be used for target sentences, but its use for source sentences is not justified as source segmentations should not be marginalized out. Accordingly, we use BPE dropout for segmenting source sentences. That is,
Figure 3: The workflow of the proposed DPE approach.
we train a mixed character-subword transformer to marginalize out the latent segmentations of a target sentence, given a randomized segmentation of the source sentence by BPE dropout. After the mixed character-subword transformer is trained, it is used to segment the target sentences as describe in section 4.2 for tokenization.
As summarized in Figure 3, we first train a mixed character-subword transformer with dynamic programming. Then, this model is frozen and used for DPE segmentation of target sentences. Finally, a standard subword transformer is trained on source sentences segmented by BPE dropout and target sentences segmented by DPE. The mixed character-subword transformer is not needed for translation inference.
Transformer Architectures. We use transformer models to train three translation models on BPE, BPE dropout, and DPE corpora. We make use of transformer base for all of the experiments.
Table 2 shows the main results. First, we see that BPE dropout consistently outperforms BPE across language pairs. In Table 2, the column labeled to  $\Delta_{1}$  shows the improvement of BPE dropout over
$^{2}$ https://github.com/moses-smt/mosesdecoder
Method	BPE	BPE dropout	This paper
Source segmentation	BPE	BPE dropout	BPE dropout
Target segmentation	BPE	BPE dropout	DPE
En→De	27.11	27.27	+0.16	27.61	+0.34
En→Ro	27.90	28.07	+0.17	28.66	+0.59
En→Et	17.64	18.20	+0.56	18.80	+0.60
En→Fi	15.88	16.18	+0.30	16.89	+0.71
En→Hu	12.80	12.94	+0.14	13.36	+0.42
De→En	30.82	30.85	+0.03	31.21	+0.36
Ro→En	31.67	32.56	+0.89	32.99	+0.43
Et→En	23.13	23.65	+0.52	24.62	+0.97
Fi→En	19.10	19.34	+0.24	19.87	+0.53
Hu→En	16.14	16.61	+0.47	17.05	+0.44
Average	22.22	22.57	+0.35	23.12	+0.55
Table 2: Average test BLEU scores (averaged over 3 independent runs) for 3 segmentation algorithms (BPE (Sennrich et al., 2016), BPE dropout (Provilkov et al., 2019) and our DPE algorithm) on 10 different WMT datasets. For each language pair, all of the segmentation techniques use the same subword dictionary with 32K tokens shared between source and target languages.  $\Delta_{1}$  shows the improvement of BPE dropout compared to BPE, and  $\Delta_{2}$  shows further improvement of our proposed DPE method compared to BPE dropout.
BPE source:Die G@@ le@@ is@@ anlage war so ausgestattet , dass dort elektr@@ isch betrie@@ bene Volkswagen eingesetzterwenden konnten .DPE target:The railway system was equipped in such a way that electrical@@ ly powered cart@@ s could be used on it .BPE target:The railway system was equipped in such a way that elect@@ r@@ ically powered car@@ ts could be used on it .
BPE source:Normalerweise wird Kok@@ ain in kleineren Mengen und nicht durch Tunnel geschm@@ ug@@ gelt .DPE target:Normal@@ ly c@@ oca@@ ine is sm@@ ugg@@ led in smaller quantities and not through tunnel@@ s .BPE target:Norm@@ ally co@@ c@@ aine is sm@@ ugg@@ led in smaller quantities and not through tun@@ nels .
Table 3: Two examples of segmentation of English sentences given German inputs.
BPE. This gain can be attributed to the robustness of the NMT model to the segmentation error on the source side, as our analysis in Section 5.3 will confirm. Second, we observe further gains resulted from DPE compared to BPE dropout. The column labeled  $\Delta_{2}$  shows the improvement of DPE over BPE dropout. DPE provides an average improvement of 0.55 BLEU over BPE dropout and BPE dropout provides an average improvement of 0.35 BLEU over BPE. As our proposal uses BPE dropout for segmenting the source, we attribute our BLEU score improvements to a better segmentation of the target language with DPE. Finally, compared to BPE for segmenting the source and target, our proposed segmentation method results in large improvements in the translation quality, up
to 1.49 BLEU score improvements in  $\mathrm{Et}\rightarrow \mathrm{En}$
5.2 Segmentation Examples
Table 3 shows examples of target sentences segmented using DPE and BPE and the corresponding source sentences. In addition, Table 4 presents the top 50 most common English words that result in a disagreement between BPE and DPE segmentations based on the  $\mathrm{Et} \rightarrow \mathrm{En}$  corpus. For DPE, for each word, we consider all segmentations produced and show the segmentation that attains the highest frequency of usage in Table 4. As can be observed, DPE produces more linguistically plausible morpheme-based subwords compared to BPE. For instance, BPE segments "carts" into "car" + "ts", as both "car" and "ts" are common subwords and