SlowGuess/NAACL_2022_OCR huggingface dataset:
Kexun Zhang $^{1*}$ , Rui Wang $^{2†}$ , Xu Tan $^{2}$ , Junliang Guo $^{2}$ , Yi Ren $^{1}$ , Tao Qin $^{2}$ , Tie-Yan Liu $^{2}$
<sup>1</sup>Zhejiang University, <sup>2</sup>Microsoft Research Asia
1{kexunz, rayeren}@zju.edu.cn
2{ruiwa,xuta,junliangguo,taoqin,tyliu}@microsoft.com
It is difficult for non-autoregressive translation (NAT) models to capture the multi-modal distribution of target translations due to their conditional independence assumption, which is known as the "multi-modality problem", including the lexical multi-modality and the syntactic multi-modality. While the first one has been well studied, the syntactic multi-modality brings severe challenge to the standard cross entropy (XE) loss in NAT and is under studied. In this paper, we conduct a systematic study on the syntactic multi-modality problem. Specifically, we decompose it into short- and long-range syntactic multi-modalities and evaluate several recent NAT algorithms with advanced loss functions on both carefully designed synthesized datasets and real datasets. We find that the Connectionist Temporal Classification (CTC) loss and the Order-Agnostic Cross Entropy (OAXE) loss can better handle short- and long-range syntactic multi-modalities respectively. Furthermore, we take the best of both and design a new loss function to better handle the complicated syntactic multi-modality in real-world datasets. To facilitate practical usage, we provide a guide to use different loss functions for different kinds of syntactic multimodality.
Traditional Neural Machine Translation (NMT) models predict each target token conditioned on previous generated tokens in an autoregressive way (Vaswani et al., 2017), resulting in high latency in inference. Non-Autoregressive Translation (NAT) models generate all the target tokens in parallel (Gu et al., 2018), significantly reducing inference latency. A disadvantage of NAT is that it suffers from the multi-modality problem (Gu et al., 2018) when a source sentence corresponds to multiple correct translations (Ott et al., 2018).
There are two types of multi-modalities: the lexical one and the syntactic one. The former one has been adequately studied (Gu et al., 2018; Zhou et al., 2020; Ding et al., 2021), while the latter one brings severe challenges to the widely used cross entropy (XE) loss in NAT. With standard XE loss, the generated tokens are required to be strictly aligned with ground truth tokens in the same positions, which fails to provide positive feedback for correctly predicted words on different positions as shown in Fig. 1a. Therefore, advanced loss functions are introduced to provide better feedback for NAT training: Connectionist Temporal Classification (CTC) loss (Libovický and Helcl, 2018) considers all possible monotonic alignments between a generated sequence and the ground truth; Aligned Cross-Entropy (AXE) loss (Ghazvininejad et al., 2020) selects the best monotonic alignment; and Order-Agnostic cross entropy (OAXE) loss (Du et al., 2021) calculates the XE loss with the best alignment based on maximum bipartite matching algorithm.
Even if with those advanced loss functions, we find they do not perform consistently across datasets and languages. In addition, diverse grammar rules in natural language (Comrie, 1989) implies the existence of different kinds of syntactic multi-modality. Inspired by Odlin (2008); Jing and Liu (2015); Liu (2007, 2010), we categorize the syntactic multi-modality into two sub types: the long-range and short-range ones. The long-range multi-modality is mainly caused by long-range word order diversity (e.g., an adverbial of place may appear at the beginning or the end of a sentence). The short-range multi-modality is mainly caused by short-range word order diversity (e.g., an adverb may appear either in front of or behind the corresponding verb) and optional words (e.g., in some languages, determiners and prepositions may be optional (Ott et al., 2018)). Based on the above categorization of syntactic multi-modality, we fur
*This work was conducted at Microsoft Research Asia.
Human Language Technologies, pages 1747 - 1757
ther ask two research questions: (1) Which kinds of syntactic multi-modality do these loss functions excel at respectively? (2) How to better address this problem by taking advantage of different loss functions?
In this paper, we conduct a systematic study to answer these questions:
- Since the short-range and long-range syntactic multi-modalities are usually entangled together in real-world datasets, we first design synthesized datasets to decouple them to better evaluate existing NAT algorithms (§3). We find that the CTC loss (Libovický and Helcl, 2018) can better handle the short-range syntactic multi-modality while the OAXE loss (Du et al., 2021) is good at the long-range one. Though carefully designed, the synthesized datasets are still different from the real-world datasets. Accordingly, we further conduct analyses on real-world datasets (§4), which also show consistent findings with that in synthesized datasets.
- We design a new loss function that takes the best of both CTC and OAXE, and performs better to handle the short- and long-range syntactic multimodilities simultaneously (§5), as verified by experiments on benchmark datasets including WMT14 EN-DE, WMT17 EN-FI, and WMT14 EN-RU. Moreover, we further provide a practical guide to use different loss functions for different kinds of syntactic multi-modality (§5).
Non-Autoregressive Translation Given the source sentence  $x = (x_{1},x_{2},\dots,x_{T_{x}})$ , traditional NMT model generates the target sentence  $y = (y_{1},y_{2},\dots,y_{T_{y}})$  from left to right and token by token:  $P(y|x) = \prod_{t=1}^{T_{y}}P(y_{t}|y_{<t},x;\theta_{\mathrm{enc}},\theta_{\mathrm{dec}})$ , where  $y_{<t}$  indicates the target tokens generated before the  $t$ -th timestep,  $T_{x}$  and  $T_{y}$  denote the length of source and target sentence,  $\theta_{\mathrm{enc}}$  and  $\theta_{\mathrm{dec}}$  denote the encoder and decoder parameters respectively. This autoregressive way suffers from high latency during inference. Non-Autoregressive Translation (NAT) (Gu et al., 2018) is proposed to reduce the inference time by generating the whole sequence in parallel,  $P(y|x) = P(T_{y}|x) \cdot \prod_{t=1}^{T_{y}}P(y_{t}|x;\theta_{\mathrm{enc}},\theta_{\mathrm{dec}})$ , where  $P(T_{y}|x)$  indicates the length prediction function. While the inference speed is boosted, the translation accuracy is
(a) XE
(b) AXE
(d) OAXE
Figure 1: The illustration of different loss functions, where "GT" stands for ground truth, "PRED" stands for predicted sequence, the green check indicates that credit is provided to the token.
(c) CTC, where solid, dash, and dot dash lines illustrate three possible alignments respectively.
sacrificed due to that target tokens are generated conditional independently.
Multi-Modality Problem The multi-modality problem (Gu et al., 2018; Zhou et al., 2020) indicates that one source sentence may have multiple correct target translations, which brings challenges to NAT models as they generate each target token independently. Specifically, we categorize the multi-modality problem into two sub-problems, i.e., lexical and syntactic multi-modalities. The lexical multi-modality indicates that a source token can be translated into different target synonym tokens (i.e., "thank you" in English can be translated into both "Danke" or "Vielen Dank" in German), while the syntactic multi-modality indicates the inconsistency of word-orders (e.g., an adverb may appear either in front of or behind the corresponding verb) and the existence of optional words between source and target languages (e.g., in some languages, determiners and prepositions may be optional) (Ott et al., 2018). The lexical multi-modality problem has been adequately studied in recent works. Sequence-level knowledge distillation (Gu et al., 2018; Zhou et al., 2020) is shown capable to reduce the lexical diversity of the dataset and thus alleviate the problem. Some works also introduce extra loss
functions such as KL-divergence (Ding et al., 2021) and bag-of-ngram (Shao et al., 2020) to alleviate the lexical multi-modality problem.
On the contrary, there still lacks a systematic study on the syntactic multi-modality problem. Generally, it is difficult to solve this problem because the order and optional words vary across different languages. For example, the word order of Russian is quite flexible (Kallestinova, 2007), thus the syntactic multi-modality may exist more frequently in Russian corpora. In contrast, the structure of English sentences is mostly subject-verb-object (SVO) (Givón, 1983), which results in less variation on word order. In this paper, we categorize the syntactic multi-modality problem into short-range and long-range instances, and provide detailed analyses accordingly.
Loss Functions in NAT Standard cross-entropy (XE) loss requires the predicted tokens to be strictly aligned with ground truth tokens, which fails to deal with the syntactic multi-modality problem. Different loss functions are proposed to solve the problem, and here we consider some most recent works. The CTC loss sums XE losses of all possible monotonic alignments and has been widely used in speech recognition (Graves et al., 2006, 2013), and the effectiveness of the CTC loss in NAT has been validated (Libovický and Helcl, 2018; Gu and Kong, 2021). AXE (Ghazvininejad et al., 2020) selects the monotonic alignment between the predicted sequence and the ground truth with the minimum XE loss. OAXE (Du et al., 2021) further relaxes the position constraint and only considers the best alignment. The illustration for each loss function is provided in Fig. 1. Though effective in different datasets, these works ignore fine-grain features of the multi-modality problem such as short/long syntactic multi-modalities. In this work, we analyse the performance of these loss functions in different syntactic scenarios, and provide a practical guide to use appropriate loss functions for different kinds of syntactic multi-modality.
3 Analyses on Synthesized Datasets
To make fine-grained analyses on the syntactic multi-modality problem, we first categorize it into long-range and short-range types, where the long-range one is mainly caused by long-range word order diversity, and the short-range one is mainly caused by short-range word order diversity and op
The extremely large dog eats the small pie ravenously
Figure 2: An illustration of generating a syntax tree for a source sentence. In the first iteration, "Sen" consists of ("NP", "VP") as the solid lines. In the second iteration, "NP" consists of ("DT", "RB", "JJ", "N") and "VP" consists of ("V", "NP", "RB") as the dash lines. In the third iteration, "NP" consists of ("DT", "JJ", "N") as the dot-and-dash lines.
tional words. Then, we would like to evaluate the accuracy of different losses on different types of syntactic multi-modality. However, in real-world corpora, the different types are usually entangled, making it difficult to control and analyse one aspect without changing the other. Thus, we construct synthesized datasets based on phrase structure rules (Chomsky, 1959) to manually control the degree of syntactic multi-modality in different aspects, and evaluate the performance of different existing techniques.
3.1 Synthesized Datasets
We first employ phrase structure rules (Chomsky, 1959) to synthesize the source sentences, where the rules are based on the syntax of languages. Considering that translation can be decomposed to word reordering and word translation (Bangalore and Riccardi, 2001; Sudoh et al., 2011), we then "translate" the synthesized source sentences to synthesized target sentences in two steps: 1) word reordering by changing its syntax tree; 2) and word translation by substituting the source words into target words.
Source Sentence Synthesis. We first generate the syntax tree of the source sentence. Specifically, we use the notations of the constituents in syntax tree according to the Penn Treebank syntactic and part of speech (POS) tag sets<sup>1</sup> (Marcus et al., 1993), and generate the syntax tree of a source sentence as following (Rosenbaum, 1967):
$$
\bullet \mathrm {S e n} \rightarrow \mathrm {N P} \mathrm {V P},
$$
$^{1}$  "Sen": sentence; "NP": noun phrase; "VP": verb phrase; "DT": determiner; "JJ": adjective; "RB": adverb; "N": noun; "V": verb.
Figure 3: An illustration of "translation", where the constituent order of "Sen" is changed to "VP NP" with probability  $1 - P^{lo}$ , the constituent order of "VP" is changed to "RB V NP" with probability  $1 - P_{1}^{so} - P_{2}^{so}$ , and the circled "DT" is removed with probability  $P^{op}$ . Meanwhile, the numbers in the source sentence are replaced with the ones in the target sentence based on mappings.
-  $\mathrm{NP} \rightarrow (\mathrm{DT})(\mathrm{RB})^{*}(\mathrm{JJ})^{*}\mathrm{N},$
-  $\mathrm{VP} \rightarrow \mathrm{V}$  (NP)  $(\mathrm{RB})^{*}$ ,
where the constituent on the left side of the arrow consists of the constituents on the right side in sequence, “(\cdot)” means that the constituent is optional, and “ $(\cdot)^{*}$ ” denotes that the constituent is not only optional but can also be repetitive. For each sentence, we start with a single constituent Sen and iteratively decompose “Sen”, “NP”, and “VP” according to the rules until all the constituents are decomposed to “DT”, “JJ”, “RB”, “V”, and “N”. An illustration of generating a syntax tree is depicted in Fig. 2. To synthesize the source sentence according to the syntax tree, we use numbers as the words in the synthesized source sentences and use different ranges of numbers to represent words with different POS, where the details of the ranges are provided in Appendix A. Then, a number is randomly sampled in the corresponding range for each word in the syntax tree.
Word Reordering. To introduce syntactic multimodality, we consider multiple possible rules for "Sen", "NP", and "VP" in the target sentences. Dependency distance is defined as the linear distance between two words with syntactical relationship (Liu et al., 2017), which can be used as a guide to select typical rules to introduce long- and short-range word order diversity. Specifically, we consider three options: 1) The word order of "Sen" is with probability  $P^{lo}$  to be the same with the source sentence (i.e., NP VP) and with probability  $\left(1 - P^{lo}\right)$  to swap the "NP" and "VP" (i.e., VP NP), which has long dependency distance and represents for the long-range word order; 2) For the word order in "VP", it is considered to be the same with the source sentence with probability  $P_{1}^{so}$ ,
Probability	Default	Effect
Plo	1	long-range word order
Pso1	1	short-range word order
Pso2	0	short-range word order
Pop	0	optional words
Table 1: Default values of the probabilities to adjust the syntactic multi-modality.
place "RB" between "V" and "NP" with probability  $P_{2}^{so}$ , and place "RB" before "V" with probability  $(1 - P_{1}^{so} - P_{2}^{so})$ , which has short dependency distance and represents for the short-range word order; 3) To introduce the syntactic multi-modality of optional words, we change the existence of "DT" in each "NP" of the source sentence with probability  $P^{op}$  (i.e., remove "DT" if it exists in the source sentence and add "DT" if it does not exist in the source sentence).
Word Translation. Same as in the source sentences, we use different range of numbers to represent words with different POS in target sentences. To do the word translation, we first randomly build mappings between the source and target words with different POS respectively. Since we focus on studying the syntactic multi-modality, we consider each source word is mapped to a single target word to eliminate the lexical multi-modality. Then, we replace the words in the source sentence based on the mappings to generate the target sentence. An illustration of "translation" is shown in Fig. 3.
3.2 Experiments and Analyses
We conduct experiments to compare existing loss functions on different kinds of syntactic multi-
modality on the synthesized datasets, by changing the probabilities (i.e.,  $P^{op}$ ,  $P_{1}^{so}$ ,  $P_{2}^{so}$ , and  $P^{lo}$ ) as listed in Table 1. In the following, we first provide the experimental settings, then show the results on the long-range and short-range syntactic multimodalities, and finally conclude the key findings.
Experimental Settings. We consider two separate vocabularies for the source and target sentences, each containing 15K words. 0.3M, 5K, and 5K synthesized sentence pairs are generated as training, validation, and test sets respectively. We use the same hyper-parameters in the transformer-base model (Vaswani et al., 2017), which is commonly used in the NAT models (Gu et al., 2018; Du et al., 2021; Sahara et al., 2020). All settings are trained on 4 Nvidia V100 GPUs with 16k tokens in a batch. For the model with OAXE loss, we train the first 50K updates with XE loss and the next 50K updates with OAXE loss (Du et al., 2021). For the other losses, we train the model for 100K updates. The length of the decoder input is set as twice the length of the source sequence for CTC loss (Saharia et al., 2020), while the golden target length is used for OAXE, AXE, and XE. To evaluate the accuracy of the predicted sequence, we first calculate the longest common sub-sequence between the predicted and the golden sequences, and then the accuracy is defined as the ratio between the length of the longest common sub-sequence and the golden sequence. The accuracy on the test set is calculated as the average accuracy among all the predicted sentences.
Long-Range Syntactic Multi-modality. To consider the effect of long-range diversity, we change the corresponding probability  $P^{lo}$ , while keeping the others unchanged to eliminate the short-range syntactic multi-modality. It is observed in Fig. 4a that CTC loss always performs better than AXE, and OAXE is the best with different degree of long-range multi-modality.
Short-Range Syntactic Multi-modality. Similarly, we only change the probabilities  $P_{1}^{so}$  and  $P_{2}^{so}$  to adjust the degree of short-range word order diversity. The results are shown in Fig. 4b, where OAXE loss performs better than AXE loss, and CTC loss outperforms all the other losses with varied degree of short-range word order diversity. In order to study the effect of optional words, we vary the probability  $P^{op}$  to change the existence of "DT". As shown in Fig. 4c, OAXE loss is slightly
(a) Effect of long-range word order.
(b) Effect of short-range word order.
(c) Effect of optional words.
Figure 4: The accuracy of different loss functions on synthesized datasets.
better than AXE loss, and CTC loss performs the best, indicating that CTC loss is superior in the syntactic multi-modality problem caused by optional words.
Analyses and Discussions. Based on the results in Fig. 4, we can get the following observations:
- OAXE loss is superior in handling the long-range syntactic multi-modality (i.e., long-range word order). OAXE loss is order-agnostic, which is able to provide fully positive feedback to the word in different positions compared to the ground truth sequence. Accordingly, OAXE is suitable for datasets with long-range word order diversity. Though it can deal with high diversity of word order, it may also incur wrong predictions on word order, which may be why OAXE is not suitable when the diversity only exists in short-range.
- CTC loss is the best choice for dealing with short-range syntactic multi-modality (i.e., short-range word order and optional words). CTC loss is
generally considered to handle monotonic matching, which seems not effective in handling the multi-modality caused by word order (Saharia et al., 2020). However, it is observed in Fig. 4a and 4b that CTC loss outperforms AXE and XE when dealing with long-range word order diversity and performs the best on the multi-modality caused by short-range word order. Since CTC considers all the monotonic alignments, it can partially provide positive feedback to the words with different order through multiple monotonic alignments. As shown in Fig. 1c, all the words can be considered in the three alignments.
Considering that AXE loss does not show superiority on any type of the syntactic multi-modality, we will only focus on CTC and OAXE losses in the following.
4 Analyses on Real Datasets
Though carefully designed, the synthesized sentence pairs consisting of numbers are still different from the real sentence pairs. Therefore, in this section, we validate the findings in Section 3 based on real datasets. Considering that different types of syntactic multi-modality are highly coupled in the real corpus, we conduct experiments on carefully selected sub-datasets from a corpus, to approximately decompose the syntactic multi-modality. In the following, we first show the details of the approach to decompose the syntactic multi-modality, and then provide the analytical results based on the real datasets.
Analytical Approach. In order to decompose the long-range and short-range types of syntactic multi-modality, we select sentences that only contain subject and verb phrases from a corpus, and divide them into two sub-datasets according to the relative order of subject and verb (i.e., subject first that denoted as "SV", or verb first that denoted as "VS"). Meanwhile, we only consider the declarative sentence pairs in the corpus to eliminate the word order difference caused by mood. Following this method, the long-range multi-modality is eliminated in each sub-dataset (i.e., "SV" and "VS"), which can be used to evaluate the effect of short-range multi-modality. To analyse the long-range multi-modality, we can adjust the degree of long-range word order diversity by sampling data from the two sub-datasets with varied ratios, while roughly keeping the degree of short-range word or
Table 2: BLEU scores of models with CTC and OAXE losses, where the models are evaluated on the WMT'19 EN-RU test set. The percentage of sentences with "RB V" among the sentences with both "RB V" and "V RB" orders are shown in column "RB V". The percentage of sentences with "JJ N" among the sentences with both "JJ N" and "N JJ" orders are shown in column "JJ N".
“SV”:“VS”	CTC	OAXE	“RB V”	“JJ N”
100% : 0%	17.7	16.5	68%	84%
75% : 25%	17.2	16.9	63%	82%
50% : 50%	16.8	17.3	70%	79%
der diversity unchanged. Specifically, considering that Russian is flexible on word order (Kallestinova, 2007) and it is feasible to select sentences on both the "SV" and "VS" order, we use an English-Russian (EN-RU) corpus from Yandex² that contains 1M EN-RU sentence pairs, from which we get 0.2M and 0.1M sentence pairs data with "SV" order and "VS" order respectively. To select the sentence pairs with different word orders, we use spaCy(Honnibal et al., 2020) to parse the dependency of Russian sentences. For the models with CTC loss, we train for 300K updates. For the models with OAXE loss, we train with XE loss for 100K updates and then train with OAXE loss for 200K updates.
Analytical Results. We keep the total number of sentence pairs in the training set as 0.2M (i.e., the number of Russian sentences in the "VS" sub-dataset), and change the ratio of sentence pairs sampled from two sub-datasets (i.e., "SV" and "VS"). The results are shown in Table 2, where the training parameters are the same as that used in Section 3. It is observed that CTC loss outperforms OAXE loss when all data samples are from the "SV" sub-dataset, which indicates that CTC loss performs better on short-range syntactic multi-modality problem. When the ratio of the data sizes on the two sub-datasets is changed to  $75\% : 25\%$ , the gap between the performance of CTC and OAXE losses diminished, while CTC loss still performs slightly better than OAXE loss. When the ratio changed to  $50\% : 50\%$ , model with OAXE loss becomes better than that with CTC loss. In summary, OAXE loss is better at handling long-range syntactic multi-modality while CTC loss is better on short-range syntactic multi-modality, which validates the key observations we obtained on the synthesized datasets in Section 3.
$^{2}$ https://translate.yandex.ru/corpus