SlowGuess/NAACL_2022_OCR huggingface dataset:
In order to demonstrate whether we have decomposed the long- and short-range syntactic multimodalities, we verify whether the degree of short-range multi-modality remains almost the same when varying the degree of long-range multimodality. We evaluate the short-range syntactic diversity based on the relative order between: 1) adverb and verb ("RB V"); 2) adjective and noun ("JJ N"). As shown in Table 2, when the ratio of the data sizes on the two sub-datasets varied from  $100\% : 0\%$  to  $50\% : 50\%$  (i.e., the ratio between "SV" and "VS" changes), the relative order on "RB V" and "RB V" (which can represent the degree of short-range word order diversity) does not vary much. These analyses verify the rationality of our analytical approach in this section.
5 Better Solving the Syntactic Multi-Modality Problem
As shown in previous sections, the CTC and the OAXE loss functions are good at dealing with short- and long-range syntactic multi-modalities respectively. While in real-world corpora, different types of multi-modalities usually occur together and vary in different languages. Accordingly, it may be better to use different loss functions for different languages. In this section, we first introduce a new loss function named Combined CTC and OAXE (CoCO), which takes advantage of both CTC and OAXE to better handle the long-range and short-range syntactic multi-modalities simultaneously, and then provide a guideline on how to choose the appropriate loss function for different scenarios.
5.1 CoCO Loss
To obtain a general loss that performs well at both types of multi-modalities, it is natural to combine the two loss functions studied above. However, the output length is mismatched between the models using CTC and OAXE, where the output length is required to be longer than the target sequence with CTC loss, and is required to be the same as the target sequence with OAXE loss. To solve this length mismatch problem, we consider using the same output length as in CTC loss, and modify the OAXE loss to make it suitable on this output length by allowing consecutive tokens in the output to be aligned with the same token in the reference sequence. The details of the modified OAXE loss are provided in Appendix B. Then, the proposed CoCO loss is defined as a linear combination of the
Figure 5: Comparing different losses on different language pairs.
CTC and modified OAXE losses as:
$$
\mathcal {L} _ {C o C O} = \lambda \mathcal {L} _ {C T C} + (1 - \lambda) \mathcal {L} _ {M - O A X E}, \tag {1}
$$
where  $\mathcal{L}_{M - OAXE}$  denotes the modified OAXE loss and  $\lambda$  is a hyper-parameter that balances the two losses.
5.2 Choosing Appropriate Loss Function
The degree of different types of multi-modalities varies among different languages. In order to find the insight to choose the appropriate loss function for different languages, we conduct experiments on several languages including Russian (RU), Finnish (FI), German (DE), Romanian (RO), and English (EN). These languages have different requirements on the positions of subject (S), verb (V), and object (O), which is one major influence factor on the large-range syntactic multi-modality. Specifically, the order in RU and FI is quite flexible, where all the 6 possible orders of "S", "V", and "O" are valid. In DE, the verb is required to be placed on the second position, which is called verb-second word order. Meanwhile, in RO and EN, the order is restricted to "SVO".
We evaluate the accuracy of different loss functions (i.e., CTC, OAXE, and CoCO) on WMT'14 EN-RU, WMT'17 EN-FI, WMT'14 EN-DE, and WMT'16 EN-RO datasets with 1.5M, 2M, 4M, and 610K sentence pairs, respectively. The  $\lambda$  in COCO loss is set as 0.1 so that  $\lambda \mathcal{L}_{CTC}$  and  $(1 - \lambda)\mathcal{L}_{M-OAXE}$  are in the same order of magnitude. Following Du et al. (2021), for the models with OAXE and CoCO loss, we first train with XE or CTC loss for 100K updates and then train with OAXE or CoCO loss for 200K updates, respectively. For CTC loss, we train for 300K updates. For decoding, we follow Gu and Kong (2021); Huang et al. (2021) to use beam search with lan
Table 3: BLEU scores of NAT models.
Model	WMT14	WMT16	WMT14	WMT17
EN-DE	DE-EN	EN-RO	EN-RU	EN-FI	Speedup
Autoregressive
Transformer	27.48	31.39	33.70	27.2	28.12	1.0 ×
Non-Autoregressive
Vanilla NAT (Gu et al., 2018)	17.69	21.47	27.29	-	-	15.0 ×
BoN (Shao et al., 2020)	20.90	24.60	28.30	-	-	10.0 ×
AXE (Ghazvininejad et al., 2020)	23.53	27.90	30.75	-	-	15.3 ×
Imputer (Saharia et al., 2020)	25.80	28.40	32.30	-	-	18.6 ×
OAXE (CMLM) (Qian et al., 2021)	26.10	30.20	32.40	-	-	15.6 ×
GLAT (Qian et al., 2021)	26.39	29.84	32.79	-	-	14.6 ×
CTC (VAE) (Gu and Kong, 2021)	27.49	30.46	33.79	-	-	16.5 ×
CTC (GLAT) (Gu and Kong, 2021)	27.20	31.39	33.71	-	-	16.8 ×
CTC (DSLP) (Huang et al., 2021)	27.02	31.61	34.17	21.38	22.83	14.8 ×
CoCO (DSLP)	27.41	31.37	34.32	21.82	23.25	14.2 ×
guage model scoring<sup>3</sup> for CTC and CoCO. The other training settings are the same as used in Section 3. We report the tokenized BLEU score to keep consistent with previous works. We show the difference values of BLEU score in Fig. 5 and provide the corresponding absolute BLEU scores in Appendix C. According to Fig. 5, we have several observations: 1) The proposed CoCO loss consistently improves the translation accuracy on all the language pairs compared to OAXE loss; 2) The CoCO loss outperforms CTC loss when the target language is with flexible word order or verb-second word order (i.e., EN-RU, EN-FI, and EN-DE); 3) CTC loss performs the best when the target language is "SVO" language (i.e., DE-EN, RO-EN, and EN-RO).
We would also like to evaluate the CoCO loss based on SOTA NAT models. Though the proposed CoCO loss can be used in both iterative and non-iterative models, we only show the results on non-iterative models in this paper and leave the iterative models as future work. We use CoCO loss on a recently proposed Deeply Supervised, Layer-wise Prediction-aware (DSLP) transformer (Huang et al., 2021), which achieves competitive results. The details of how CoCO loss is applied on DSLP are provided in Appendix D. The results are shown in Table 3. Compared to DSLP with CTC loss (Huang et al., 2021), DSLP with CoCO loss consistently improves the BLEU scores on three language pairs, including EN-RU, EN-FI, and EN-DE. On the contrary, DSLP with CTC loss is better or comparable
to DSLP with CoCO loss when the target language is restricted to the "SVO" word order, including EN-RO and DE-EN.
According to the experiments on language pairs with different kinds of requirements on word order, we suggest to: 1) use CoCO loss when the word order of the target language is relatively flexible ( e.g., RU and FI, where word order on "S" "V" "O" is free, and DE, where the verb is required to be placed on the second position); 2) use CTC loss when the target language is with relatively strict word order (e.g., RO and EN, which are "SVO" languages).
In this paper, we conduct a systematic study on the syntactic multi-modality problem in non-autoregressive machine translation. We first categorize this problem into long-range and short-range types and study the effectiveness of different loss functions on each type. Considering the different types are usually entangled in real-world datasets, we design and construct synthesized datasets to control the degree of one type of multi-modality without changing another for analyses. We find that CTC loss is good at short-range syntactic multimodality while OAXE loss is better at the long-range one. These findings are further verified on real-world datasets with our designed analytical approach. Based on these analyses, we propose a CoCO loss that can better handle the complicated syntactic multi-modality in real-world datasets, and a practical guide to use different loss functions for
<sup>3</sup>https://github.com/kpu/kenlm
different kinds of syntactic multi-modality: CoCO loss is preferred when the word order of target language is relatively flexible while CTC loss is preferred when target language is with strict word order. Our study in this paper can facilitate better understanding of the multi-modality problem and provide insights to better solve this problem in non-autoregressive translation. Besides, there still remain some open problems that need future investigation. For example, we generally consider long-range and short-range types for syntactic multimodality, while there may be more fine-gained categorizations on the syntactic multi-modality due to the complex syntax of natural language.
Srinivas Bangalore and Giuseppe Riccardi. 2001. A finite-state approach to machine translation. In IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU'01., pages 381-388. IEEE.
Noam Chomsky. 1959. On certain formal properties of grammars. Information and control, 2(2):137-167.
Bernard Comrie. 1989. Language universals and linguistic typology: Syntax and morphology. University of Chicago press.
Liang Ding, Longyue Wang, Xuebo Liu, Derek F. Wong, Dacheng Tao, and Zhaopeng Tu. 2021. Understanding and improving lexical choice in non-autoregressive translation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021. Order-agnostic cross entropy for non-autoregressive machine translation. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 2849-2859. PMLR.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Aligned cross entropy for non-autoregressive machine translation. In International Conference on Machine Learning, pages 3515-3523. PMLR.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 6111-6120. Association for Computational Linguistics.
Talmy Givón. 1983. Topic continuity in spoken english. *Topic continuity in discourse: A quantitative cross-language study*, 3:347-363.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376.
Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013. Hybrid speech recognition with deep bidirectional LSTM. In 2013 IEEE workshop on automatic speech recognition and understanding, pages 273-278. IEEE.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Conference on Learning Representations.
Jiatao Gu and Xiang Kong. 2021. Fully non-autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 120-133. Association for Computational Linguistics.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python.
Chenyang Huang, Hao Zhou, Osmar R. Zaïane, Lili Mou, and Lei Li. 2021. Non-autoregressive translation with layer-wise prediction and deep supervision. CoRR, abs/2110.07515.
Yingqi Jing and Haitao Liu. 2015. Mean hierarchical distance augmenting mean dependency distance. In Proceedings of the third international conference on dependency linguistics (Depling 2015), pages 161-170.
Elena Dmitrievna Kallestinova. 2007. Aspects of word order in Russian. The University of Iowa.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97.
Jindrich Libovicky and Jindrich Helcl. 2018. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3016-3021. Association for Computational Linguistics.
Haitao Liu. 2007. Probability distribution of dependency distance. Glottometrics.
Haitao Liu. 2010. Dependency direction as a means of word-order typology: A method based on dependency treebanks. *Lingua*, 120(6):1567-1578.
Haitao Liu, Chunshan Xu, and Junying Liang. 2017. Dependency distance: A new perspective on syntactic patterns in natural languages. Physics of life reviews, 21:171-193.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank.
Terence Odlin. 2008. A handbook of varieties of english. Language, 84(1):193-196.
Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3956-3965. PMLR.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2021. Glancing transformer for non-autoregressive neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 1993-2003. Association for Computational Linguistics.
Peter S Rosenbaum. 1967. Phrase structure principles of english complex sentence formation. Journal of Linguistics, 3(1):103-118.
Chitwan Sahara, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Non-autoregressive machine translation with latent alignments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 1098-1108. Association for Computational Linguistics.
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. 2020. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 198-205.
Katsuhito Sudoh, Xianchao Wu, Kevin Duh, Hajime Tsukada, and Masaaki Nagata. 2011. Post-ordering in statistical machine translation. In Proc. MT Summit.
Chunting Zhou, Jiatao Gu, and Graham Neubig. 2020. Understanding knowledge distillation in non-autoregressive machine translation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
A Number Ranges to Synthesis the Source and Target Sentences
We use [1, 5000], [5001, 10000], [10001, 12500], [12501, 15000], and 15001, 15002, 15003 to represent “N” “V” “JJ” “RB” “DT” in the source sentences, and [15004, 20003], [20004, 25003], [25004, 27503], [27504, 30003], and 30004, 30005, 30006 to represent “N” “V” “JJ” “RB” “DT” in the target sentences.
B Modified OAXE Loss
Step 1
α*	Tokens	Probability distribution of the output
P1	P2	P3	P4	P5	P6
6	A	0.2	0.3	0.3	0.1	0.4	0.5
4	B	0.2	0.3	0.5	0.7	0.2	0.2
1	C	0.6	0.4	0.2	0.2	0.4	0.3
β	3	-1	-1	2	-1	1
Step 2-1
Possible end tokens of “A”	P1	P2	P3
Select the best
β	3	3	2	2	-1	1
Step 2-2
Possible end tokens of “B”	P4	P5
Select the best
β	3	3	2	2	2	1
Figure 6: An illustration of the modified OAXE loss.
Specifically, we consider one training pair  $(X,Y)$ , where there are  $n$  tokens in the ground truth sequence, denoted as  $Y = (y_{1},y_{2},\ldots ,y_{n})$ . The corresponding output sequence has  $m$  tokens with probability distributions  $P_{1},P_{2},\ldots ,P_{m}$ , where  $m > n$ . According to OAXE, we first get the alignment between the ground truth sequence and the output sequence that minimizes the cross entropy loss based on maximum bipartite matching algorithm (Kuhn, 1955):
$$
\alpha^ {\star} = \underset {\alpha \in \gamma (\alpha)} {\arg \min } \left(- \sum_ {i} \log P _ {\alpha (i)} \left(y _ {i} \mid X, \theta\right)\right), \tag {2}
$$
where  $\alpha$  denotes the alignment from the ground truth sequence to the output sequence,  $\gamma (\alpha)$  is the set of all possible alignments, and  $y_{i}$  is aligned with the  $\alpha (i)$ -th token of the output. We consider each output token can only be aligned to one ground
Table 4: BLEU scores of models with different losses on different language pairs.
Loss	EN-RU	EN-FI	EN-DE	DE-EN	RO-EN	EN-RO
CTC	20.84	22.86	26.10	30.36	33.68	33.06
OAXE	21.23	23.13	26.16	30.07	33.25	32.31
CoCO	21.45	23.27	26.25	30.19	33.31	32.67
truth token (i.e.,  $\alpha(i) \neq \alpha(j)$  if  $i \neq j$ ). Then, we can get the alignment from the output sequence to the ground truth sequence, based on  $\alpha^{\star}$ :
$$
\beta (k) = \left\{ \begin{array}{l l} i & \text {i f} \alpha^ {\star} (i) = k, \\ - 1 & \text {i f} \forall i \in [ 1, n ], \alpha^ {\star} (i) \neq k, \end{array} \right. \tag {3}
$$
where the  $k$ -th token of the output is aligned to  $y_{\beta(k)}$  and  $\beta(k) = -1$  denotes the token has not been aligned. We provide an illustration as the "step 1" in Fig. 6, where we consider 3 tokens in the target sequence and 6 tokens in the output and the best alignment is "A"-“ $P_6$ ”, “B"-“ $P_4$ ”, and "C"-“ $P_1$ ”. Since consecutive repetitive tokens are merged when decoding with CTC loss, we consider that consecutive tokens in the output can be aligned to the same ground truth token. Accordingly, we enumerate the end of each ground truth token in the output sequence respectively, and select the one that minimize the cross entropy loss. For example, given  $\beta(k_1) = i$ ,  $\beta(k_2) = j$  and  $\beta(k) = -1$  when  $k_1 \leq k \leq k_2$ , we select  $k^{\star}$  according to:
$$
\begin{array}{l} k ^ {\star} = \operatorname * {a r g   m i n} _ {k _ {1} \leq k ^ {\prime} <   k _ {2}} \left(- \sum_ {k _ {1} \leq k \leq k ^ {\prime}} \log P _ {k} (y _ {i} | X, \theta) \right. \\ \left. - \sum_ {k ^ {\prime} <   k \leq k _ {2}} \log P _ {k} \left(y _ {j} \mid X, \theta\right)\right), \tag {4} \\ \end{array}
$$
and align the  $(k_{1}, k^{\star}]$ -th output token to  $i$  and the  $(k^{\star}, k_{2})$ -th output token to  $j$  as:
$$
\beta (k) = \left\{ \begin{array}{l l} i & \text {i f} k \in \left(k _ {1}, k ^ {\star} \right] \\ j & \text {i f} k \in \left(k ^ {\star}, k _ {2}\right). \end{array} \right. \tag {5}
$$
As the illustration in Fig. 6, we enumerate all the possible end tokens of 'A' and 'B' to find the best one. Then, we get the modified OAXE loss as:
$$
\mathcal {L} _ {M - O A X E} = - \sum_ {1 \leq k \leq m} \log P _ {k} \left(y _ {\beta (k)} | X, \theta\right). \tag {6}
$$
C BLEU Scores of Different Losses on Different Language Pairs.
The BLEU scores of models with CTC, OAXE and CoCO loss on different languages pairs are shown
in Table 4.
D Use CoCO Loss in DSLP
Partially feeding ground truth tokens to the decoder during training shows promising performance in NAT (Ghazvininejad et al., 2019; Sahara et al., 2020; Qian et al., 2021; Huang et al., 2021). For the models training with golden length of the ground truth sentence using XE loss, the ground truth token embedding is placed to the position of the corresponding input (Qian et al., 2021). When using CTC loss, the inputs of the decoder are always longer than the ground truth sentences, where Gu and Kong (2021) proposes to use the best monotonic alignment between the ground truth and output sequences, and provides the ground truth to the corresponding input position of the decoder. With the proposed CoCO loss, we use the best alignment which is not required to be monotonous. In addition, DSLP requires deep supervision on each layer of the decoder. We find that only replacing CTC loss with CoCO loss on the first layer is better than using CoCO loss on all layers. Accordingly, when using CoCO loss in DSLP transformer, we use CoCO loss in the first layer and CTC loss for all the other layers in the DSLP transformer.