SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhuoyuan Mao  $\mu$  Chenhui Chu  $\mu$  Raj Dabre  $\nu$  Haiyue Song  $\mu$  Zhen Wan  $\mu$  Sadao Kurohashi  $\mu$
$^{\mu}$  Kyoto University, Japan  $\nu$  NICT, Japan
{zhuoyuanmao, chu, song, zhenwan, kuro}@nlp.ist.i.kyoto-u.ac.jp raj.dabre@nict.go.jp
Word alignment has proven to benefit many-to-many neural machine translation (NMT). However, high-quality ground-truth bilingual dictionaries were used for pre-editing in previous methods, which are unavailable for most language pairs. Meanwhile, the contrastive objective can implicitly utilize automatically learned word alignment, which has not been explored in many-to-many NMT. This work proposes a word-level contrastive objective to leverage word alignments for many-to-many NMT. Empirical results show that this leads to 0.8 BLEU gains for several language pairs. Analyses reveal that in many-to-many NMT, the encoder's sentence retrieval performance highly correlates with the translation quality, which explains when the proposed method impacts translation. This motivates future exploration for many-to-many NMT to improve the encoder's sentence retrieval performance.
Many-to-many neural machine translation (NMT) (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Sen et al., 2019; Arivazhagan et al., 2019b; Lin et al., 2020; Pan et al., 2021b) jointly trains a translation system for multiple language pairs and obtain significant gains consistently across many translation directions. Previous work (Lin et al., 2020) shows that word alignment information helps improve pre-training for many-to-many NMT. However, manually cleaned high-quality ground-truth bilingual dictionaries are used to pre-edit the source sentences, which are unavailable for most language pairs.
Recently, contrastive objectives (Clark et al., 2020; Gunel et al., 2021; Giorgi et al., 2021; Wei et al., 2021; Mao et al., 2021) have been shown to be superior at leveraging alignment knowledge in various NLP tasks by contrasting the representations of positive and negative samples in a discriminative manner. This objective, which should
be able to utilize word alignment learned by any toolkit, which in turn will remove the constraints of using manually constructed dictionaries, has not been explored in the context of leveraging word alignment for many-to-many NMT.
An existing contrastive method (Pan et al., 2021b) for many-to-many NMT relies on sentence-level alignments. Given that the incorporation of word alignments has led to improvements in previous work, we believe that fine-grained contrastive objectives focusing on word alignments should help improve translation. Therefore, this paper proposes word-level contrastive learning for many-to-many NMT using the word alignment extracted by automatic aligners. We conduct experiments on three many-to-many NMT systems covering general and spoken language domains. Results show that our proposed method achieves significant gains of 0.8 BLEU in the general domain compared to previous word alignment based methods and the sentence-level contrastive method.
We then analyze how the word-level contrastive objective affects NMT training. Inspired by previous work (Artetxe and Schwenk, 2019) that train sentence retrieval models using many-to-many NMT, we speculate that our contrastive objectives affect the sentence retrieval performance and subsequently impact the translation quality. Further investigation reveals that in many-to-many NMT, the sentence retrieval precision of the multilingual encoder for a language pair strongly correlates with its translation quality (BLEU), which provides insight about when contrastive alignment improves translation. This revelation emphasizes the importance of improving the retrieval performance of the encoder for many-to-many NMT.
2 Word-level Contrastive Learning for Many-to-many NMT
Inspired by the contrastive learning framework (Chen et al., 2020) and the sentence-level con
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1766 - 1775 July 10-15, 2022 ©2022 Association for Computational Linguistics
trastive learning objective (Pan et al., 2021b), we propose a word-level contrastive learning objective to explicitly guide the training of the multilingual encoder to obtain well-aligned cross-lingual representations. Specifically, we use word alignments, obtained using automatic word aligners, to supervise the training of the multilingual encoder by a contrastive objective alongside the NMT objective.
Alignment Extraction Two main approaches for automatically extracting aligned words from a sentence pair are: using a bilingual dictionary and using unsupervised word aligners. The former extracts fewer but precise alignments, whereas the latter extracts more but noisy alignments. We extract word-level alignments by both methods and explore how they impact NMT training. For the former approach, we use word2word (Choe et al., 2020) to construct bilingual lexicons and then extract word pairs from parallel sentences. The extracted word pairs are combined to form a phrase if words are consecutive in the source and target sentence. For the latter approach, we use FastAlign (Dyer et al., 2013) and use only 1-to-1 mappings for training.
Word-level Contrastive Learning With the extracted alignments, we propose a word-level contrastive learning objective for the multilingual encoder by the motivation that the aligned words within a sentence pair should have a similar contextual representation. We expect the supervision of the contrastive objective on the corresponding contextual word representation leads to a robust multilingual encoder. Assume that the tokenized source and target parallel sentences in the  $i - th$  batch are  $\mathcal{D}_i = \{src_{ij}, tgt_{ij}\}_{j=1}^B$ , and the extracted alignments from all the sentence pairs in each batch are  $\mathcal{A}_i = \{s_{ik}, t_{ik}\}_{k=1}^N$ , where  $B$  and  $N$  denote the batch-size and the number of alignments, respectively. Note that  $s_{ik}$  and  $t_{ik}$  may contain several tokens after the word combination for word2word or subword tokenization for NMT. Then the word-level contrastive loss in a batch is:
$$
\begin{array}{l} \mathcal {L} _ {a l i g n} ^ {(i)} = - \sum_ {k = 1} ^ {N} (\log \frac {\exp (s i m (s _ {i k} , t _ {i k}) / \mathcal {T})}{\sum_ {m = 1} ^ {N} \exp (s i m (s _ {i k} , t _ {i m}) / \mathcal {T})} \\ + \log \frac {\exp (s i m \left(s _ {i k} , t _ {i k}\right) / \mathcal {T})}{\sum_ {m = 1} ^ {N} \exp (s i m \left(s _ {i m} , t _ {i k}\right) / \mathcal {T})} \tag {1} \\ \end{array}
$$
where  $\mathcal{T}$  denotes a similarity scaling temperature. The similarity between two words is measured by:
$$
\operatorname {s i m} \left(\operatorname {w o r d} _ {x}, \operatorname {w o r d} _ {y}\right) = \cos (g (\bar {\mathbf {x}}), g (\bar {\mathbf {y}})) \tag {2}
$$
La. pair	Source	Size	N (w2w)	N (FA)
en-et	WMT18	1.9M	5,762,977	38,454,477
en-it	IWSLT17	231k	603,032	3,000,011
en-ja	IWSLT17	223k	684,583	2,797,882
en-kk	WMT19	124k	124,511	279,429
en-my	ALT	18k	75,383	377,392
en-nl	IWSLT17	237k	564,697	2,836,873
en-ro	WMT16	612k	3,271,848	13,092,240
en-tr	WMT17	207k	770,873	2,885,102
en-vi	IWSLT15	133k	354,167	2,120,755
Table 1: Data Source and number of the extracted word pairs. La. pair, N (w2w) and N (FA) denote the language pair, the number of the word pairs extracted by word2word and FastAlign, respectively. Refer to Appendix B for details of the dataset splits.
where  $g(\mathbf{x}) = \mathbf{W}_2\sigma (\mathbf{W}_1\mathbf{x})$  and  $\bar{\mathbf{x}}$  denotes the average of contextual hidden states of the corresponding subword positions on top of the multilingual encoder. Following (Chen et al., 2020), we use an MLP between contrastive loss and the contextual representation for NMT loss. ReLU activation is used for  $\sigma$ ,  $\mathbf{W}_1$  is  $d\times d$  and  $\mathbf{W}_2$  is  $d\times d'$ , where  $d$  is the encoder's hidden dimension and  $d' < d$ .
Finally, to jointly train with the NMT loss, we use the following equation to combine our proposed word-level contrastive loss for a batch:
$$
\mathcal {L} ^ {(i)} = \frac {1}{B} \left(\mathcal {L} _ {N M T} ^ {(i)} + w \frac {N _ {T}}{2 N} \mathcal {L} _ {\text {a l i g n}} ^ {(i)}\right) \tag {3}
$$
where  $N_{T}$  is the number of the tokens within a batch,  $\frac{N_T}{2N}$  is a multiplier that scales the contrastive loss to be consistent with NMT loss, and  $w$  is a weight to balance the joint training.
Datasets and Preprocessing We selected ten languages, including English (en), Estonian (et), Italian (it), Japanese (ja), Kazakh (kk), Burmese (my), Dutch (nl), Romanian (ro), Turkish (tr), Vietnamese (vi) from different language families to train the NMT systems. We used the parallel datasets from different domains for the selected nine language pairs, including IWSLT, WMT, and ALT. We followed mBART (Liu et al., 2020) for tokenization. Details are given in Appendix A. For each parallel dataset, we implemented two approaches as stated in Section 2 to extract word pairs for the contrastive training objective. Data source and the number of the extracted word pairs are shown in Table 1. To ensure high alignment
Methods	en-tr	en-ro	en-et	en-kk	en-my
→	←	→	←	→	←	→	←	→	←
MLSC	9.3	12.6	25.0	26.2	10.8	15.1	0.5	5.3	15.1	15.6
+align	9.0	12.4	24.6	26.5	10.7	14.6	0.4	5.4	15.0	15.3
+w2w (ours)	9.4	12.6	24.8	26.8	10.8	15.1	0.5	5.8	15.2	15.9
+FA (ours)	9.1	12.2	24.8	26.7	10.7	14.8	0.3	5.6	15.0	15.6
mBART FT	17.7	22.2	33.8	37.1	14.5	24.3	1.8	14.1	17.8	23.1
+align	17.5	21.9	33.8	36.7	15.2	24.3	1.8	14.0	16.9	22.1
+w2w (ours)	17.6	22.2	34.2	37.5	15.0	25.0	1.2	14.1	18.3	23.8
+FA (ours)	17.5	22.2	34.3	37.5	14.9	25.1	1.3	14.4	17.9	23.6
Table 2: BLEU scores of 626_en-tr-ro-et-my-kk system. Significantly better scores (Koehn, 2004) are in cyan, and marginal improvements are in lightcyan.
Methods	222_en-ja	626_I	626-II
MLSC	13.90	23.76	13.55
+align	13.90	23.67	13.39
+w2w (ours)	13.85	23.44	13.69
+FA (ours)	13.30	23.68	13.48
mBART FT	18.90	29.11	20.64
+align	18.55	28.87	20.42
+w2w (ours)	18.80	29.08	20.89
+FA (ours)	18.65	29.01	20.87
Table 3: Overall average BLEU of all the systems. 626_I and 626_II denote 626_en-it-ja-nl-tr-vi and 626_en-tr-ro-et-my-kk, respectively. Results better than MLSC or mBART FT are marked bold. Refer to Appendix D for the detailed scores of all the systems.
quality, we used large-scale out-of-domain (see Appendix B) parallel corpora with FastAlign.
Many-to-many NMT systems We established three many-to-many NMT systems as follows:
222_en-ja: Bidirectional en-ja NMT model using en-ja parallel corpus.
626_en-it-ja-nl-tr-vi: 6-to-6 multilingual NMT model using spoken language domain corpora for en-it, en-ja, en-nl, en-tr and en-vi.
626_en-tr-ro-et-my-kk: 6-to-6 multilingual NMT model using general domain corpora for en-tr, en-ro, en-et, en-my and en-kk.
Baselines and Ours For each language group setting above, we conducted NMT experiments on both the multilingual training from scratch (MLSC) (Johnson et al., 2017; Aharoni et al., 2019) and the mBART multilingual fine-tuning (mBART FT) (Tang et al., 2020) as baselines. We applied our proposed word-level contrastive learning in both MLSC and mBART FT, and compared with another strong baseline, word alignment based joint
Figure 1: NMT loss, sentence retrieval  $\mathbf{P}@\mathbf{1}$  of the encoder in MLSC and mBART FT. The average of the contextual embeddings on top of the encoder is used as the sentence embedding. We report the average in-batch retrieval precision of both directions of each language pair.
NMT training (+align) (Garg et al., 2019). For applying our method, we investigated the performance of joint training with word pairs extracted by both word2word (+w2w) and FastAlign (+FA). We omitted Lin et al. (2020) as a baseline because their method can not be applied to mBART fine-tuning, and they used high-quality ground-truth dictionaries, which are unavailable for most languages pairs. Implementation We used mBART-large (mBART-25) for mBART FT and transformer-base (Vaswani et al., 2017) for MLSC. See Appendix C for details.
BLEU Results We report case-sensitive tokenized BLEU (Papineni et al., 2002) results in Table 3 and 2. In Table 3, we observe that with our pro
Figure 2: Sentence retrieval  $\mathbf{P}@\mathbf{1}$  on the validation set for each language pair. Left and middle are the results on 626_en-tr-ro-et-my-kk MLSC and mBART FT, respectively. "626" in right subfigure denote 626_en-it-ja-nl-tr-vi. Refer to Appendix E for setup and results in details.
posed training objectives, BLEU scores are comparable in 222_en-ja and 626_en-it-ja-nl-tr-vi while they are slightly improved in 626_en-tr-ro-et-mykk. However,  $" + \text{align}"$  performs comparable or even worse compared with the baseline. Referring to Table 2 for specific BLEUs on each language pair, we find that with our methods, translation performances are significantly improved for mBART FT while nontrivial improvements can merely be observed on en-ro and en-kk direction for MLSC. This indicates that NMT fine-tuning on monolingual pre-trained models (mBART) may benefit more from our proposed methods. Note that the BLEU improvements for MLSC are not significant, and we explain why this happens in the "Word Retrieval P@1 is improved" part.
Latent Encoder Alignment Property We now inspect which aspect of alignment-based methods impacts the translation performance. Previous work (Artetxe and Schwenk, 2019) showed that the encoder of a strong multilingual NMT system is an ideal model for the bilingual sentence retrieval task. In addition, Arivazhagan et al. (2019a) introduced the correlation between the encoder-side sentence representation<sup>1</sup> and the translation quality. Inspired by these, we speculate that alignment-based objectives affect sentence retrieval performance, which further impacts the translation quality. We train MLSC and mBART FT and report the sentence retrieval precision and NMT loss during the training. Results are reported in Figure 1. We observe that the validation retrieval precision show similar trends as the NMT loss. This indicates that during many-to-many NMT training from scratch, encoder-side sentence-level retrieval precision is optimized along with the NMT loss.
Sentence Retrieval P@1 Correlates with BLEU According to the investigation of the encoder alignment property above, we verify the relationship
Figure 3: Average Word retrieval  $\mathbf{P}@\mathbf{1}$  on the validation set for each language pair. "626-*-1" and "626-*-2" indicate 626_en-it-ja-nl-tr-vi and 626_en-tr-ro-et-my-kk, respectively. Refer to Appendix F for setup and results in details.
between BLEU score and sentence retrieval precision on the validation set for each language pair. Results are shown in Figure 2. Cross-referencing the BLEU score in Table 2, we found that BLEU scores are improved when the encoder achieves gains on the sentence retrieval precision. For example, we see increases of the retrieval P@1 on en-ro, en-et, and en-my on mBART FT (the middle of Figure 2) while BLEU scores are significantly improved on these three language pairs (Table 2). We further calculate the Pearson correlation coefficient between the BLEU changes and sentence retrieval P@1 changes for mBART+align, mBART+w2w, and mBART+FA in the 626_en-tr-ro-et-my-kk setting. Results are 0.79, 0.93, 0.90, respectively, demonstrating a strong correlation between translation quality and sentence retrieval precision.
Word Retrieval  $\mathbf{P}@\mathbf{1}$  is Improved We probe the trained contextualized word representations on top of the encoder. As shown in Figure 3, we observe that the word retrieval precision is improved in all
222_en-ja MLSC setting can hardly learn a well-aligned encoder while our methods improve the encoder sentence-level alignment quality without sacrificing BLEU scores.
Usually a pooled encoder output.
the settings. This demonstrates that the encoder parameters of the NMT system trained with our proposed objective are of a rather different distribution. By just changing the random seed, we can expect similar BLEU results, but we cannot obtain a better aligned encoder. However, the improvement of the word retrieval precision does not directly contribute to the translation quality, which we explain next.
Word-level Contrastive Objective and Sentence Retrieval P@1 With the word-level contrastive objective, we observed significant BLEU score improvements on language pairs such as en-ro, en-et and en-my for mBART FT as presented in Table 2. However, noisy word pairs (Pan et al., 2021a) extracted via word alignment toolkits leads to poor supervision signals for improving sentence retrieval P@1, which in turn prevents some language pairs such as en-kk from exhibiting BLEU improvements. We found that for en-kk, the numbers of extracted word pairs per sentence by word2word and FastAlign are 1.0 and 2.2, respectively. In contrast, these numbers are 4.2 and 20.7 for improved language pairs, calculated from Table 1. Although better extracted word alignments for the word-level contrastive objective leads to BLEU improvements, its contribution towards improvements varies for MLSC and mBART FT, as shown in Table 2. We expect these findings to provide new perspectives for improving many-to-many NMT.
Sentence-level Contrastive Objective We conducted the experiments for the sentence-level contrastive objective (Pan et al., 2021b) on all two six-to-six settings and compared it against our proposed approach. The average BLEUs of our methods significantly outperform those of sentence-level contrastive objectives (see Table 8 and 9), clearly showing the sentence-level objective's limitation. Moreover, we checked the sentence retrieval P@1 for Pan et al. (2021b) (Table 10 and 11) and found that it correlates with BLEU changes, indicating that sentence-level contrastive objective is suboptimal for language pairs with decreased retrieval precision. $^3$
We proposed a word-level contrastive learning objective for many-to-many NMT. Experimental re
sults showed that our proposed method leads to significantly better translation for several language pairs, which is then explained by analyses showing the relationship between BLEU scores and sentence retrieval performance of the NMT encoder. Future work can focus on: (1) further improving the encoder's retrieval performance in many-to-many NMT; (2) contrastive objective's feasibility in a massively multilingual scenario.
All the corpora we used in this paper are publicly available resources without the issue of the copyright. The technique this paper proposed is for NMT models, so it can not circumvent the issues that NMT models have. Since our automatically dictionaries are extracted from potentially biased data, the translations may also contain biases. However, we expect that these issues may be resolved by using unbiased data or the addition of debiasing objectives.
This work was supported by Grant-in-Aid for Young Scientists #19K20343, JSPS and Information/AI/Data Science Doctoral Fellowship of Kyoto University.
Roee Aharoni, Melvin Johnson, and Orhan First. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguistics.
Naveen Arivazhagan, Ankur Bapna, Orhan First, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019a. The missing ingredient in zero-shot neural machine translation. CoRR, abs/1903.07091.
Naveen Arivazhagan, Ankur Bapna, Orhan First, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019b. Massively multilingual neural machine translation in the wild: Findings and challenges. CoRR, abs/1907.05019.
Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Note that the sentence-level contrastive objective incorporates sentences in multiple languages for contrastive loss. It does not necessarily improve the pair-wise retrieval precision.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Yo Joong Choe, Kyubyong Park, and Dongwoo Kim. 2020. word2word: A collection of bilingual lexicons for 3,564 language pairs. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3036-3045, Marseille, France. European Language Resources Association.
Christos Christodoulopoulos and Mark Steedman. 2015. A massively parallel corpus: the bible in 100 languages. Lang. Resour. Evaluation, 49(2):375-395.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Chenchen Ding, Hnin Thu Zar Aye, Win Pa Pa, Khin Thandar Nwet, Khin Mar Soe, Masao Utiyama, and Eiichiro Sumita. 2020. Towards burmese (myanmar) morphological analysis: Syllable-based tokenization and part-of-speech tagging. ACM Trans. Asian Low Resour. Lang. Inf. Process., 19(1):5:1-5:34.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Orhan First, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866-875, San Diego, California. Association for Computational Linguistics.
Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. 2019. Jointly learning to align and translate with transformer models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4453-4462, Hong Kong, China. Association for Computational Linguistics.
John M. Giorgi, Osvald Nitski, Bo Wang, and Gary D. Bader. 2021. Declutr: Deep contrastive learning for unsupervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for