SlowGuess/NAACL_2022_OCR huggingface dataset:
Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 879-895. Association for Computational Linguistics.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. 2020. Pretraining multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2649-2663, Online. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Zhuoyuan Mao, Prakhar Gupta, Chenhui Chu, Martin Jaggi, and Sadao Kurohashi. 2021. Lightweight cross-lingual sentence representation learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2902–2913, Online. Association for Computational Linguistics.
Hajime Morita, Daisuke Kawahara, and Sadao Kurohashi. 2015. Morphological analysis for unsegmented languages using recurrent neural network language model. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language
Processing, pages 2292-2297, Lisbon, Portugal. Association for Computational Linguistics.
Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. 2016. ASPEC: Asian scientific paper excerpt corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 2204-2208, PortoRož, Slovenia. European Language Resources Association (ELRA).
Lin Pan, Chung-Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, and Mo Yu. 2021a. Multilingual BERT post-pretraining alignment. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 210–219, Online. Association for Computational Linguistics.
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021b. Contrastive learning for many-to-many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 244-258, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Multilingual unsupervised NMT using shared encoder and language-specific decoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3083-3089, Florence, Italy. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 371-376, Berlin, Germany. Association for Computational Linguistics.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. CoRR, abs/2008.00401.
Arseny Tolmachev, Daisuke Kawahara, and Sadao Kurohashi. 2018. Juman++: A morphological analysis toolkit for scriptio continua. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 54-59, Brussels, Belgium. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021. On learning universal representations across languages. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
La. pair	Train	Valid	Test	OD Size
en-et	WMT18	WMT18	WMT18	10.7M
en-it	IWSLT17	IWSLT15	IWSLT16	13.6M
en-ja	IWSLT17	IWSLT15	IWSLT16	10.7M
en-kk	WMT19	WMT19	WMT19	851k
en-my	ALT	ALT	ALT	446k
en-nl	IWSLT17	IWSLT15	IWSLT16	12.7M
en-ro	WMT16	WWT16	WMT16	11.0M
en-tr	WMT17	WWT16	WMT16	11.1M
en-vi	IWSLT15	IWSLT13	IWSLT14	11.9M
Table 4: Dataset statistics for each language pair. "La. pair" means language pair and "OD Size" denotes the number of the out-of-domain sentence pairs used for training FastAlign.
Methods	en-ja	ja-en
MLSC	15.9	11.9
+align	16.3	11.5
+w2w (ours)	16.0	11.7
+FA (ours)	15.6	11.0
mBART FT	19.8	18.0
+align	19.6	17.5
+w2w (ours)	19.4	18.2
+FA (ours)	19.5	17.8
A Tokenization Settings
For Japanese, we use Jumanpp (Morita et al., 2015; Tolmachev et al., 2018) for segmentation, and we follow the same settings as in mBART (Liu et al., 2020) for other languages: myseg.py (Ding et al., 2020) is used for Burmese, Moses tokenization and special normalization is used for Romanian following (Sennrich et al., 2016),<sup>4</sup> and Moses tokenization for other languages.<sup>5</sup> Following mBART, we apply SentencePiece (Kudo and Richardson, 2018) to further segment sentences into subwords.<sup>6</sup>
B Datasets and Alignment Extraction
The datasets used for NMT training, validation and test are shown in Table 4. For the word align-
Table 5: BLEU scores of 222_en-ja system. Significantly better scores are in cyan, and marginal improvements are in lightcyan. The significance test is done with Koehn (2004).
Methods	en-ja
MLSC	3.3
+align	3.5
+w2w (ours)	73.5
+FA (ours)	69.6
mBART FT	88.9
+align	87.4
+w2w (ours)	85.2
+FA (ours)	84.8
Table 6: Sentence retrieval  $\mathbf{P}@\mathbf{1}$  on the validation set for 222_en-ja.
Methods	en-ja
MLSC	20.1
+align	22.5
+w2w (ours)	68.3
+FA (ours)	67.6
mBART FT	65.2
+align	64.3
+w2w (ours)	71.5
+FA (ours)	70.7
Table 7: Word retrieval  $\mathbf{P}@\mathbf{1}$  on the validation set for 222_en-ja.
ment extraction using FastAlign, we also use out-of-domain parallel corpora to train the FastAlign jointly, aiming to obtain word alignments with less noise. The out-of-domain corpora for all the language pairs contain Tatoeba, Europarl, GlobalVoices, NewsCommentary, OpenSubtitles, TED, WikiMatrix, QED, GNOME, bible-uedin, and ASPEC (Nakazawa et al., 2016). We collect them from the OPUS project (Christodoulopoulos and Steedman, 2015) and WAT. $^{7}$  The number of the out-of-domain parallel sentences for each language pair is shown in Table 4.
Following Tang et al. (2020), we set the oversampling temperature of 1.5 for all the settings. For MLSC, we set the dropout of 0.3 to avoid overfitting on small-scale training data. We used the batch size of 1,024 tokens for all the settings. For our word-level contrastive learning, we set the weight of 0.1, the temperature of 0.2,  $d'$  of 128, and a
4https://github.com/rsennrich/wmt16-scripts
5https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/ tokenizertokenizer.perl
https://github.com/google/ sentencepiece
$^{7}$ https://lotus.kuee.kyoto-u.ac.jp/WAT/WAT2021/index.html
Methods	en-ja	en-vi	en-it	en-nl	en-tr	Avg.
→	←	→	←	→	←	→	←	→	←
MLSC	15.4	11.8	29.6	28.6	27.5	32.7	29.1	36.4	11.6	14.9	23.76
+align	15.1	11.4	29.4	28.3	27.7	33.0	28.9	36.0	11.8	15.1	23.67
+w2w (ours)	15.3	11.6	29.7	28.2	27.6	32.4	28.6	35.8	10.8	14.4	23.44
+FA (ours)	15.5	11.6	29.6	28.0	27.8	33.2	29.1	35.9	11.2	14.9	23.68
+sent	15.1	11.6	29.6	28.3	27.3	32.7	28.1	36.6	11.3	14.7	23.53
mBART FT	17.8	17.0	34.1	35.7	32.5	38.0	32.6	41.6	18.7	23.1	29.11
+align	17.6	16.7	33.7	35.6	32.0	37.7	32.5	41.3	18.7	22.9	28.87
+w2w (ours)	17.6	17.2	34.2	35.7	32.5	38.2	32.1	41.7	18.7	22.9	29.08
+FA (ours)	17.5	17.7	34.0	35.2	32.4	37.9	32.3	41.4	18.6	23.1	29.01
+sent	17.8	16.5	33.7	35.6	32.2	38.1	32.5	41.2	18.1	22.9	28.86
Table 8: BLEU scores of 626_en-it-ja-nl-tr-vi system. Significantly better scores are in cyan, and marginal improvements are in lightcyan. The significance test is done with Koehn (2004).
Methods	en-tr	en-ro	en-et	en-kk	en-my	Avg.
→	←	→	←	→	←	→	←	→	←
MLSC	9.3	12.6	25.0	26.2	10.8	15.1	0.5	5.3	15.1	15.6	13.55
+align	9.0	12.4	24.6	26.5	10.7	14.6	0.4	5.4	15.0	15.3	13.39
+w2w (ours)	9.4	12.6	24.8	26.8	10.8	15.1	0.5	5.8	15.2	15.9	13.69
+FA (ours)	9.1	12.2	24.8	26.7	10.7	14.8	0.3	5.6	15.0	15.6	13.48
+sent	8.7	12.1	24.5	26.0	10.4	14.5	0.4	5.3	13.8	14.6	13.03
mBART FT	17.7	22.2	33.8	37.1	14.5	24.3	1.8	14.1	17.8	23.1	20.64
+align	17.5	21.9	33.8	36.7	15.2	24.3	1.8	14.0	16.9	22.1	20.42
+w2w (ours)	17.6	22.2	34.2	37.5	15.0	25.0	1.2	14.1	18.3	23.8	20.89
+FA (ours)	17.5	22.2	34.3	37.5	14.9	25.1	1.3	14.4	17.9	23.6	20.87
+sent	17.2	22.1	34.2	37.0	14.2	24.1	1.6	14.0	17.7	23.4	20.55
Table 9: BLEU scores of 626_en-tr-ro-et-my-kk system. Significantly better scores are in cyan, and marginal improvements are in lightcyan. The significance test is done with Koehn (2004).
Methods	en-ja	en-vi	en-it	en-nl	en-tr	Avg.
MLSC	52.7	84.6	91.0	85.7	89.7	80.9
+align	53.5	82.8	91.2	86.4	88.9	80.6
+w2w (ours)	73.4	85.7	91.4	84.7	83.1	83.7
+FA (ours)	71.3	84.9	91.3	83.8	82.0	82.7
+sent	87.2	84.7	91.1	87.7	86.6	87.5
mBART FT	87.1	96.2	97.3	94.6	98.5	94.7
+align	85.1	95.8	97.3	94.2	98.5	94.2
+w2w (ours)	81.6	91.4	94.7	90.8	89.6	89.6
+FA (ours)	82.6	92.3	95.0	91.7	90.4	90.4
+sent	76.2	88.3	93.6	88.7	89.8	87.3
Table 10: Sentence retrieval  $\mathbf{P}@\mathbf{1}$  on the validation set for 626_en-it-ja-nl-tr-vi.
Methods	en-tr	en-ro	en-et	en-kk	en-my	Avg.
MLSC	86.2	84.0	85.4	64.4	72.4	78.5
+align	85.9	82.4	84.0	61.3	61.8	75.1
+w2w (ours)	79.6	88.1	76.8	77.4	83.7	81.1
+FA (ours)	77.0	86.1	69.8	75.7	73.4	76.4
+sent	76.3	77.6	55.2	63.8	71.4	68.9
mBART FT	98.0	92.7	96.0	92.9	94.7	94.9
+align	97.4	92.5	97.0	92.1	93.7	94.5
+w2w (ours)	94.3	95.6	96.8	86.0	96.2	93.8
+FA (ours)	94.3	96.3	97.3	87.9	96.2	94.4
+sent	94.6	97.3	95.4	93.1	95.7	95.2
Table 11: Sentence retrieval  $\mathbf{P}@\mathbf{1}$  on the validation set for 626_en-tr-ro-et-my-kk.
smaller dropout of 0.2 because our proposed objective serves as a regularization part. We followed the hyperparameter setting of Garg et al. (2019) for word alignment-based joint NMT training. We
used 8 NVIDIA A100 for mBART FT and 8 TI-TAN Xp for MLSC model training. The model is validated every 1000 steps for 222_en-ja and 2000 steps for both two 626 settings. We do the early
Methods	en-ja	en-vi	en-it	en-nl	en-tr	Avg.
MLSC	61.8	54.6	42.8	42.1	42.7	48.8
+align	61.9	54.1	43.7	42.0	42.3	48.8
+w2w (ours)	64.0	64.7	55.8	57.7	52.8	59.0
+FA (ours)	58.2	65.2	59.2	60.1	48.1	58.2
mBART FT	64.5	57.2	47.4	45.9	47.2	52.4
+align	64.0	56.8	47.3	45.7	46.8	52.1
+w2w (ours)	71.3	70.1	60.6	62.9	57.8	64.5
+FA (ours)	68.6	69.4	63.2	64.7	57.4	64.7
Table 12: Word retrieval P@1 on the validation set for 626_en-it-ja-nl-tr-vi.
Methods	en-tr	en-ro	en-et	en-kk	en-my	Avg.
MLSC	41.9	63.2	64.4	63.4	65.8	59.7
+align	40.9	63.2	63.9	63.4	66.2	59.5
+w2w (ours)	50.1	66.5	67.6	68.8	71.3	64.9
+FA (ours)	47.2	66.7	65.7	65.4	66.3	62.3
mBART FT	46.8	66.1	68.0	68.7	71.7	64.3
+align	46.4	65.9	67.8	68.5	71.1	63.9
+w2w (ours)	55.6	70.3	72.8	74.7	74.4	69.6
+FA (ours)	55.3	70.1	73.0	74.0	74.0	69.3
Table 13: Word retrieval  $\mathbf{P}@\mathbf{1}$  on the validation set for 626_en-tr-ro-et-my-kk.
stopping if no improvement of the validation loss is observed for 8 checkpoints. The model with the best validation loss was used for evaluation.
D BLEU Scores
We report all the BLEU results of 222_en-ja, 626_en-it-ja-nl-tr-vi, and 626_en-tr-ro-et-my-kk in Table 5, 8 and 9, respectively.
E Sentence Retrieval Precision
We report the sentence retrieval precision for all the systems in Tables 6, 10 and 11. The sentence retrieval previsions are evaluated by using the validation dataset of each language pair. The mean pooled encoder output is used as the sentence embedding. We use cosine similarity to conduct the retrieval task, and report the average retrieval precision of both directions of each language pair.
F Word Retrieval Precision
We report the word retrieval precision for all the systems in Tables 7, 12, and 13. The word retrieval precision are computed by using the validation dataset and the word2word alignments on it. The mean pooled encoder output on corresponding positions is used as the contextualized word embedding. We use cosine similarity to implement the retrieval for word pairs in a batch, and present
the average in-batch retrieval precision of both directions of each language pair. Batch size is set as 512 tokens.