SlowGuess/ACL_2020_OCR huggingface dataset:
UNMT obtains great improvement when combined with XLM, achieving state-of-the-art unsupervised MT performance better than Unsupervised SMT (Artetxe et al., 2019) and Unsupervised NMT (Lample et al., 2018b) across close and distant language pairs.
Figure 2: Visualization of the bilingual word embeddings after Bi-view AT.
ACP+AT/Bi-view AT performs consistently superior to XLM+UNMT. Especially on distant language pairs, ACP+Bi-view AT gains 2.7-9.4 BLEU improvements over the strong XLM+UNMT. This indicates that AT/Bi-view AT with ACP builds closer language spaces via anchored pretraining and anchored training. We present such advantage in the analyses of Section 4.6.
Comparison with Supervised SMT
To check the ability of our system using only the dictionary and non-parallel corpora, we make the comparison to supervised SMT trained on over 4M parallel sentences, which are from WMT19 for English-Russian and from LDC for English-Chinese. We use Moses<sup>9</sup> as the supervised SMT system with a 5-gram language model trained on the target language part of the parallel corpora.
The bottom part of Table 2 shows that ACP+Bi-view AT performs comparable to supervised SMT, and performs even better on English-to-Russian and English-to-Chinese.
4.6 Analyses
We analyze the cross-lingual property of our approaches in both word level and sentence level. We also compare the performances between the ground-truth dictionary and the artificial dictionary. In the end, we vary the size of the bilingual dictionary and report its impact on the AT training.
Effect on Bilingual Word Embeddings
As shown in Figure 2, we depict the word embeddings of some sampled words in English-Chinese after our Bi-view AT. The dimensions of the embedding vectors are reduced to two by using T-SNE and are visualized by the visualization tool in Tensorflow<sup>10</sup>.
We sample the English words that are not covered by the dictionary at first, then search their nearest Chinese neighbors in the embedding space. It shows that the words which constitute a new ground-truth translation pair do appear as neighboring points in the 2-dimensional visualization of Figure 2.
Precision of New Word Pairs
We go on with studying bilingual word embedding by quantitative analysis of the new word pairs, which are detected by searching bilingual words that are neighbors in the word embedding space, and evaluate them using the ground-truth bilingual dictionary. In particular, we split the Muse dictionary of Chinese-to-English into standard training set and test set as in BLI (Artetxe et al., 2018a). The training set is used for the dictionary-based systems, including our AT/Bi-view AT, UNMT+SWET, and Muse, which is a BLI toolkit. The test set is used to evaluate these systems by computing the precision of discovered translation words given the source words in the test set. The neighborhood is computed by CSLS distance (Conneau et al., 2018).
Table 3 shows the precision, where precision@k indicates the accuracy of top-k predicted candidate. Muse induces new word pairs through either the supervised way or the unsupervised way. MuseSupervised is better than MuseUnsupervised since it is supervised by the ground-truth bilingual dictionary. Our AT/Bi-view AT surpasses MuseSupervised by a large margin. UNMT+SWET/uwET also obtains good performance through the word embedding transformation. Bi-view AT significantly surpasses UNMT+SWET/uwET in precision@5 and precision@10, while is worse than them in precision@1. This indicates that Bi-view AT can produce better  $n$ -best translation words that are beneficial for NMT beam decoding to find better translations.
Through the word level analysis, we can see that AT/Bi-view AT leads to more consistent word em
<sup>9</sup>http://www.statmt.org/moses/. We use the default setting of Moses.
$^{10}$ https://projector.tensorflow.org/
MuseUnsupervised	MuseSupervised	UNMT+SWET	UNMT+UWET	AT	Bi-view AT
Precision@1	30.51	35.38	48.01	45.85	43.32	45.49
Precision@5	55.42	58.48	68.05	67.15	68.23	72.02
Precision@10	62.45	63.18	72.02	72.20	73.83	76.71
Figure 3: Sentence level cosine similarity of the parallel sentences on each encoder layer.
bedding space shared by both languages, making the translation between both languages easier.
Sentence Level Similarity of Parallel Sentences
We check the sentence level representational invariance across languages for the cross-lingual pretraining methods. In detail, following Arivazhagan et al. (2018), we adopt max-pooling operation to collect the sentence representation of each encoder layer for all Chinese-to-English sentence pairs in the test set. Then we calculate the cosine similarity for each sentence pair and average all cosine scores.
Figure 3 shows the sentence level cosine similarity. ACP+Bi-view AT consistently has a higher similarity for parallel sentences than XLM+UNMT on all encoder layers. When compare Bi-view AT and AT, the Bi-view AT is better on more encoder layers.
We can see that in both word level and sentence level analysis, our AT methods achieve better crosslingual invariance, significantly reduce the gap between the source language space and the target language space, leading to decreased translation difficulty between both languages.
Ground-Truth Dictionary Vs Artificial Dictionary
Table 4 presents the comparison in English-Chinese. The ground-truth dictionary is from the Muse dictionary deposit, and the artificial dictio
Table 3: Precision of Discovered New Word Pairs.
Ground-Truth Dict.	Artificial Dict.
zh→en	en→zh	zh→en	en→zh
AT	19.83	9.18	16.7	6.98
Bi-view AT	21.16	11.23	18.23	8.50
Table 4: BLEU of AT methods using either the groundtruth dictionary or the artificial dictionary.
XLM+UNMT	20.68
ACP+AT with 1/4 of the dictionary	22.84
ACP+AT with 1/2 of the dictionary	24.32
ACP+AT with the full dictionary	26.80
Table 5: BLEU of ACP+AT using different size of the dictionary in  $\mathrm{{zh}} \rightarrow  \mathrm{{en}}$  translation.
nary is generated by unsupervised BLI (Conneau et al., 2018). We extract top- $n$  word pairs as the artificial dictionary, where  $n$  is the same as the number of entries in the ground-truth dictionary.
Both dictionaries use AT methods for translation. As shown in Table 4, the ground-truth dictionary performs significantly better than the artificial dictionary in both methods and both translation directions.
The Effect of The Dictionary Size
We randomly select a portion of the ground-truth bilingual dictionary to study the effect of the dictionary size on the performance. Table 5 reports the performances of  $\mathrm{ACP + AT}$  using a quarter or a half of the  $\mathrm{zh} \rightarrow \mathrm{en}$  dictionary.
It shows that, in comparison to the baseline of XLM+UNMT that does not use a dictionary, a quarter of the dictionary consisting of around 3k word pairs is capable of improving the performance significantly. More word pairs in the dictionary lead to better translation results, suggesting that expanding the size of the current Muse dictionary via collecting various dictionaries built by human experts may improve the translation performance further.
In the literature of unsupervised MT that only uses non-parallel corpora, Unsupervised SMT (USMT) and Unsupervised NMT (UNMT) are
complementary to each other. Combining them (USMT+UNMT) achieves significant improvement over the individual system, and performs comparable to XLM+UNMT (Lample et al., 2018b; Artetxe et al., 2019).
We have set XLM+UNMT as a stronger baseline, and our ACP+AT/Bi-view AT surpasses it significantly. By referring to the literature of unsupervised MT, we can opt to combine ACP+AT/Bi-view AT with SMT. We leave it as a future work.
In this paper, we explore how much potential an MT system can achieve when only using a bilingual dictionary and large-scale monolingual corpora. This task simulates people acquiring translation ability via looking up the dictionary and depending on no parallel sentence examples. We propose to tackle the task by injecting the bilingual dictionary into MT via anchored training that drives both language spaces closer so that the translation becomes easier. Experiments show that, on both close language pairs and distant language pairs, our proposed approach effectively reduces the gap between the source language space and the target language space, leading to significant improvement of translation quality over the MT approaches that do not use the dictionary and the approaches that use the dictionary to supervise the cross-lingual word embedding transformation.
The authors would like to thank the anonymous reviewers for the helpful comments. This work was supported by National Natural Science Foundation of China (Grant No. 61525205, 61673289), National Key R&D Program of China (Grant No. 2016YFE0132100), and was also partially supported by the joint research project of Alibaba and Soochow University.
Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith. 2016. Massively multilingual word embeddings. ArXiv:1602.01925.
Naveen Arivazhagan, Ankur Bapna, Orhan First, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2018. The missing ingredient in zero-shot neural machine translation. ArXiv:1903.07091.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proc. of EMNLP.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proc. of ACL.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. Unsupervised statistical machine translation. In Proc. of EMNLP.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. In Proc. of ACL.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018c. Unsupervised neural machine translation. In Proc. of ICLR.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018d. Unsupervised neural machine translation. In Proc. of ICLR.
Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In Proc. of EMNLP.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Michael Schuster, Zhi-Feng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In Proc. of ACL.
Trevor Cohn, Steven Bird, Graham Neubig, Oliver Adams, and Adam J. Makarucha. 2017. Crosslingual word embeddings for low-resource language modeling. In Proc. of EACL.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Proc. of ICLR.
Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2016. Learning crosslingual word embeddings without bilingual corpora. In Proc. of EMNLP.
Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proc. of EACL.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. Convolutional sequence to sequence learning. In Proc. of ICML.
Stephan Gouws and Anders Søgaard. 2015. Simple task-specific bilingual word embeddings. In Proc. of HLT-NAACL.
Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. ArXiv:1901.07291.
Guillaume Lample, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In Proc. of ICLR.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. In Proc. of EMNLP.
Angeliki Lazaridou, Georgiana Dinu, and Marco Baroni. 2015. Hubness and pollution: Delving into cross-space mapping for zero-shot learning. In Proc. of ACL.
Ang Lu, Weiran Wang, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Deep multilingual correlation for improved word embeddings. In Proc. of HLT-NAACL.
Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. ArXiv:1309.4168.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proc. of ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proc. of ACL.
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proc. of ICLR.
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2019. Unsupervised bilingual word embedding agreement for unsupervised neural machine translation. In Proc. of ACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NIPS.
Min Xiao and Yuhong Guo. 2014. Distributed word representation learning for cross-lingual dependency parsing. In Proc. of CoNLL.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proc. of HLT-NAACL.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2018. Unsupervised neural machine translation with weight sharing. In Proc. of ACL.
Jiajun Zhang and Chengqing Zong. 2016. Bridging neural machine translation and bilingual dictionaries. ArXiv:1610.07272.
Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi S. Jaakkola. 2016. Ten pairs to tag - multilingual pos tagging via coarse mapping between embeddings. In Proc. of HLT-NAACL.