SlowGuess/ACL_2020_OCR huggingface dataset:
Xiangyu Duan $^{1}$ , Baijun Ji $^{1}$ , Hao Jia $^{1}$ , Min Tan $^{1}$ , Min Zhang $^{1*}$ , Boxing Chen $^{2}$ , Weihua Luo $^{2}$ , Yue Zhang $^{3}$
$^{1}$  Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow university
2 Alibaba DAMO Academy
$^{3}$  School of Engineering, Westlake University
{xiangyuduan,minzhang}  $@$  );suda.edu.cn; {bjihjia,mtan2017}  $@$ stu.suda.edu.cn;
{boxing.cbx,weihua.luowh}  $@$  alibaba-inc.com; yue.zhang@wias.org.cn
In this paper, we propose a new task of machine translation (MT), which is based on no parallel sentences but can refer to a ground-truth bilingual dictionary. Motivated by the ability of a monolingual speaker learning to translate via looking up the bilingual dictionary, we propose the task to see how much potential an MT system can attain using the bilingual dictionary and large scale monolingual corpora, while is independent on parallel sentences. We propose anchored training (AT) to tackle the task. AT uses the bilingual dictionary to establish anchoring points for closing the gap between source language and target language. Experiments on various language pairs show that our approaches are significantly better than various baselines, including dictionary-based word-by-word translation, dictionary-supervised cross-lingual word embedding transformation, and unsupervised MT. On distant language pairs that are hard for unsupervised MT to perform well, AT performs remarkably better, achieving performances comparable to supervised SMT trained on more than 4M parallel sentences<sup>1</sup>.
Motivated by a monolingual speaker acquiring translation ability by referring to a bilingual dictionary, we propose a novel MT task that no parallel sentences are available, while a ground-truth bilingual dictionary and large-scale monolingual corpora can be utilized. This task departs from unsupervised MT task that no parallel resources, including the ground-truth bilingual dictionary, are allowed to utilize (Artetxe et al., 2018c; Lample et al., 2018b). This task is also distinct to
supervised/semi-supervised MT task that mainly depends on parallel sentences (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018; Sennrich et al., 2016a).
The bilingual dictionary is often utilized as a seed in bilingual lexicon induction (BLI) that aims to induce more word pairs within the language pair (Mikolov et al., 2013). Another utilization of the bilingual dictionary is for translating low-frequency words in supervised NMT (Arthur et al., 2016; Zhang and Zong, 2016). We are the first to utilize the bilingual dictionary and the large scale monolingual corpora to see how much potential an MT system can achieve without using parallel sentences. This is different from using artificial bilingual dictionaries generated by unsupervised BLI for initializing an unsupervised MT system (Artetxe et al., 2018c,b; Lample et al., 2018a), we use the ground-truth bilingual dictionary and apply it throughout the training process.
We propose Anchored Training (AT) to tackle this task. Since word representations are learned over monolingual corpora without any parallel sentence supervision, the representation distances between source language and target language are often quite large, leading to significant translation difficulty. As one solution, AT selects words covered by the bilingual dictionary as anchoring points to drive the distance between the source language space and the target language space closer so that translation between the two languages becomes easier. Furthermore, we propose Bi-view AT that places anchors based on either source language view or target language view, and combines both views to enhance the translation quality.
Experiments on various language pairs show that AT performs significantly better than various baselines, including word-by-word translation through looking up the dictionary, unsupervised MT, and dictionary-supervised cross-lingual word embed
$^{1}$ Code is available at https://github.com/mttravel/Dictionary-based-MT
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1570-1579 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ding transformation to make distances between both languages closer. Bi-view AT further improves AT performance due to mutual strengthening of both views of the monolingual data. When combined with cross-lingual pretraining (Lample and Conneau, 2019), Bi-view AT achieves performances comparable to traditional SMT systems trained on more than 4M parallel sentences. The main contributions of this paper are as follows:
- A novel MT task is proposed which can only use the ground-truth bilingual dictionary and monolingual corpora, while is independent on parallel sentences.
- AT is proposed as a solution to the task. AT uses the bilingual dictionary to place anchors that can encourage monolingual spaces of both languages to become closer so that translation becomes easier.
- The detailed evaluation on various language pairs shows that AT, especially Bi-view AT, performs significantly better than various methods, including word-by-word translation, unsupervised MT, and cross-lingual embedding transformation. On distant language pairs that unsupervised MT struggled to be effective, AT and Bi-view AT perform remarkably better.
The bilingual dictionaries used in previous works are mainly for bilingual lexicon induction (BLI), which independently learns the embedding in each language using monolingual corpora, and then learns a transformation from one embedding space to another by minimizing squared euclidean distances between all word pairs in the dictionary (Mikolov et al., 2013; Artetxe et al., 2016). Later efforts for BLI include optimizing the transformation further through new training objectives, constraints, or normalizations (Xing et al., 2015; Lazaridou et al., 2015; Zhang et al., 2016; Artetxe et al., 2016; Smith et al., 2017; Faruqui and Dyer, 2014; Lu et al., 2015). Besides, the bilingual dictionary is also used for supervised NMT which requires large-scale parallel sentences (Arthur et al., 2016; Zhang and Zong, 2016). To our knowledge, we are the first to use the bilingual dictionary for MT without using any parallel sentences.
Our work is closely related to unsupervised NMT (UNMT) (Artetxe et al., 2018c; Lample
et al., 2018b; Yang et al., 2018; Sun et al., 2019), which does not use parallel sentences neither. The difference is that UNMT may use the artificial dictionary generated by unsupervised BLI for initialization (Artetxe et al., 2018c; Lample et al., 2018a) or abandon the artificial dictionary by using joint BPE so that multiple BPE units can be shared by both languages (Lample et al., 2018b). We use the ground-truth dictionary instead and apply it throughout a novel training process. UNMT works well on close language pairs such as English-French, while performs remarkably bad on distant language pairs in which aligning the embeddings of both side languages is quite challenging. We use the ground-truth dictionary to alleviate such problem, and experiments on distant language pairs show the necessity of using the bilingual dictionary.
Other utilizations of the bilingual dictionary for tasks beyond MT include cross-lingual dependency parsing (Xiao and Guo, 2014), unsupervised cross-lingual part-of-speech tagging and semi-supervised cross-lingual super sense tagging (Gouws and Søgaard, 2015), multilingual word embedding training (Ammar et al., 2016; Duong et al., 2016), and transfer learning for low-resource language modeling (Cohn et al., 2017).
There are multiple freely available bilingual dictionaries such as Muse dictionary $^2$  (Conneau et al., 2018), Wiktionary $^3$ , and PanLex $^4$ . We adopt Muse dictionary which contains 110 large-scale groundtruth bilingual dictionaries.
We propose to inject the bilingual dictionary into the MT training by placing anchoring points on the large scale monolingual corpora to drive the semantic spaces of both languages becoming closer so that MT training without parallel sentences becomes easier. We present the proposed Anchored Training (AT) and Bi-view AT in the following.
3.1 Anchored Training (AT)
Since word embeddings are trained on monolingual corpora independently, the embedding spaces of both languages are quite different, leading to significant translation difficulty. AT forces words of a translation pair to share the same word embedding as an anchor. We place multiple anchors by
$^{2}$ https://github.com/facebookresearch/MUSE
<sup>3</sup>https://en.wiktionary.org/wiki/Wiktionary:Main_Page
<sup>4</sup>https://panlex.org/
Figure 1: Illustration of (a) AT and (b) Bi-view AT. We use a source language sentence “ $s_1s_2s_3s_4$ ” and a target language sentence “ $t_1t_2t_3t_4t_5$ ” from the large-scale monolingual corpora as an example.  $\triangleright$  denotes an anchoring point which replaces a word with its translation based on the bilingual dictionary. Thin arrows of  $\downarrow$  denote NMT decoding, thick arrows of  $\Downarrow$  denote training an NMT model,  $\dashrightarrow$  and  $\leftarrow\leftarrow$  denote generating the anchored sentence based on the dictionary. Words with primes such as  $t_1'$  denote the decoding output of a thin arrow.
selecting words covered by the bilingual dictionary. With stable anchors, the embedding spaces of both languages become more and more close during the AT process.
As illustrated in Figure 1 (a), given the source sentence " $s_1s_2s_3s_4$ " with words of  $s_2$  and  $s_3$  being covered by the bilingual dictionary, we replace the two words with their translation words according to the dictionary. This results in the source sentence " $s_1s_{2 \triangleright t}s_{3 \triangleright t}s_4$ ", of which  $s_{2 \triangleright t}$  and  $s_{3 \triangleright t}$  serve as the anchors which are actually the target language words obtained by translating  $s_2$  and  $s_3$  according to the dictionary, respectively. Through the anchors, some words on the source side share the same word embeddings with the corresponding words on the target side. The AT process will strengthen the consistency of embedding spaces of both languages based on these anchors.
The training process illustrated in Figure 1 (a) consists of a mutual back-translation procedure. The anchored source sentence  $s_1 s_{2 \triangleright t} s_{3 \triangleright t} s_4$  is translated into target sentence  $t_1' t_2' t_3''$  by using source-to-target decoding, then  $t_1' t_2' t_3''$  and  $s_1 s_{2 \triangleright t} s_{3 \triangleright t} s_4$  constitute a sentence pair for training the target-to-source translation model. In contrast, the target sentence  $t_1 t_2 t_3 t_4 t_5$  is translated into anchored source sentence  $s_1' s_2' s_{3 \triangleright t}' s_4''$  by using target-to-source decoding, then both sen
tences constitute a sentence pair for training the source-to-target translation model. Note that during training the translation model, the input sentences are always pseudo sentences generated by decoding an MT model, while the output sentences are always true or anchored true sentences. Beside this mutual back-translation procedure, a denoising procedure used in unsupervised MT (Lample et al., 2018b) is also adopted. The deletion and permutation noises are added to the source/target sentence, and the translation model is also trained to denoise them into the original source/target sentence.
During testing, a source sentence is transformed into an anchored sentence at first by looking up the bilingual dictionary. Then we use the source-to-target model trained in the AT process to decode the anchored sentence.
We use Transformer architecture (Vaswani et al., 2017) as our translation model with four stacked layers in both encoder and decoder. In the encoder, we force the last three layers shared by both languages, and leave the first layer not shared. In the decoder, we force the first three layers shared by both languages, and leave the last layer not shared. Such architecture is designed to capture both common and specific characteristics of the two languages in one model for the training.
3.2 Bi-view AT
AT as illustrated in Figure 1 (a) actually tries to model the sentences of both languages in the target language view with partial source words replaced with the target words and the full target language sentence. Bi-view AT enhances AT by adding another language view. Figure 1 (b) adds the source language view shown in the right part to accompany with the target language view of Figure 1 (a). In particular, the target language sentence  $t_1t_2t_3t_4t_5$  is in the form of  $t_1t_2t_{3\succ s}t_4t_{5\succ s}$  after looking up the bilingual dictionary. Such partial target words replaced with the source words and the full source language sentence  $s_1s_2s_3s_4$  constitute the source language view.
Based on the target language view shown in the left part and the source language view shown in the right part, we further combine both views through the pseudo sentences denoted by primes in Figure 1 (b). As shown by “ $\dashrightarrow$ ” in Figure 1 (b), “ $t_1't_2't_3''$ ” is further transformed into “ $t_1't_{2\succ s}'t_{3\succ s}'$ ” by looking up the bilingual dictionary. Similarly, “ $s_1's_2's_3''$ ” is further transformed into “ $s_1's_2's_{3\succ t}'$ ” as shown by “ $\leftarrow$ ”. Finally, solid line box represents training the source-to-target model on data from both views, and dashed line box represents training the target-to-source model on data from both views.
Bi-view AT starts from training both views in parallel. After both views converge, we generate pseudo sentences in both the solid line box and the dashed line box, and pair these pseudo sentences (as input) with genuine sentences (as output) to train the corresponding translation model. This generation and training process iterates until Bi-view AT converges. Through such rich views, the translation models of both directions are mutually strengthened.
3.3 Anchored Cross-lingual Pretraining (ACP)
Cross-lingual pretraining has demonstrated effectiveness on tasks such as cross-lingual classification, unsupervised MT (Lample and Conneau, 2019). It is conducted over large monolingual corpora by masking random words and training to predict them as a cloze task. Instead, we propose ACP to pretrain on data that is obtained by transforming the genuine monolingual corpora of both languages into the anchored version. For example, words in the source language corpus that are cov
ered by the bilingual dictionary are replaced with their translation words respectively. Such words are anchoring points that can drive the pretraining to close the gap between the source language space and the target language space better than the original pretraining method of Lample and Conneau (2019) does as evidenced by the experiments in section 4.5. Such anchored source language corpus and the genuine target language corpus constitute the target language view for ACP.
ACP can be conducted in either the source language view or the target language view. After ACP, each of them is used to initialize the encoder of the corresponding AT system.
3.4 Training Procedure
For AT, the pseudo sentence generation step and NMT training step are interleaved. Take the target language view AT shown in Figure 1 (a) for example, we extract anchored source sentences as one batch, and decode them into pseudo target sentences; then we use the same batch to train the NMT model of target-to-anchored source. In the meantime, a batch of target sentences are decoded into pseudo anchored source sentences, and then we use the same batch to train the NMT model of anchored source-to-target. The above process repeats until AT converges.
For Bi-view AT, after each mono-view AT converging, we set larger batch for generating pseudo sentences as shown in solid/dashed line boxes in Figure 1 (b), and train the corresponding NMT model using the same batch.
For ACP, we follow XLM procedure (Lample and Conneau, 2019), and conduct pretraining on the anchored monolingual corpora concatenated with the genuine corpora of the other language.
We conduct experiments on English-French, English-Russian, and English-Chinese translation to check the potential of our MT system with only bilingual dictionary and large scale monolingual corpora. The English-French task deals with the translation between close-related languages, while the English-Russian and English-Chinese tasks deal with the translation between distant languages that do not share the same alphabets.
For English-French translation task, we use the monolingual data released by XLM (Lample and Conneau, 2019). For English-Russian translation task, we use the monolingual data identical to Lample et al. (2018a), which uses all available sentences for the WMT monolingual News Crawl datasets from years 2007 to 2017. For English-Chinese translation task, we extract Chinese sentences from half of the 4.4M parallel sentences from LDC, and extract English sentences from the complementary half. We use WMT newstest-2013/2014, WMT newstest-2015/2016, and NIST2006/NIST2002 as validation/test sets for English-French, English-Russian, and English-Chinese, respectively.
For cross-lingual pretraining, we extract raw sentences from Wikipedia dumps, which contain 80M, 60M, 13M, 5.5M monolingual sentences for English, French, Russian, and Chinese, respectively.
Muse ground-truth bilingual dictionaries are used for our dictionary-related experiments. If a word has multiple translations, we select the translation word that appears most frequently in the monolingual corpus. Table 1 summarizes the number of word pairs and their coverage on the monolingual corpora on the source side.
entry no.	coverage
fr→en	97,046	60.91%
en→fr	94,719	69.77%
ru→en	45,065	65.43%
en→ru	42,725	88.77%
zh→en	13,749	50.20%
en→zh	32,495	47.02%
Table 1: Statistics of Muse bilingual dictionaries.
For AT/Bi-view AT without cross-lingual pretraining, we use Transformer with 4 layers, 512 embedding/hidden units, and 2048 feed-forward filter size, for fair comparison to UNMT (Lample et al., 2018b). For AT/Bi-view AT with ACP, we set Transformer with 6 layers, 1024 embedding/hidden units, and 4096 feed-forward filter size for a fair comparison to XLM (Lample and Conneau, 2019).
We conduct joint byte-pair encoding (BPE) on the monolingual corpora of both languages with a shared vocabulary of 60k tokens for both English-French and English-Russian tasks, and 40k tokens for English-Chinese task (Sennrich et al., 2016b).
During training, we set the batch size to 32 and limit the sentence length to 100 BPE tokens. We employ the Adam optimizer with  $lr = 0.0001$ ,  $t_{warm\_up} = 4000$  and dropout  $= 0.1$ . At decoding time, we generate greedily with length penalty  $\alpha = 1.0$ .
4.3 Baselines
- Word-by-word translation by looking up the ground truth dictionary or the artificial dictionary generated by Conneau et al. (2018).
- Unsupervised NMT (UNMT) that does not rely on any parallel resources (Lample et al., 2018b) $^6$ . Besides, cross-lingual pretraining (XLM) based UNMT (Lample and Conneau, 2019) $^7$ , is also set as a stronger baseline (XLM+UNMT).
- We implement a UNMT initialized by Unsupervised Word Embedding Transformation (UNMT+UWET) as a baseline(Artetxe et al., 2018d). The transformation function is learned in an unsupervised way without using any ground-truth bilingual dictionaries (Conneau et al., 2018)<sup>8</sup>.
- We also implement a UNMT system initialized by Supervised Word Embedding Transformation (UNMT+SWET) as a baseline. Instead of UWET used in Artetxe et al. (2018d), we use the ground-truth bilingual dictionary as the supervision signal to train the transformation function for transforming the source word embeddings into the target language space (Conneau et al., 2018). After such initialization, the gap between the embedding spaces of both languages is narrowed for easy UNMT training.
4.4 Experimental Results: without Cross-lingual Pretraining
The upper part of Table 2 presents the results of various baselines and our AT approaches. AT and Bi-view AT significantly outperform the baselines, and Bi-view AT is consistently better than AT. Detailed comparisons are listed as below:
Results of Word-by-word Translation
<sup>5</sup>https://github.com/facebookresearch/XLM/blob/master/get-data-nmt.sh
$^{6}$ https://github.com/facebookresearch/UnsupervisedMT
<sup>7</sup>https://github.com/facebookresearch/XLM
$^{8}$ https://github.com/facebookresearch/MUSE
system	fr → en	en → fr	ru → en	en → ru	zh → en	en → zh
Without Cross-lingual Pre-training
Word-by-word using artificial dictionary	7.76	4.88	3.05	1.60	1.99	1.14
Word-by-word using ground-truth dictionary	7.97	6.61	4.17	2.81	2.68	1.79
UNMT (Lample et al., 2018b)	24.02	25.10	9.09	7.98	1.50	0.45
UNMT+SWET	21.11	21.22	9.79	4.07	19.78	7.84
UNMT+UWET	19.80	21.27	8.79	6.21	15.54	6.62
AT	25.07	26.36	10.20	9.91	19.83	9.18
Bi-view AT	27.11	27.54	12.85	10.64	21.16	11.23
With Cross-lingual Pre-training
XLM+UNMT (Lample and Conneau, 2019)	33.28	35.10	17.39	13.29	20.68	11.28
ACP+AT	33.51	36.15	16.41	15.43	26.80	13.91
ACP+Bi-view AT	34.05	36.56	20.09	17.62	30.12	17.05
Supervised SMT	-	-	21.48	14.54	31.86	16.55
Table 2: Experiment results evaluated by BLEU using the multi-bleu script.
It shows that using the ground-truth dictionary is slightly better than using the artificial one generated by Conneau et al. (2018). Both performances are remarkably bad, indicating that simple word-by-word translation is not qualified as an MT method. More effective utilization of the bilingual dictionary is needed to improve the translation performance.
Comparison between UNMT and UNMT with WET Initialization
UNMT-related systems generally improves the performance of the word-by-word translation. On the close-related language pair of English-French, UNMT is better than UNMT+UWET/SWET. This is partly because there are numerous BPE units shared by both English and French, enabling easy establishing the shared word embedding space of both languages. In contrast, WET that transforms the source word embedding into the target language space seems not a necessary initialization step since shared BPE units already establish the shared space.
On distant language pairs, UNMT does not have an advantage over UNMT with WET initialization. Especially on English-Chinese, UNMT performs extremely bad, even worse than the word-by-word translation method. We argue that this is because the BPE units shared by both languages are so few that UNMT fails to align the language spaces. In contrast, using the bilingual dictionary greatly alleviate such problem for distant language pairs. UNMT+SWET, which transforms the source word embedding into the target word embedding space supervised by the bilingual dictionary, outperforms UNMT by more than 18 BLEU points on Chinese-to-English and more than 7 BLEU points
on English-to-Chinese. This indicates the necessity of the bilingual dictionary for translation between distant language pairs.
Comparison between AT/Bi-view AT and The Baselines
Our proposed AT approaches significantly outperform the baselines. The baselines of using the ground-truth bilingual dictionary, i.e., word-by-word translation using the dictionary and UNMT+SWET that uses the dictionary to supervise the word embedding transformation, are inferior to our AT approaches.
The AT approaches consistently improves the performances over both close-related language pair of English-French and distant language pairs of English-Russian and English-Chinese. Our Bi-view AT achieves the best performance on all language pairs.
4.5 Experimental Results: with Cross-lingual Pretraining
The bottom part of Table 2 reports performances of UNMT with XLM, which conducts the cross-lingual pretraining on concatenated non-parallel corpora (Lample and Conneau, 2019), and performances of our AT/Bi-view AT with the anchored cross-lingual pretraining, i.e., ACP. The results show that our proposed AT approaches are still superior when equipped with the cross-lingual pretraining.