SlowGuess/ACL_2020_OCR huggingface dataset:
Haipeng Sun $^{1*}$ , Rui Wang $^{2}$ , Kehai Chen $^{2}$ , Masao Utiyama $^{2}$ , Eiichiro Sumita $^{2}$ , and Tiejun Zhao $^{1}$
$^{2}$ National Institute of Information and Communications Technology (NICT), Kyoto, Japan
hpsun@hit-mtlab.net,tjzhao@hit.edu.cn
{wangrui, khchen, mutiyama, eiichiro-sumita}@nict.go.jp
Unsupervised neural machine translation (UNMT) has recently achieved remarkable results for several language pairs. However, it can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time. That is, research on multilingual UNMT has been limited. In this paper, we empirically introduce a simple method to translate between thirteen languages using a single encoder and a single decoder, making use of multilingual data to improve UNMT for all language pairs. On the basis of the empirical findings, we propose two knowledge distillation methods to further enhance multilingual UNMT performance. Our experiments on a dataset with English translated to and from twelve other languages (including three language families and six language branches) show remarkable results, surpassing strong unsupervised individual baselines while achieving promising performance between non-English language pairs in zero-shot translation scenarios and alleviating poor performance in low-resource language pairs.
Recently, neural machine translation (NMT) has been adapted to the unsupervised scenario in which NMT is trained without any bilingual data. Unsupervised NMT (UNMT) (Artetxe et al., 2018; Lample et al., 2018a) requires only monolingual corpora. UNMT achieves remarkable results by using a combination of diverse mechanisms (Lample et al., 2018b) such as an initialization with bilingual word embeddings, denoising auto-encoder (Vincent et al., 2010), back-translation (Sennrich et al., 2016a), and shared latent representation. More recently, Lample and Conneau (2019) achieves better
UNMT performance by introducing the pretrained language model. However, conventional UNMT can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time (Wang et al., 2020).
Multilingual UNMT (MUNMT) translating multiple languages at the same time can save substantial training time and resources. Moreover, the performance of MUNMT in similar languages can promote each other. Research on MUNMT has been limited and there are only a few pioneer studies. For example, Xu et al. (2019) and Sen et al. (2019) proposed a multilingual scheme that jointly trains multiple languages with multiple decoders. However, the performance of their MUNMT is much worse than our re-implemented individual baselines (shown in Tables 2 and 3) and the scale of their study is modest (i.e., 4-5 languages).
In this paper, we empirically introduce an unified framework to translate among thirteen languages (including three language families and six language branches) using a single encoder and single decoder, making use of multilingual data to improve UNMT for all languages. On the basis of these empirical findings, we propose two knowledge distillation methods, i.e., self-knowledge distillation and language branch knowledge distillation, to further enhance MUNMT performance. Our experiments on a dataset with English translated to and from twelve other languages show remarkable results, surpassing strong unsupervised individual baselines. This paper primarily makes the following contributions:
- We propose a unified MUNMT framework to translate between thirteen languages using a single encoder and single decoder. This paper is the first step of multilingual UNMT training on a large scale of European languages.
- We propose two knowledge distillation meth
* Haipeng Sun was an internship research fellow at NICT when conducting this work.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3525-3535 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ods for MUNMT and our proposed knowledge distillation methods consider linguistic knowledge in the specific translation task.
- Our proposed MUNMT system achieves state-of-the-art performance on the thirteen languages. It also achieves promising performance in zero-shot translation scenarios and alleviates poor performance in low-resource language pairs.
2 Background of UNMT
UNMT can be decomposed into four components: cross-lingual language model pretraining, denoising auto-encoder, back-translation, and shared latent representations. For UNMT, two monolingual corpora  $X^{1} = \{X_{i}^{1}\}$  and  $X^{2} = \{X_{i}^{2}\}$  in two languages  $L_{1}$  and  $L_{2}$  are given.  $|X^{1}|$  and  $|X^{2}|$  are the number of sentences in monolingual corpora  $\{X_{i}^{1}\}$  and  $\{X_{i}^{2}\}$  respectively.
2.1 Cross-lingual Language Model Pretraining
A cross-lingual masked language model, which can encode two monolingual sentences into a shared latent space, is first trained. The pretrained cross-lingual encoder is then used to initialize the whole UNMT model (Lample and Conneau, 2019). Compared with previous bilingual embedding pretraining (Artetxe et al., 2018; Lample et al., 2018a; Yang et al., 2018; Lample et al., 2018b; Sun et al., 2019), this pretraining can provide much more cross-lingual information, causing the UNMT model to achieve better performance and faster convergence.
2.2 Denoising Auto-encoder
Noise obtained by randomly performing local substitutions and word reorderings (Vincent et al., 2010; Hill et al., 2016; He et al., 2016), is added to the input sentences to improve model learning ability and regularization. Consequently, the input data are continuously modified and are different at each epoch. The denoising auto-encoder model objective function can be minimized by encoding a noisy sentence and reconstructing it with the decoder in the same language:
$$
\begin{array}{l} \mathcal {L} _ {D} = \sum_ {i = 1} ^ {| X ^ {1} |} - l o g P _ {L _ {1} \to L _ {1}} (X _ {i} ^ {1} | C (X _ {i} ^ {1})) \\ + \sum_ {i = 1} ^ {| X ^ {2} |} - \log P _ {L _ {2} \rightarrow L _ {2}} \left(X _ {i} ^ {2} \mid C \left(X _ {i} ^ {2}\right)\right), \\ \end{array}
$$
where  $\{C(X_i^1)\}$  and  $\{C(X_i^2)\}$  are noisy sentences.  $P_{L_1\to L_1}$  and  $P_{L_2\to L_2}$  denote the reconstruction probability in language  $L_{1}$  and  $L_{2}$ , respectively.
2.3 Back-translation
Back-translation (Sennrich et al., 2016a) plays a key role in achieving unsupervised translation that relies only on monolingual corpora in each language. The pseudo-parallel sentence pairs  $\{(M^2(X_i^1), X_i^1)\}$  and  $\{(M^1(X_i^2), X_i^2)\}$  produced by the model in the previous iteration are used to train the new translation model. Therefore, the back-translation objective function can be optimized by minimizing:
$$
\begin{array}{l} \mathcal {L} _ {B} = \sum_ {i = 1} ^ {| X ^ {1} |} - \log P _ {L _ {2} \rightarrow L _ {1}} \left(X _ {i} ^ {1} \mid M ^ {2} \left(X _ {i} ^ {1}\right)\right) \tag {2} \\ + \sum_ {i = 1} ^ {| X ^ {2} |} - l o g P _ {L _ {1} \rightarrow L _ {2}} \left(X _ {i} ^ {2} \mid M ^ {1} \left(X _ {i} ^ {2}\right)\right), \\ \end{array}
$$
where  $P_{L_1\to L_2}$  and  $P_{L_2\to L_1}$  denote the translation probability across the two languages.
2.4 Sharing Latent Representations
Encoders and decoders are (partially) shared between  $L_{1}$  and  $L_{2}$ . Therefore,  $L_{1}$  and  $L_{2}$  must use the same vocabulary. The entire training of UNMT needs to consider back-translation between the two languages and their respective denoising processes. In summary, the entire UNMT model can be optimized by minimizing:
$$
\mathcal {L} _ {\text {a l l}} = \mathcal {L} _ {D} + \mathcal {L} _ {B}. \tag {3}
$$
3 Multilingual UNMT (MUNMT)
3.1 Multilingual Pretraining
Motivated by Lample and Conneau (2019), we construct a multilingual masked language model, using a single encoder. For each language, the language model is trained by encoding the masked input and reverting it with this encoder. This pretrained multilingual language model is used to initialize the full set of parameters of MUNMT.
3.2 Multilingual UNMT Training
We have established a MUNMT model on  $N$  languages with a single encoder and single decoder. We denote a sentence in language  $L_{j}$  as  $X_{i}^{j}$ . For example,  $L_{1}$  indicates English.  $|X^{j}|$  is the number of sentences in the corpus  $X^{j} = \{X_{i}^{j}\}$ .
Figure 1: MUNMT architecture. We take  $L_{1} \leftrightarrow L_{j}$  time-step as an example. The grey symbols indicate that the corresponding data are not used or generated during this time-step.
As Figure 1 shows, the entire training process of the MUNMT model is performed through the denoising and back-translation mechanisms, between English and non-English language pairs, by minimizing:
$$
\mathcal {L} _ {\text {M U N M T}} = \mathcal {L} _ {M D} + \mathcal {L} _ {M B}, \tag {4}
$$
where  $\mathcal{L}_{MD}$  denotes the denoising function and  $\mathcal{L}_{MB}$  denotes the back-translation function.
In the denoising training, noise (in the form of random token deletion and swapping) is introduced into the input sentences for any language  $L_{j}$ . The denoising auto-encoder, which encodes a noisy version and reconstructs it with the decoder in the same language, is optimized by minimizing:
$$
\mathcal {L} _ {M D} = \sum_ {j = 1} ^ {N} \sum_ {i = 1} ^ {| X ^ {j} |} - \log P _ {L _ {j} \rightarrow L _ {j}} \left(X _ {i} ^ {j} \mid C \left(X _ {i} ^ {j}\right)\right), \tag {5}
$$
where  $\{C(X_i^j)\}$  is a set of noisy sentences for language  $L_{j}$ .  $P_{L_j\to L_j}$  denotes the reconstruction probability in  $L_{j}$ .
In this paper, we primarily focus on the translation from English to other languages or from other languages to English. This is because most test dataset contains English. In the process of back-translation training, we only conduct back-translation from language  $L_{1}$  (English) to other languages and back-translation from other languages to language  $L_{1}$ . For any non-English language  $L_{j}$ , the pseudo-parallel sentence pairs  $\{(M^{j}(X_{i}^{1}), X_{i}^{1})\}$  and  $\{(M^{1}(X_{i}^{j}), X_{i}^{j})\}$  are obtained by the previous model in the  $L_{1} \rightarrow L_{j}$
and  $L_{j}\rightarrow L_{1}$  direction, respectively. Therefore, the back-translation objective function can be optimized on these pseudo-parallel sentence pairs by minimizing:
$$
\begin{array}{l} \mathcal {L} _ {M B} = \sum_ {j = 2} ^ {N} \sum_ {i = 1} ^ {| X ^ {1} |} - \log P _ {L _ {j} \rightarrow L _ {1}} \left(X _ {i} ^ {1} \mid M ^ {j} \left(X _ {i} ^ {1}\right)\right) \tag {6} \\ + \sum_ {j = 2} ^ {N} \sum_ {i = 1} ^ {| X ^ {j} |} - l o g P _ {L _ {1} \to L _ {j}} (X _ {i} ^ {j} | M ^ {1} (X _ {i} ^ {j})), \\ \end{array}
$$
where  $P_{L_1\to L_j}$  and  $P_{L_j\to L_1}$  denote the translation probabilities, in each direction, between any non-English language and English.
4 Knowledge Distillation for MUNMT
To further enhance the performance of our proposed MUNMT described in Section 3, we propose two knowledge distillation methods: self-knowledge distillation (Algorithm 1) and language branch knowledge distillation (Algorithm 2). Figure 2 illustrates the architecture of MUNMT and the proposed knowledge distillation methods.
Generally, during UNMT training, an objective function  $\mathcal{L}_{KD}$  is added, to enhance the generalization ability of the MUNMT model. The general
(b)
Figure 2: (a) Architecture of MUNMT with self-knowledge distillation; (b) Architecture of MUNMT with language branch knowledge distillation. Similar as Figure 1, we take  $L_{1} \leftrightarrow L_{j}$  time-step as an example. The blue lines denote our proposed knowledge distillation methods are added in the MUNMT training.
MUNMT objective function can be reformulated as follows:
$$
\mathcal {L} _ {M U N M T} = \mathcal {L} _ {M D} + \mathcal {L} _ {M B ^ {\prime}},
$$
$$
\mathcal {L} _ {M B ^ {\prime}} = (1 - \alpha) \mathcal {L} _ {M B} + \alpha T ^ {2} \mathcal {L} _ {K D}, \tag {7}
$$
where  $\alpha$  is a hyper-parameter that adjusts the weight of the two loss functions during backtranslation.  $T$  denotes the temperature used on the softmax layer. If the temperature is higher, the probability distribution obtained would be softer (Hinton et al., 2015).
4.1 Self-knowledge Distillation
On the basis of the existing architecture of MUNMT, we introduce self-knowledge distillation (Hahn and Choi, 2019) (SKD) during backtranslation, to enhance the generalization ability of the MUNMT model, as shown in Figure 2(a). Unlike Hahn and Choi (2019)'s method, using two soft target probabilities that are based on the word embedding space, we make full use of multilingual information via self-knowledge distillation.
During back-translation, only language  $L_{j}$  sentences  $M^{j}(X_{i}^{1})$  are generated before training the MUNMT model in the  $L_{j} \rightarrow L_{1}$  direction. However, other languages, which have substantial multilingual information, are not used during this training. Motivated by this, we propose to introduce another language  $L_{z}$  (randomly chosen but dis
tinct from  $L_{1}$  and  $L_{j}$ ) during this training. We argue that the translation from the source sentences through different paths,  $L_{1} \rightarrow L_{j} \rightarrow L_{1}$  and  $L_{1} \rightarrow L_{z} \rightarrow L_{1}$ , should be similar. The MUNMT model matches not only the ground-truth output of language  $L_{j}$  sentences  $M^{j}(X_{i}^{1})$ , but also the soft probability output of language  $L_{z}$  sentences  $M^{z}(X_{i}^{1})$ . The opposite direction is similar. Therefore, this MUNMT model is optimized by minimizing the objective function:
$$
\begin{array}{l} \mathcal {L} _ {M B ^ {\prime}} = (1 - \alpha) \mathcal {L} _ {M B} + \alpha T ^ {2} \mathcal {L} _ {S K D}, \\ \mathcal {L} _ {S K D} = \sum_ {j = 2} ^ {N} \sum_ {i = 1} ^ {| X ^ {1} |} K L (X ^ {1} (M ^ {j} (X _ {i} ^ {1})), X ^ {1} (M ^ {z} (X _ {i} ^ {1}))) \\ + \sum_ {j = 2} ^ {N} \sum_ {i = 1} ^ {| X ^ {j} |} K L \left(X ^ {j} \left(M ^ {1} \left(X _ {i} ^ {j}\right)\right), X ^ {j} \left(M ^ {z} \left(X _ {i} ^ {j}\right)\right)\right), \tag {8} \\ \end{array}
$$
where  $KL(\cdot)$  denotes the KL divergence. It is computed over full output distributions to keep these two probability distributions similar. For any language  $L_{j},X^{1}(M^{j}(X_{i}^{1}))$  and  $X^{1}(M^{z}(X_{i}^{1}))$  denote the softened  $L_{1}$  sentence probability distribution after encoding  $M^{j}(X_{i}^{1})$  and  $M^{z}(X_{i}^{1})$  respectively.  $M^{j}(X_{i}^{1})$  and  $M^{z}(X_{i}^{1})$  were generated by the previous model in the  $L_{1}\rightarrow L_{j}$  and  $L_{1}\to L_{z}$  directions, respectively.  $X^{j}(M^{1}(X_{i}^{j}))$  and  $X^{j}(M^{z}(X_{i}^{j}))$  denote the softened  $L_{j}$  sentence probability distribution after encoding  $M^{1}(X_{i}^{j})$
Figure 3: The language distribution of our selected languages.
Algorithm 2 The LBKD algorithm
Monolingual training data  $X^{1},X^{2},\dots ,X^{N}$  LBUNMT models  $\theta_1^{LB},\theta_2^{LB},\dots ,\theta_M^{LB};$  The pretrained model  $\theta_0$  Number of steps  $K$
1: Initialize  $\theta \gets \theta_0$
2: while Step  $q \leq \max \text{step } K$  do
3: for  $j = 1$ ;  $j < N$ ;  $j + +$  do
4: Sample batch  $\{X_{j}^{j}\}$  from  $X^{j}$
5: Compute denoising loss  $\mathcal{L}_{MD}$
6: Update  $\theta \gets$  optimizer  $(\mathcal{L}_{MD})$
7: end for
8: for  $j = 2$ ;  $j < N$ ;  $j + +$  do
9: Sample batch  $\{X_{i}^{1}\}$  from  $X^{1}$
10: Compute back-translation loss  $\mathcal{L}_{MB}$
11: Select LBUNMT language  $L_{1}$  belongs and compute distillation loss  $\mathcal{L}_{LBKD}$
12: Update  $\theta \gets$  optimizer  $(\mathcal{L}_{MB} + \mathcal{L}_{LBKD})$
13: Sample batch  $\{X_{j}^{j}\}$  from  $X^{j}$
14: Compute back-translation loss  $\mathcal{L}_{MB}$
15: Select LBUNMT language  $L_{j}$  belongs and compute distillation loss  $\mathcal{L}_{LBKD}$
16: Update  $\theta \gets$  optimizer  $(\mathcal{L}_{MB} + \mathcal{L}_{LBKD})$
17: end for
18: end while
and  $M^z (X_i^j)$ , respectively.  $M^{1}(X_{i}^{j})$  and  $M^{z}(X_{i}^{j})$  were generated by the previous model in the  $L_{j}\rightarrow L_{1}$  and  $L_{j}\to L_{z}$  directions, respectively. Note that zero-shot translation was used to translate language  $L_{j}$  to language  $L_{z}$ . The direction  $L_{j}\rightarrow L_{z}$  was not trained during MUNMT training.
4.2 Language Branch Knowledge Distillation
We consider thirteen languages: Czech (Cs), German (De), English (En), Spanish (Es), Estonian (Et), Finnish (Fi), French (Fr), Hungarian
(Hu), Lithuanian (Lt), Latvian (Lv), Italian (It), Romanian (Ro), and Turkish (Tr), which belong to three language families including several language branches (Lewis, 2009) as shown in Figure 3.
As shown in Figure 2(b), we propose knowledge distillation within a language branch (LBKD), to improve MUNMT performance through the existing teacher models. To the best of our knowledge, this is the first proposal that aims to distill knowledge within a language branch. As the number of languages increases, the cost of training time and resources to train an individual model on any two languages increases rapidly. An alternative knowledge distillation method within a language branch can avoid this prohibitive computational cost. Because languages in the same language branch are similar, we first train small multilingual models across all languages in the same language branch (LBUNMT) before training MUNMT. The LBUNMT model trained in the same language branch performed better than the single model because similar languages have a positive interaction during the training process as shown in Tables 2 and 3. Therefore, the distilled information of LBUNMT is used to guide the MUNMT model during backtranslation. The MUNMT model matches both the ground-truth output and the soft probability output of LBUNMT. Therefore, this MUNMT model is optimized by minimizing the objective function:
$$
\begin{array}{l} \mathcal {L} _ {M B ^ {\prime}} = (1 - \alpha) \mathcal {L} _ {M B} + \alpha T ^ {2} \mathcal {L} _ {L B K D}, \\ \mathcal {L} _ {L B K D} = \sum_ {j = 2} ^ {N} \sum_ {i = 1} ^ {| X ^ {1} |} K L \left(X ^ {1} \left(M ^ {j} \left(X _ {i} ^ {1}\right)\right), L B ^ {1} \left(M ^ {j} \left(X _ {i} ^ {1}\right)\right)\right) \\ + \sum_ {j = 2} ^ {N} \sum_ {i = 1} ^ {| X ^ {j} |} K L \left(X ^ {j} \left(M ^ {1} \left(X _ {i} ^ {j}\right)\right), L B ^ {j} \left(M ^ {1} \left(X _ {i} ^ {j}\right)\right)\right), \tag {9} \\ \end{array}
$$
where  $X^{1}(M^{j}(X_{i}^{1}))$  and  $LB^{1}(M^{j}(X_{i}^{1}))$  denote
the softened  $L_{1}$  sentence probability distribution of the MUNMT and LBUNMT models, respectively, after encoding  $M^{j}(X_{i}^{1})$  generated by the previous MUNMT model in the  $L_{1} \rightarrow L_{j}$  direction.  $X^{j}(M^{1}(X_{i}^{j}))$  and  $LB^{j}(M^{1}(X_{i}^{j}))$  denote the softened  $L_{j}$  sentence probability distribution of the MUNMT and LBUNMT models, respectively, after encoding  $M^{1}(X_{i}^{j})$  generated by the previous MUNMT model in the  $L_{j} \rightarrow L_{1}$  direction.
To establish an MUNMT system, we considered 13 languages from WMT monolingual news crawl datasets: Cs, De, En, Es, Et, Fi, Fr, Hu, It, Lt, Lv, Ro, and Tr. For preprocessing, we used the Moses tokenizer (Koehn et al., 2007). For cleaning, we only applied the Moses script clean-corpus-n.perl to remove lines in the monolingual data containing more than 50 words. We then used a shared vocabulary for all languages, with 80,000 sub-word tokens based on BPE (Sennrich et al., 2016b). The statistics of the data are presented in Table 1. For Cs,De,En, we randomly extracted 50M monolingual news crawl data after cleaning; For other languages, we used all news crawl data after cleaning as shown in Table 1.
Language	Sentences	Words	Sub-words
Cs	50.00M	860.36M	1.16B
De	50.00M	887.37M	1.19B
En	50.00M	1.15B	1.32B
Es	36.33M	1.01B	1.19B
Et	3.00M	51.39M	101.43M
Fi	15.31M	189.39M	359.78M
Fr	50.00M	1.19B	1.38B
Hu	34.35M	708.13M	1.03B
It	30.82M	755.56M	911.51M
Lt	0.34M	6.38M	14.64M
Lv	8.60M	172.56M	281.54M
Ro	8.92M	207.07M	279.95M
Tr	9.14M	153.03M	254.70M
Table 1: Statistics of monolingual corpora.
We report the results for WMT newstest2013 for Cs-En, De-En, Es-En, and Fr-En. We can evaluate the translation performance between pairs of non-English languages because newstest2013 includes these five languages parallel to each other. For other language pairs, we chose the newest WMT newstest set. That is, we reported the results on WMT newstest2019 for Fi-En and Lt-En; WMT
newstest2018 for Et-En and Tr-En; WMT newstest2017 for Lv-En; WMT newstest2016 for Ro-En; and WMT newstest2009 for Hu-En and It-En. Note that the versions of newstest2019 on Fi/Lt  $\rightarrow$  En and  $\mathrm{En}\to \mathrm{Fi} / \mathrm{Lt}$  are different. We chose the corresponding newstest2019 for each direction.
5.2 Language Model and UNMT Settings
We used a transformer-based XLM toolkit to train a multilingual masked language model and followed the settings used in Lample and Conneau (2019): six layers were used for the encoder. The dimension of hidden layers was set to 1024. The Adam optimizer (Kingma and Ba, 2015) was used to optimize the model parameters. The initial learning rate was  $0.0001$ ,  $\beta_{1} = 0.9$ , and  $\beta_{2} = 0.98$ .
We used the same toolkit and followed the settings of UNMT used in (Lample and Conneau, 2019): six layers were used for the encoder and decoder. The batch size was set to 2000 tokens. The other parameters were the same as those used for training language model. For our proposed knowledge distillation method,  $\alpha$  was set to 0.1 and  $T$  was set to 2 (the parameters are empirically selected by small-scale experiments and most of the settings achieved good results). The cross-lingual language model was used to pretrain the encoder and decoder of the whole UNMT model. All monolingual data, described in Table 1, were used in the pretraining and MUNMT training phase. The parameters of the multilingual and single models were the same.
For evaluation, we used the case-sensitive BLEU scores computed by the Moses script multi-bleu.perl. We executed a single model (two languages) for 60,000 iterations, a small multilingual model (three to five languages) for 30,000 iterations, and a large multilingual model (13 languages) for 15,000 iterations. Eight V100 GPUs were used to train all UNMT models. The single model was trained for approximately two days; the multilingual model (13 languages) costs approximately six days since 13 languages participated in the training.
Tables 2 and 3 present the detailed BLEU scores of all systems on the English and non-English language pairs, in each direction<sup>1</sup>. Our observations
The translation quality of pretrained model was not presented in the Tables 2 and 3. The result was poor because the pretrained model (cross-lingual language model) was trained within an encoder. The encoder and decoder of UNMT was
Corpus	SNMT	Sen et al. (2019)	Xu et al. (2019)	SM	LBUNMT	MUNMT	SKD	LBKD
En-Cs	19.20	-	6.79	14.54	14.54	14.40	14.89	15.47
En-De	20.30	8.09	13.25	18.26	18.26	17.58	18.47	19.28
En-Es	30.40	14.82	20.43	25.14	25.40	25.05	25.61	26.79
En-Et	25.20	-	-	14.86	15.02	14.09	15.03	15.62
En-Fi	27.40	-	-	9.87	9.99	9.75	10.70	10.57
En-Fr	30.60	13.71	20.27	26.02	26.36	25.84	26.45	27.78
En-Hu	-	-	-	11.32	11.40	10.90	11.64	12.03
En-It	-	-	-	24.19	24.30	23.80	24.69	25.52
En-Lt	20.10	-	-	0.79	8.29	10.07	11.15	11.11
En-Lv	21.10	-	-	1.02	11.55	13.09	13.90	14.33
En-Ro	28.90	-	-	29.44	29.58	28.82	29.65	31.28
En-Tr	20.00	-	-	11.87	11.87	12.41	13.24	13.83
Average	-	-	-	15.61	17.21	17.15	17.95	18.63