SlowGuess/ACL_2020_OCR huggingface dataset:
Table 3: Scores of baseline systems.
BLEU↑	TER↓	MET.↑	CHRF3↑
EU-ES	+RBMTbt	23.27	62.67	48.02	56.51
+PB-SMTbt	22.51	64.57	45.97	54.53
+LSTMbt	24.74	63.55	47.58	55.59
BT. +Transformerbt	25.70	60.29	48.53	57.08
+Allbt	26.18	59.10	49.19	57.31
Auth. FromAll	25.93	59.76	48.66	56.69
BT. EachFromAll	25.85	58.92	48.83	57.17
+ EachFromAll x4	24.59	61.15	48.10	56.19
DS EachFromAll RS	25.77	59.86	48.59	56.92
DE-EN	+RBMTbt	39.02	42.27	37.32	62.72
Auth. +PB-SMTbt	42.32	39.21	39.37	65.91
+LSTMbt	40.97	39.75	38.45	64.81
BT +Transformerbt	42.75	38.73	39.35	66.05
+Allbt	42.69	38.45	39.65	65.99
Auth. FromAll	43.66	37.71	40.10	67.01
+ BT EachFromAll	43.45	38.24	39.81	66.44
+ DS EachFromAll RS	43.98	37.79	39.91	67.10
Table 4: Scores for systems trained on authentic (Auth.) and backtranslated (BT) data, and after data selection (DS). MET. abbreviates METEOR.
We observe from Table 4 that for both language pairs the inclusion of backtranslated data clearly improves the results of the baseline systems. For EU-ES the ordering of the systems from best to
worse is Transformer  $>$  RBMT  $>$  LSTM  $>$  PB-SMT for all metrics except BLEU, where the order is Transformer  $>$  LSTM  $>$  RBMT  $>$  PB-SMT. The EU-ES system trained on (authentic data and) data translated by all systems  $(+\mathrm{All}_{bt})$ , thus using 4 times more backtranslated data than the rest, obtains the best results; however, the observed improvements are not as high as those for the other systems, e.g. the best  $(+\mathrm{Transformer}_{bt})$  has a 0.96 BLEU point improvement over the second best  $(+\mathrm{LSTM}_{bt})$ , while the  $+\mathrm{All}_{bt}$  system is only 0.48 BLEU points better than  $+\mathrm{Transformer}_{bt}$ . This tendency is the same for the other metrics too. For the DE-EN use-case the score differences between the best systems  $(+\mathrm{Transformer}_{bt}$  or  $+\mathrm{PB - SMT}_{bt}$  depending on the metric) and  $+\mathrm{All}_{bt}$  are even smaller, with BLEU and chrF3 favouring the former, and TER and METEOR the latter.
For EU-ES, all systems trained with 2M sentence pairs selected from the backtranslated data according to the basic DS methods and the newly proposed method with rescoring obtain better results than any system trained with backtranslated data originating from a single system. Furthermore, according to all metrics except BLEU, the EachFromAll system outperforms FromAll. Compared to the system including the data translated by all systems  $(+\mathrm{All}_{bt})$ , EachFromAll is better only in terms of TER. These results show that either the quantity of data leads to differences in performance (comparing the best system after data selection, i.e. EachFromAll, to  $+\mathrm{All}_{bt}$ ), or that the data selection method fails to retrieve those sentence pairs that would lead to better performance. In order to test these two assumptions, we first train a system with the EachFromAll data repeated 4 times resulting in the same number of sentence pairs as in the  $+\mathrm{All}_{bt}$  case. According to the resulting evaluation scores, this system is worse than  $+\mathrm{All}_{bt}$ , but also worse than any of the basic data selection configurations. This indicates that the diversity (among the source sentences) gained by using 4 different systems for backtranslation is more important than the quantity of the data in terms of automatic scores. While for EU-ES the EachFromAll selection configuration achieves the best results, for DE-EN the FromAll configuration leads to better scores. Furthermore, this configuration outperforms the system with all backtranslated data  $(+\mathrm{All}_{bt})$ .
Next, we train a system with data selected from the backtranslated data after the original FDA
scores have been rescored using the quality and lexical diversity/richness scores. These systems are shown in Table 4 with the suffix RS (i.e. ReScored). While for EU-ES this system does not outperform the rest, in the DE-EN case we observe that it does. With the exception of the TER and METEOR scores, the EachFromAll RS for the DE-EN language pair is the best system. These experiments show different outcomes for each language pair and thus disagree with respect to our hypothesis of rescoring the data selection scores being beneficial for MT. Accordingly, more experiments are needed to specify how to perform this rescoring, as well as in which settings our rescoring proposal is beneficial. Further analysis and a discussion on lexical diversity/richness, data selection and sentence length follow in the rest of this section.
6.2 Lexical Diversity/Richness
We analyse the lexical diversity/richness of the corpora of both language pairs based on the Yule's I, MTLD and TTR metrics. We calculate these scores for the corpora resulting from backtranslation by the different systems (BT), for the corpora resulting from applying the basic data selection approaches (DS), and the development and test sets used for evaluation (EV). We show these scores in Table 5 and Table 6 for EU-ES and DE-EN, respectively.
Regarding the different systems used for backtranslation, we observe that for EU-ES the sentences translated by the RBMT system are much more diverse than the rest according to all metrics, while Transformer obtains the highest scores among the other three. For the DE-EN corpora, this is not the case, and the data from the Transformer system is more diverse according to Yule's I and TTR, but not according to MTLD.
We note that Yule's I and TTR depend on the amount of sentences in the assessed corpora. As such, we can see that for the development and test sets the scores are quite a bit higher than the rest. Accordingly, comparisons should be only be conducted for corpora with the same number of sentences.
Following the analysis and discussion in Vanmassenhove et al. (2019), we decided to use MTLD as the lexical diversity metric for our rescoring data selection approach, as defined in Section 3.
6.3 Systems Selected by Data Selection
We first analyse how the basic data selection methods choose different numbers of sentences from
Type	Corpus	Yule's EU	I*100 ES	MTLD EU	ES	TTR * 100 EU	ES
BT	\( RBMT_{bt} \)	74.3	15.33	3.70
\( PB-SMT_{bt} \)	0.40	0.91	13.76	14.06	1.01	1.01
\( LSTM_{bt} \)	3.23	13.20	2.77
\( Trans_{bt} \)	8.19	13.79
DS	FA	2.81	0.16	13.73	13.91	2.26	0.42
EFA	5.78	0.91	13.88	14.03	3.08	1.01
EFA RS	9.54	0.91	13.84	14.03	3.67	1.01
EV	Dev.	626	456	13.72	13.92	32.90	27.50
Test	663	491	13.63	13.75	32.80	27.50
Table 5: Lexical diversity scores of the backtranslation (BT), data selection (DS) and evaluation (EV) corpora for the ES-EU and EU-ES systems. Trans. = Transformer, FA = ForAll, EFA = EachFromAll, EFA RS = EachFromAll Rescored.
Type	Corpus	Yule's I*100	MTLD	TTR * 100
DE	EN	DE	EN	DE	EN
BT	\( RBMT_{bt} \)	4.55	48.50	1.64
\( PB-SMT_{bt} \)	0.66	2.68	74.90	37.50	0.80	1.56
\( LSTM_{bt} \)	2.31	40.00	1.90
\( Trans_{bt} \)	5.62	53.70	2.61
DS	FA	2.49	0.11	107.00	50	1.44	0.36
EFA	3.96	0.39	103.00	46.00	1.83	0.69
EFA RS	5.39	0.39	105.00	45.60	2.56	0.69
EV	Dev	386	282	108.15	61.06	20.00	15.59
Test	528	301	117.90	59.63	23.83	18.11
Table 6: Lexical diversity scores of the backtranslation (BT), data selection (DS) and evaluation (EV) corpora for the EN-DE and DE-EN systems. Trans. = Transformer, FA = ForAll, EFA = EachFromAll, EFA RS = EachFromAll Rescored.
each system used for backtranslation, and then we compare them with the rescoring method. Figures 1 and 2 show the portion of selected sentences per backtranslation system that form the training sets for the systems listed in Table 4.
For EU-ES, we observe that the EachFromAll configuration (the one with the highest scores according to the evaluation metrics in Table 4) selects more sentences from Transformer (649,312) in contrast to the ForAll approach that prefers PB-SMT (657,543). For DE-EN, FromAll and EachFromAll tend to select a higher number of sentences backtranslated by the PB-SMT model (820,765 and 924,694, respectively). However, for both language pairs, both ForAll and EachFromAll distributions are very similar as can be seen in Figures 1 and 2. Given that the DE-EN system trained with backtranslated data from PB-SMT  $(+PB - SMT_{bt})$  obtains the worst results while the one from Transformer  $(+Transformer_{bt})$  performs the best, we correlate the two measurements and hypothesise that a
Figure 1: Amount of sentences selected from each system by the data selection approaches for EU-ES. FA = FromAll, EFA = EachFromAll, EFA RS = EachFromAll Rescored.
Figure 2: Amount of sentences selected from each system by the data selection approaches for EN-DE. FA = FromAll, EFA = EachFromAll, EFA RS = EachFromAll Rescored.
distribution where more sentences originating from Transformer are selected would yield better results. Our  $\phi$  rescoring (cf. Equation (2)) shifts the preferred selection system to Transformer. For EU-ES, the EachFromAll Rescored selects 1,720,736 out of the total of 1,985,227 sentences (about  $87\%$ ); for DE-EN, it selects 2,131,227 out of the total of 2,284,800 sentences  $(93\%)$ .
For a more in-depth view of the distribution of selected sentence pairs per backtranslation system, we present the amount of selected sentences per system in bins of 100,000 for the FromAll systems. We show the results for EU-ES in Figure 3 and for DE-EN in Figure 4. For EU-ES, we observe that Transformer is the most selected system for the first bins, but the number of sentences sharply decreases until the middle of the corpus and then stabilises. In contrast, the number of sentences originating from PB-SMT increases in the first half and slowly
Figure 3: Number of sentences selected from each system by the FromAll data selection approach for EU-ES language pair in subsequent bins of 100,000 sentences (extrapolated for the last bin).
Figure 4: Number of sentences selected from each system by the FromAll data selection approach for DE-EN language pair in subsequent bins of 100,000 sentences (extrapolated for the last bin).
decreases afterwards. The number of sentences from RBMT and LSTM seams more stable, with a slight tendency to increase, peaking in the last bins. For DE-EN, we observe that PB-SMT is always the preferred system, but with a decreasing tendency; and the number of sentences originating from LSTM increases towards the last bins.
6.4 Sentence Length
We also analyse how the average sentence length varies during the data selection process in the FromAll configuration, as we did in Section 6.3 when analysing the selected systems.
Table 7 shows the average sentence lengths of the EU-ES and DE-EN data from the different reverse systems (BT), of the corpora resulting after data selection (DS) and of the test and the development sets (EV). We note that the sentences translated by PB-SMT are longer than those translated
by any other system for both language pairs. Correlating these results with those presented in Table 4 and in Figures 3 and 4, we can assert that in FDA the length penalty has a weaker effect than  $n$ -gram overlap and as such FDA has a preference towards  $n$ -gram MT paradigms, i.e. PB-SMT. However, data selection that results in more Transformer sentences would appear to be a better option.
Type	Corpus	EU	ES	DE	EN
BT	\( RBMT_{bt} \)	10.56	16.16	33.64	34.30
\( PB-SMT_{bt} \)	16.09	16.16	39.04	34.30
\( LSTM_{bt} \)	12.53	16.16	29.55	34.30
\( Transformer_{bt} \)	12.62	16.16	23.37	34.30
DS	FromAll	17.60	21.21	41.61	51.84
EachFromAll	13.67	16.16	32.94	34.30
EV	Dev.	10.85	10.34	15.09	14.34
Test	11.64	11.04	21.27	20.79
Table 7: Average sentence length of the backtranslation (BT), data selection (DS) and evaluation sets (EV).
We evaluated several approaches to data selection over the data backtranslated by RBMT, PB-SMT, LSTM and Transformer systems for two language pairs (EU-ES and DE-EN) from the clinical/biomedical domain. The former is a low-resource language pair, and the latter a well researched, high-resource language pair. Furthermore, in terms of the two target languages, English is a morphologically less rich language than Spanish, which creates a different setting again in which to evaluate our methodology. We use these two different use-cases to better understand both data selection and backtranslation.
We show how the different FDA data selection configurations tend to select different numbers of sentences coming from different systems, resulting in MT systems with different performance.
Under the assumption that FDA's performance is hindered by the fact that the data originates from MT systems, and as such contains errors and is of lower lexical richness, we rescored the data selection scores for each sentence by a factor depending on the BLEU, TER and MTLD values of the system used to backtranslate it. By doing so, we managed to improve the results for the DE-EN system, while for EU-ES we obtained similar performance to the other MT systems; this allows us to use just  $25\%$  of the data. Further investigation is required to study under which conditions our proposed rescoring method is beneficial, but our experiments with
both low- and high-resource language pairs suggest that if the systems used for backtranslation are poor, then this technique will be of little value; clearly this is closely related to the amount of resources available for the language pair under study.
In the future, we plan to investigate ways to directly incorporate the rescoring metrics into the data selection process itself, so that penalising similar sentences can also be taken into account. We also aim to conduct a human evaluation of the translated sentences in order to obtain a better understanding of the effects of data selection and backtranslation on the overall quality. Finally, we intend to analyse the effect of these measures in a wider range of language pairs and settings, in order to propose a more general solution.
Xabier Soto's work was supported by the Spanish Ministry of Economy and Competitiveness (MINECO) FPI grant number BES-2017-081045. This work was mostly done during an internship at the ADAPT Centre in DCU.
The ADAPT Centre for Digital Content Technology is funded under the Science Foundation Ireland (SFI) Research Centres Programme (Grant No. 13/RC/2106) and is co-funded under the European Regional Development Fund.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 194-203, Florence, Italy.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, USA. 15pp.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan.
Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, Antonio Jimeno Yepes, Madeleine Kittner, Martin Krallinger, Nancy Mah, Aurelie Neveol, Mariana Neves, Felipe Soares, Amy Siu, Karin Verspoor, and Maika Vicente Navarro. 2019. Findings of the
WMT 2019 biomedical translation shared task: Evaluation for MEDLINE abstracts and biomedical terminologies. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 29-53, Florence, Italy.
Ergun Biçici and Deniz Yuret. 2015. Optimizing instance selection for statistical machine translation with feature decay algorithms. Transactions on Audio, Speech & Language Processing, 23(2):339-350.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 1-46, Lisbon, Portugal.
Franck Burlot and François Yvon. 2018. Using monolingual data in neural machine translation: a systematic study. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 144-155, Belgium, Brussels.
Mara Chinea-Rios, Alvaro Peris, and Francisco Casacuberta. 2017. Adapting neural machine translation with parallel synthetic data. In Proceedings of the Second Conference on Machine Translation, pages 138-147, Copenhagen, Denmark.
Meghan Dowling, Teresa Lynn, and Andy Way. 2019. Investigating backtranslation for the improvement of English-Irish machine translation. TEANGA, the Journal of the Irish Association for Applied Linguistics, 26:1-25.
Thierry Etchegoyhen, Andoni Azpeitia, and Naiara Pérez. 2016. Exploiting a large strongly comparable corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3523-3529, Portoroz, Slovenia.
Mikel L. Forcada, Mireia Ginesti-Rosell, Jacob Nordfalk, Jim O'Regan, Sergio Ortiz-Rojas, Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Gema Ramírez-Sánchez, and Francis M. Tyers. 2011. Apertium: A free/open-source platform for rule-based machine translation. Neural Computation, 25(2):127-144.
Kenneth Heafield. 2011. KenLM: Faster and Smaller Language Model Queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, UK.
Chris Hokamp. 2017. Ensembling factored neural machine translation models for automatic post-editing and quality estimation. In Proceedings of the Second Conference on Machine Translation, pages 647-654, Copenhagen, Denmark.
Joanes Etxeberri Saria V. Edizioa. 2014. Donostia unibertsitate ospitaleko alta-txostenak. Donostiako Unibertsitate Ospitalea, Komunikazio Unitatea.
Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016. Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 751-758, Berlin, Germany.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700-1709, Seattle, Washington, USA.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, USA. 15pp.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics-System Demonstrations, pages 67-72, Vancouver, Canada.
Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: The tenth Machine Translation Summit, pages 79-86, Phuket, Thailand.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver, Canada.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. Computing Research Repository, arXiv:1901.07291.
Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020. When does unsupervised machine translation work? Computing Research Repository, arXiv:2004.05516.
Aingeru Mayor. 2007. Erregeletan oinarritutako itzulpen automatikokoSYSTEMa baten eraikuntza estaldura handiko baliabide linguistikkoak berrerabiliz. Ph.D. thesis, University of the Basque Country, Donostia, Spain.
Philip M McCarthy. 2005. An Assessment of the Range and Usefulness of Lexical Diversity Measures and the Potential of the Measure of Textual, Lexical Diversity. Ph.D. thesis, University of Memphis, TN.
Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160-167, Sapporo, Japan.
Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19-51.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA.
Olatz Perez-de-Viñaspre. 2017. Automatic medical term generation for a low-resource language: translation of SNOMED CT into Basque. Ph.D. thesis, University of the Basque Country, Donostia, Spain.
Alberto Poncelas, Gideon Maillette de Buy Wenniger, and Andy Way. 2018a. Feature decay algorithms for neural machine translation. In 21st Annual Conference of the European Association for Machine Translation, pages 239-248, Alicante, Spain.
Alberto Poncelas, Maja Popovic, Dimitar Shterionov, Gideon Maillette de Buy Wenniger, and Andy Way. 2019. Combining SMT and NMT Back-Translated Data for Efficient NMT. In Proceedings of Recent Advances in Natural Language Processing, pages 922–931, Varna, Bulgaria.
Alberto Poncelas, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018b. Investigating backtranslation in neural machine translation. In 21st Annual Conference of the European Association for Machine Translation, pages 249-258, Alicante, Spain.
Alberto Poncelas and Andy Way. 2019. Selecting Artificially-Generated Sentences for Fine-Tuning Neural Machine Translation. In Proceedings of the 12th International Conference on Natural Language Generation, pages 219–228, Tokyo, Japan.
Maja Popovic. 2015. chrF: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany.