SlowGuess/ACL_2020_OCR huggingface dataset:
Xabier Soto, $^{1}$  Dimitar Shterionov, $^{2}$  Alberto Poncelas, $^{2}$  and Andy Way $^{2}$
Ixa NLP Group, HiTZ Center, University of the Basque Country (UPV/EHU)
$^{2}$ ADAPT Centre, School of Computing, Dublin City University
'xabier.soto@ehu.eus
$^{2}\{\text {firstname.lastname}\} \text{@adaptcentre.ie}$
Machine translation (MT) has benefited from using synthetic training data originating from translating monolingual corpora, a technique known as backtranslation. Combining backtranslated data from different sources has led to better results than when using such data in isolation. In this work we analyse the impact that data translated with rule-based, phrase-based statistical and neural MT systems has on new MT systems. We use a real-world low-resource use-case (Basque-to-Spanish in the clinical domain) as well as a high-resource language pair (German-to-English) to test different scenarios with backtranslation and employ data selection to optimise the synthetic corpora. We exploit different data selection strategies in order to reduce the amount of data used, while at the same time maintaining high-quality MT systems. We further tune the data selection method by taking into account the quality of the MT systems used for backtranslation and lexical diversity of the resulting corpora. Our experiments show that incorporating backtranslated data from different sources can be beneficial, and that availing of data selection can yield improved performance.
The use of supplementary backtranslated text has led to improved results in several tasks such as automatic post-editing (Junczys-Dowmunt and Grundkiewicz, 2016; Hokamp, 2017), machine translation (MT) (Sennrich et al., 2016a; Poncelas et al., 2018b), and quality estimation (Yankovskaya et al., 2019). Backtranslated text is a translation of a monolingual corpus in the target language (L2) into the source language (L1) via an already existing MT system, so that the aligned monolingual corpus and its translation can form an L1-L2 parallel corpus. This corpus of synthetic parallel data can then be used for training, typically alongside authentic
human-translated data. For MT, backtranslation has become a standard approach to improving the performance of systems when additional monolingual data in the target language is available.
While Sennrich et al. (2016a) show that any form of source-side data (even using dummy tokens on the source side) can improve MT performance, both the quality and quantity of the backtranslated data play a significant role in practice. Accordingly, the choice of systems to be used for backtranslation is crucial. In Poncelas et al. (2019), different combinations of backtranslated data originating from phrase-based statistical MT (PB-SMT) and neural MT (NMT) were shown to have different impacts on the quality of MT systems.
In this work we conduct a systematic study of the effects of backtranslated data from different sources, as well as how to optimally select subsets of this data taking into account the loss in quality and lexical richness when data is translated with different MT systems. That is, we aim to (i) provide a systematic analysis of backtranslated data from different sources; and (ii) to exploit a reduction in the amount of training data while maintaining high translation quality. To achieve these objectives we analyse backtranslated data from several MT systems and investigate multiple approaches to data selection for backtranslated data based on the Feature Decay Algorithms (FDA: Biçici and Yuret (2015); Poncelas et al. (2018a)) method. We exploit different ways of ranking the data and extracting parallel sentences; we also interleave quality evaluation and lexical diversity/richness information into the ranking process. While our empirical evaluation shows different results for the tested language pairs, this is the first work in this direction and lays a firm foundation for future research.
Nowadays, NMT (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015), and in particular Transformer (Vaswani et al., 2017)
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3898-3908 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
achieves state-of-the-art results for many domains and language pairs. However, NMT requires a lot more data than other paradigms (Koehn and Knowles, 2017), which makes it harder to adapt to low-resource scenarios (Sennrich and Zhang, 2019). Using synthetic parallel data via backtranslation has been helpful in some low-resource use-cases (Dowling et al., 2019). For extreme cases with no bilingual parallel corpora, unsupervised MT can obtain reasonable results (Artetxe et al., 2019; Lample and Conneau, 2019). However, its application to real low-resource scenarios is still a matter of study (Marchisio et al., 2020). In this work we are motivated by a real-world low-resource use-case, namely the translation of clinical texts from Basque to Spanish (EU-ES). Basque is a minority language, so most of the Electronic Health Records (EHR) are written in Spanish so that any doctor from the Basque public health service can understand them. The development of a system for translating clinical texts from Basque to Spanish could allow Basque-speaking doctors to write EHRs in Basque, thus contributing to the normalisation of the language in specialised areas.
We conduct our analysis in the scope of the EU-ES translation of EHR use-case, as well as on a language pair and a data set that have been well studied in the literature - German to English (DEEN) data used in the WMT Biomedical Translation Shared Task (Bawden et al., 2019). As the EU-ES medical data cannot be made publicly available due to privacy regulations, using the DE-EN data is a way to allow for the replicability of our work.
One of the first papers comparing the performance of different systems for backtranslation was Burlot and Yvon (2018). The authors compared SMT and NMT systems, obtaining similar results. Closer to our work, Soto et al. (2019) also try RBMT, PB-SMT and NMT systems for backtranslating EHRs from Spanish into Basque. However, both papers are limited to comparing the performance of systems trained with backtranslated data originating from a single source, without examining whether a combination might be more effective.
More recently Poncelas et al. (2019) combined the outputs of PB-SMT and NMT systems used for backtranslation, showing that the combination of synthetic data originating from different sources was useful in improving translation performance.
In this work we extend these ideas by combining backtranslated data from RBMT, PB-SMT, NMT (LSTM) and NMT (Transformer); in addition, we use FDA to select sentences translated by different systems and analyse the impact of data selection of backtranslated data on the overall translation performance. Regarding the use of data-selection techniques in conjunction with synthetic data, Poncelas and Way (2019) fine-tune NMT models with sentences selected from a backtranslated set, and Chinaa-Rios et al. (2017) select monolingual source-side sentences to generate synthetic target strings to improve the translation model.
While the most common approach to assessing the translation capabilities of a MT system is via evaluation scores such as BLEU (Papineni et al., 2002), TER (Snover et al., 2006), chrF (Popovic, 2015), and METEOR (Banerjee and Lavie, 2005), recently research has begun to address another side of quality of translated text, namely lexical richness and diversity. In a recent paper, Vanmassenhove et al. (2019) study the loss of lexical diversity and richness of the same corpora translated with PB-SMT and NMT systems. Vanmassenhove et al. (2019) investigate the problem for seen (during MT training) and unseen text using MT systems trained on the Europarl corpus (Koehn, 2005), with original (human-produced and translated) text as well as in a round-trip-translation setting. In this work we calculate the same lexical diversity metrics as Vanmassenhove et al. (2019), and further use those metrics to improve the data selection process applied to backtranslated data.
3 Data Selection for Backtranslation from Multiple Sources
FDA (Biçici and Yuret, 2015; Poncelas et al., 2018a) is a data selection technique that retrieves sentences from a corpus based on the number of  $n$ -grams overlapping with those present in an indomain data set referred to as  $S_{seed}$ . FDA scores each candidate sentence  $s$  according to: (i) the number of  $n$ -grams that are shared with the seed  $S_{seed}$ ; and (ii) the  $n$ -grams already present in a set  $L$  of
<sup>1</sup>In their experiments, Vanmassenhove et al. (2019) backtranslate the training data via an MT system trained on the same data, then train yet another system with this data and analyse its performance. They assess how errors propagate through repeated translation, thereby investigating the extent of inherent algorithm bias in MT models.
selected sentences, as defined in (1):
$$
[ t ] \operatorname {s c o r e} (s, S _ {\text {s e e d}}, L) = \frac {\sum_ {n g r \in \{s \cap S _ {\text {s e e d}} \}} 0 . 5 ^ {C _ {L} (n g r)}}{\operatorname {l e n g t h} (s)} \tag {1}
$$
where  $length(s)$  is the number of words in the sentence  $s$  and  $C_L(ngr)$  is the number of occurrences of the  $n$ -gram  $ngr$  in  $L$ . The score is then used to rank sentences, with the one with the highest score being selected and added to  $L$ . This process is repeated iteratively. To avoid selecting sentences containing the same  $n$ -grams,  $score(s, S_{seed}, L)$  applies a penalty to the  $n$ -grams (up to order three in the default configuration) proportional to the occurrences that have been already selected. In (1), the term  $0.5^{C_L(ngr)}$  is used as the penalty.
In the context of MT, FDA has been shown to obtain better results than other methods for data selection (Silva et al., 2018). Accordingly, in this work we too focus on FDA, although our rescoring idea is more general and can be applied to other selection methods based on  $n$ -gram overlap.
Related work on quality and lexical diversity and richness of MT demonstrates that (i) regardless of the overall performance of an MT system (as measured by both automatic and human evaluation), in general machine-translated text is error-prone and cannot reach human quality (Toral et al., 2018)); and (ii) machine-translated text lacks the lexical richness and diversity of human-translated (or post-edited) text (Vanmassenhove et al., 2019).
In its operation, FDA compares two types of text – the seed and the candidate sentences – without taking into account the quality or the lexical diversity/richness of the candidate text. Our hypothesis is that when selecting data from different sources, FDA cannot account for the differences in quality and lexical diversity/richness of these texts, with the consequence that the selected set  $(L)$  is sub-optimal.
We test our hypothesis by assessing the quality and lexical diversity/richness of the backtranslated data with the four different systems as well as with different selected subsets of training data.
To tackle the problem of sub-optimal FDA-selected datasets, we propose to rescore FDA scores based on quality evaluation and lexical diversity/ richness scores.2 That is, for each sentence
$s_i^{BT}$  from a backtranslated corpus  $D_{i}^{BT}$  originating from the  $i^{th}$  MT system, we factor in the quality expressed by the evaluation metrics,  $q(D_{i}^{BT})$  and the lexical diversity/richness expressed by the diversity metrics,  $d(D_{i}^{BT})$  as shown in (2):
$$
\begin{array}{l} \operatorname {s c o r e} \left(s _ {i} ^ {B T}, S _ {\text {s e e d}}, L\right) = \\ \frac {\sum_ {n g r \in \{s \cap S _ {s e e d} \}} 0 . 5 ^ {C _ {L} (n g r)}}{\text {l e n g t h} (\mathrm {s})} \cdot \phi \left(q \left(D _ {i} ^ {B T}\right), d \left(D _ {i} ^ {B T}\right)\right) \tag {2} \\ \end{array}
$$
where  $\phi$  is a function over quality and lexical diversity metrics producing a non-negative real number.
We note three considerations with respect to our approach to Equation (2).
1. Sentence-level selection versus document-level quality and lexical diversity/richness evaluation. The FDA algorithm works on a sentence level, while our approach rescores the FDA scores using document-level metrics. As our goal is to differentiate between the output of different MT systems, we consider metrics that reflect the overall quality of each system. Furthermore, metrics for lexical diversity/richness as type-token ratio (TTR) (Templin, 1975), Yule's I (Yule, 1944), and the measure of textual lexical diversity (MTLD) (McCarthy, 2005) are to be calculated on a document-level; the same is valid for automatic evaluation metrics such as BLEU and TER.
2. Combined metrics. We conduct our analysis using the quality metrics BLEU, TER, ME-TEOR and chrF; and TTR, MTLD and Yule's I for lexical diversity/richness. For rescoring we use only BLEU, TER and MTLD as a factor:  $\phi = \log (BLEU*(100 - TER)*MTLD)$ . We decided on this rescoring formula based on preliminary experiments, as it led to the selection of more sentence pairs originating from models trained with backtranslated data from the system that performs best (for both ES-EU and EN-DE); we chose MTLD based on the findings of Vanmassenhove et al. (2019) which show this metric to be more suitable for comparative analysis, as well as mitigating issues related to sentence length typical for TTR and Yule's I (McCarthy, 2005).
3. Use of devset as a seed. Using a development set in MT aims to test whether the performance of the MT system has reached a certain level. In
factors dependent on MT quality and lexical diversity (right part in equation (2)).
2We talk about "rescoring" as if we compare equations (1) and (2), the only difference is the rescoring produced by multiplying equation (1) (left part in equation (2)) by the
FDA for MT, we use a devset as the seed. In our method we compute BLEU and TER on the devset also used as a seed; MTLD is computed on the backtranslated text, i.e. the synthetic source text.
4 Language Pairs - Challenges and Objectives
As a challenging low-resource scenario, we chose the translation of clinical texts from Basque to Spanish, for which there is no in-domain bilingual corpora. We make use of available EHRs in Spanish coming from the hospital of Galdakao-Umansolo to create a synthetic parallel corpus via backtranslation. The Galdakao-Umansolo EHR corpus consists of 142,154 documents compiled between 2008 and 2012. After dedduplication, we end up with a total of 2,023,811 sentences. $^{3}$
As a basis for training the MT systems for backtranslation, we use a bilingual out-of-domain corpus of 4.5M sentence pairs: 2.3M sentence pairs from the news domain (Etchegoyhen et al., 2016), and 2.2M from administrative texts, web-crawling and specialised magazines.
In order to adapt the systems to the clinical domain, we used a bilingual dictionary previously used for automatic clinical term generation in Basque (Perez-de-Viñaspre, 2017), consisting of 151,111 terms in Basque corresponding to 83,360 unique terms in Spanish.
To evaluate our EU-ES systems, we use EHR templates in Basque written with academic purposes (Joanes Etxeberri Saria V. Edizioa, 2014) together with their manual translations into Spanish produced by a bilingual doctor. These 42 templates correspond to diverse specializations, and were written by doctors of the Donostia Hospital. After dedduplication, we obtain 1,648 sentence pairs that are randomly divided into 824 sentence pairs for validation (devset) and 824 for testing.
In order to test the generalisability of our idea, we use a well-researched language pair, German-to-English. As our out-of-domain corpus, we used the DE-EN parallel data provided in the WMT 2015 (Bojar et al., 2015) news translation task.
The adaptation of systems to the medical domain with backtranslated data is performed using
the UFAL data collection. $^{4}$  We selected the following subsets: ECDC, EMEA, EMEA_new_crawl, MuchMore, PatTR_Medical and Subtitles. The total amount of sentences was 2,555,138 which after dedduplication was reduced to 2,335,892. After filtering misaligned and empty lines, $^{5}$  the resulting amount was 2,322,599 sentences. We used the EN monolingual side. For development and test sets we used the Cochrane and NHS 24 subsets from the Himl 2017 set. $^{6}$
Table 1 provides the statistics of our corpora.
Desc.	Sent.	Tokens
src	trg
EU-ES	out-of-domain	4.5M	73M	102M
clinical terms	151K	271K	258K
EHRs	2M	33M
EHR templates	1.6K	18.5K	17.6K
DE-EN	out-of-domain	4.5M	110M	116M
in-domain	2.3M	97M
devset	1K	16K	15K
test set	467	10K	9.7K
Table 1: Description and statistics of the used corpora.
Via a set of experiments, we (i) investigate the differences in the backtranslated data originating from the four different MT systems and their impact on the performance of MT systems using this backtranslated data, and (ii) test our hypothesis as well as different approaches to rescoring the data selection algorithm.
5.1 Systems Used for Backtranslation
First, we train PB-SMT, LSTM and Transformer models for the ES-EU and EN-DE (i.e. reverse) language directions. Then we backtranslate the monolingual corpus into the target language (EU and DE, respectively) using those systems, as well as a RBMT one.
RBMT: We use Apertium (Forcada et al., 2011) for the EN-DE language pair, and Matxin (Mayor, 2007) for ES-EU, adapted to the clinical domain by the inclusion of the same dictionaries used to train the other systems.
PB-SMT: We use Moses with default parameters, using MIGZA for word alignment (Och and Ney,
<sup>3</sup>Due to privacy requirements, this corpus is not publicly available. Prior to use, it was de-identified by reordering sentences, and only authors who had previously signed a nondisclosure commitment had access to it.
$^{4}$ https://ufal.mff.cuni.cz/ufalmedical Corpus  
 $^{5}$ We used the clean-corpus-n.pl script provided with the Moses toolkit (Koehn et al., 2007).  
 $^{6}$ http://www.himl.eu/test-sets
2003), an "msd-bidirectional-fe" lexicalised reordering model and a KenLM (Heafield, 2011) 5-gram target language model. We tuned the model using Minimum Error Rate Training (Och, 2003) with an n-best list of length 100.
LSTM: We use an RNN of 4 layers, with LSTM units of size 512, dropout of 0.2 and a batch-size of 128. We use Adam (Kingma and Ba, 2015) as the learning optimiser, with a learning rate of 0.0001 and 2,000 warmup steps.
Transformer: We train a Transformer model with the hyperparameters recommended by OpenNMT, halving the batch-size so that it could fit in 2 GPUs, and accordingly doubling the value for gradient accumulation.
We train all NMT systems using OpenNMT (Klein et al., 2017) for a maximum of 200,000 steps, and select the model that obtains the highest BLEU score on the devset; note that the final systems trained after applying data selection use early stopping with perplexity not decreasing in 3 consecutive steps as our stopping criterion. Backtranslation is performed with the default hyperparameters, including a beam-width of 5 and a batch-size of 30.
We use Moses scripts to tokenise and truecase all the corpora to be used for statistical or neural systems. For the NMT systems, we apply BPE (Sennrich et al., 2016b) on the concatenated bilingual corpora with 90,000 merge operations for EU-ES and 89,500 for DE-EN, using subword-nmt. $^{8}$
5.2 Systems with Data Selected via Backtranslation
For each language pair we train four Transformer models with the authentic and backtranslated data, as well as a fifth system with all four back translated versions concatenated to the authentic data. These we refer to as  $+S_{bt}$ , where  $S$  is one of RBMT, PB-SMT, LSTM or Transformer and indicates the origin of the backtranslation, and  $+\mathrm{All}_{bt}$  to refer to the system trained with all backtranslated data.
Next, we use the devset as a seed for the data selection algorithm. Given that FDA does not score sentences that have no  $n$ -gram overlaps with any sentence from the seed, for the 'EachFromAll' configuration presented later, which is constrained to
select one sentence for each sentence in the monolingual corpus, we randomly select one sentence among those produced by the 4 different systems used for backtranslation, in case none of them overlap with any sentence from the seed. We obtain the FDA scores and use them to order the sentence pairs in descending order. Next, we apply the following different data selection configurations:
1. Top from all sentences (referred to as FromAll henceforth): concatenate the data backtranslated with all the systems and select the top ranking 2M (for EU-ES) or 2.3M (for DE-EN) sentence pairs with the possibility of selecting the same target sentence more than once, i.e. translated by different systems.
2. Top for each (target) sentence (henceforth, EachFromAll): concatenate the data backtranslated with all the systems and select the optimal sentence pairs avoiding the selection of the same target sentence more than once. That is, each selected target sentence will have only one associated source sentence originating from one specific system.
3. Top for each (target) sentence  $x4$  (henceforth, EachFromAll  $x4$ ): same as EachFromAll, but repeating the selected backtranslated data four times (only for EU-ES).
4. Top for each (target) sentence rescored (henceforth, EachFromAll RS): use MT evaluation and lexical diversity metrics to rescore the FDA ranks and perform an EachFromAll selection.
We selected the Transformer architecture as the basis of our backtranslation models because (i) it has obtained the best performance for many use-cases and language pairs which we also aim at, and (ii) it has been shown that Transformer's performance is strongly impacted by the quantity of data, which can act as an indicator as to whether our improvements originate from the quantity or the quality of the data. That is why we compare EachFromAll systems to systems trained with all backtranslated data (i.e. all 8M sentence pairs), to verify that it is not only the amount of data that impacts performance.
We use the automatic evaluation metrics BLEU, TER, METEOR and chrF (in its chrF3 variant) to assess the translation quality of our systems. In Table 2 we show the scores on the test set of the
$^{7}$ http://opennmt.net/OpenNMT-py/FAQ. html#how-do-i-use-the-transformer-model (Accessed on December 9, 2019.)
$^{8}$ https://github.com/rsennrich/subword-nmt (Accessed on December 9, 2019.)
reverse systems used for backtranslation (the best are marked in bold). For EU-ES, since we only use clinical terms as in-domain training data, the results are poor overall. However, we observe that Transformer obtains the best results according to all metrics for both EU-ES and DE-EN. Table 3 shows the results of our baseline (forward) systems. It shows that Transformer systems perform best for both language pairs. Evaluation scores for the systems trained on authentic and backtranslated data, and for the systems trained after data selection for EU-ES and DE-EN, are shown in Table 4.
BLEU↑	TER↓	METEOR↑	CHRF3↑
ES-EU	RBMT	11.37	75.52	19.80	41.35
PB-SMT	9.38	70.70	25.36	44.07
LSTM	7.01	72.29	20.46	33.94
Transformer	12.21	66.53	26.96	44.42
EN-DE	RBMT	8.21	72.26	25.70	41.40
PB-SMT	14.85	74.00	35.62	48.92
LSTM	24.65	54.60	43.30	53.51
Transformer	32.24	46.83	50.25	60.29
Table 2: Scores of reverse systems for backtranslation.
BLEU↑	TER↓	METEOR↑	CHRF3↑
DE-EN-UES	LSTM	10.84	85.00	32.79	41.36
Transformer	19.64	69.11	43.84	53.03
LSTM	LSTM	28.15	51.95	32.19	55.40
Transformer	Transformer	38.27	42.87	37.02	62.37