SlowGuess/NAACL_2021_OCR huggingface dataset:
Christian Herold Jan Rosendahl Joris Vanvinckenroye Hermann Ney
Human Language Technology and Pattern Recognition Group
{surname}@i6.informatik.rwth-aachen.de
Data filtering for machine translation (MT) describes the task of selecting a subset of a given, possibly noisy corpus with the aim to maximize the performance of an MT system trained on this selected data. Over the years, many different filtering approaches have been proposed. However, varying task definitions and data conditions make it difficult to draw a meaningful comparison. In the present work, we aim for a more systematic approach to the task at hand. First, we analyze the performance of language identification, a tool commonly used for data filtering in the MT community and identify specific weaknesses. Based on our findings, we then propose several novel methods for data filtering, based on cross-lingual word embeddings. We compare our approaches to one of the winning methods from the WMT 2018 shared task on parallel corpus filtering on three real-life, high resource MT tasks. We find that said method, which was performing very strong in the WMT shared task, does not perform well within our more realistic task conditions. While we find that our approaches come out at the top on all three tasks, different variants perform best on different tasks. Further experiments on the WMT 2020 shared task for parallel corpus filtering show that our methods achieve comparable results to the strongest submissions of this campaign.
In recent years, neural machine translation (NMT) systems have greatly improved the quality of automatically generated translations, some argue even to the point of human parity (Hassan et al., 2018). While there most definitely have been advancements in designing the NMT system architectures (Bahdanau et al., 2015; Vaswani et al., 2017), arguably the best (and easiest) way to improve an NMT system is to use more training data. With an ever increasing amount of parallel data for NMT
training, which often comes from web-crawling<sup>1</sup> and is quite 'noisy', the task of data filtering becomes increasingly important (Khayrallah and Koehn, 2018).
Data filtering in the context of machine translation (MT) describes a collection of approaches which select a subset of a given, possibly noisy corpus with the aim to maximize the performance of an MT system trained on this data. There exist very simple approaches, the most prominent being based on language identification tools, to detect certain types of noise, e.g. sentences that are from a wrong language. However, other types of noise are much harder to detect, for example when both source and target sentence are well formulated and in the correct language but are not translations of one another.
In some formulations of the data filtering task, for example in the WMT shared task for parallel corpus filtering (Koehn et al., 2018, 2019, 2020), the assumption is that there already exists a large amount of 'clean' data which can be used to detect bad training samples in a separated 'noisy' corpus. However, such an assumption does typically not hold true in real-life scenarios. Therefore, in this work, we make no such distinction between 'known-to-be-clean' and 'noisy' data. We present novel approaches that use all the available data to filter that very same data in order to improve translation performance.
In the proposed methods, we use the structure of cross-lingual word embeddings to compare the words in a given source-target sentence pair to determine if the pair is of 'good' quality. This is done in a variety of ways, including nearest neighbor search in the embedding space and an explicit calculation of alignment scores. All proposed methods are specifically designed to detect the types of noise which cannot be detected by language identification tools. Furthermore, we design our approaches
<sup>1</sup> http://opus.nlpl.eu
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 162-172 June 6-11, 2021. ©2021 Association for Computational Linguistics
to not rely on the quality of the sentence pair alignments between the source and the target side of the data, since this information might be highly unreliable in a 'noisy' corpus.
The main contributions of this paper are summarized below:
- We perform a systematic analysis of 'noisetypes' for a commonly used MT task and identify specific weaknesses of the commonly used filtering by language identification.
- Building on our findings, we propose novel data filtering approaches using cross-lingual word embeddings.
- We compare our approaches to other strong filtering systems from the literature on three real-life, high resource MT tasks and the WMT 2020 task on parallel corpus filtering.
Recently, a number of shared tasks for data filtering have been held, giving a good overview of current state-of-the-art methods. Best known is the WMT shared task for parallel corpus filtering, which was held in 2018 (Koehn et al., 2018), 2019 (Koehn et al., 2019) and 2020 (Koehn et al., 2020) respectively. In these tasks, the participants are asked to provide scores for every sentence pair in a noisy corpus. Afterwards, a fixed amount of sentence pairs is selected according to that score.
The best performing submissions from past years use language identification tools as the first part of their setup (Junczys-Dowmunt, 2018; Chaudhary et al., 2019; Lu et al., 2020), removing sentence pairs where the language of either source or target sentence does not match the expectation. Rossenbach et al. (2018) and Junczys-Dowmunt (2018) use a combination of language model and translation model scores to sort the sentence pairs by quality. Chaudhary et al. (2019) use the cosine distance between cross-lingual sentence embeddings of source and target sentence as score. Wang et al. (2017) estimate the quality of a sentence pair using the euclidean distance between each sentence vector and two vectors representing in-domain and out-domain data. Hangya and Fraser (2018) score the similarity between source and target sentence by averaging the word-pair similarity, which is calculated from cross-lingual word embeddings.
Since the above mentioned methods are evaluated on different tasks with very different data
conditions, one can not easily make a statement about which approach works best. However, all approaches have in common that they use 'known-to-be-clean' parallel data in order to train the models of their filtering pipeline.
Creating cross-lingual word embeddings from parallel and/or monolingual data is an active field of research (Ruder et al., 2019). In addition to capturing semantic relationships within each language, these representations should be aligned in such a way that the embeddings of the same word in different languages are close together in the embedding space. The standard approach for creating such embeddings is to first train embeddings for each language pair separately (Mikolov et al., 2013; Pennington et al., 2014) and then projecting them into the same vector space (Conneau et al., 2017; Artetxe et al., 2018), which is possible with or without the help of parallel data.
Word alignments between a source and a target sentence were an integral part in count-based statistical machine translation systems (Brown et al., 1993; Koehn et al., 2007) and it has been shown that they can be used to help certain aspects of NMT systems as well (Alkhouli et al., 2018). For a long time, IBM-model-based frameworks like GIZA++ (Och and Ney, 2003) or fastalign (Dyer et al., 2013) produced the best word alignments. However, recently Sabet et al. (2020) report equally good results by using a word similarity matrix calculated from cross-lingual word embeddings.
3 Detecting Different Types of Noise
Applying language identification (language ID) is a well established first step in most high performing data filtering approaches. During this step, all sentence pairs for which either the source or target sentence is not mapped to the correct language are discarded. It can be argued that this step does not only remove sentence pairs in the wrong language, but also that language-agnostic noise, e.g. sequences of numbers, is almost completely removed.
In order to evaluate the effectiveness of the filtering by language ID approach, we decide to test the method on the popular De  $\rightarrow$  En data filtering task. By manually checking the noisy corpus (see Section 5.1 for details) we find different types of 'noise patterns'. For each of these 'noise patterns', we create a synthetic corpus (50k lines each), only consisting of sentence pairs with this specific noise.
We find/create the following 'noise patterns':
trg to src: The source and target side of a valid sentence pair are swapped.
trg to trg: Both source and target side contain different sentences from the target language.
src to src: Both source and target side contain different sentences from the source language.
src to other: The sentence on the source side is from the correct language. The sentence on the target side is a random sentence from a third language.
other to trg: The sentence on the source side is a random sentence from a third language. The sentence on the target side is from the correct language.
other to other: Both sentences on the source and target side are random sentences from a third language.
sentence misalign: Both sentences on the source and target side are from the correct language, but they are not translations of one another.
overtranslation: Both sentences on the source and target side are from the correct language and translations of one another, but parts of the source sentence are missing.
undertranslation: Both sentences on the source and target side are from the correct language and translations of one another, but parts of the target sentence are missing.
random digits: The source and target sentences each consist of random number sequences.
For the unrelated third language (other) we choose French.
Next, we use the langid.py toolkit (Lui and Baldwin, 2012) to filter each of these synthetic corpora and check which percentage of noise (ideally  $100.0\%$  gets removed. The results are shown in Table 1.
We find that the language identification filtering approach does an outstanding job in detecting noise that comes from wrong language alignment. Furthermore it also removes basically all of the random noise, represented by the random digits corpus. However, we also see where this approach
Noise Type	Percentage removed
trg to src	100.0%
trg to trg	100.0%
src to src	100.0%
src to other	99.5%
other to trg	99.8%
other to other	100.0%
sentence misalign	0.0%
overtranslation	7.8%
undertranslation	6.7%
random digits	100.0%
Table 1: Removal rate of different noise types by the language identification filtering method.
fails: it can not detect noise resulting from a semantic mismatch between source and target sentence.
Two conclusions can be drawn from this experiment: First, the filtering methods applied after language identification filtering can be language-agnostic, since all types of noise which originate from wrong languages can be detected by language identification very reliably. Second, downstream filtering methods should focus on the alignment between source and target sentence, since this is where language identification filtering predictably fails.
4 Data Filtering Methods
Intuitively a bilingual sentence pair is appropriate for training if a) both the source and the target sentence belong to the corresponding languages and b) they are translations of each other. We rely on established language identification methods (see Section 5.1) to verify the first condition. Following state of the art filtering systems (Junczys-Dowmunt, 2018; Chaudhary et al., 2019) we predict the language for source and target sentence and keep the sentence only if both match the requirements of the task. To check whether the sentences of a training pair  $(f_1^J, e_1^I)$  are indeed translations of each other we propose several approaches based on cross-lingual word embeddings. For the details of how the cross-lingual word embeddings are constructed we refer to Section 5.1. Here we assume that we are given a cross-lingual word embedding  $E: V_{\mathrm{src}} \cup V_{\mathrm{trg}} \to \mathbb{R}^{d_{\mathrm{embd}}}$  that maps each word from the source vocabulary  $V_{\mathrm{src}}$  or the target vocabulary  $V_{\mathrm{trg}}$  to a joint space  $\mathbb{R}^{d_{\mathrm{embd}}}$  with a similarity measure  $\rho$ . For convenience we use  $E_w := E(w)$ . In practice all embedding vectors are length normalized,
i.e.  $||E_w|| = 1$
4.1 Nearest Neighbour based
Many works investigate distances in the embedding space as an indicator of relatedness between words of the same language. However we are interested in the relation between the words of the source sentence and the target sentence. Specifically, we want to know whether the two sentences are translations of each other. We assume a source word  $f$  is explained by a word  $e$  in the target sentence, if  $E(f)$  is one of the  $k$  nearest neighbours of  $E(e)$  i.e. if:
$$
\rho \left(E _ {f}, E _ {e}\right) \geq \max  - k \left\{\rho \left(E _ {\hat {f}}, E _ {e}\right) \mid \hat {f} \in V _ {\mathrm {s r c}} \right\}
$$
where max-k yields the  $k$ -th biggest value. Note that we only consider the source nearest neighbourhood around  $e$ . To score a sentence pair  $(f_1^J, e_1^I)$  we calculate:
$\mathrm{explain}(f_1^J \mid e_1^I) := \left|\{f_j \mid \exists e_i : e_i \text{ explains } f_j\}\right|$ .
For data filtering we consider different variants of combining the forward and backward score:
Accumulated Explanation Score:
$$
\frac {\operatorname {e x p l a i n} (e _ {1} ^ {I} | f _ {1} ^ {J}) + \operatorname {e x p l a i n} (f _ {1} ^ {J} | e _ {1} ^ {I})}{I + J}
$$
Explanation Disagreement Score: Note that being nearest neighbours in a multilingual embedding space is not a symmetric relation. We compute the agreement of the forward and the backward score:
$$
\left| \frac {\operatorname {e x p l a i n} (e _ {1} ^ {I} | f _ {1} ^ {J})}{I} - \frac {\operatorname {e x p l a i n} (f _ {1} ^ {J} | e _ {1} ^ {I})}{J} \right|
$$
Explanation Disagreement + Pre-Filtering: A sentence pair is removed if its score for either direction falls below a threshold  $\gamma$ :
$$
\min  \left\{\operatorname {e x p l a i n} \left(e _ {1} ^ {I} \mid f _ {1} ^ {J}\right), \operatorname {e x p l a i n} \left(f _ {1} ^ {J} \mid e _ {1} ^ {I}\right) \right\} <   \gamma
$$
the remaining sentences are scored via explanation disagreement score
As similarity measure  $\rho$  we choose cross-domain-similarity-scaling (CSLS) (Conneau et al., 2017):
$$
\begin{array}{l} \operatorname {C S L S} \left(E _ {f}, E _ {e}\right) = 2 \cos \left(E _ {f}, E _ {e}\right) \\ - \frac {1}{n} \sum_ {f ^ {\prime} \in N _ {f} (e, n)} \cos (E _ {f ^ {\prime}}, E _ {e}) \\ - \frac {1}{n} \sum_ {e ^ {\prime} \in N _ {e} (f, n)} \cos \left(E _ {f}, E _ {e ^ {\prime}}\right) \\ \end{array}
$$
where  $N_{f}(e,n)$  is the neighborhood of size  $n$  across the word  $e$  in the space of the language of  $f$ .
4.2 Source  $\leftrightarrow$  Target Embedding Similarity
The methods described so far are based on the neighbourhood of size  $k$  around each word to create a source  $\rightarrow$  target and a distinct target  $\rightarrow$  source alignment. Alternatively we consider the source  $\leftrightarrow$  target similarity matrix:
$$
A _ {i, j} := A (f _ {1} ^ {J}, e _ {1} ^ {I}) _ {i, j} := E _ {e _ {i}} ^ {\top} E _ {f _ {j}}
$$
where each entry expresses the similarity of a word pair from the source and target sentence. Note that due to the construction of the cross-lingual word embeddings (see Section 5.1) all word embeddings are normalized. This means that the scalar product above is equivalent to the cosine similarity. We consider several options to compute a source  $\leftrightarrow$  target similarity score:
Argmax Agreement: Considers alignment points where  $\mathrm{src} \rightarrow \mathrm{trg}$  and the  $\mathrm{trg} \rightarrow \mathrm{src}$  argmax are the same:
$$
\begin{array}{l} M := \left\{(i, j) \mid i = \operatorname {a r g m a x} _ {\hat {i}} A _ {\hat {i}, j} \text {a n d} \right. \\ j = \operatorname {a r g m a x} _ {\hat {j}} A _ {i, \hat {j}} \} \\ \end{array}
$$
and sums up the corresponding weights
$$
\frac {1}{\max  \{I , J \}} \sum_ {(i, j) \in M} A _ {i, j}.
$$
Maximum Matching (Score): On the complete bipartite graph induced from the similarity matrix  $A$ , i.e. the bipartite graph with vertices  $V \coloneqq f_1^J \cup e_1^I$  and edge weight function  $f \coloneqq I \times J \to \mathbb{R} : (i, j) \mapsto A_{i,j}$ . We use the total weight of the maximum-weight matching divided by  $\max \{I, J\}$  as a score.
Maximum Matching (Count): We construct a maximum-weight matching on the bipartite graph with vertices  $V$  and edge weights  $f$  however we prune the edges if the corresponding word similarity is below a threshold  $t$ , keeping only the edges
$$
\mathcal {E} := \left\{(i, j) \in I \times J \mid A _ {i, j} \geq t \right\}.
$$
The number of matching points divided by  $\max \{I,J\}$  is used as score for the sentence pair.
Average similarity: The score is defined as the average over the similarity matrix, i.e.
$$
\frac {1}{I \cdot J} \sum_ {I \times J} A _ {i, j}.
$$
We would like to point out that parallel to the present work, Sabet et al. (2020) also introduced the first two of the four methods. Since they aim to extract an explicit alignment between source and target they do not construct a score for a sentence pair and do not consider the use in a data filtering task.
Since we are interested in aligning the source and target sentence to obtain a score for data filtering we also use the IBM4 alignment scores provided from GIZA++ (Och and Ney, 2003) for filtering as a comparison.
4.3 Data Selection and Score Transformation
We consider different ways to select training data given a noisy corpus where each sentence pair  $(f_1^J, e_1^I)$  has an associated score  $s(f_1^J, e_1^I) \in \mathbb{R}$ :
(1) Top X%: Selecting the  $X\%$  sentence pairs with the best score  $s$ .
(2) Top X% Transformed: Selecting the X% sentence pairs with the best transformed score:
$$
s _ {\mathfrak {t}} \left(f _ {1} ^ {J}, e _ {1} ^ {I}\right) = \left| s \left(f _ {1} ^ {J}, e _ {1} ^ {I}\right) - \sum_ {(F, E) \in \operatorname {d e v}} \frac {s (F , E)}{\left| \operatorname {d e v} \right|} \right|.
$$
(3) Dev set distribution: We score the dev set using  $s$ . Empirically this yields a Gaussian distribution where some scores are more frequent than others. We fit a Gaussian distribution and select a lower and an upper threshold such that  $95\%$  of the dev set distribution are selected. All sentence pairs from the training corpus whose score falls between the two thresholds are selected.
We introduce Variants (2) and (3) since we observe that often the best scored sentence pairs exhibit a pattern that is easy to learn but not representative for translation at all, e.g. sentence pairs that are dominated by long dates on both sides, etc. In particular the sentence pairs from the dev set are our best approximation of what 'valid training data' should look like. A sentence pair that scores significantly better than the dev set is just as suspicious than one that scores significantly worse.
#trg tokens	# lines
De→En	743M	37M
En→Tr	332M	50M
En→Cs	668M	57M
Table 2: Training data size of the three translation tasks.
We evaluate the performance of the data filtering systems on three high-resource tasks, namely German  $\rightarrow$  English, English  $\rightarrow$  Turkish and English  $\rightarrow$  Czech. The De  $\rightarrow$  En training data consists of the corpora Commoncrawl, Europarl, Rapid and ParaCrawl from the WMT 2019 news translation task $^{2}$ . We use the czeng 1.7 corpus $^{3}$  from the WMT 2018 news translation task for En  $\rightarrow$  Cs. For En  $\rightarrow$  Tr we test our systems on a real world corpus with a focus on the entertainment domain provided by a company. We select these three data conditions because they provide high resource data that originates from very different sources and, hence, should express rather different data biases and noise patterns. We choose to test the proposed methods in two settings of the WMT news translation task and not in the conditions defined by the WMT parallel corpus filtering task because we experienced in the past, that performance gains from data filtering on the very noisy corpora of the data filtering task do not carry over to the news translation task. For the corpus data statistics, please refer to Table 2.
Following state of the art filtering systems (Junczys-Dowmunt, 2018), we use the langid.py toolkit (Lui and Baldwin, 2012) as the first step in our filtering pipeline by removing source and target sentences where at least one side is not classified to be the correct language. In order to obtain cross-lingual word embeddings we follow the method proposed by Artetxe et al. (2018). In particular we first train GloVe Word Embeddings (Pennington et al., 2014) with a fixed vector size of 300 on the respective monolingual corpora after applying langid.py. From these we select the embeddings of the 200k most common words in each language. They form the base
3https://ufal.mff.cuni.cz/czeng/ czeng17
Filter Method	Data Selection Method	Training Data	dev BLEU	test BLEU
#trg tokens	#sent. pairs
De→En Accu. Expl. Scores	Top 50	374M	18M	33.1	34.1
Top 50 Transformed	360M	14M	34.2	35.6
Dev Set Distribution	612M	26M	34.0	35.4
De→En Matching (score) (BPE)	Top 50	366M	16M	33.7	34.9
Top 50 Transformed	366M	17M	33.7	34.8
Dev Set Distribution	559M	25M	33.8	34.8
En→Tr Matching (score) (BPE)	Top 50	164M	21M	14.8	14.7
Top 50 Transformed	162M	19M	17.6	20.5
Dev Set Distribution	273M	35M	15.0	15.3
Table 3: Comparison of different data selection methods and the resulting translation performance. As test set we use: newstest2017 (De→En) and newstest2018 (En→Tr). BLEU and TER are reported in percentage.
for the cross-lingual word embeddings, also with a fixed vector size of 300, which are created using the VecMap toolkit (Artetxe et al., 2018). All of the cross-lingual word embeddings are normalized. To be consistent with our filtering task definition, we do not use an initial seed dictionary to train the cross-lingual word embeddings. For nearest neighbor search we set  $k$  equal to five and use cross-domain-similarity-scaling (Conneau et al., 2017) as the distance metric when computing the sentence pair scores. The threshold  $\gamma$  is set to 0.1 for the prefiltering step of the explanation disagreement score. We compare our methods to another strong filtering method, that scores all sentence pairs by averaging the log probabilities of two language models (LMs) and two translation models (TMs) (Rossenbach et al., 2018). Each method creates a subset from the corpus, which is used to train a base transformer model (Vaswani et al., 2017) with six encoder and decoder layers implemented using the RETURNN toolkit (Zeyer et al., 2018). Machine translation performance is measured using BLEU scores (Papineni et al., 2002) and TER scores (Snover et al., 2006) using the MtEval tool from the Moses toolkit (Koehn et al., 2007). The development sets we use are newstest2015 for De→En, newstest2016 for En→Cs and a concatenation of development sets from multiple domains for En→Tr.
In a first step we investigate the data selection strategies described in Section 4.3. We consider two variants that select a fixed amount of training data plus an additional variant where the amount of selected data is dynamically determined in an automatic