SlowGuess/ACL_2020_OCR huggingface dataset:
Boliang Zhang, Ajay Nagesh, and Kevin Knight
DiDi Labs
{boliangzhang, ajaynagesh, kevinknight}@didiglobal.com
Web-crawled data provides a good source of parallel corpora for training machine translation models. It is automatically obtained, but extremely noisy, and recent work shows that neural machine translation systems are more sensitive to noise than traditional statistical machine translation methods. In this paper, we propose a novel approach to filter out noisy sentence pairs from web-crawled corpora via pre-trained language models. We measure sentence parallelism by leveraging the multilingual capability of BERT and use the Generative Pre-training (GPT) language model as a domain filter to balance data domains. We evaluate the proposed method on the WMT 2018 Parallel Corpus Filtering shared task, and on our own web-crawled Japanese-Chinese parallel corpus. Our method significantly outperforms baselines and achieves a new state-of-the-art. In an unsupervised setting, our method achieves comparable performance to the top-1 supervised method. We also evaluate on a web-crawled Japanese-Chinese parallel corpus that we make publicly available.
Training modern neural machine translation (NMT) systems requires large parallel-text resources. Publicly-available parallel corpora are mostly paired with English, such as German-English, French-English, Chinese-English, etc., and their domains are limited. For building machine translation systems between non-English language pairs, such as Chinese and Japanese, existing parallel corpora are insufficient and often low quality. To address this problem, system builders have trained NMT systems on web-crawled data and achieved promising results (Xu and Koehn, 2017; Junczys-Dowmunt, 2018; Schwenk, 2018; Schwenk et al., 2019). However, data automatically crawled from the web is extremely noisy. Khayrallah and Koehn
(2018) and Belinkov and Bisk (2018) show that neural translation models are far more sensitive to noisy parallel training data than statistical machine translation. Data selection methods that can filter noisy parallel sentences from large-scale web crawled resources are in demand.
In this paper, we study the problem in a real-world scenario where we crawl a large Japanese-Chinese parallel corpus from various websites and build open-domain machine translation systems between Japanese and Chinese, by filtering the web crawled parallel corpus. In addition, a small amount of clean parallel data is available, in the software domain. In order to confirm our results on a public data, we also apply our filter to the WMT 2018 German-English Parallel Corpus Filtering shared task.
Previous work on parallel corpus filtering performs poorly in our scenario as it either requires large clean parallel corpora or dictionaries (Xu and Koehn, 2017; Artetxe and Schwenk, 2019; Junczys-Dowmunt, 2018; Chaudhary et al., 2019), or relies on multilingual word embeddings and neglects context when measuring translation parallelism (Hangya and Fraser, 2018).
In this paper, we propose a simple but effective parallel corpus filtering method. Multilingual BERT (Devlin et al., 2019) projects multilingual sentences into a shared space and has shown a great potential for cross-lingual model transfer (Pires et al., 2019). We use pre-trained multilingual BERT as prior knowledge and fine-tune it on a synthetic dataset. This multilingual BERT-based classifier forms an acceptability filter that determines whether or not a sentence pair consists of a bona-fide translation.
As the domain of training data largely affects machine translation model performance, we also introduce a domain filter. It uses the pre-trained Generative Pre-training (GPT) as in-domain language
8545
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8545-8554 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
model and is an extension of the existing cross-entropy difference based domain filter (Moore and Lewis, 2010; Junczys-Dowmunt, 2018).
We evaluate our proposed method on the WMT 2018 German-English Parallel Corpus Filtering shared task and achieve a new state-of-the-art. Our unsupervised method achieves comparable performance to the top system that is trained on millions of clean parallel sentence pairs. Our proposed methods also significantly outperform baselines in our own Japanese-Chinese parallel corpus filtering task.
- We propose a novel approach to filter noisy parallel corpora by using pre-trained language models. Our approach outperforms strong baselines and achieves a new state-of-the-art.
- We devise an unsupervised filtering approach that does not require an identifiable clean subset of parallel segments. Our unsupervised method matches the results of previous supervised methods.
- We release a large web-crawled Japanese-Chinese parallel corpus which can be a useful resource for machine translation research on non-English language pairs.
Several recent works address parallel corpus filtering. Denkowski et al. (2012), Dyer et al. (2010) and Heafield (2011) use language models and word alignments to determine how likely sentences are to be a good translation of another. Xu and Koehn (2017) introduce a noise filtering tool, Zipporah, that discriminates parallel and non-parallel sentences based on word-frequency vectors and a dictionary. Junczys-Dowmunt (2018) proposes a dual conditional cross-entropy filtering method, which achieved first place in the WMT 2018 German-English Parallel Corpus Filtering shared task. They train two translation models in inverse directions on millions of parallel sentences and score sentence pairs based on the word-normalized conditional cross-entropy from the translation models. Artetxe and Schwenk (2019) and Schwenk (2018) propose a margin-based scoring method that compares the
similarity of the source and target sentence representations. The sentence representations are produced by a sentence encoder trained on clean parallel data via a neural encoder-decoder architecture. Other works based on sentence embeddings include Hangya and Fraser (2018) and Littell et al. (2018), as well as Schwenk et al. (2019), which mines millions of parallel sentences in 1620 language pairs from Wikipedia. These encoder-decoder based methods require large amounts of clean parallel training data and are not applicable in our scenario where available data is noisy. Ondrej Bojar (2020) organize an open domain translation challenge where participants are provided a large, noisy set of Japanese-Chinese segment pairs built from web data, and the task is to clean the noisy data and build an end-to-end machine translation system.
Work on data selection is also related. Moore and Lewis (2010); Junczys-Dowmunt (2018) select domain-related data by computing the cross-entropy difference between in-domain and out-domain language models. Duh et al. (2013) use neural language models for data selection. Axelrod et al. (2011) and Axelrod et al. (2015) expand cross-entropy difference filtering to both sides of the parallel corpus. Since we aim to build a general machine translation system, instead of selecting data that are relevant to a specific domain, we select data whose domains are as general as possible, by using Generative Pre-training (GPT) models trained on large and diverse corpora.
In this section we introduce a language detection filter, a translation-acceptability filter, and a domain filter. Each filter produces a score for every candidate source/target sentence pair. The partial score produced by each filter ranges from 0 to 1. Values beyond this range are normalized by min-max normalization:  $\hat{y} = (y - \min) / (\max -\min)$ . The final score is the product of the partial scores.
3.1 Language Detection Filter
Targeting a web-crawler at a given language pair still results in many pages written in the wrong language. For example, while a URL pair may clearly indicate translation (e.g.,“.jp” and“.zh”), it may happen that the text content is simply copied rather than translated. We observe this in both our Japanese-Chinese data and the German-English Paracrawl data set. It is necessary to filter out sen
http://iwslt.org/doku.php?id=open_domain Translation
8546
tence pairs with undesired languages.
We adopt the fastText (Joulin et al., 2017, 2016) language identification toolkit in our language detection filter. For each sentence, the toolkit produces a list of language candidates and their corresponding confidence scores. We select the language that has the highest confidence score from fastText as the language of the sentence. Sentence pairs that have both of the elements detected as the desired language are assigned score 1 and otherwise 0. By discarding sentence pairs with undesired language IDs, we filter out  $27\%$  of our Chinese-Japanese parallel sentences and nearly  $70\%$  of the German-English parallel sentences from Paracrawl data set.
3.2 Acceptability Filter
In this section, we introduce our translation acceptability filter, one of the main contributions in the paper. It aims to measure the parallelism of sentence pairs and filter out sentence pairs that are not mutual translations.
The pre-trained language model BERT (Devlin et al., 2019) has been shown to be effective in many NLP tasks as it produces better and meaningful contextualized word representations. Multilingual BERT, a transformer Masked Language Model pre-trained on Wikipedia dumps of 104 languages, shows remarkable multilingual capability, given that it is not exposed to any multilingual signals, such as parallel data or dictionaries. A thorough study by Pires et al. (2019) shows the promising zero-shot cross-lingual model transfer ability of multilingual BERT on named entity recognition and part-of-speech tagging tasks. They hypothesize that having language-universal word pieces, such as numbers and URLs, mapped to a shared space forces the co-occurring pieces to also be mapped to a shared space, thus spreading the effect to other word pieces, until different languages are close in the shared space.
We use pre-trained multilingual BERT to encode a sentence pair  $(s, t)$  and create the sentence embeddings  $v_{s}$  and  $v_{t}$  by using the representations of the [CLS] token of  $s$  and  $t$ . We find that the cosine similarity between  $v_{s}$  and  $v_{t}$  does not necessarily reflect the parallelism of sentence  $s$  and  $t$ . We suspect that the word representations from multilingual BERT are loosely aligned across languages as there is no parallel data or dictionary used during the pre-training. A similar observation was made in
Lample et al. (2018), where the cross-lingual word embeddings learned in an unsupervised manner are loosely aligned. However, after fine-tuning on a few anchor pairs (word translations), they become more aligned.
Similarly, we use an unsupervised synthetic training set as anchors to fine-tune multilingual BERT with a binary classification objective. Xu and Koehn (2017) did similar work to train a filtering classifier on synthetic data, but via bag-of-words translation features.
Synthetic Training Set. In cases where a small number of clean parallel sentence pairs are available, we use them as positive training samples for our classifier. In Japanese-Chinese filtering, we use around 300k sentence pairs, mostly from open-source software documentation,[2] as our positive samples. In extreme cases where no identifiable, clean parallel data is available, we sub-select high quality parallel sentences, which are used as positive samples, from the noisy parallel corpus based on the Hunalign (Varga et al., 2007) sentence-alignment score. We sample negative instances by simulating the noise produced by web crawling and alignment. Given a positive pair  $(s,t)$ , we create a negative sample by randomly choosing one of the following options:
- Randomly select a target sentence from its adjacent sentences within a window size of  $k$  (where  $k = 2$  in our experiments).
- Randomly truncate  $30\% - 70\%$  of the source or target sentence.
- Swap the order of  $30\% - 70\%$  words of the source or target sentence.
To balance the training set, we create the same number of positive instances and sampled negative instances.
Binary Classification Objective. We feed the sentence pair  $(s, t)$  into multilingual BERT, which accepts two-sentence input due to its next-sentence prediction objective (Devlin et al., 2019). Instead of using the [CLS] token representation, we use a Convolutional Network (CNN) layer that takes the BERT output and generates the final representation of the pair. Our experiments show that using CNN layer pooling achieves marginal gains over [CLS] pooling. The final layer is a feed-forward network
$^{2}$ GNOME, Ubuntu, OpenOffice, and KDE data set, from http://opus.nlpl.eu/
8547
with a softmax activation function to produce label probabilities. We use the softmax probability as the degree of parallelism.
3.3 Domain Filter
Web-crawled data contains noise of various types, due to the complicated structure of web pages. By inspecting the training data generated by the above methods, we notice much of the content is not well-formed, e.g., concatenated lists of months and dates, randomly mixed content from tables, series of emojis and punctuation marks, etc. These are certainly written in the desired language, thus not filtered out by language detection. The translation acceptability filter also accepts them. However, such malformed data is not helpful to machine translation models, and we prefer a training corpus to contain meaningful content.
For our domain filter, we adopt the cross-entropy difference scoring method proposed by Moore and Lewis (2010) and Junczys-Dowmunt (2018). More specifically, we treat a general domain monolingual corpus as our in-domain data set  $\mathbf{I}$ , and the noisy parallel corpus without any filtering as our non-domain data set  $\mathbf{N}$ . We train two language models  $\mathbf{L}_I$  and  $\mathbf{L}_N$  and measure how the target sentence  $\mathbf{t}$  is domain-related to  $\mathbf{I}$  and less domain-related to  $\mathbf{N}$  by a perplexity ratio, which is a transformation of cross-entropy difference:
$$
\hat {f} _ {\mathrm {d o m}} (s, t) = \frac {\mathbf {P P L} _ {N} (t)}{\mathbf {P P L} _ {I} (t)}
$$
where  $\mathbf{PPL}_M(x)$  is the word-normalized perplexity of the sentence  $x$  defined by the language model  $\mathbf{L}_M$ :
$$
\mathbf {P P L} _ {M} (x) = \exp (\frac {1}{| x |} \sum_ {i = 1} ^ {| x |} \log P _ {M} (x _ {i} | x _ {<   i}))
$$
The intuition is fairly straightforward: the higher the perplexity of the sentence to the non-domain corpus and the lower the perplexity of the sentence to the in-domain corpus, the more likely the sentence is meaningful.
Our contribution is to use GPT (Radford et al., 2019) as our in-domain language model, instead of news domain text (Junczys-Dowmunt, 2018). This minor yet crucial change yields non-trivial performance gains in our experiments for German-English parallel corpus filtering. As GPT is trained on data from various sources, such as Wikipedia, Reddit, news websites, etc., it covers a wide range
of domains, so our filtered data is more diverse and performs better on multi-domain test sets, as well as in the real world application.
For our in-domain language model, we use pre-trained Chinese  $\mathrm{GPT}^3$  for Japanese-Chinese and pre-trained GPT-24 for German-English. We randomly sample 4 million sentences from the unfiltered noisy parallel corpus and use KenLM (Heafield, 2011) to train the non-domain language model. Perplexity scores from different language models are compatible.
Following Junczys-Dowmunt (2018), we introduce two operations, clip and cutoff, to postprocess the domain filter score  $\hat{f}_{\mathrm{dom}}(s,t)$ . The clip operation clips the maximum value of the domain score to a threshold  $\tau_{\mathrm{clip}}$ :
$$
f _ {\mathrm {c l i p}} (x, \tau_ {\mathrm {c l i p}}) = \mathbf {m i n} (x, \tau_ {\mathrm {c l i p}})
$$
and the cutoff operation modifies scores below a threshold  $\tau_{\mathrm{cutoff}}$  and changes them to 0:
$$
f _ {\text {c u t o f f}} (x, \tau_ {\text {c u t o f f}}) = \left\{ \begin{array}{l l} x, & \text {i f} x > \tau_ {\text {c u t o f f}} \\ 0, & \text {o t h e r w i s e} \end{array} \right.
$$
$\tau_{\mathrm{clip}}$  prevents a high monolingual in-domain score from overwriting scores from other filters.  $\tau_{\mathrm{cutoff}}$  eliminates out-domain sentence pairs and ensures that highly parallel sentence pairs are at least somewhat in-domain. We tune  $\tau_{\mathrm{clip}}$  and  $\tau_{\mathrm{cutoff}}$  on the development set.
The scoring method of our final domain filter becomes:
$$
f _ {\text {d o m}} (s, t) = f _ {\text {c l i p}} \left(f _ {\text {c u t o f f}} \left(\hat {f} _ {\text {d o m}} (s, t), \tau_ {\text {c u t o f f}}\right), \tau_ {\text {c l i p}}\right)
$$
4.1 WMT 2018 Parallel Corpus Filtering
We use the WMT 2018 Parallel Corpus Filtering shared task (Koehn et al., 2018) as a benchmark to evaluate our methods. Participants in the shared task are provided a very noisy 1 billion word (English token count) German-English corpus crawled from the web by the Paracrawl project. The task is to sub-select clean sentence pairs amounting to (a) 10 million words, and (b) 100 million words, counted on the English side. The quality of the
3https://github.com/dbiir/UER-py
$^{4}$ https://github.com/huggingface/transformers
<sup>5</sup>https://paracrawl.eu
8548
resulting subsets is determined by training a neural machine translation system (Marian)<sup>6</sup> (Junczys-Dowmunt et al., 2018) on this data. The quality of the machine translation system is measured by BLEU score on six test sets from various domains. As the task is to address the challenge of the data quality and not domain-relatedness of the data for a particular use, sub-sampling the corpus for relevance to the news domain is not encouraged by the shared task organizers. All parameters used for training Marian machine translation models are the same as described in Koehn et al. (2018). We use CLIP = 5 and CUTOFF = 1.5 in the experiments. We use 4 GPUs for training.
4.2 Web-Crawled Japanese-Chinese Parallel Corpus Filtering
Due to the lack of publicly available Japanese-Chinese parallel corpus, we build a data harvesting pipeline to fetch Japanese-Chinese parallel text from the Internet. The crawled bi-text are extremely noisy, but we rely on the proposed parallel corpus filtering method to clean up the data and eventually train a satisfactory machine translation system. In this paper, we use these crawled data as another test bed to evaluate our proposed method.
A single run of the data harvesting pipeline is the following. We first identify Japanese-Chinese parallel webpages by programmatically analyzing the URL structure of the 5 billion URLs from CommonCrawl, for example, https://www.gotokyo.org/jp/ and https://www.gotokyo.org/cn/ only differ by jp and cn. Then we download the webpages and conduct a series of cascaded data cleaning methods, including removing HTML markings, sentence segmentation, etc. Finally we perform segment alignment and filtering. Our workflow consists of several runs of the data harvesting pipeline with entry points at different modules (for instance, a more targeted crawling of higher quality material from a previous run).
We also integrate existing Japanese-Chinese parallel datasets from other publicly available sources for a final parallel data size of  $527\mathrm{m}$  characters in 20.9M parallel segments.
We include all details of our data harvesting
pipeline, as well as the statistics of the obtained dataset, in Appendix A.
Test and Development Dataset. We curate two parallel test sets by manually processing web data involving daily expressions (337 parallel segments) and news (437 parallel segments). For our development set, we use 5304 Japanese-Chinese basic expressions.
WMT 2018 Parallel Corpus Filtering. Table 1 presents the BLEU scores of neural machine translation systems trained on 10 million and 100 million words of training data, selected by different filtering methods. In the table, we list the top three performers from the shared task, as well as another two work that are similar to ours. Junczys-Dowmunt (2018) has a dual conditional cross-entropy adequacy filter and a domain filter trained on news corpora. Hangya and Fraser (2018) generate sentence embeddings by using unsupervised word embedding alignment and measure parallelism via multilingual sentence embedding similarity. Chaudhary et al. (2019) leverage massive publicly available English-German parallel corpora to train multilingual sentence embeddings via bidirectional Long Short Term Memory (LSTM) encoder-decoder network.
We replicate the adequacy and domain-news filters from Junczys-Dowmunt (2018) and obtain similar results. By replacing the domain-news filter with our domain-GPT filter, we achieve new state-of-the-art scores on 10M and 100M word data sets (bold scores in the table). Given the very compact score range in the shared task (Koehn et al., 2018), we consider this gain very successful. It is stated in the shared task that the test sets are from multiple domains. Domain-news filter in Junczys-Dowmunt (2018) tends to select sentence pairs from news domain as the filter is trained on news domain data, and this leads to a biased parallel corpus for training machine translation system. Our proposed domain-GPT filter is trained from various sources and thus covers a wide range of domains, so our filtered data is more diverse and performs better on multi-domain test sets.
For our supervised acceptability filter, we train a multilingual BERT classifier on clean parallel sentences as positive examples and randomly sampling negative instances, using the method described in Section 3.2. For our unsupervised acceptabil
$^{6}$ https://github.com/marian-nmt/marian (We do not evaluate our method using Moses, the statistical machine translation system provided by WMT, as neural machine translation better fits our real world scenario.)  ${}^{7}$ https://commoncrawl.org/
8549
Method	Supervised	Unsupervised	10M	100M
Junczys-Dowmunt (2018) top-1	x	28.62	32.05
Lu et al. (2018) top-2	x	27.60	31.93
Lo et al. (2018) top-3	x	27.41	31.88
Hangya and Fraser (2018)	x	22.96	30.54
Chaudhary et al. (2019)	x	26.98	30.77
adequacy (our replication of J-D 2018)	x	27.12	31.20
+ domain-news (our replication of J-D 2018)	x	28.66	32.01
+ domain-GPT	x	†29.09	†32.11
supervised acceptability	x	27.09	31.56
+ domain-GPT	x	28.94	32.03
unsupervised acceptability	x	27.03	30.65
+ domain-GPT	x	‡28.68	‡32.02
- all methods above apply language detection filter beforehand.
† our new state-of-the-art combines adequacy (Junczys-Dowmunt, 2018) + our proposed domain-GPT.
$\ddagger$  our unsupervised acceptability  $^+$  domain-GPT is comparable to top supervised method.
Table 1: BLEU scores of German-English neural MT systems trained on 10 million and 100 million word training data selected by different methods. The scores are averaged BLEU scores across the six test sets from WMT 2018 parallel corpus filtering task. domain-news trains an in-domain language model on news corpus, while domain-GPT uses the pre-trained GPT language model.
Methods	JA-ZH	%*	ZH-JA	%*
unfiltered	22.92	100	22.27	100
Chaudhary et al. (2019)	23.46	75	26.22	70
adequacy (our replication of J-D 2018)	23.91	90	24.51	90
+ domain-GPT	24.00	65	-	-
acceptability	25.53	75	28.54	50
+ domain-GPT	25.49	50	-	-
- all methods above apply language detection filter beforehand.
* percentage of raw parallel sentences used for MT training.
Table 2: BLEU scores of Japanese-Chinese and Chinese-Japanese MT systems trained on data sets generated by various filtering methods. We rank sentence pairs by filtering scores and train an MT system on  $N$  percent of the top ranked data.  $N$  is selected based on the development set and we report the best BLEU score. domain-GPT is the domain filter whose in-domain language model is the pre-trained GPT language model; note that for ZH-JA, we do not have access to pre-trained Japanese GPT.