SlowGuess/ACL_2020_OCR huggingface dataset:
ity filter, we rank noisy parallel sentences by (a) the alignment score from Hunalign, and (b) the GPT domain filter score. We then select the top 10M words (counted on English side) worth of sentence pairs as positive examples. This makes the method completely unsupervised, not requiring any identifiable clean parallel data. With finetuning multilingual BERT on sentences pairs aligned by Hunalign, the unsupervised acceptability already achieves comparable performance to Chaudhary et al. (2019) which use massive public parallel data. After applying the unsupervised domain-GPT filter, we achieve a surprisingly good result (underlined scores in the table), comparable to the best super
vised method.
Japanese-Chinese Parallel Corpus Filtering. In Table 2, we evaluate machine translation systems trained on data generated by different filtering methods. Unfiltered refers to data generated by Hunalign without any filtering. Chaudhary et al. (2019) refer to LASER, the top performing filtering system in WMT 2019 Parallel Corpus Filtering shared task. We use the pretrained 93-language LASER model to generate sentence pair scores. The model is trained on a large parallel corpus that contains 3.2M English-Japanese and 8.2M English-Chinese sentence pairs (English is used as pivot to connect Japanese and
8550
Chinese during their training). Adequacy refers to the dual conditional cross-entropy filtering method that we replicate from Junczys-Dowmunt (2018). It is trained on around 300k high quality software-domain parallel sentences from Microsoft Developer Network (MSDN) and Ubuntu. The GPT domain filter uses a pre-trained Chinese  $\mathrm{GPT}^8$  as the in-domain language model and trains a four-gram KenLM (Heafield, 2011) language model on the Chinese side of our 4 million unfiltered noisy parallel sentences as a non-domain language model. Acceptability is our proposed multilingual BERT based filtering method, which is trained on a synthetic dataset, where we use 300k high-quality software domain parallel sentences as positive examples and sample equal-sized negative sentence pairs, using the sampling methods described in Section 3.2.
Chaudhary et al. (2019) train a multilingual sentence encoder on various English-Foreign-Language parallel corpus and prove the zero-shot cross-lingual transfer capability between non-English pairs, such as Japanese and Chinese. However, when English is used as the pivot, the distance between Japanese and Chinese become larger, resulting in not effectively capturing the correlation between them. The conditional cross-entropy metric in adequacy relies on the quality of machine translation system. Due to the difficulty of training high-quality machine translation systems on 300k sentence pairs, the adequacy filter cannot produce accurate conditional cross-entropy. The GPT domain filter assigns higher score to sentences that are more like human natural language and downgrades malformed sentence pairs. It is effective in the German-English filtering task, where a fixed-size subset is selected and we want to fill the subset with as much domain relevant data as possible. However, to best fit the real world scenario where the goal is to have the best machine translation system, we do not limit the amount of data to select for training machine translation system and let the system decide the amount of the data to select, according to each filtering method. We rank sentence pairs by their filtering scores and train a MT system on  $N$  percentage of the top ranked data.  $N$  is selected based on the development set and we report the best BLEU score. Under this setting, adding a domain filter makes the model use less data ( $N = 50\%$ )
Figure 1: Precision and recall curves of the acceptability filter on our internal JA-ZH filtering test set. The threshold is based on the classifier probability produced by the softmax layer. When threshold set to 0.9, we obtain  $97.7\%$  precision parallel sentence pairs at  $66.9\%$  recall.
vs  $N = 75\%$ ), but we do not observe any performance gain, as we suspect that the malformed but parallel sentence pairs are neither harmful or helpful to the model, and filtering them out makes no difference in performance of the model.
High Precision Parallel Corpus Filtering. For analysis purposes, we manually annotate a small set of 320 sentence pairs randomly selected from our original web crawled Japanese-Chinese data set.  $24\%$  of the sentence pairs are labeled "not mutual translations." As stated in Khayrallah and Koehn (2018), neural machine translation models are more sensitive to noise than statistical machine translation models, so having high precision filtering results as training data is necessary. In Figure 1, we show precision and recall curves for our proposed filtering method on this labeled test set, under different threshold settings. The threshold is selected based on the filtering classifier probability produced by the softmax layer. By setting the threshold to 0.9, we are able to obtain  $97.7\%$  precision high-quality parallel sentences, while still having  $66.9\%$  recall.
In this paper, we address the parallel corpus filtering problem in machine translation. We propose a novel filtering method using pre-trained language models. Our method outperforms strong baselines and achieves a new state-of-the-art. We release a large Japanese-Chinese web crawled parallel corpus for the research purposes. Because it is artificial to use synthetic data for training a filter classifier, future work can focus on a better objective that
pre-trained Mixedlarge corpus + GptEncoder + LmTarget Model in https://github.com/dbiar/UER-py
8551
models parallelism more smoothly. Future work also includes extending the method to low-resource languages not covered by multilingual BERT.
We would like to thank the anonymous reviewers for their constructive feedback.
Mikel Artetxe and Holger Schwenk. 2019. Margin-based parallel corpus mining with multilingual sentence embeddings. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Amittai Axelrod, Yogarshi Vyas, Marianna Martindale, and Marine Carpuat. 2015. Class-based n-gram language difference models for data selection. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT).
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of the Sixth International Conference on Learning Representations (ICLR).
Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation.
Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi. 2015. Integrated parallel sentence and fragment extraction from comparable corpora: A case study on Chinese-Japanese Wikipedia. ACM Transactions on Asian and Low-Resource Language Information Processing.
Raj Dabre and Sadao Kurohashi. 2017. MMCR4NLP: multilingual multiway corpora repository for natural language processing. CoRR, abs/1710.01025.
Michael Denkowski, Greg Hanneman, and Alon Lavie. 2012. The CMU-Avenue French-English translation system. In Proceedings of the Seventh Workshop on Statistical Machine Translation.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Kevin Duh, Graham Neubig, Katsuhito Sudoh, and Hajime Tsukada. 2013. Adaptation data selection using neural language models: Experiments in machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Chris Dyer, Jonathan Weese, Hendra Setiawan, Adam Lopez, Ferhan Ture, Vladimir Eidelman, Juri Ganitkevitch, Phil Blunsom, and Philip Resnik. 2010. cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), System Demonstrations.
Viktor Hangya and Alexander Fraser. 2018. An unsupervised system for parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers.
Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hévre Jégou, and Tomas Mikolov. 2016. FastText.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL).
Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andre F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in  $\mathbf{C} + +$ . In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), System Demonstrations.
Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the Workshop on Neural Machine Translation and Generation.
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L Forcada. 2018. Findings of the WMT 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In Proceedings of the 6th International Conference on Learning Representations (ICLR).
8552
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC).
Patrick Littell, Samuel Larkin, Darlene Stewart, Michel Simard, Cyril Goutte, and Chi-kiu Lo. 2018. Measuring sentence parallelism using Mahalanobis distances: The NRC unsupervised submissions to the WMT18 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers.
Chi-kiu Lo, Michel Simard, Darlene Stewart, Samuel Larkin, Cyril Goutte, and Patrick Littell. 2018. Accurate semantic textual similarity for cleaning noisy parallel corpora using semantic machine translation evaluation metric: The NRC supervised submissions to the parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers.
Jun Lu, Xiaoyu Lv, Yangbin Shi, and Boxing Chen. 2018. Alibaba submission to the WMT18 parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers.
Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Christian Federmann Jiatao Gu Fei Huang Ajay Nagesh Jan Niehues Elizabeth Salesky Sebastian St uker Marco Turchi Ondrej Bojar, Marcello Federico. 2020. Findings of the iwslt 2020 evaluation campaign. In Proceedings of the 2020 International Conference on Spoken Language Translation (IWSLT).
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from Wikipedia. arXiv preprint arXiv:1907.05791.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC).
Daniel Varga, Péter Halácsy, András Kornai, Viktor Nagy, László Nemeth, and Viktor Trón. 2007. Parallel corpora for medium density languages. Amsterdam Studies in the Theory and History of Linguistic Science.
Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
8553
A Web-Crawled Parallel Data for Japanese-Chinese
Figure 2: Our Japanese-Chinese parallel data harvesting pipeline. It consists of several modules, each of them numbered. The inputs to and outputs from each module are depicted in orange. The example entry points to the data pipeline are shown at the bottom of the diagram.
Source	# Segment-pairs	# Characters (zh side)	Reference
Web-crawled (pipeline)	18,966,595	493,902,539	-
Linux documentation	92,250	1,549,964	Tiedemann (2012)
Open Subtitles	914,355	10,932,722	Lison and Tiedemann (2016)
TED	376,441	5,345,867	Dabre and Kurohashi (2017)
Global Voices	16,848	337,194	Tiedemann (2012)
Wikipedia	228,565	5,067,489	Chu et al. (2015)
Wiktionary	62,557	222,562	wiktionary.org
News Commentary	570	65,038	Tiedemann (2012)
Tatoeba	4,243	50,846	tatoeba.org
Facebook	267,409	9,950,657	Schwenk et al. (2019)
Total	20,929,833	527,424,878	-
Table 3: Japanese-Chinese parallel data assembled for our experiments.
This appendix describes our pipeline to extract parallel Japanese-Chinese parallel sentence fragments from the Internet (Figure 2). We start with 5 billion URLs from CommonCrawl. We identify Japanese-Chinese parallel webpages by looking at URL structure (step 2). For example, https://www.gotokyo.org/jp/ and https://www.gotokyo.org/cn/ only differ by jp and cn. We download these potentially parallel page pairs (step 3), remove HTML and other markup metadata (step 4), and split into sentence segments. We use off-the-shelf Hunalign for segment alignment (step 5). We filter segment pairs by rough language ID and length ratio (step 6). We obtain 227k URL pairs, 1.4m segment pairs, and 28.7m characters of parallel data (measured on the Chinese side).
From the 227k URL pairs above, we trace which site pairs yielded the most parallel data. We then run a deep-crawling module on each of the 6000 most-promising sites, $^{12}$  and we process the resulting URLs using the rest of the pipeline. Concatenating parallel data from all runs (step 7) and running a simple post-processing filter to remove objectionable content in the text gathered, we obtain around  $494\mathrm{m}$  characters of parallel data (measured on the Chinese side).
We also integrate existing Japanese-Chinese parallel datasets from other publicly available sources for a final parallel data size  $527\mathrm{m}$  characters in  $20.9\mathrm{m}$  parallel segments. Table 3 describes the various components of this dataset.
$^{9}$ https://commoncrawl.org/
10Using Python module BeautifulSoup
<sup>11</sup> http://mokk.bme.hu/en/resources/hunalign/
12Using the Python-based scrapy tool
8554