SlowGuess/NAACL_2021_OCR huggingface dataset:
way. Note that the amount of data is measured in target positions on the raw text. However since for each MT training we train and apply a new subword splitting, the amount of target subwords in training varies slightly (we observe changes of less than  $5\%$ ). Results for the different data selection schemes can be found in Table 3. We observe that transforming the scores can be extremely helpful to get good filtering performance. Selecting based on a dev set distribution yields similar strong results but is not as stable. We select data corresponding to the Top  $50\%$  of target tokens according to the transformed score except for the GIZA method where we use the non-transformed score because the transformation resulted in unreliable scores due to precision issues.
German→English
First we consider the De→En WMT 2019 news translation task. Note that most of the training data comes from the news translation task ParaCrawl corpus which is smaller and of better quality than the ParaCrawl corpus used in the WMT 2018 parallel corpus filtering task. We start with all the training data and apply language ID as initial filtering, i.e. if either the source or the target sentence of a training pair is not classified with the correct language we drop the sentence pair. The result of this filtering can be seen in Table 4, Line 2. All further filtering methods are trained and applied on this pre-filtered corpus.
It is interesting to point out that the LM & TM comparison system does not even beat the language identification baseline. For LM & TM we employ a slight simplification of a system that improved
Filter Method	Training Data Ratio	dev (newstest2015)	newstest2017
BLEU	TER	BLEU	TER
None (baseline)	1.00	33.5	53.3	34.6	52.7
Language ID	0.89	33.7	53.0	35.0	52.0
LM & TM (Rossenbach et al., 2018)	0.49	33.6	53.7	34.5	52.9
Accum. Expl. Scores	0.48	34.2	52.6	35.6	51.6
Expl. Disagreement Score	0.50	33.5	53.3	35.1	52.1
+ pre-filtering	0.49	33.6	52.8	35.2	51.8
Argmax Agreement	0.49	34.1	52.9	35.2	52.0
Maximum Matching (score)	0.49	33.9	53.2	35.1	52.2
+ BPE level	0.49	33.7	53.2	34.8	52.7
Maximum Matching (count)	0.49	33.8	53.0	34.9	52.2
+ BPE level	0.50	33.5	53.8	34.9	52.4
Average similarity	0.49	33.9	52.8	35.2	52.2
GIZA	0.50	32.6	53.9	33.5	53.1
Table 4: Comparing filtering methods on De→En WMT 2019 news translation task. All filtering methods are trained and applied on a corpus that is pre-filtered with language identification (Line 2). Amount of training data is given as ratio of the original corpus. BLEU and TER are reported in percentage.
translation performance by more than 8.0 BLEU and performed among the best on the WMT 2018 data filtering task (Rossenbach et al., 2018). There are two crucial differences to consider: (1) We train the filtering system on the same data that it needs to filter afterwards. This means the filtering pipeline might learn typical patterns from the data that are not actually relevant for translation, like copying the input sentence. (2) The ParaCrawl corpus used here is a newer version of better quality and we add the established training data for the WMT news translation task so that the complete training data is generally of significantly higher quality. Note that the ParaCrawl corpus still provides  $80\%$  of the training data and the benefits of doing data filtering diminish quite clearly. We conclude that it is highly important how exactly the data filtering task is phrased.
The best performance on the De→En WMT task is achieved by the 'Accumulated Explanation Scores' method which yields an average improvement of  $0.5\%$  with respect to both BLEU and TER across the dev and test set. All other methods except for 'GIZA' are on par with the language identification baseline, however they achieve a significant reduction of the training data. We experiment with a variant of the Maximum Matching method for scores and counts that is built on top of cross-lingual subword embeddings without any effect in translation performance.
English  $\rightarrow$  Turkish
The behaviour of the filtering systems is quite different for the company data set of the  $\mathrm{En} \rightarrow \mathrm{Tr}$  task. We report results on three openly available test sets from different domains. In this scenario language identification helps quite clearly on two out of three data sets while LM & TM data filtering significantly reduces the translation performance.
With our methods, we observe very clear improvements on the TED test set as well as newstest2018. The Explanation Disagreement Score with pre-filtering gains an average of 0.7 BLEU [‰] over the language identification filtering. If we apply Maximum Matching filtering on BPE level we even observe improvements of 2.2 and 5.1 BLEU [‰] on TED and newstest2018, however we lose 0.9 BLEU [‰] and 0.7 TER [‰] on the OpenSubtitles test set. In practice, this minor degradation is out weighted by the significantly stronger performance on the other domains, proofing the usefulness of data filtering in this scenario.
The scores based on GIZA alignments result in a very poor performance on all domains except subtitles. By analyzing the selected data, we find that the 'GIZA' method selects on average shorter sequences than other methods which is detrimental for the news and talks domain but not so much for subtitles.
Filter Method	Training Data Ratio	TED	newstest2018	Opensubtitles
BLEU	TER	BLEU	TER	BLEU	TER
None (baseline)	1.00	14.8	77.2	16.2	73.5	20.9	71.4
Language ID	0.84	16.1	76.1	19.3	70.6	20.4	74.6
LM & TM (Rossenbach et al., 2018)	0.50	10.4	82.1	9.2	83.5	19.4	76.1
Accum. Expl. Scores	0.48	15.6	75.6	22.1	66.6	19.0	76.1
Expl. Disagreement Score	0.48	16.8	73.9	21.8	67.6	19.3	76.4
+ pre-filtering	0.48	17.3	73.1	22.7	65.9	20.0	75.6
Maximum Matching (score)	0.48	16.3	74.8	20.5	68.7	19.5	76.0
+ BPE level	0.52	18.2	71.2	24.4	64.1	19.5	75.3
Maximum Matching (count)	0.48	16.1	74.0	19.4	69.5	18.7	77.8
+ BPE level	0.49	14.7	75.3	15.7	73.7	18.6	77.7
GIZA	0.50	7.4	82.9	5.9	84.6	18.7	76.7
Table 5: Comparing filtering methods on  $\mathrm{En} \rightarrow \mathrm{Tr}$  WMT 2019 news translation task. All filtering methods are trained and applied on a corpus that is pre-filtered with language identification (Line 2). Amount of training data is given as ratio of the original corpus. BLEU and TER are reported in percentage.
Filter Method	Training Data Ratio	dev (newstest2016)	newstest2019
BLEU	TER	BLEU	TER
None (baseline)	1.00	25.7	63.3	22.3	66.8
Language ID	0.90	25.9	63.4	22.7	66.7
LM & TM (Rossenbach et al., 2018)	0.50	24.7	64.7	21.3	68.4
Accum. Expl. Scores	0.49	25.7	63.4	23.0	66.3
Expl. Disagreement Score	0.49	25.7	63.2	22.3	66.9
Maximum Matching (score)	0.48	25.6	63.5	22.3	66.9
+ BPE level	0.48	25.5	63.7	22.5	66.7
GIZA	0.50	24.8	64.0	20.6	68.4
Table 6: Comparing filtering methods on  $\mathrm{En}\rightarrow \mathrm{Cs}$  WMT 2019 news translation task. All filtering methods are trained and applied on a corpus that is pre-filtered with language identification (Line 2). Amount of training data is given as ratio of the original corpus. BLEU and TER are reported in percentage.
English  $\rightarrow$  Czech
For the  $\mathrm{En} \rightarrow \mathrm{Cs}$  task we observe no significant improvement with any of the methods over even the training on the full training data, even though  $10\%$  of the data is removed by simple language identification filtering. Here we observe that LM & TM filtering becomes actively hurtful to the translation performance while the methods proposed in this paper reduce the training data by a factor of two without losing in translation performance. The proposed filtering methods all provide very similar filtering performances except for the scores based on GIZA alignments which decrease the system performance by more than one  $\mathrm{BLEU}^{[\%]}$ .
5.3 WMT 2020: Khmer→English
As an additional experiment, we also test our methods on the WMT 2020 shared task for parallel corpus filtering in the Khmer  $\rightarrow$  English setting. Although some conditions of this task are quite artificial as discussed before, it provides the opportunity to compare different filtering approaches in the same framework.
The task consists of selecting sentence pairs that amount to 5.0M English words from a noisy parallel corpus with a total of 58.3M English words. The quality of the selected data is evaluated by training an NMT system (Ott et al., 2019) on this data and evaluating the system on unseen test sets labeled 'devt' and 'test' (Koehn et al., 2020). For
Filter Method	BLEU
devt	test
LASER (2019 winner)	7.1	8.4
Alibaba system (2020 winner)	8.9	11.0
Maximum Matching (score)	8.2	10.3
Accum. Expl. Scores	9.0	10.9
Table 7: Final performance of NMT systems trained on the selected data (5.0M English tokens) of the WMT 2020 Khmer→English data filtering task.
training the filtering system, around 123k clean parallel sentences are given as well as large monolingual corpora for both languages (14M sentences for Khmer and 1.9B sentences for English).
As a first step, we apply filtering using language identification as described in Section 3 to sort out sentence pairs with wrong language on source and/or target side. Based on the previous findings, we use our 'Accum. Expl. Scores' and our 'Maximum Matching (score)' methods on the BPE level for scoring. Since the parallel data is very small and of questionable quality we only use the monolingual data for the training of our word embeddings. We use all the available monolingual Khmer data while subsampling 14M English sentences. We use the polyglot tokenizer<sup>4</sup> on the Khmer data and train BPE models for Khmer and English separately. The performance of the resulting NMT system is shown in Table 7.
Also shown in the table are the results of the LASER filtering system (Chaudhary et al., 2019) which won the WMT 2019 data-filtering evaluation as well of the Alibaba filtering system (Lu et al., 2020) which won the WMT 2020 data-filtering evaluation for Khmer→English. We find that our filtering methods performs strongly on this task as well, with our 'Accum. Expl. Scores' method performing on par with the strongest submission of the latest WMT campaign while not relying on any parallel data.
In this work we focus on data filtering for machine translation. We define this task as the selection of a subset of a given, possibly noisy corpus, without the help of additional large-scale 'clean' corpora. In order to develop a helpful filtering method, we first analyze the commonly used 'filtering by lan
guage identification' approach by applying it to synthetically generated noisy data. We find that while 'filtering by language identification' does an outstanding job in detecting noise that comes from wrong language alignment, it fails to detect noise resulting from a semantic mismatch between source and target sentence.
Building on these findings, we develop several approaches - based on cross-lingual word embeddings - specifically targeting the word alignments between source and target sentence. Furthermore, we conduct a systematic comparison on data selection methods in an effort to uncouple the scoring and selection parts of any data filtering pipeline. We compare our approaches to one of the winning methods from the WMT 2018 shared task on parallel corpus filtering on three real-life, high resource tasks as well as on the recent WMT 2020 shared task on parallel corpus filtering. We find that the existing approach does not perform well in our more realistic scenario, leading to a degradation in performance in most cases. Our methods result in improvements over the baseline on all three three tasks. However, different variants of our methods perform best on different tasks and we can not identify a single best approach.
Finally, we compare our methods to state-of-the-art data-filtering systems on the WMT 2020 shared task on parallel corpus filtering. Here, our proposed approaches yield comparable results to aforementioned state-of-the-art methods while not relying on any parallel training data.
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 re
search and innovation programme (grant agreement No 694537, project "SEQCLAS"). The work reflects only the authors' views and the European Research Council Executive Agency (ERCEA) is not responsible for any use that may be made of the information it contains.
Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney. 2018. On the alignment problem in multi-head attention-based neural machine translation. In Pro
<sup>4</sup>https://github.com/aboSamoor/polyglot
ceedings of the Third Conference on Machine Translation: Research Papers, pages 177-185.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 789-798. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Peter F. Brown, Stephen Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Comput. Linguistics, 19(2):263-311.
Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019 - Volume 3: Shared Task Papers, Day 2, pages 261-266.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. CoRR, abs/1710.04087.
Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648.
Viktor Hangya and Alexander M. Fraser. 2018. An unsupervised system for parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, WMT 2018, Belgium, Brussels, October 31 - November 1, 2018, pages 882-887.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving human parity on automatic chinese to english news translation. CoRR, abs/1803.05567.
Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine
Translation: Shared Task Papers, WMT 2018, Belgium, Brussels, October 31 - November 1, 2018, pages 888-895.
Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, NMT@ACL 2018, Melbourne, Australia, July 20, 2018, pages 74-83. Association for Computational Linguistics.
Philipp Koehn, Vishrav Chaudhary, Ahmed El-Kishky, Naman Goyal, Peng-Jen Chen, and Francisco Guzmán. 2020. Findings of the WMT 2020 shared task on parallel corpus filtering and alignment. In Proceedings of the Fifth Conference on Machine Translation, pages 726-742.
Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. 2019. Findings of the WMT 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019 - Volume 3: Shared Task Papers, Day 2, pages 54-72.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic. The Association for Computational Linguistics.
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L Forcada. 2018. Findings of the WMT 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 726-739.
Jun Lu, Xin Ge, Yangbin Shi, and Yuqi Zhang. 2020. Alibaba submission to the WMT20 parallel corpus filtering task. In Proceedings of the Fifth Conference on Machine Translation, pages 979-984.
Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proceedings of the ACL 2012 System Demonstrations, pages 25-30, Jeju Island, Korea. Association for Computational Linguistics.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Comput. Linguistics, 29(1):19-51.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Nick Rossenbach, Jan Rosendahl, Yunsu Kim, Miguel Graça, Aman Gokrani, and Hermann Ney. 2018. The RWTH Aachen University filtering system for the WMT 2018 parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 946-954, Belgium, Brussels. Association for Computational Linguistics.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. J. Artif. Intell. Res., 65:569-631.
Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020, pages 1627-1643. Association for Computational Linguistics.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas. Cambridge, MA.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
Rui Wang, Andrew M. Finch, Masao Utiyama, and Eichiro Sumita. 2017. Sentence embedding for neural machine translation domain adaptation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 2: Short Papers, pages 560-566.
Albert Zeyer, Tamer Alkhouli, and Hermann Ney. 2018. RETURNNN as a generic flexible neural toolkit with application to translation and speech recognition. In
Annual Meeting of the Assoc. for Computational Linguistics, volume abs/1805.05225.