SlowGuess/NAACL_2022_OCR huggingface dataset:
First, we examine how models trained only on good quality data (Figure 4) behave as we vary their quantity. We experiment with English-Greek EN-EL CCMatrix bittexts and simulate various resource settings via downsampling. In low-resource settings (i.e.,  $|A| < 1\mathrm{M}$ ), translation quality exhibits rapid improvements, with an increase from 100K to 500K training samples boosting BLEU, by approximately 10 points. In medium-resource scenarios (i.e.,  $1 < \mathrm{M}|A| < 5\mathrm{M}$ ), a proportional increase in the quantity of good quality bittexts yields smaller—yet, significant—translation quality improvements (i.e., moving from 1M to 5M bittexts yields +2 BLEU). Finally, in high-resource settings (i.e.,  $|A| > 5$ ), translation quality reaches a saturation point, with BLEU increases being small and insignificant (i.e.,  $\sim +0.2$ ) as we move from 10M to 15M training samples.
Figure 3: Translation quality (i.e., BLEU) of EN  $\rightarrow$  EL NMT models trained on different amounts of Pool A and Pool B data (i.e.,  $|A|$  given by  $x$ -axis). Across settings, bitext refinement (i.e.,  $A \cup r(B)$ ) performs better or comparably to training on the original CCMatrix (i.e.,  $A \cup B$ ) or its filtered version (i.e.,  $A$ ).
Figure 4: BLEU for EN  $\rightarrow$  EL NMT trained on varying size of CCMatrix data (Pool A).
Second, we present a controlled analysis experiment on how bitext refinement impacts the translation quality of NMT systems under different resource settings (Figure 3). Starting from a high resource language-pair in CCMatrix (here, EN-EL) we sample good and poor quality bitexts (i.e.,  $A$  and  $B$ , respectively) representing low- to high-data scenarios (e.g., 500K up to 15M sentence-pairs). Then, we train EN→EL NMT systems on  $A \cup B$  while varying their distribution to represent three settings: (a) good quality bitexts overwhelm the training data (i.e.,  $|B| = |A| / 2$ ), (b) good and poor quality bitext are equally represented (i.e.,  $|B| = |A|$ ), and (c) poor quality bitexts overwhelm the training data (i.e.,  $|B| = 2|A|$ ). We include more details on experimental settings in Appendix B.
Across distribution conditions, adding imperfect translations (i.e.,  $B$ ) to the original good quality data yields improvements for low-to-medium resource settings (i.e.,  $|A| < 5$ ). This results complement the earlier observations of §5 that question the appropriateness of a filtering framework in settings where data is scarce. On the other hand, when moving to high resource scenarios, the additional signal that results from imperfect references can have either insignificant (i.e., Figure 3a) or negative impact (i.e., Figures 3b and 3c) on translation quality. The latter depends on whether the good quality data is underrepresented in the training samples.
→	[EN] CCMatrix	After that time the whole group would talk for 5 minutes.
[EL] CCMatrix	Aργότερα, η σαδα μελέτης ζήτησε από δλούςν αν διαλογίστουν για πέντε λεπά.
[GLOSS	Later, the study group asked everyone to meditate for 5 minutes.
[EN] BITEXTEDIT	Later, the study group asked everyone to meditate for five minutes.
→	[EN] CCMatrix	We should, however, always be striving to live a sustainable and kind life.
[EL] CCMatrix	Πάντα πρέπεινα παλέουμε για διακη καβιώσμη εἰρήνη.
[GLOSS	We must always fight for a just and lasting peace.
[EN] BITEXTEDIT	We must always fight for just and sustainable peace.
→	[EN] CCMatrix	“The western influence came from film and television”, he later explained.
[EN] CCMatrix	«H ρλοχρία ευτένετα δλο καπερισότepo στον κανηματογράρο κατητην τηλέρωση», εἰξηγησο εἰδιος.
[GLOSS	“Censorship is intensifying in cinema and television”, he explained.
[EL] BITEXTEDIT	«H δυτήειρρού ἡρθε από την ταυία κατητην τηλέρωση», εἰξηγησο αργότερα.
[GLOSS	“The western influence came from form and television”, as their later explained.
→	[EN] CCMatrix	I could work with a hospital specialist as a clinical assistant (as I have done).
[EL] CCMatrix	Δούλενε δός βοηδός ερευνητή παράληλα με το διακτοριχό (όπως κα εγώ)
[GLOSS	They were working as an assistant researcher in parallel with their doctorate (as I have done).
[EL] BITEXTEDIT	Θα μρούσαν αν δουλέψω με εναν εἰδιχό στονονοχομείος ως κλινηός βοηδός (όπως ἐχω κάνει).
[GLOSS	I could work with a hospital specialist as a clinical assistant (as I have done).
Table 3: Examples of CCMatrix bittexts along with refined sides generated by BITEXTEDIT.  $\rightarrow$  denotes the side ([EL] or [EN]) that the model edits, while highlighted segments indicate the meaning mismatches in the original CCMatrix sentence that gets edited. Greek sentences are glossed to help understanding their meaning.
Third, starting from good quality bitexts of varying sizes, we train separate bitext refinement models and edit the corresponding poor quality samples (i.e.,  $r(.))$  defined earlier. Across the board, NMT models that are trained on  $A \cup r(B)$  yield the best translation quality results compared to both filtering and training on original CCMatrix. However, we observe that the magnitude of the improvements depends on the data settings. Concretely, bitext refinement yields significant improvements on low-to-medium resource settings (i.e.,  $\sim +2$  BLUE points). On the other hand, in high resource scenarios bitext refinement helps mitigate the negative impact of overwhelming poor quality instances and performs comparably to filtering. The latter suggests that our refinement strategy improves bitexts quality across low- to high- resource settings.
6.2 Qualitative analysis
We conduct a qualitative study to confirm that BITEXTEDIT improves the quality of CCMatrix. We include details on the annotation in Appendix C. One of the authors manually evaluates a random sample of 200 EN-EL sentence-pairs where we compare the original bittexts against the refined ones. Here, we present results on bitext refinement models that use 0.5M PoolA samples. Manual inspection on refined outputs of models trained on larger pools showed similar performance. As shown in
Figure 5: Number of bittexts manually rated as perfect translations (i.e., No difference), partial translations (i.e., some meaning difference), and wrong translations (i.e., unrelated) for a random sample of original vs. refined CCMatrix EN-EL data.
Figure 5, our models performs edits that refine meaning mismatches found in the original CCMatrix data. While only  $\sim 38\%$  of the original samples contain parallel texts that are perfect translations of each other, the revised sample contains  $\sim 70\%$  perfect translations. Finally—apart from evaluating meaning differences—we also rate fluency of the edited translations. We find that our model does not suffer from major fluency issues with  $84.5\%$  of their outputs rated as flawless and  $15.5\%$  as good. Table 3 presents example outputs of our BITEXTEDIT approach for English-Greek. More examples can be found in Appendix E.
CORPUS	EDITED SENT.	C	S	D	I	C	S	D	I
ALL (%)	ALL \ COPIES (%)
Tatoeba	29.80%	97.47	1.88	0.29	0.34	86.38	10.16	1.56	1.88
OpenSubtitles	65.63%	90.46	5.53	1.27	2.73	74.51	14.79	3.39	7.29
ParaCrawl	88.11%	96.30	2.25	0.39	1.04	85.42	8.89	1.55	4.12
Percentage of edited bitexts Table 5 presents coarse statistics on the percentage of refined bitexts that exhibit at least one edit compared to the original ones. First, we observe that the percentage of edited bitexts varies across the languages-pairs studied. This reflects the varying quality of PoolB samples in different languages and also connects to the varying magnitude of improvements we show in Table 2. The biggest improvements are given for IT-OC, where  $\sim 76\%$  of the bitexts are edited by our refinement models. On the other hand, the smallest improvements are found for EN-SW, with only  $\sim 36\%$  of its bitext being revised, probably due to the already good quality of the initial CC-Matrix sentence pairs.
Editing EN-EL OPUS corpora Broadly speaking, a good bitext refinement model should be able to rewrite bitext in a way that improves potential errors in the original references. At the same time though, it should avoid over-editing (i.e., avoid editing an already good translation-pair). We perform a quantitative analysis on EN-EL corpora from OPUS that vary in their quality and extract Translation Error Rate (TER) label (Snover et al., 2006) token-level statistics to study both the frequency and the types of edits that our bitext refinement models perform. Table 4 presents results on random samples ( $\sim$ 100K) of three popular corpora: (a) the Tatoeba corpus (Tiedemann, 2020) consisting of human translations, (b) the OpenSubtitles corpus (Lison and Tiedemann, 2016) consisting of sentence-aligned subtitles of movie series $^{6}$ , and (c) the ParaCrawl corpus (Esplà et al., 2019) consisting of automatically crawled translations from translations of European Parliament Proceedings.
As expected, our model performs minimal editing on the high-quality Tatoeba bittexts. Concretely,
Table 4: TER statistics for bitext refinement of random samples of EN-EL OPUS bittexts. Second column gives the % of bittexts that get at least one edit operation; the last two columns present the percentage of correct (C), substituted (S), deleted (D), and inserted (I) tokens for all the bittexts (i.e., ALL) and the subset of bittexts that receive revisions compared to the original (i.e., ALL\ COPIES).
PAIR	SRC	TGT	BOTH
EN-OC	34.06%	66.58%	67.48%
IT-OC	34.76%	41.11%	75.78%
EN-MR	58.35%	19.90%	68.07%
BE-EN	21.01%	28.06%	49.06%
EN-SW	14.52%	21.05%	35.57%
Table 5: Percentage of sentences with at least one edit operation compared to the original for: source-side (SRC), target-side (TGT), and both sides (BOTH).
only  $\sim 30\%$  of it gets revised, while as suggested by the token-level TER statistics even the revised sentence-pairs mostly consist of substituted tokens. Further manual inspection reveals that most of those tokens depict subtle spelling differences between Greek words. On the other hand, when editing the samples of automatically extracted bittexts our refinement model performs more frequent edits: it revises  $\sim 65\%$  of OpenSubtitles and  $\sim 88\%$  of ParaCrawl bittexts. Interestingly, although a greater amount of ParaCrawl texts get revised compared to OpenSubtitles, edits on the latter are more aggressive as it consists of at least  $10\%$  fewer correct (i.e., C) tokens than the former. A break down on the types of operations further reveals that editing OpenSubtitles requires more deletion (i.e., D) and insertion (i.e., I) operations compared to the other two. This observation connects to prior efforts on auditing OpenSubtitles that found sentence segmentation errors (i.e, added extra leading/trailing words in one side) to be a frequent type error for this corpus (Vyas et al., 2018).
Automatic Post-Editing APE aims at automatically correcting the output of a black-box MT system. Recent approaches on APE (Chatterjee et al., 2019, 2020) fine-tune pre-trained multilingual models models (Lopes et al., 2019) or translation models (Yang et al., 2020) on a combination of gold-standard APE data and artificially aug
http://www.opensubtitles.org/,https:// opus.nlpl.eu/OpenSubtitles-v2018.php
mented candidates resulting from external translations. BITEXTEDIT aims instead, at editing imperfect translations representing human generated texts in two languages, without assuming access to gold-standard training data.
Low-resource MT Haddow et al. (2021) structure the diverse set of approaches to low-resource MT to (a) efforts for increasing the amounts of available bittexts (i.e., data collection; Schwenk et al. (2021a,b)), (b) methods that explore how other types of data can be incorporated into MT (i.e., data exploitation; Baziotis et al. (2020); Zoph et al. (2016); García-Martínez et al. (2017)), and (c) advances in modeling (i.e., model choices; Vaswani et al. (2017)). BITEXTEDIT is an alternative data exploitation approach that does not require further bilingual data or other sources of supervision.
Synthetic Bitext Generating synthetic bitext has mainly been studied as a means of data augmentation for NMT through forward translation (Zhang and Zong, 2016), backtranslation (Sennrich et al., 2016a; Marie et al., 2020; Hoang et al., 2018), or round-trip translation (Ahmadnia and Dorr, 2019) of monolingual resources. Moreover, recent line of works use the predictions of forward and backward translation models to induce the creation of new versions of the parallel data: Nguyen et al. (2020) diversify the parallel data via translating both sides using multiple models and then merge them with the original to train a final NMT model; Jiao et al. (2020) employ a similar approach to rejuvenate the most inactive examples that contributes less to the model performance; Kim and Rush (2016) propose to train a student model of smaller capacity on sequence-level interpolated data generated by a teacher model of higher capacity. Using synthetic translations to augment or revise real bitexts assumes access to NMT systems of sufficient quality. Recent works propose methods to automatically revise noisy synthetic bitexts (Cheng et al., 2020; Wei et al., 2020). By contrast, our work accounts for imperfect references in real bitext and is tailored to low-resource settings where NMT quality is too low to provide reliable candidate translations.
Retrieve & Edit Approaches Retrieve and edit approaches have been integrated at inference time for several tasks, such as NMT (Gu et al., 2018; Zhang et al., 2018; Cao and Xiong, 2018; Hossain et al., 2020), APE (Hokamp, 2017), dialogue generation (Weston et al., 2018), among others.
We introduce an alternative approach for bitext quality improvement that we show is better suited for low-resource language pairs. Instead of filtering out imperfect translation references that result from automatic bitext mining, we instead edit them with the goal of improving their quality. Our editing models are trained using only synthetic supervision, which can be gathered at scale for any language pair that support bitext mining. Extensive quantitative analysis suggests that our approach successfully improves bitext quality for a variety of language-pairs and different resource conditions. Furthermore, extrinsic experiments on 10 low-resource NMT tasks suggest that bitext refinement constitutes a successful approach to improving NMT translation quality in low data regimes. Those findings highlight the importance of the good quality bittexts in scenarios where large quantities cannot be guaranteed and motivate future research on improving low-resource NMT further.
Benyamin Ahmadnia and B. Dorr. 2019. Augmenting neural machine translation through round-trip training approach. Open Computer Science, 9:268 - 278.
Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Tom Ash, Remi Francis, and Will Williams. 2018. The speechmatics parallel corpus filtering system for WMT18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 853-859, Belgium, Brussels. Association for Computational Linguistics.
Marta Banón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. ParaCrawl: Web-scale acquisition of parallel corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4555-4567, Online. Association for Computational Linguistics.
Christos Baziotis, Barry Haddow, and Alexandra Birch. 2020. Language model prior for low-resource neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language
Processing (EMNLP), pages 7622-7634, Online. Association for Computational Linguistics.
Eleftheria Briakou and Marine Carpuat. 2020. Detecting Fine-Grained Cross-Linguial Semantic Divergences without Supervision by Learning to Rank. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1563–1580, Online. Association for Computational Linguistics.
Eleftheria Briakou and Marine Carpuat. 2021. Beyond noise: Mitigating the impact of fine-grained semantic divergences on neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7236-7249, Online. Association for Computational Linguistics.
Eleftheria Briakou and Marine Carpuat. 2022. Can synthetic translations improve bitext quality? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (to appear), Online. Association for Computational Linguistics.
Qian Cao and Deyi Xiong. 2018. Encoding gated translation memory into neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3042-3047, Brussels, Belgium. Association for Computational Linguistics.
Rajen Chatterjee, Christian Federmann, Matteo Negri, and Marco Turchi. 2019. Findings of the WMT 2019 shared task on automatic post-editing. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 11-28, Florence, Italy. Association for Computational Linguistics.
Rajen Chatterjee, Markus Freitag, Matteo Negri, and Marco Turchi. 2020. Findings of the WMT 2020 shared task on automatic post-editing. In Proceedings of the Fifth Conference on Machine Translation, pages 646-659, Online. Association for Computational Linguistics.
Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019. Lowresource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 261-266, Florence, Italy. Association for Computational Linguistics.
Shanbo Cheng, Shaohui Kuang, Rongxiang Weng, Heng Yu, Changfeng Zhu, and Weihua Luo. 2020. Ar: Auto-repair the synthetic data for neural machine translation. ArXiv, abs/2004.02196.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020. CCAligned: A massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5960-5969, Online. Association for Computational Linguistics.
Miquel Esplà, Mikel Forcada, Gema Ramírez-Sánchez, and Hieu Hoang. 2019. ParaCrawl: Web-scale parallel corpora for the languages of the EU. In Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks, pages 118-119, Dublin, Ireland. European Association for Machine Translation.
Mercedes García-Martínez, Loïc Barrault, and Fethi Bougares. 2017. Neural machine translation by generating multiple linguistic factors. CoRR, abs/1712.01821.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O.K. Li. 2018. Search engine guided neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali-English and Sinhala-English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6098-6111, Hong Kong, China. Association for Computational Linguistics.
Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindrich Helcl, and Alexandra Birch. 2021. Survey of low-resource machine translation.
Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. 2014. Predicting grammaticality on an ordinal scale. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 174-180, Baltimore, Maryland. Association for Computational Linguistics.
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative backtranslation for neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 18-24, Melbourne, Australia. Association for Computational Linguistics.
Chris Hokamp. 2017. Ensembling factored neural machine translation models for automatic post-editing and quality estimation. In Proceedings of the Second Conference on Machine Translation, pages 647-654, Copenhagen, Denmark. Association for Computational Linguistics.
Nabil Hossain, Marjan Ghazvininejad, and Luke Zettlemoyer. 2020. Simple and effective retrieve-edit-terank text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2532–2538, Online. Association for Computational Linguistics.
Wenxiang Jiao, Xing Wang, Shilin He, Irwin King, Michael Lyu, and Zhaopeng Tu. 2020. Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2255-2266, Online. Association for Computational Linguistics.
Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74-83, Melbourne, Australia. Association for Computational Linguistics.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Philipp Koehn, Vishrav Chaudhary, Ahmed El-Kishky, Naman Goyal, Peng-Jen Chen, and Francisco Guzmán. 2020. Findings of the WMT 2020 shared task on parallel corpus filtering and alignment. In Proceedings of the Fifth Conference on Machine Translation, pages 726-742, Online. Association for Computational Linguistics.
Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. 2019. Findings of the WMT 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 54-72, Florence, Italy. Association for Computational Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018. Findings of the WMT 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 726-739, Belgium, Brussels. Association for Computational Linguistics.