SlowGuess/NAACL_2022_OCR huggingface dataset:
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computational Linguistics.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoit Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Muller, Andre Muller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Cabuk Balli, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Association for Computational Linguistics, 10:50-72.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 923-929, Portož, Slovenia. European Language Resources Association (ELRA).
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
António V. Lopes, M. Amin Farajian, Gonçalo M. Correia, Jonay Trénous, and André F. T. Martins. 2019. Unbabel's submission to the WMT2019 APE shared task: BERT-based encoder-decoder for automatic post-editing. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 118-123, Florence, Italy. Association for Computational Linguistics.
Jun Lu, Xiaoyu Lv, Yangbin Shi, and Boxing Chen. 2018. Alibaba submission to the WMT18 parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 917-922, Belgium, Brussels. Association for Computational Linguistics.
Benjamin Marie, Raphael Rubino, and Atsushi Fujita. 2020. Tagged back-translation revisited: Why does it really work? In Proceedings of the 58th Annual
Meeting of the Association for Computational Linguistics, pages 5990-5997, Online. Association for Computational Linguistics.
Xuan-Phi Nguyen, Shafiq Joty, Wu Kui, and Ai Ti Aw. 2020. Data diversification: A simple strategy for neural machine translation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA. Curran Associates Inc.
Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3956-3965. PMLR.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Vassilis Papavassiliou, Sokratis Sofianopoulos, Prokopis Prokopidis, and Stelios Piperidis. 2018. The ILSP/ARC submission to the WMT 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 928-933, Belgium, Brussels. Association for Computational Linguistics.
Maja Popovic. 2015. *chrF: character n-gram F-score* for automatic MT evaluation. In *Proceedings of the Tenth Workshop on Statistical Machine Translation*, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Philip Resnik. 1999. Mining the web for bilingual text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 527-534, College Park, Maryland, USA. Association for Computational Linguistics.
Nick Rossenbach, Jan Rosendahl, Yunsu Kim, Miguel Graça, Aman Gokrani, and Hermann Ney. 2018. The RWTH Aachen University filtering system for the WMT 2018 parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 946-954, Belgium, Brussels. Association for Computational Linguistics.
Víctor M. Sánchez-Cartagena, Marta Bañón, Sergio Ortiz-Rojas, and Gema Ramírez. 2018. Prompsit's submission to WMT 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 955-962, Belgium, Brussels. Association for Computational Linguistics.
Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 228-234, Melbourne, Australia. Association for Computational Linguistics.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021a. WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1351-1361, Online. Association for Computational Linguistics.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021b. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6490-6500, Online. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223-231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
Jörg Tiedemann. 2020. The tatoeba translation challenge - realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, pages 1174-1182, Online. Association for Computational Linguistics.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12), Istanbul, Turkey. European Language Resources Association (ELRA).
Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. 2020. Cross-lingual retrieval for iterative self-supervised
training. In Advances in Neural Information Processing Systems, volume 33, pages 2207-2219. Curran Associates, Inc.
Eva Vanmassenhove, Dimitar Shterionov, and Andy Way. 2019. Lost in translation: Loss and decay of linguistic richness in machine translation. In Proceedings of Machine Translation Summit XVII: Research Track, pages 222-232, Dublin, Ireland. European Association for Machine Translation.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Yogarshi Vyas, Xing Niu, and Marine Carpuat. 2018. Identifying semantic divergences in parallel text without annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1503-1515, New Orleans, Louisiana. Association for Computational Linguistics.
Hao-Ran Wei, Zhirui Zhang, Boxing Chen, and Weihua Luo. 2020. Iterative domain-repaired backtranslation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5884-5893, Online. Association for Computational Linguistics.
Jason Weston, Emily Dinan, and Alexander Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, pages 87-92, Brussels, Belgium. Association for Computational Linguistics.
Hao Yang, Minghan Wang, Daimeng Wei, Hengchao Shang, Jiaxin Guo, Zongyao Li, Lizhi Lei, Ying Qin, Shimin Tao, Shiliang Sun, and Yimeng Chen. 2020. HW-TSC's participation at WMT 2020 automatic post editing shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 797-802, Online. Association for Computational Linguistics.
Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1535-1545, Austin, Texas. Association for Computational Linguistics.
Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1325-1335, New Orleans, Louisiana. Association for Computational Linguistics.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393-1404, Online. Association for Computational Linguistics.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568-1575, Austin, Texas. Association for Computational Linguistics.
A Results on Second Evaluation Metric
Table A presents results on NMT tasks for a second evaluation metric.
EN→OC	IT→OC	EN→BE	EN→MR	EN→SW
1:	41.59	30.92	29.28	31.19	59.17
2:	39.73	32.26	28.24	31.90	58.76
3:	42.34	40.62	30.96	35.41	58.60
4:	47.40	42.83	31.21	35.01	59.02
5:	44.66	39.01	30.66	35.20	59.50
6:	47.74	43.03	31.08	34.65	59.49
OC→EN	OC→IT	BE→EN	MR→EN	SW→EN
7:	48.04	32.90	37.13	37.84	57.10
8:	42.10	33.73	33.51	36.55	57.07
9:	50.99	42.42	37.13	39.99	56.74
10:	52.13	42.42	39.20	42.45	57.96
11:	51.63	38.99	36.99	40.29	58.74
12:	53.86	44.05	39.18	42.71	58.29
Table 6: Results on NMT tasks for the chrF metric (rows follow the enumeration of Table 2).
B Scaling-Up Settings
Tables 7, 8, and 9 present training data sizes for experiments in Figure 3.
A	0.5M	1.0M	5.0M	10.0M	15.0M
AUB	0.75M	1.5M	7.5M	15.0M	22.5M
AUR(B)	0.75M	1.5M	7.5M	15.0M	22.5M
Table 7: Training data size for experiments in Figure 3(a), where  $|B| = |A| / 2$ .
A	0.5M	1.0M	5.0M	10.0M	15.0M
A∪B	1.0M	2.0M	10.0M	20.0M	30.0M
A∪r(B)	1.0M	2.0M	10.0M	20.0M	30.0M
Table 8: Training data size for experiments in Figure 3(b), where  $|B| = |A|$ .
A	0.5M	1.0M	5.0M	10.0M	15.0M
AUB	1.5M	3.0M	15.0M	30.0M	70M
AUr(B)	1.0M	2.0M	10.0M	20.0M	70.0M
Table 9: Training data size for experiments in Figure 3(c), where  $|B| = 2|A|$ .
C Manual Annotation Details
For each bitext (i.e., original CCMatrix sample or refined sample edited by a bitext refinement model) we rate the degree of equivalence between the two
sentences following the protocol of semantic divergences (Briakou and Carpuat, 2020). Concretely, a bitext is annotated as having no meaning difference if it corresponds to perfect translations, some meaning differences if the sentences share important content in common but differ by few tokens (e.g., small added content, or phrasal mistranslation), and unrelated if the sentences are only topically or structurally related. For rating fluency we evaluate the output sentence of the bitext refinement models in isolation on a discrete scale of 1 to 5, following Heilman et al. (2014) (Other  $\rightarrow$  Incomprehensible  $\rightarrow$  Somewhat Comprehensible  $\rightarrow$  Comprehensible  $\rightarrow$  Perfect).
D Fairseq configuration details
Table 10 presents details of NMT training with fairseq. The same parameters are used to train BTEXTEDIT models.
Table 10: Fairseq configuration used for NMT training.
E BITEXTEDIT: Model outputs
Table 11 presents model outputs samples edited by our model for EN-EL CCMatrix instances.
→	[EN] CCMatrix	Respect the dignity of all people, regardless of their age.
→	[EL] CCMatrix	Ποτέω στην αυαγκαιδιτητα Αξοποίησης δλων των ἀχων αὐρώπων Ανεζάρτητα πάτην Μλίανια τους.
→	[GLOSS]	I believe in the importance of using all skilled people, regardless of their age.
→	[EL] BTEXTEDIT	Σεβαμός στην αξοπρέπεια δλων των αὐρώπων, ανεζάρτητα πάτην Μλίανια τους.
→	[EN] CCMatrix	After that time the whole group would talk for 5 minutes.
→	[EL] CCMatrix	Αργότερα, η Μαύδα μελέτης Βήτησε Από δλούςν αἰδλογιστουν για πέντε λεπά.
→	[GLOSS]	Later, the study group asked everyone to meditate for 5 minutes.
→	[EN] BTEXTEDIT	Later, the study group asked everyone to meditate for five minutes.
→	[EN] CCMatrix	Say no to fake products and scams.
→	[EL] CCMatrix	Εἰπατέ οχι στις Φέμιτες έυσχέσεις και στη Συναλλαγή.
→	[GLOSS]	You said no to fake products and transactions.
→	[EL] BTEXTEDIT	Πετέ οχι στις Φέμιτα προίντα και απάτες.
→	[EN] CCMatrix	We're all part of a larger system.
→	[EL] CCMatrix	Τα πάντα εἶνα μέρος εύς μεγαλίθερου Συστήματος.
→	[GLOSS]	Everything is part of a larger System.
→	[EN] BTEXTEDIT	Everything is part of a larger system.
→	[EN] CCMatrix	Currently, no equivalent technology exists on the market.
→	[EN] CCMatrix	Δεν Φάρχει αντίσοιχην αντάγωνιστίχη τέχονούγία στου όόσμο αὐτήτη στιγμή.
→	[GLOSS]	There is no corresponding competing technology in the world right now.
→	[EN] BTEXTEDIT	There is no competitive technology in the world right now.
→	[EN] CCMatrix	"The western influence came from film and television", he later explained.
→	[EN] CCMatrix	«H λογοχρία ευτένεται δόλ κα περισόθερο στου όνηματοράρον και τήν τηλέραση», εύγησος o δίος.
→	[GLOSS]	"Censorship is intensifying in cinema and television", he explained.
→	[EL] BTEXTEDIT	«H δυιχή ευρρόη φρεθε αύτην ταυλίακα τήν τηλέραση», εύγησος αργότερα.
→	[EN] CCMatrix	Then he paused, surveying the surreal scene.
→	[EN] CCMatrix	Κα πράγματι ἔφυγε, προσερνώντας τον ἐπληκτο Kόλι.
→	[GLOSS]	And indeed he left, passing Keli, who was surprised.
→	[EL] BTEXTEDIT	Στη συνέχεια σταμάτησε, εἰνθερώντας τήν σουρελιστίχη σχήνή.
→	[EN] CCMatrix	Device installation error is a frequent error.
→	[EL] CCMatrix	H αλατάληλη φράμα βιογραρχού, εἶνα ενα πολύ συχό λάθος.
→	[GLOSS]	An improper resume form, is a very frequent mistake.
→	[EN] BTEXTEDIT	The inappropriate biographical form is a very frequent mistake.
→	[EN] CCMatrix	I could work with a hospital specialist as a clinical assistant (as I have done).
→	[EL] CCMatrix	δόμλενε δς βόγδς ερευμήτα παράληλα με το δίδακτορικό (όπως και εγύ)
→	[GLOSS]	They were working as an assistant researcher in parallel with their doctorate (as I have done).
→	[EL] BTEXTEDIT	Θα μιρούσανανούλεμω με εναν εἰδιχό στονούσομείος ως κλινίας βόγός (όπως εχώ καύνι).
→	[EN] CCMatrix	We should, however, always be striving to live a sustainable and kind life.
→	[EL] CCMatrix	Πάντα πρέπεινα παλέουμε για δίακαη και βιώσημι εμρήνη.
→	[GLOSS]	We must always fight for a just and lasting peace.
→	[EN] BTEXTEDIT	We must always fight for just and sustainable peace.
Table 11: Examples of CCMatrix bittexts along with refined sides generated by BTEXTEDIT.  $\rightarrow$  denotes the side ([EL] or [EN]) that the model edits, while highlighted segments indicate the meaning mismatches in the original CCMatrix sentence that gets edited. Greek sentences are glossed to help understanding their meaning.
Corpus	Version	License	Citation	Link
CCmatrix	v2	-	Schwenk et al. (2021b)	https://data.statmt.org/cc-matrix/
FLORES	v1	CC-BY-SA	Guzmán et al. (2019)	https://github.com/facebookresearch/flores
OpenSubtitles	v2018	-	Lison and Tiedemann (2016)	https://opus.nlp1.eu/OpenSubtitles-v2018.php
Tatoeba	v2	CC-BY 2.0 FR	Tiedemann (2012)	https://opus.nlp1.eu/Tatoeba.php
ParaCrawl	v7.1	Creative Commons CC0	Esplà et al. (2019)	https://opus.nlp1.eu/ParaCrawl.php
F Details on Scientific Artifacts
Statistics on Training Examples Tables 13 and 14 include detailed statistics on training and dev samples used to train each of the NMT and BITEXTEDIT models discussed in the paper.
Table 12: Additional documentation of scientific artifacts used in our paper.
Training	Dev	Test
Pair	|A|	|A ∪ B|
EN-OC	242,982	365,399	997	1,012
IT-OC	309,703	440,283	997	1,012
EN-BE	659,430	3,944,412	997	1,012
EN-MR	1,503,477	3,611,336	997	1,012
EN-SW	1,721,801	2,641,234	997	1,012
Table 13: Number of training/dev/test examples used to train NMT models in Table 2.
Pair (src-tgt)	All	Mined (src)	Mined (tgt)
Training samples
EN-OC	3,822,800	965,184	946,216
IT-OC	4,743,350	1,228,328	1,143,347
EN-BE	10,152,596	2,637,575	2,544,460
EN-MR	17,764,241	5,640,928	5,991,336
EN-SW	16,232,991	6,734,355	6,859,214
Dev samples
EN-OC	15,908	3,988	3,966
IT-OC	15,952	3,988	3,988
EN-BE	15,952	3,988	3,988
EN-MR	15,952	3,988	3,988
EN-SW	15,952	3,988	3,988
Table 14: Number of training/dev examples used to train BTEXTEDIT models in Table 2. The two last columns (i.e., mined) include further statistics on the number of mined bittexts consumed by the edit-based reconstruction loss; the rest of the training samples correspond to machine-translation samples upweighted to match the number of mined bittexts (i.e., equal contribution of two losses).
License details We use data derived from OPUS (https://opus.nlpl.eu/) corpora as summarized in Table 12. All data are solely used for research purposes.
Original	Edited
# Tokens
SRC	TGT	SRC	TGT
EN-OC	3,591,876	3,995,351	3,601,179	3,978,861
OC-IT	5,717,341	5,496,704	5,763,860	5,428,767
BE-EN	97,172,691	43,007,326	16,806,977	19,002,607
EN-MR	36,468,349	32,934,460	36,411,035	3,2830,479
EN-SW	4,1855,796	40,666,513	41,978,701	40,472,724
# Types
SRC	TGT	SRC	TGT
EN-OC	165,310	234,252	169,191	235,503
OC-IT	277,397	278,727	292,357	283,656
BE-EN	531,309	526,289	533,224	381,666
EN-MR	407,977	956,589	379,015	922,184
EN-SW	414,873	802,292	409,0224	791,853
Type-Token ratio
SRC	TGT	SRC	TGT
EN-OC	4.6%	5.9%	4.7%	5.9%
OC-IT	5.9%	5.1%	5.1%	5.2%
BE-EN	0.5%	1.2%	3.2%	2.0%
EN-MR	1.1%	2.9%	1.0%	2.8%
EN-SW	2.0%	2.0%	1.0%	2.0%
Table 15: Lexical characteristics of Original vs. Edited version of CCMatrix bittexts.
G Compute Infrastructure & Run time
Each experiment runs on a single machine with 8 GPUs. NMT models require less than 3.5 hours (e.g., EN-OC on  $A \cup B$  requires  $\sim 20$  minutes to train). Similarly, BITEXTEDIT models require less than 13.5 hours to train (e.g., EN-OC requires  $\sim 5$  hours). All models follow the transformer architecture detailed in Appendix D with a total of 165M parameters.
H Potential Risks
Hallucination detection Our approach introduces synthetic samples (i.e., edited references that replace the originally human generated samples) that are later consumed as training instances by NMT models. One concern of using synthetic instances highlighted by recent work (Zhou et al., 2021), is the generation of hallucinations (i.e., fluent text that is not tight to the source segment). To understand whether our method potentially contributes to the issue of hallucinations, one of the authors examined a small sample of 20 outputs
generated by three NMT models for EN→EL translation: 1. a model trained only on 1M of PoolA data; 2. a model trained on the concatenation of 1M PoolA and 2M PoolB data; 3. a model trained on the concatenation of 1M PooA and 2M edited PoolB data. The NMT outputs are annotated labeled as: incomprehensible, faithful, or contains hallucinations following the protocol of Zhou et al. (2021). All annotated instances are found to be faithful to the source.
Lexical Richness Synthetically generated data (e.g., machine-translated instances) are known to exhibit a decay in lexical richness when compared to human written texts (Vanmassenhove et al., 2019). To confirm that our approach does not potentially contribute to this issue, we report more detailed statistics on how the original and edited CCMatrix texts differ in terms of lexical features (i.e., #tokens, #types, and type-token ratio). As presented in Table 15 the edited text does exhibit a decrease in the type-token ratio percentage when compared to the original one.