SlowGuess/NAACL_2022_OCR huggingface dataset:
BITEXTEDIT: Automatic Bitext Editing for Improved Low-Resource Machine Translation
Eleftheria Briakou<sup>1*</sup>, Sida I. Wang<sup>2</sup>, Luke Zettlemoyer<sup>2</sup>, Marjan Ghazvininejad<sup>2</sup>
<sup>1</sup> University of Maryland, <sup>2</sup> Facebook AI Research
ebriakou@cs.umd.edu,{sida,lsz,ghazvini}@fb.com
Mined bittexts can contain imperfect translations that yield unreliable training signals for Neural Machine Translation (NMT). While filtering such pairs out is known to improve final model quality, we argue that it is suboptimal in low-resource conditions where even mined data can be limited. In our work, we propose instead, to refine the mined bittexts via automatic editing: given a sentence in a language  $\mathbf{x}_{\mathrm{f}}$ , and a possibly imperfect translation of it  $\mathbf{x}_{\mathrm{e}}$ , our model generates a revised version  $\mathbf{x}_{\mathrm{f}}^{\prime}$  or  $\mathbf{x}_{\mathrm{e}}^{\prime}$  that yields a more equivalent translation pair (i.e.,  $< \mathbf{x}_{\mathrm{f}}, \mathbf{x}_{\mathrm{e}}^{\prime} >$  or  $< \mathbf{x}_{\mathrm{f}}^{\prime}, \mathbf{x}_{\mathrm{e}} >$ ). We use a simple editing strategy by (1) mining potentially imperfect translations for each sentence in a given bitext, (2) learning a model to reconstruct the original translations and translate, in a multi-task fashion. Experiments demonstrate that our approach successfully improves the quality of CCMatrix mined bittext for 5 low-resource language-pairs and 10 translation directions by up to 8 BLEU points, in most cases improving upon a competitive translation-based baseline.
Neural Machine Translation (NMT) for low-resource languages is challenging due to the scarcity of bittexts, i.e., translated text in two languages (Koehn and Knowles, 2017). Models are often trained on heuristically aligned (Resnik, 1999; Banon et al., 2020; Esplà et al., 2019) or automatically mined data (Schwenk et al., 2021a,b), which can be low quality (Briakou and Carpuat, 2020; Kreutzer et al., 2022). This data can include errors that range from small meaning differences in sentences that overlap in content to major differences that yield completely incorrect translations and random noise, e.g., empty sequences, text in the wrong language, non-linguistic content, among others.
Figure 1: Noisy bittexts consist of a mixture of good-quality, imperfect, and poor-quality translations. Filtering decreases the size of training samples which is crucial for low-resource NMT. Our approach, alternatively, revises noisy bittexts via utilizing imperfect translations in a more effective way, while keeps the size of training data untouched.
Filtering out noisy samples from web-crawled bittexts is therefor standard practice for building high quality models (Koehn et al., 2018), and is particularly helpful in low-resource settings (Koehn et al., 2019, 2020). Despite the popularity of this approach, we argue it has two key limitations. First, partially correct translations provide signal that is lost if the entire example is dropped (see first sample bitext in Figure 1). Second, filtering out samples exacerbates the data scarcity problem for the long-tail of low-resource language-pairs.
In this paper, we instead aim to make use of as much of the signal from the mined bitext as possible. We propose an editing approach to bitext quality improvement. Our model takes as input a bitext (i.e.,  $(\mathbf{x_f},\mathbf{x_e}))$ , and edits one of the two sentences to generate a refined version of the original (i.e.,  $\mathbf{x_f}'$  or  $\mathbf{x_e}'$ ) as necessary. By framing the problem as a bitext editing (BITEXTEDIT) task, we can perform a wide range of operations from
*Work done during internship at Facebook AI Research.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1469 - 1485
copying good-quality bitext, to partial editing of small meaning mismatches, and translating from scratch incorrect references. Following previous extrinsic evaluations of bitext quality (Koehn et al., 2019, 2020; Schwenk et al., 2021b,a), we compare NMT models trained on the original and revised versions of CCMatrix bittexts. Concretely, we report consistent improvements in translation quality for 10 low-resource NMT translation tasks: EN  $\leftrightarrow$  OC, IT  $\leftrightarrow$  OC, EN  $\leftrightarrow$  BE, EN  $\leftrightarrow$  MR, and EN  $\leftrightarrow$  SW, while in most cases we even improve upon a competitive translation-based baseline. Crucially, BITEXTEDIT yields from  $4 - 8$  BLEU point improvements in the more data-scarce settings (i.e., EN-OC, IT-OC). Additionally, our quantitative and qualitative analyses indicate that BITEXTEDIT improves bitext quality in higher-resource settings with lighter editing that targets more fine-grained meaning differences.
Bitext Mining The idea of using the web as a source of parallel texts has a long history (Resnik, 1999). Recent advances in multilingual representation learning (Artetxe and Schwenk, 2019; Liu et al., 2020) enable the curation of mined bittexts across multiple languages at scale. For instance, combining LASER (Artetxe and Schwenk, 2019) embeddings with nearest neighbor search allows for effective bittext mining from Wikipedia, i.e., WikiMatrix (Schwenk et al., 2021a) and Common-Crawl monolingual texts, i.e., CCMatrix (Schwenk et al., 2021b). While the latter approach requires parallel text supervision to train the multilingual sentence representation encoder, Tran et al. (2020) shows that it can be extended to an unsupervised framework via iterative self-supervised training.
Issues in Bitext Quality Kreutzer et al. (2022) manually audit the quality of multilingual datasets in 205 language-specific corpora that result from automatic curation pipelines, including bittexts from CCAigned (El-Kishky et al., 2020), WikiMatrix (Schwenk et al., 2021a), and ParaCrawl (Banon et al., 2020; Esplà et al., 2019). All have systematic issues, especially for low-resource languages. The vast majority of low-resource pairs contain less than  $50\%$  valid translations. However, they do often share structural similarity and partial content. Briakou and Carpuat (2020)—in a more fine-grained annotation study—highlight that small content mismatches are even found in high resource pairs:  $40\%$  of English-French WikiMa
trix sentence-pairs have small meaning mismatches. Our work aims at improving bitext quality via eliminating their systematic issues via editing.
Bitext Quality vs. NMT Training Khayrallah and Koehn (2018) demonstrate the often significant impact of various types of noise on NMT, via increasing the percentage of 5 types of artificially injected errors on a clean English-German corpus—mimicking frequent issues in parallel texts (i.e., copying, wrong language, non-linguistic content, short segments, empty sequences). Ott et al. (2018) also argue that data uncertainty resulting from noisy references contributes to the miscalibration of NMT models. Apart from noisy references, small meaning mismatches have also a measurable impact on various aspects of NMT: Briakou and Carpuat (2021) show that models trained on synthetic divergences output degenerated text more frequently and are less confident in their predictions. In contrast with prior studies that discuss how imperfect references interact with NMT training solely for high-resource pairs, we primarily focus on low-resource settings and improve NMT models by improving their training bittexts.
Bitext Quality Improvement The most standardized approach to improving bitext either discards an example or treats it as a perfect training instance (Koehn et al., 2018). Past submissions to the Parallel Corpus Filtering WMT shared task employ a diverse set of approaches covering simple pre-filtering rules based on language identifiers and sentence features (Rossenbach et al., 2018; Lu et al., 2018; Ash et al., 2018), learning to weight scoring functions based on language models, extracting features from neural translation models and lexical translation probabilities (Sánchez-Cartagena et al., 2018), combining pre-trained embeddings (Papavassiliou et al., 2018), and dualcross entropy (Chaudhary et al., 2019). In contrast to prior work, and similar to ours, Briakou and Carpuat (2022) propose to revise imperfect translations in bitext via selectively replace them with synthetic translations generated by NMT of sufficient quality. Our work builds on top of prior work and instead of filtering out all the imperfect bittexts, we selectively edit them and keep them in the pool of training data targeting low-resource NMT.
Figure 2: BITEXTEDIT training strategy: Our multi-task model is trained using synthetic supervision from mined bittexts. Starting from an original bitext  $(x_{e}, x_{f})$ , we mine imperfect translations  $x_{f}^{\prime}$  and  $x_{e}^{\prime}$  for each reference using LASER (Bitext Mining). A sequence-to-sequence Transformer model is trained to translate and reconstruct the original references given synthetically extracted bittexts representing imperfect translations (Bitext Editing).
3 Approach: BITEXTEDIT
We frame bitext refinement as an editing task (i.e, BITEXTEDIT) that takes two input sentences: a sentence  $\mathbf{x}_{\mathrm{f}}$  in language  $\mathbf{f}$  and a sentence  $\mathbf{x}_{\mathrm{e}}$  in language  $\mathbf{e}$ , and aims at editing one of them (i.e., it outputs  $\mathbf{x}_{\mathrm{f}}^{\prime}$  or  $\mathbf{x}_{\mathrm{e}}^{\prime}$ ) with the goal of yielding a more equivalent translation pair (i.e.,  $< \mathbf{x}_{\mathrm{f}}, \mathbf{x}_{\mathrm{e}}^{\prime} >$  or  $< \mathbf{x}_{\mathrm{f}}^{\prime}, \mathbf{x}_{\mathrm{e}} >$ ). Figure 2 gives an overview of our approach while below we describe the bitext refinement model (§3.1) and the curation of data needed to train our model based on bitext mining (§3.2).
3.1 Bitext Editing
Architecture Our bitext editing model is a transformer sequence-to-sequence architecture. Each bitext  $(\mathbf{x_f},\mathbf{x_e})$  is encoded via adding position embeddings that are reset for each input sentence to facilitate their alignment (Conneau and Lample, 2019) and two language embeddings, initialized at random, to indicate the two languages for the editing model. The decoder generates autoregressively a refined version of  $\mathbf{x_f}$  or  $\mathbf{x_e}$ , where the first generated token indicates which of the two input sentences is edited, as described below.
Learning During training, we optimize the multitask loss presented in Equation 1, which has two components. The first represents a edit-based reconstruction loss (i.e.,  $\mathcal{L}_{\mathrm{EDIT}}$ ) that reconstructs one of the two sentences, e.g.,  $\mathbf{x_f}$  started from a noised version of the original bittexts e.g.,  $\mathbf{x_f}'$  and  $\mathbf{x_e}$ . We make this loss bi-directional via adding a symmetrical loss that reconstructs  $\mathbf{x_e}$  from  $\mathbf{x_f}$  and  $\mathbf{x_e}'$ , respectively. The second component, is implemented as a bi-directional translation loss (i.e.,  $\mathcal{L}_{\mathrm{MT}}$ ) via masking the inputs of the target translation directions
(e.g., generate  $\mathbf{x_e}$  given  $\mathbf{x_f}$  and  $<\mathrm{MASK}>$ ). Finally, in both losses a language identification symbol (i.e.,  $<\mathrm{f}>$  or  $<\mathrm{e}>$ ) is used as the initial token to predict the language of the output text.
$$
\begin{array}{l} \mathcal {L} = \sum_ {\left(\mathbf {x _ {f}}, \mathbf {x _ {e}}\right)} \left(\underbrace {\log p \left([ <   e > \mathbf {x _ {e}} ] \mid \left(\mathbf {x _ {f}} , \mathbf {x _ {e} ^ {\prime}}\right)\right)} _ {\mathcal {L} _ {\mathrm {E D I T}}} + \log p \left([ <   f > \mathbf {x _ {f}} ] \mid \left(\mathbf {x _ {f} ^ {\prime}}, \mathbf {x _ {e}}\right)\right) + \right. \\ \underbrace {\log \left(\left[ <   \mathrm {e} > \mathbf {x} _ {\mathbf {e}} \right] \mid \left(\mathbf {x} _ {\mathbf {f}} , <   \text {M A S K} >\right)\right) + \log \left(\left[ <   \mathrm {f} > \mathbf {x} _ {\mathbf {f}} \right] \mid \left(<   \text {M A S K} >, \mathbf {x} _ {\mathbf {e}}\right)\right)} _ {\mathcal {L} _ {\mathrm {M T}}} \tag {1} \\ \end{array}
$$
Inference At test time, our model takes as input a possibly imperfect bitext and edits one of the reference translations, while first generating the language identification token. The latter is used to infer which of the two reference translations gets revised. Finally, we pair the edited output sequence with the original input that does not get revised, yielding a refined bitext.
3.2 Bitext Mining
Our model requires access to  $\mathbf{x}_{\mathrm{f}}^{\prime}$  and  $\mathbf{x}_{\mathrm{e}}^{\prime}$  training instances that are treated as noised versions of  $\mathbf{x}_{\mathrm{f}}$  and  $\mathbf{x}_{\mathrm{e}}$ , respectively. Since our goal is to develop a model that can refine mismatches found in mined bittexts at inference time, we want our noised training instances to share similar properties with the mined ones, e.g., fluent text in the target language, possibly imperfect translations of the source text. To this direction, we take a distance-based mining approach to construct the noised samples similar to Schwenk (2018). Unlike Artetxe and Schwenk (2019) we do not use a margin score on the normalized cosine distance of sentence-pairs to keep the computation cost low and encourage mining of more diverse imperfect translations. Concretely,
given the mined bitext  $(\mathbf{x_f},\mathbf{x_e})$  and two pools of monolingual sentences  $\mathcal{F}$  and  $\mathcal{E}$ , in language  $\mathbf{f}$  and  $\mathbf{e}$ , we extract  $\mathbf{x}_{\mathbf{f}}^{\prime}$  and  $\mathbf{x}_{\mathbf{e}}^{\prime}$  as follows:
$$
\begin{array}{l} \mathbf {x} _ {\mathbf {f}} ^ {\prime} = \operatorname {a r g m a x} _ {\mathbf {z} \in \mathcal {F}} \cos (\text {L A S E R} (\mathbf {z}), \text {L A S E R} (\mathbf {x} _ {\mathbf {e}})) \tag {2} \\ \mathbf {x} _ {\mathbf {e}} ^ {\prime} = \operatorname {a r g m a x} _ {\mathbf {z} \in \mathcal {E}} \cos (\operatorname {L A S E R} (\mathbf {x} _ {\mathbf {f}}), \operatorname {L A S E R} (\mathbf {z})) \\ \end{array}
$$
where LASER (Artetxe and Schwenk, 2019) represents a multilingual encoder used to extract sentence embeddings for each sentence, while the most similar sentence is returned based on nearest neighbor retrieval. Furthermore, this formula is extended to retrieval of top  $k$  sentences, while we also allow mining of the original CCMatrix translations. The latter happens to expose the model to good translations at training time, that should not be edited.
Bittexts We focus on CCMatrix data for two main reasons: a) it constitutes the only large-scale available resource for a lot of low-resource language pairs and b) recent efforts of auditing this corpus raise concerns regarding the quality of mined bitext of low-resource pairs. CCMatrix is mined using LASER embeddings following the max-strategy approach: a margin score is computed for all monolingual sentences in two languages, then the union of forward and backward candidates is built and pairs that score above a pre-defined threshold are treated as translations. Schwenk et al. (2021b) set the threshold globally for all languages at 1.06.
Our primary goal is to explore whether bittexts that are typically discarded by filtering can be refined by our model and thus benefit low-resource NMT. For this purpose, we define two pools of CC-Matrix data: Pool A corresponds to CCMatrix data with LASER scores greater than 1.06, while Pool B contains bittexts with scores lower than 1.06 and greater than 1.05. The latter threshold is primarily chosen since CCMatrix bittexts is only available above this value. Editing bittexts with even smaller scores is an interesting area for future work.
Training data Our models are trained based on procedures described in §3.2, where we use Pool A to seed the generation of noised training samples  $\mathbf{x}_{\mathbf{f}}^{\prime}$  and  $\mathbf{x}_{\mathbf{e}}^{\prime}$ . We mine  $k$  samples  $\mathbf{x}_{\mathbf{f}}^{\prime}$  for each  $\mathbf{x}_{\mathbf{e}}$  and  $k$  samples  $\mathbf{x}_{\mathbf{e}}^{\prime}$  for each  $\mathbf{x}_{\mathbf{f}}$ , respectively. We set  $k$  to 4 and include detailed statistics in Appendix F.
Language-pairs We experiment with the following languages: English-Occitan (EN-OC), Italian-Occitan (IT-OC), English-Belarusian (EN-BE),
PAIR	SCRIPTS	Pool A	Pool B
EN-OC	Latin-Latin	0.2M	0.1M
IT-OC	Latin-Latin	0.3M	0.1M
EN-BE	Latin-Cyrillic	0.7M	1.1M
EN-MR	Latin-Devanagari	1.5M	2.1M
EN-SW	Latin-Latin	1.7M	0.9M
Table 1: Statistics of CCMatrix bittexts.
English-Marathi (EN-MR), and English-Swahili (EN-SW). The 5 language pairs are chosen to include diverse low-resource pairs, which differ either in training data size or language similarity. Table1 summarizes the data conditions.
Comparisons We run several extrinsic evaluations using NMT trained on different versions of CCMatrix data. First, we train NMT models on two versions of original CCMatrix data: Pool A (Schwenk et al., 2021b) and Pool  $A \cup B$ . Second, we aim at revising Pool B via a) a translation-based approach that revisits the source-side of the bittexts via back-translating their target-side with a model trained on original CCMatrix, (i.e.,  $b(.))$  and b) via editing either the source or the target side of it using our proposed approach (i.e.,  $r(.))$ .
Model details Our models are implemented on top of fairseq (Ott et al., 2019).<sup>1</sup> We use the same Transformer architecture as in Schwenk et al. (2021b), with embedding size 512, 4,096 transformer hidden size, 8 attention heads, 6 transformer layers, and dropout 0.4. We train with 0.2 label smoothing and Adam optimizer with a batch size of 4,000 tokens per GPU. We include more model details in Appendices D and G. We train for 100 epochs and select best checkpoint based on validation perplexity. We report single run results.
Data Preprocessing We use the standard Moses scripts (Koehn et al., 2007) for tokenization of EN, OC, IT, BE and SW and the Indic NLP library² for MR. For each language-pair, we learn 60K BPES using subword-nmt (Sennrich et al., 2016b).³
Evaluation We evaluate our models on the devtest of flores (Guzmán et al., 2019). We report spm-bleu<sup>4</sup> on detokenized outputs and chrF (Popović, 2015) as our second evaluation metric.<sup>5</sup>
$^{1}$ https://github.com/pytorch/fairseq $^{2}$ https://anoopkunchukuttan.github.io/indic_nlp_library/
3https://github.com/rsennrich/subword-nmt
4https://github.com/facebookresearch/ flores
Results on chrF are included in Appendix A.
EN→OC	IT→OC	EN→BE	EN→MR	EN→SW
1:	CCMatrix	A∪B	20.5	11.5	11.0	12.2	38.1
2:	Filtering	A	18.1	-2.4	11.7	+0.2	9.8	-0.2	12.2	0.0	37.6
3:	Translation-based	b(A∪B)	20.8	+0.3	17.0	+5.5	12.3	+1.3	15.5	+3.2	37.6
4:	BITEXTEDIT	r(A∪B)	25.4	+4.9	19.8	+8.3	12.8	+1.7	15.8	+3.6	37.8
5:	Translation-based	A∪b(B)	23.0	+2.5	17.0	+5.5	12.1	+1.1	15.4	+3.2	38.8
6:	BITEXTEDIT	A∪r(B)	26.0	+5.5	19.9	+8.4	13.0	+2.0	15.3	+3.1	38.3
OC→EN	OC→IT	BE→EN	MR→EN	SW→EN
7:	CCMatrix	A∪B	24.3	11.6	9.8	13.0	34.8
8:	Filtering	A	17.8	-6.5	11.1	-0.5	7.8	-2.0	11.3	-1.07	34.8
9:	Translation-based	b(A∪B)	26.6	+2.3	17.3	+5.7	9.9	+0.1	13.6	+0.6	33.8
10:	BITEXTEDIT	r(A∪B)	28.2	+3.9	18.5	+6.9	10.7	+0.9	16.4	+3.4	35.8
11:	Translation-based	A∪b(B)	27.7	+3.4	15.6	+4.0	9.6	-0.2	15.1	+2.1	36.8
12:	BITEXTEDIT	A∪r(B)	28.7	+4.4	18.3	+6.7	10.8	+1.0	16.7	+3.7	36.2
Table 2: Results on NMT tasks for models trained on different versions of CCMatrix. For each task the first column denotes spm-BLEU; the second columns (highlighted scores) give the difference of each row with the original CCMatrix. Models trained on the refined bittexts improve NMT for low-resource language-pairs.
Bitext filtering revisited We first provide empirical evidence that bitext filtering might be a suboptimal solution to low-resource NMT. Table 2 shows that filtering out sentence pairs that score below the predefined threshold of 1.06 (i.e., Filtering) surprisingly hurts translation quality in almost all translation tasks (rows 2 vs. 1 and 8 vs. 7). This result is likely because the threshold was optimized for specific language-pairs, and the fact that—under low-resource regimes—increasing the amounts of possibly imperfect translation data might still benefit NMT. Furthermore, this experiment gives us insights on the quality of the training data our bitext editing model uses: for IT-OC, BE-EN, and EN-MR we expect Pool A to provide more noisy training signals (as BLEU scores of NMT models trained on it are  $\sim 11$ ), compared to EN-OC and EN-SW where the quality of the given bitext is expected to be significantly better (BLEU scores  $\sim 18$  and  $\sim 37$ , respectively).
Editing Pool B Applying BITEXTEDIT to edit erroneous translations in Pool B (i.e.,  $A \cup r(B)$ ) improves the quality of NMT systems over the ones trained on the original CCMatrix corpus (rows 6 vs. 1 and 12 vs. 7). Among the language-pairs considered, the largest improvements are reported for IT-OC translation tasks (i.e.,  $+8.4 / +6.7$ ), followed by EN-OC (i.e.,  $+5.5 / +4.4$ ). The magnitude of improvements might be explained by the relatedness of the two languages which facilitates editing with simpler operations (e.g., copying instead of translating).
Our approach also brings significant improvements over the original data for distant languagepairs written in different scripts, despite being trained on more noisy data, as discussed above. For example, we see improvements  $+2.0 / + 1.0$  for EN-BE and  $+3.1 / + 3.7$  for EN-MR. On the other hand, improvements on EN-SW are smaller (i.e.,  $+0.5 / + 1.8$ ). This is expected given the high BLEU scores that the original CCMatrix data yields.
Comparison with Translation-based Baseline Since Pool B bittexts are typically filtered out from the pool of NMT training instances, one reasonable way of incorporating them in NMT training is via treating them as monolingual samples. We experiment with a translation-based model that uses back-translation—the most popular approach to employ data augmentation for NMT. Comparing NMT models trained on CCMatrix augmented with back-translated Pool B against our revised Pool B version (i.e., rows 5 vs. 6 and 11 vs. 12) shows that editing outperforms the translation-based model for  $7/10$  tasks, while it yields comparable results to it for the rest 3.
Editing Pool A and Pool B Since the editing framework gives us the potential to generalize all types of operations that might be needed to refine bittexts, it is also important that it does not perform overediting (i.e., editing already good quality bittexts). For this reason, we also attempt to revise the entire CCMatrix corpus (i.e.,  $r(A \cup B)$ ), using our bitext refinement models (i.e., rows 4 and 9). To better understand the importance of performing conservative editing on good quality bittexts, we
(a)  $|B| = |A| / 2$
(b)  $|B| = |A|$
(c)  $|B| = 2|A|$
also compare against the translation-based baseline (i.e,  $b(A \cup B)$  in rows 3 and 9). First, we observe that our approach yields consistently significant improvements over CCMatrix with the exception of EN→SW where it performs comparably to it. Second, for most tasks the improvements are comparable to those reported when revising only Pool B, while it is consistently better than the translation-based approach. It, overall, provides a universal method that works well in every case.
We now turn into analysis with a focus on understanding the broader space where BITEXTEDIT can be applied. We experiment with scaling-up bitext refinement to higher-resource settings in §6.1, we perform qualitative analysis on the edited bittexts in §6.2, and quantitative analysis on the types and intensity of edits in different corpora in §6.3.
6.1 Scaling-up BITEXTEDIT