xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages
Year: 2023

Abstract:
We introduce a new proxy score for evaluating bitext mining based on similarity in a multilingual embedding space: xsim++. In comparison to xsim, this improved proxy leverages rulebased approaches to extend English sentences in any evaluation set with synthetic, hard-todistinguish examples which more closely mirror the scenarios we encounter during largescale mining. We validate this proxy by running a significant number of bitext mining experiments for a set of low-resource languages, and subsequently train NMT systems on the mined data. In comparison to xsim, we show that xsim++ is better correlated with the downstream BLEU scores of translation systems trained on mined bitexts, providing a reliable proxy of bitext mining performance without needing to run expensive bitext mining pipelines. xsim++ also reports performance for different error types, offering more fine-grained feedback for model development.

heading: 1 Introduction
text: When training neural machine translation (NMT) systems, it has been shown in prior works that generally, the quality of such systems increases with the availability of high-quality training data (Koehn and Knowles, 2017). However, for many low-resource languages there are few public corpora available, posing many challenges. In order to address this sparsity, one approach is to supplement existing datasets with automatically created parallel corpora, and a technique which has shown to be successful for such issues is the task of bitext mining (Schwenk et al., 2021b).
In bitext mining, the aim is to find pairs of sentences with the same sentence meaning across collections of monolingual corpora. In this work, we adopt a global mining approach (Schwenk et al., 2021a), which has shown recent success in provid-
∗Equal contribution
ing high-quality data for low-resourced languages (NLLB Team et al., 2022).
In order to evaluate any bitext mining method, a natural approach is to train a NMT system on the automatically created alignments. However, this is extremely costly. As an alternative, the BUCC task (Zweigenbaum et al., 2018) offers a method for evaluating bitext mining algorithms by embedding known alignments within monolingual corpora, and then reporting on the number of correctly aligned pairs. However, this task currently only covers 5 high-resourced languages (English, French, Russian, German and Chinese), and so is not applicable to the low-resource domain. In order to address this, another approach to evaluate bitext mining is to align existing multilingual parallel test sets. Two such test sets are Tatoeba1 and FLORES200.2 However, as shown by Heffernan et al. (2022), the Tatoeba corpus is not very reliable given that for some sentence pairs there are only a few hundred sentences. Therefore, we opt to use FLORES200, which is also n-way parallel.
One existing method for evaluating bitext mining on parallel test sets is xsim.3 This method reports the error rate of misaligned sentences, and follows a margin-based global mining approach (Artetxe and Schwenk, 2019a). However, although using xsim on test sets such as FLORES200 has been shown to be useful as a proxy metric for bitext mining (NLLB Team et al., 2022), it has the following limitations:
1. Using FLORES200 alone has proven to not be difficult enough as for many language pairs, existing approaches quickly saturate at 0% error (NLLB Team et al., 2022).
1https://github.com/facebookresearch/LASER/ tree/main/data/tatoeba/v1
2https://github.com/facebookresearch/flores/ tree/main/flores200
3https://github.com/facebookresearch/LASER/ tree/main/tasks/xsim
101
2. As the dev and devtest sets are quite small (997/1012 respectively), this is arguably not a good approximation for performance when mining against billions of possible candidate sentences.
3. We have observed that there is not a significant overlap in the semantics between candidate sentences, meaning that it is not possible to test difficult scenarios that arise in bitext mining when choosing between multiple (similar) candidate pairs.
In order to address these limitations, in this work we introduce xsim++. This is an improved proxy for bitext mining performance which expands the dev and devtest sets of FLORES200 to include both more data points, and also difficult to distinguish cases which provide far greater challenges to the models. Our contributions can be summarised as follows:
1. We create a more semantically challenging and expanded English test set for FLORES200.
2. We validate this new test set by independently performing 110 bitext mining runs, training 110 NMT systems on the output mined bitexts, and then determining both the correlation and statistical significance between xsim++ and the resulting BLEU scores.
3. We open-source the expanded FLORES200 dev and devtest sets, and also the xsim++ code to evaluate them4.
2 Methodology
2.1 Background: xsim Given two lists of sentences in different languages, xsim seeks to align each sentence in the source
4https://github.com/facebookresearch/LASER
language to a corresponding sentence in the target language based on a margin-based5 similarity (Artetxe and Schwenk, 2019a). In doing so, xsim leverages the mining approach described in Artetxe and Schwenk (2019b) to first encode sentences into embedding vectors, assign pairwise scores between sentences in the lists, and then take the sentence in the target language that achieves the maximum score as the final prediction. xsim relies on humanannotated parallel corpora and measures the performance of bitext mining using the fraction of misaligned source sentences, i.e., error rates.
2.2 xsim++
As the effectiveness of xsim is limited by the availability of parallel corpora, we choose to create xsim++ by automatically expanding the English sentences, and evaluate the sentence encoders on into-English language directions, following prior work on low-resource bitext mining (Heffernan et al., 2022). Aside from the expanded candidate set, xsim++ follows the same procedure as xsim. xsim++ seeks to capture more subtle improvements in bitext mining by adding challenging negative examples. The examples are humanwritten sentences transformed by various operations. These operations intend to perturb semantics through minimal alternations in the surface text. In particular, we use the following categories of transformations: causality alternation, entity replacement, and number replacement. We focus on these three transformation types only as they easily allow us to create negative examples. Examples of the transformed sentences are shown in Table 1. For these transformations, we adapt the implementation in Dhole et al. (2021)6 and describe the details
5In this work we report all results using the absolute margin
6Although this library has additional transformation methods available, many would create positive examples in this use case (e.g. paraphrases).
of these transformations below.
Causality Alternation. To alter causality in a sentence, we (1) replace adjectives with their antonyms; (2) negate the meaning of sentences by adding or removing negation function words (e.g. “did not” and “was not”) to the sentences; or (3) leverage the negation strengthening approach (Tan et al., 2021), which changes the causal relationships through more assertive function words (e.g. replacing “may” with “will”). For example, as shown in Table 1 we replace “good” with the antonym “bad”.
Entity Replacement. We collect candidate entities from large amounts of monolingual data. Then we replace entities in sentences with the ones randomly sampled from the candidate set. For both stages, we use the named entity recognizer from NLTK (Bird et al., 2009).
Number Replacement. We use spaCy (Honnibal and Montani, 2017) to detect dates, ordinals, cardinals, times, numbers, and percentages and then randomly replace their values.
Given the strategies above, for each sentence we create multiple transformations (i.e. negative examples) of that source sentence. For example, consider Table 1. In the “Entity Replacement” example we create a transformation by replacing two named entities. We can then continue this process by replacing these with other named entities until we have reached the desired number of total transformations7. Note that since the opportunity to change each category is dependent on the frequency of that category in the evaluation sets, some transformations occurred more than others (e.g. entities were more frequent than numbers). We summarize the data statistics for xsim++ on the FLORES200 dev
7We set a maximum threshold of 100 transformations per category per sentence.
set in Table 2. Results for the devtest set are in appendix A.

heading: 3 Experiment
text: In order to establish xsim++ as a proxy for bitext mining performance, we measure the correlation between both xsim and xsim++ error rates, and the BLEU scores resulting from NMT systems trained on mined bitexts. More specifically, for each language we choose a sentence encoder model, followed by bitext mining using each respective encoder, and then train and evaluate bilingual NMT systems on the resulting mined bitexts. We use the FLORES200 development sets when computing the BLEU scores.
In order to validate xsim++ against varied embedding spaces, we encode (and mine) using two different multilingual encoder methods: LASER (Artetxe and Schwenk, 2019b) and LaBSE (Feng et al., 2022). For LASER, we trained our own custom encoders (details below). For LaBSE, we used a publicly available model8 as the code and data for training LaBSE are not publicly available.
We randomly choose 10 low-resource languages to perform both encoder training (if applicable) and bitext mining. The languages are: Faroese (fao), Kabuverdianu (kea), Tok Pisin (tpi), Kikuyu (kik), Friulian (fur), Igbo (ibo), Luxembourgish (ltz), Swahili (swh), Zulu (zul), Bemba (bem).
Encoder Training. We trained LASER encoders using the teacher-student approach described in Heffernan et al. (2022). We choose a LASER model (Artetxe and Schwenk, 2019b) as our teacher, and then trained specialised students for each language. In order to train each student, we used both publicly available code9 and bitexts (e.g. OPUS10)
Bitext Mining. For each chosen encoder model, we perform bitext mining against approximately 3.7 billion sentences of English. For low-resource languages, the sizes of monolingual data range from 140k to 124 million. Details are in the appendix. We make use of monolingual data available from both Commoncrawl and Paracrawl11, and operationalize the mining using the stopes library (An-
8https://huggingface.co/sentence-transformers/LaBSE 9https://github.com/facebookresearch/fairseq/
tree/nllb/examples/nllb/laser_distillation 10https://opus.nlpl.eu 11https://paracrawl.eu
drews et al., 2022).12 For LASER, we use 1.06 as the margin threshold following Heffernan et al. (2022) and for LaBSE, we use 1.16.13 Following mining, for each language we concatenate publicly available bitexts and the mined bitext as training data for NMT bilingual models using fairseq,14 translating from each foreign text into English. For all NMT systems, we keep the hyperparameters fixed (details in Appendix).
Evaluation. Model selection involves two use cases: comparisons within a model and across different models. For the former comparison, given our custom encoders, we choose to compare 10 checkpoints from each model.15 For cross model comparisons, we compare each chosen encoder checkpoint against another existing system. In this case, the LaBSE encoder. To quantitatively measure these two cases, we report pairwise ranking accuracy (Kocmi et al., 2021) for xsim and xsim++. Formally, the accuracy is computed as follows
|s(proxy∆) = s(mining∆) for all system pairs| |all system pairs|
where proxy∆ is the difference of the xsim or xsim++ scores, mining∆ is the difference of the BLEU scores, s(·) is the sign function, and | · | returns the cardinal number of the input.
In this work, we have 550 system pairs with 55 pairs per language direction (i.e. ( 11 2 ) pairs given 10 custom LASER encoder checkpoints + LaBSE). We always compare systems within a language direction as the scores for system pairs across different directions are not comparable.16

heading: 3.1 Results
text: As shown in Table 3, xsim++ significantly outperforms xsim on the pairwise ranking accuracy. Additionally, when comparing the computational cost to mining, xsim++ costs over 99.9% less GPU hours and saves approximately 3 metric tons of carbon
12https://github.com/facebookresearch/stopes 13We did grid search on threshold values from 1.11 to 1.25 on three languages (swh, ltz, and zul), decided the optimal one based on the BLEU scores, and used the threshold for the rest of languages.
14https://github.com/facebookresearch/fairseq 15Evenly spaced between epochs 1 and 30. 16There are factors varied across language directions that are unrelated to the quality of sentence encoders but could affect mining performance, such as amounts of monolingual data available for mining.
emissions, but still manages to achieve a competitive accuracy. We observe similar trends for the within a model and across models use cases and report their separate accuracies in the appendix.
To better understand the contributions of each transformation category (cf. subsection 2.1) in measuring the final mining performance, we report accuracies for different combinations of categories in Table 4. In cases where an incorrect bitext alignment do does not map to any of the augmented sentences of the true alignment, we denote these as “misaligned”. We find that entity replacement helps most in improving the accuracy and combing all the transformations gives the best performance.

heading: 4 Related Work
text: As xsim++ uses rule-based data augmentation, it is related to work in other areas that also employ similar data augmentation methods, such as part-ofspeech tagging (Şahin and Steedman, 2018), contrastive learning (Tang et al., 2022), text classification (Kobayashi, 2018; Wei and Zou, 2019), dialogue generation (Niu and Bansal, 2018) and summarization (Chen and Yang, 2021).

heading: 5 Conclusion and Future Work
text: We proposed a proxy score xsim++ for bitext mining performance using three kinds of data augmentation techniques: causality alternation, entity replacement, and number replacement. To validate its effectiveness, we conducted large-scale bitext mining experiments for 10 low-resource languages, and reported pairwise ranking accuracies. We found that xsim++ significantly improves over xsim, doubling the accuracies. Analysis reveals that entity replacement helps most in the improvement. In future work, we plan to extend xsim++ to non-English language pairs.

heading: 6 Limitations
text: We highlight three limitations of our work. The first is that xsim++ is automatically constructed. There could be noisy sentences leading to errors that are irrelevant to the quality of encoders. The second is that xsim++ applies transformations solely to English sentences. Generalizing it to non-English language pairs requires additional research. Finally, we have experimented with the two most popular multilingual encoders: LASER and LaBSE. There are other available approaches which would be interesting to also validate xsim++ against.

heading: B Sizes of Monolingual data for Low-Resource Languages
text: We report the sizes of monolingual data for each language in Table 6.
C Hyperparameters for NMT systems
We report hyperparameters for NMT evaluations in Table 7.

heading: D Within and Across Model Accuracies
text: We report accuracies for within a model (i.e., LASER) and across different models (i.e., the 10 LASER checkpoints vs LaBSE) in Table 8.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2
3 B1. Did you cite the creators of artifacts you used? Section 2
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2
C 3 Did you run computational experiments? Section 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3 and Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3 and Appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we plan to extend xsim++ to non-English language pairs