"A Little is Enough": Few-Shot Quality Estimation based Corpus Filtering improves Machine Translation
Year: 2023

Abstract:
Quality Estimation (QE) is the task of evaluating the quality of a translation when reference translation is not available. The goal of QE aligns with the task of corpus filtering, where we assign the quality score to the sentence pairs present in the pseudo-parallel corpus. We propose a Quality Estimation based Filtering approach to extract high-quality parallel data from the pseudo-parallel corpus. To the best of our knowledge, this is a novel adaptation of the QE framework to extract quality parallel corpus from the pseudo-parallel corpus. By training with this filtered corpus, we observe an improvement in the Machine Translation (MT) system’s performance by up to 1.8 BLEU points, for English-Marathi, Chinese-English, and Hindi-Bengali language pairs, over the baseline model. The baseline model is the one that is trained on the whole pseudo-parallel corpus. Our Few-shot QE model transfer learned from the English-Marathi QE model and finetuned on only 500 Hindi-Bengali training instances, shows an improvement of up to 0.6 BLEU points for Hindi-Bengali language pair, compared to the baseline model. This demonstrates the promise of transfer learning in the setting under discussion. QE systems typically require in the order of (7K-25K) of training data. Our Hindi-Bengali QE is trained on only 500 instances of training that is 1/40 of the normal requirement and achieves comparable performance. All the scripts and datasets utilized in this study will be publicly available.

heading: 1 Introduction
text: In recent times, Neural MT has shown excellent performance, having been trained on a large amount of parallel corpora (Dabre et al., 2020). However, not all language pairs have a substantial amount of parallel data. Hence, we have to rely on the noisy web-crawled corpora for low-resource languages. The task of Parallel Corpus Filtering aims to provide a scoring mechanism that helps extract goodquality parallel corpus from a noisy pseudo-parallel
corpus. The task of Quality Estimation (QE) aims to provide a quality score for a translation when the reference translation is unavailable. We use Quality Estimation to assign the quality scores to the sentence pairs present in pseudo-parallel corpora and extract good-quality parallel sentences. We aim to improve the quality of Machine Translation for English(En)-Marathi(Mr), Hindi(Hi)-Bengali(Bn) and Chinese(Zh)-English(En) language pairs by using sentence-level QE-based corpus filtering. We observe that QE-based corpus filtering performs better than previously proposed methods.
Our contributions are:
1. Adaptation of the QE framework, which is normally used for MT evaluation, to extract highquality parallel corpus from pseudo-parallel corpus; to the best of our knowledge, this is a novel adaptation of the QE framework to extracting quality parallel corpus from the pseudo-parallel corpus.
2. Demonstrating the promise of Few-Shot QE technique to generate training data for MT; a Hindi-Bengali QE model is trained with only 500 training instances transfer learned from an English-Marathi trained QE model; the filtered parallel data using this Hindi-Bengali QE system gives 0.6 BLEU point improvement over Hi-Bn MT system trained on the pseudo-parallel corpus.
3. Demonstrating performance improvement of the Machine Translation systems by up to 1.8 BLEU points for English-Marathi, HindiBengali and Chinese-English language pairs, over the model trained on the whole pseudoparallel corpus.
14175

heading: 2 Related work

heading: 2.1 Parallel Corpus Filtering
text: Neural Machine Translation (NMT) is extremely data hungry (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017). Recently, there has been a growing interest in the process of filtering noisy parallel corpora to enhance the data used for training machine translation systems. The Conference on Machine Translation (WMT) has organized annual Shared Tasks on Parallel Corpus Filtering (WMT 2018, WMT 2019, WMT 2020). Lu et al. (2020) proposed an approach that uses the Dual Bilingual GPT-2 model and the Dual Conditional CrossEntropy Model to evaluate the quality of the parallel corpus. Feng et al. (2020) proposed the LaBSE model, which is a multilingual sentence embedding model trained on 109 languages, including some Indic languages. Herold et al. (2022) mentioned different types of noise that can be injected in a parallel corpus and investigated whether state-of-the-art filtering models are capable of removing all the noise types proposed by Khayrallah and Koehn (2018).
Most recently, Batheja and Bhattacharyya (2022) used a combination of Phrase Pair Injection and LaBSE (Feng et al., 2020) based Corpus Filtering to extract high-quality parallel data from a noisy parallel corpus. In contrast, we use QE-based filtering to extract high-quality data from noisy pseudoparallel data. We observe that QE quality scores are superior to the LaBSE quality scores.

heading: 2.2 Quality Estimation
text: Quality Estimation (QE) is the task of evaluating the quality of a translation when reference translation is not available. The state-of-the-art MonoTransQuest architecture, proposed by Ranasinghe et al. (2020), builds upon XLM-R, a widely-used pretrained cross-lingual language model known for its ability to generalize to low-resource languages (Conneau et al., 2020). (Kocyigit et al., 2022) proposed a combination of multitask training, data augmentation and contrastive learning to achieve better and more robust QE in a Parallel Corpus Mining setting. The Parallel Corpus Mining task aims to detect the most similar texts in a large multilingual collection and perform sentence alignment. This motivates us to use QE in the Parallel Corpus Filtering task.

heading: 3 Approaches
text: We first discuss methods to extract good-quality parallel sentences from the pseudo-parallel corpus. Then we discuss a transfer learning-based filtering approach in few-shot settings.

heading: 3.1 LaBSE based Filtering
text: Language Agnostic BERT Sentence Embedding model (Feng et al., 2020) is a multilingual embedding model that supports 109 languages, including some Indic languages. We generate the sentence embeddings for the source and target sides of the pseudo-parallel corpora using the LaBSE 1 model. Then, we compute the cosine similarity between the source and target sentence embeddings. After that, we extract good-quality parallel sentences based on a threshold value of the similarity scores.

heading: 3.2 Phrase Pair Injection (PPI) with LaBSE-based Filtering
text: Batheja and Bhattacharyya (2022) proposed a combination of Phrase Pair Injection (Sen et al., 2021) and LaBSE-based Corpus Filtering to extract highquality parallel data from a noisy parallel corpus. We train a PBSMT model on the noisy pseudoparallel corpus using the Moses 2 decoder. Then, we extract phrase pairs with the highest translation probability. Finally, we perform LaBSE-based filtering on these phrase pairs to remove poor-quality phrase pairs. We augment these high-quality phrase pairs with LaBSE-filtered parallel sentences.

heading: 3.3 Quality Estimation based Filtering
text: In this approach, we train the MonoTransQuest3 (Ranasinghe et al., 2020) model and use
1https://huggingface.co/sentence-transformers/ LaBSE
2http://www2.statmt.org/moses/?n=Development. GetStarted
3https://github.com/TharinduDR/TransQuest
it to generate the quality scores for the pseudoparallel corpus of the corresponding language pair. Then, we extract high-quality parallel sentences from the pseudo-parallel corpus using a threshold quality score value.

heading: 3.4 Few-shot Quality Estimation
text: The Quality Estimation task requires humanannotated Direct Assessment scores for the corresponding language pairs. In few-shot settings, we fine-tune a pre-trained QE model for a highresource language pair on QE data for the corresponding low-resource language pair to obtain a QE model for the low-resource language pair.

heading: 4 Mathematical Preliminaries
text: LaBSE scoring Let D = {(xi, yi)}Ni=1 be a pseudo-parallel corpus with N examples, where xi and yi represents ith source and target sentence respectively. We first feed all the source sentences present in the pseudo parallel corpus as input to the LaBSE4 (Feng et al., 2020) model, which is a Dual encoder model with BERT-based encoding modules to obtain source sentence embeddings (Si). The sentence embeddings are extracted as the l2 normalized [CLS] token representations from the last transformer block. Then, we feed all the target sentences as input to the LaBSE model to obtain target sentence embeddings (Ti). We then compute cosine similarity (scorei) between the source and the corresponding target sentence embeddings.
Si = LaBSE (xi) (1)
Ti = LaBSE (yi) (2)
scorei = cosine_similarity (Si, Ti) (3)
QE scoring We feed “xi[SEP ]yi" as an input to the MonoTransQuest (Ranasinghe et al., 2020) architecture which uses a single XLM-R model. The output of the [CLS] token is used as the input of a softmax layer that predicts the quality score (scorei) of the ith sentence pair < xi, yi >.
scorei = MonoTransQuest (xi, yi) (4)

heading: 5 Experimental Setup

heading: 5.1 Dataset
text: In all NMT experiments, we use two sets of corpus, namely, Parallel and Pseudo-Parallel corpus. The
4https://huggingface.co/sentence-transformers/ LaBSE
Parallel corpus consists of high-quality sentence pairs, while the Pseudo-Parallel corpus contains sentence pairs of varying quality.
The En-Mr Parallel Corpus consists of the ILCI phase 1, Bible, PIB and PM-India corpus (Jha, 2010; Christos Christodouloupoulos, 2015; Haddow and Kirefu, 2020). The Zh-En Parallel corpus consists of ParaMed5 corpus. The Hi-Bn Parallel corpus is obtained from the OPUS6 corpus repository. The En-Mr and Zh-En Pseudo-Parallel Corpus consist of the Samanantar (Ramesh et al., 2021) and WMT18 Zh-En7 corpus, respectively. The HiBn Pseudo-Parallel Corpus consists of Samanantar and Tatoeba (Tiedemann, 2020) corpus. The detailed data statistics are mentioned in table 1.
In QE experiments, we create a small corpus (500 instances) for Hindi-Bengali language pair that consists of human-annotated Domain Adaptation scores for each sentence pair annotated by three annotators. The pairwise Pearson Correlation between the three annotators of Hindi-Bengali QE is 0.68, 0.61 and 0.67. This indicates a good agreement between the three annotators. Please refer to Appendix A.3 for further annotation details. We use the QE data provided by Ranasinghe et al. (2020) and Deoghare and Bhattacharyya (2022) for the Zh-En and En-Mr language pairs, respectively. The detailed QE data statistics are mentioned in
5https://github.com/boxiangliu/ParaMed 6https://opus.nlpl.eu/ 7http://data.statmt.org/wmt18/
translation-task/preprocessed/zh-en/
table 2. For evaluation, we use the FLORES 101 test set which contains 1,012 sentence pairs for each language pair.

heading: 5.2 Models
text: We use MonoTransQuest model architecture to train the QE models. We use the Indic NLP library for preprocessing the Indic language data and Moses for preprocessing the English language data. For Indic languages, we normalize and tokenize the data. For English, we lowercase and tokenize the data. We use a Transformer based architecture provided by OpenNMT-py library to train the NMT models for all our experiments. The optimizer used was adam with betas (0.9, 0.98). The initial learning rate used was 5e-4 with the inverse square root learning rate scheduler. We use 8000 warmup updates. The dropout probability value used was 0.1 and the criterion used was label smoothed cross entropy with label smoothing of 0.1. We use a batch size of 4096 tokens. All the models were trained for 200,000 training steps. We use MonoTransquest8 model to train the sentence-level QE model. We start with a learning rate of 2e-5 and use 5% of training data for warm-up. We use early patience over ten steps. We use a batch size of eight. The model architecture is mentioned in Appendix A.2. Baseline We train the baseline NMT models on
8https://github.com/TharinduDR/TransQuest
the whole pseudo-parallel corpus augmented with the parallel corpus for the corresponding language pairs. LaBSE based Filtering In this model, we use the LaBSE filtering with threshold 0.8 to extract good quality parallel sentences from the En-Mr, Hi-Bn and Zh-En pseudo-parallel corpus. Then we augment the parallel corpus with the LaBSEfiltered parallel sentences and train the respective NMT models. LaBSE + PPI-LaBSE based Filtering We extract LaBSE Filtered parallel sentences and phrases from the pseudo-parallel corpus and augment them with the parallel corpora to train the respective NMT models. Our Model, QE based Filtering We train the sentence-level QE model from scratch for En-Mr and Zh-En language pairs using their respective training data, Table 2. We use the English-Marathi pre-trained QE model for the Hi-Bn language pair and finetune it on Hi-Bn training data, Table 2. We compute quality scores for the noisy pseudoparallel corpora using the trained QE models. Then, we extract high-quality sentence pairs from the pseudo-parallel corpus using the threshold values of -0.5, -0.4, and 0 for En-Mr, Zh-En, and Hi-Bn language pairs, respectively. We augment the extracted high-quality sentence pairs with the parallel corpus and train the respective NMT models.

heading: 6 Results and Analysis
text: We evaluate our NMT models using BLEU (Papineni et al., 2002). We use sacrebleu (Post, 2018) python library to calculate the BLEU scores. Table 5 shows that QE based filtering model outperforms all other models for Hi-Bn, En-Mr and Zh-En language pairs. The QE based Filtering model improves the MT system’s performance by 0.85, 0.6, 1.8, 0.37 and 0.63 BLEU points over the baseline model for Zh→En, En→Mr, Mr→En,Hi→Bn and Bn→Hi, respectively. It also outperforms LaBSE + PPI-LaBSE based Filtering model by up to 0.7 BLEU points for Zh-En, En-Mr and Hi-Bn language pairs. The LaBSE + PPI-LaBSE based Filtering model performs better than QE based Filtering model for En→Mr language direction. The LaBSE + PPI-LaBSE model, which is trained on nearly twice the amount of training data compared to the QE-based filtering model, can be a contributing factor to its better performance in En→Mr.
The improvement in the performance of the Bn→Hi QE-based filtered MT system is comparable to the En→Mr and Zh→En QE-based filtered MT model. The Hi-Bn QE model is trained with only 500 training instances transfer learned from En-Mr trained QE models. This demonstrates the promise of the few-shot QE technique to generate training data for MT.
We compute Pearson Correlation between human annotated quality scores and quality scores computed using LaBSE and QE, shown in Table
6. The QE quality scores have a higher correlation with human annotated quality scores, compared to LaBSE quality scores for all 3 language pairs. Table 7 shows the Pearson Correlation between LaBSE and QE quality scores for all 3 language pairs. We observe that the LaBSE quality score has a low correlation with the QE quality score and the QE quality score has a high correlation with the human annotated quality score. This establishes the superiority of QE over the LaBSE quality score.

heading: 7 Conclusion and Future Work
text: We introduced a simple Quality Estimation based corpus filtering approach to extract high-quality parallel data from the noisy pseudo-parallel corpora. The takeaway from our work is that sentence-level QE-based filtering performs better than LaBSE-based filtering and helps improve the performance of NMT systems. We also show that few-shot QE models trained using a transfer learning-based approach can be used to extract good-quality parallel corpus from the pseudoparallel corpus. Only 1/40th of the normal data requirement (7K-25K) of QE training data achieves comparable performance for the Hindi-Bengali language pair. We also show that the QE quality score is superior to the LaBSE quality score.
In the future, we plan to use the proposed corpus filtering technique for other language pairs. This will provide us with a general overview of how this filtering technique performs for multiple languages.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their insightful feedback. We also express our gratitude towards Shivam Mhaskar, Sourabh Deoghare and other members of the Machine Translation group at CFILT, IIT Bombay, for their interesting and insightful comments.
Limitations
Although our primary effort in this work was to extract as much parallel corpora as possible, the improvement in the performance has been found to be only marginal. The LaBSE and QE-based filtering experiments involve a hyper-parameter called "threshold quality score." To achieve optimal results, we conduct experiments with different values of this hyper-parameter. The proposed few-shot transfer learning technique requires a small amount of data that needs to be annotated by multiple annotators.
Ethics Statement
The aim of our work is to extract high-quality parallel corpus from a noisy pseudo-parallel corpus. The datasets that we used in this work are publicly available and we have cited the sources of all the datasets that we have used. Publicly available datasets can contain biased sentences. We have also created a dataset for Hindi-Bengali few-shot QE. We briefly discuss the annotation guideline given to the annotators for the task in the Appendix A.3.

heading: A Appendix
text: A.1 Instances of Translations (Referred from Table 5)
The instances of translations for all 3 language pairs are provided in Table 6, 7, 8, 9 and 10.
A.2 Model Architecture (Referred from section 5.2)
We use a Transformer based architecture to train English-Marathi, Hindi-Bengali and ChineseEnglish NMT models for all our experiments. The encoder of the Transformer consists of 6 encoder layers and 8 encoder attention heads. The encoder uses embeddings of dimension 512. The decoder
of the Transformer also consists of 6 decoder layers and 8 decoder attention heads. We use MonoTransQuest architecture to train English-Marathi, Hindi-Bengali and Chinese-English QE models for all our experiments. We use a single Nvidia A100 GPU with 40 GB memory to train our NMT and QE models
A.3 Annotation Details (Referred from section 5)
A.3.1 Annotator Demographic For the Direct Assessment score annotation, we requested three native language speakers of Bengali who are well-versed in Hindi and have attended their graduate degrees in the Hindi language. They were aged between 25 to 42 and were paid for the time they spent on annotations.
A.3.2 Guidelines The guidelines provided to the annotators for the Quality Estimation task are shown in Figure 2.
A.3.3 Dataset We create Hi-Bn QE data for our few-shot settings. We use 500 high-quality Hindi sentences from IIT Bombay English-Hindi parallel corpus (Kunchukuttan et al., 2018). We use the HindiBengali NMT model to generate translations for the 500 Hindi sentences. We provide this HindiBengali parallel data to the annotators for the Direct Assessment Task. The Direct Assessment tasks require the annotators to score the MT translations as per the guidelines provided, Figure A.3.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section: Limitations
7 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section: Abstract and Section: 1 (Introduction)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5.2: Models; Section 5.1 Dataset
3 B1. Did you cite the creators of artifacts you used? Section 2: Related Work; Section 5.1: Dataset
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 5: Experimental Setup; Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.2: Models
C 3 Did you run computational experiments? Section 5: Experimental Setup; Appendix A.2: Model Architecture; A.3: Training Details
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A.2: Model Architecture; A.3: Training Details
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5: Experimental Setup
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6: Results and Analysis
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 6: Results and Analysis
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5.1: Dataset
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section: Appendix
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Section: Appendix

heading: Future Work
text: In the future, we plan to use the proposed corpus filtering technique for other language pairs. This will provide us with a general overview of how this filtering technique performs for multiple languages