How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages? (2024)

Abstract: While machine translation evaluation has been studied primarily for high-resource languages, there has been a recent interest in evaluation for low-resource languages due to the increasing availability of data and models. In this paper, we focus on a zero-shot evaluation setting focusing on low-resource Indian languages, namely Assamese, Kannada, Maithili, and Punjabi. We collect sufficient Multi-Dimensional Quality Metrics (MQM) and Direct Assessment (DA) annotations to create test sets and metaevaluate a plethora of automatic evaluation metrics. We observe that even for learned metrics, which are known to exhibit zero-shot performance, the Kendall Tau and Pearson correlations with human annotations are only as high as 0.32 and 0.45. Synthetic data approaches show mixed results and overall do not help close the gap by much for these languages. This indicates that there is still a long way to go for low-resource evaluation. The dataset and evaluation metrics are publicly accessible online.1

heading: 1 Introduction
text: While there has been a meteoric rise in the amount of data and improvements in architectures for machine translation (MT) models (Gala et al., 2023; Costa-jussà et al., 2024), in order to scientifically establish whether the translation quality has improved, it is important to have reliable evaluation metrics. However, most of the evaluation metrics were developed with English and a few select other languages in mind. It has been shown that such metrics do not necessarily generalize to other languages and have to be separately meta-evaluated (Sai B et al., 2023; Rivera-Trigueros and OlveraLobo, 2021). The reasons behind this include linguistic aspects that vary across languages, along with factors like the diversity of outputs produced by the models for each language. Such qualitative
1https://github.com/AI4Bharat/IndicMT-Eval
differences will be exacerbated in low-resource languages due to the prominent reliance on extensive data resources by today’s models.
In this work, we delve deeper into the evaluation of low-resource Indian languages, namely Assamese, Maithili, Punjabi, and Kannada, belonging to 2 different language families. Our goal is to establish the reliability of MT evaluation metrics for low-resource languages. To facilitate this, we collect human scores on the candidate translations using the MQM approach (Lommel et al., 2014). We make use of 5 large multilingual models and APIs that can output text in these languages to generate candidate translations for evaluation. We then collect 250 annotations per language, amounting to a total of 1000 MQM annotations for low-resource languages.
Using the data we created, we evaluate multiple existing evaluation metrics of different types, both automatic and learned. In the case of learned metrics, since we do not have training data, we leverage data for related Indic languages from Sai B et al. (2023) for fine-tuning and performing zeroshot meta-evaluations. We observe that for these learned metrics, despite studies finding decent to good performance in other languages, there is a huge margin for improvement in evaluating lowresource languages. We also explore the influence of the base model and synthetic data generation for low-resource languages.
In summary, our contributions are as follows: (i) MQM dataset for 4 low resource languages for evaluation (ii) Meta-evaluation of existing metrics on low-resource languages (iii) Analysis of potential techniques to improve the metrics, including (a) exposure of metrics to related languages, (b) different base models, and (c) usage of synthetic data. We show that evaluation for low-resource languages is still far behind other languages.
640

heading: 2 Related Work
text: The effectiveness of evaluation metrics has been studied for various languages. Most of the existing MT evaluation metrics are typically analyzed for language pairs where English serves as either the source or target language. That has led to several criticism works (Ananthakrishnan et al., 2006; Callison-Burch et al., 2006; Post, 2018) followed by improvements. However, models are getting increasingly multilingual and slowly evaluation metrics are being studied for other languages (Sai B et al., 2023; Freitag et al., 2021; RiveraTrigueros and Olvera-Lobo, 2021; Cahyawijaya et al., 2021). Metrics like chrF (Popović, 2015) and chrF++ (Popovic, 2017) were proposed for character-based, morphologically-rich languages. While some of these criteria hold for the languages we consider, there is no publicly available open study of such metrics for the specific case of low resource languages. On the other hand, different evaluation metrics are being used to evaluate models in these languages. WMT23 (Pal et al., 2023) had a special task track for low resource Indic languages for which BLEU, ChrF, RIBES, TER, and COMET metrics were used apart from human evaluation. However, to the best of our knowledge, there are no studies analyzing whether these metrics correlate with human judgments or not for these languages. Additionally, there is no publicly available data with human scores to study this. Mohtashami et al. (2023) used synthetic data augmentation to build a BLEURT-like metric for low resource languages. The only Indian language in their set is Punjabi (2k size, not publicly released), which initially had a poor Pearson correlation of 0.184. This was slightly improved to a value of 0.194 when adding synthetic data to their baseline data, although it is still a poor correlation value.

heading: 3 Methodology
text: We collect MQM annotations as well as direct assessment (DA) scores and also create synthetic data for 4 languages, viz., Assamese, Punjabi, Kannada, and Maithili. We use the human-curated data as test data to benchmark the performance of various metrics on these low resource languages. The synthetic data is used to investigate the use of such strategies for augmenting resources in these languages for potential improvements in performance. We design experiments to understand the role of other related languages and the base model on the
performance. The following subsections provide the details of the data we create and the strategies explored in our experiments.

heading: 3.1 MQM Data Annotation
text: Following Sai B et al. (2023), for each of the 4 languages, we hired 2 language experts who are native speakers of that language with bilingual proficiency in English. We provided them the English source segment, the translation to be evaluated, and the MQM annotation guidelines (Lommel et al., 2014; Sai B et al., 2023) for identifying error types and their severities in the translations. These annotations were later used to calculate MQM scores. In addition to identifying errors, the annotators were also asked to assign a score to the translation in the range of 0-25, which we refer to as DA score since these are directly assigned by the annotator.
For quality assurance, we initially gave 50 common segments to both annotators to mark the errors and indicate their scores. For any disagreements in annotations, the reasons were independently discussed with the annotators. Most of these disagreements were slight differences in marking severity, which we found to be subjective and difficult to standardize. Later, we computed the interannotator agreements (IAA) using the Pearson correlation of their scores. We employed a different annotator and repeated the validation process whenever this was below a threshold of 0.5 (which was the case for one language in our set - Punjabi). The final IAA is as follows for the 4 languages considered - Maithili - 0.7, Punjabi - 0.7, Assamese - 0.65, and Kannada - 0.68.
We obtained the translations from 5 state-of-theart multilingual models and APIs including IndicTrans (Ramesh et al., 2022), NLLB2 (Costa-jussà et al., 2024), NLLB-MoE, Microsoft Azure Cognitive Services API 3 and Google translation API4. The source segments fed to these models are sampled from the FLORES-101 dataset (Goyal et al., 2022), and each segment is translated by each of the 5 models. These sources and translated segments are presented to the language expert in a random order without details regarding the model / API that generated the translation. The language expert is asked to highlight the text containing the error and indicate the type and severity of the error. We obtain such detailed annotations on 250
2We use the 1.3 B parameter version of the NLLB models. 3Bing API 4Google API
segments per language.

heading: 3.2 Synthetic Data Creation
text: As human annotation data is expensive and timeconsuming to collect, we follow Geng et al. (2023) and Geng et al. (2022) and generate synthetic data for the aforementioned languages to reflect the variety of error types and severities in translations. Since we only have test sets, we obtain error type and severity distributions from datasets of related Indic languages in Sai B et al. (2023). We generate similar proportions of the error types and severities that can be both synthetically recreated and have a significant occurrence count in the distribution. To generate synthetic examples, we utilized BPCC-seed dataset containing data in all these lan-
guages without any overlap with the FLORES test set. More details about the synthetic data creation are presented in the Appendix B. Specifically, we created synthetic data with around 44k sentences for Assamese, 32k for Kannada, 24k for Maithili, and 6k for Punjabi based on the size of the available data in these languages.

heading: 3.3 Evaluation Metrics Considered
text: We investigate the performance of multiple metrics of different categories. We consider (i) Wordoverlap based metrics of BLEU (Papineni et al., 2002) variants, SacreBLEU (Post, 2018), ROUGE (Lin, 2004), (ii) Character-based metric of chrF++ (Popovic, 2017), (iii) Edit-distance based metric of TER (Snover et al., 2006), (iv) Embedding-based
metrics of LabSE (Feng et al., 2022), LASER (Artetxe and Schwenk, 2019) (v) BERTScore computed using mBERT (Zhang et al., 2020), IndicBERT (Kakwani et al., 2020) and MuRIL (Khanuja et al., 2021), (vi) Trained metrics of BLEURT (Sellam et al., 2020) and COMET variants (Rei et al., 2020). Additionally, we also assess GEMBA-MQM (Kocmi and Federmann, 2023), a GPT-based reference-free evaluation metric. We experiment with replacing the English-focused examples in the prompt with examples from various Indian languages. We do this by selecting samples in en-hi, en-ta, and en-gu directions from the Indic MT Eval dataset. Further details on our adaptation are provided in Appendix A.

heading: 3.4 Zero Shot-Evaluation Approach
text: Since the focus of this paper is zero-shot evaluation of our languages of interest, for learned metrics like COMET-DA and COMET-MQM, we leverage training data containing MQM and DA annotations, for all 5 related Indic languages, henceforth called related data, from Sai B et al. (2023). The related languages include Hindi, Gujarati, Marathi belonging to Indo-aryan family and Tamil and Malayalam belonging to Dravidian language family. There are 1,476 annotated examples in total per language which we split into train, validation and test set containing 1000, 200 and 276 examples respectively for each language. We found that some of the references were mismatched with the source sentences, which we corrected for our fine-tuning experiments. The validation data is used for early stopping, and the models performing best on the 5 related languages are used for zero-shot evaluation on our 4 languages of interest. We consider fine-tuning existing COMET-DA and COMET-MQM models which are language agnostic and compare them against fine-tuned variants using IndicBERTv2 (Doddapaneni et al., 2022) which is Indic focused. Note that XLM-Roberta and hence COMET models has 24 layers while IndicBERT v2 has 12 layers making the latter efficient. Using synthetic data: Regarding the use of synthetic data, created as described in section 3.2, henceforth called synthetic, we consider the following configurations on COMET-DA:
1. Single Stage: jointly-trained model on a randomly shuffled mix of related data and synthetic data.
2. 2-Stage S/R: training on synthetic data fol-
lowed by related data with a reduced learning rate.
3. 2-Stage R/S: training on related data followed by synthetic data with a reduced learning rate.

heading: 4 Results
text: We present the results for the following research questions to find ways to potentially improve performance on these models: (RQ0) How do existing metrics fare on lowresource languages? (RQ1) Does fine-tuning on related languages help? (RQ2) Does replacing the underlying model of a trained evaluation metric with an alternate backbone model trained on related languages help? (RQ3) Does synthetic training data help? We report Kendall-tau and Pearson correlations with human annotations.

heading: 4.1 Meta-Evaluation of Existing Metrics
text: Table 1 shows that, among the word or character overlap-based metrics, chrF++ performs the best on most of the languages. In the embeddingbased approach, we find LabSE performs better than LASER embeddings. However, overall the word-based, character-based, and embeddingbased metrics are outperformed by the trained metrics. Among the trained metrics, the COMET model variants perform the best. Specifically, the recently proposed referenceless COMET-Kiwi and COMET-Kiwi-xl models have the best correlations with human judgments. However, most of the COMET-variants, except for COMET-Kiwi-xl, perform poorly in the Maithili language. This is despite the COMET*-DA variants having seen Hindi language data during training, which is closely related to Maithili and shares the same script. We observe that the GPT-4 based evaluation exhibited significantly lower performance on these languages. This could be attributed to the limited exposure of the underlying model to Indian languages, potentially hindering its ability to effectively identify translation errors in this context. All the analysis above presents observations of the relatively better performing metrics. Overall, we find that none of the evaluation metrics have good correlations with human judgments on these low resource languages.

heading: 4.2 Impact of Related languages
text: In the 6th block of Table 1, specifically in the first two rows, we observe that fine-tuning on the 5
related languages improves correlations with human judgments(detailed results in Table 4 of Appendix C). We find that it also enhances performance of COMET-DA ("Indic-COMET-DA" row) on the low-resource languages belonging to the same or a close language family. However, we did not observe the same trend for COMET-MQM ("Indic-COMET-MQM" row).
Our findings suggest that fine-tuning on related languages using supervised data can be a promising technique for improving performance on low resource languages. However, its effectiveness may vary depending on the underlying model and training configuration.

heading: 4.2.1 Does the Backbone Model Matter?
text: To assess the role of the backbone model on the zero-shot performance, we perform experiments by replacing the XLM-Roberta base model of COMET with the IndicBERT v2 model5. The IndicBERT v2 model is a pretrained multilingual masked language model that was trained on 23 Indian languages including the low-resource languages in our evaluation set. However, note that it used a different dataset namely IndicCorp v2 for training.
The rows of ‘Base-IndicBert(DA)’ and ‘BaseIndicBert(MQM)" in 6th block of Table 1, show what happens when we switch from the COMET backbone to IndicBERT v2. Comparing with nonfine-tuned as well as fine-tuned COMET variants, latter being Indic-COMET, we find that fine-tuning with an Indic-languages-specific base model like IndicBERT v2, which has prior exposure to these languages, leads to an improvement in performance.

heading: 4.3 Training with Synthetic Data
text: Following the synthetic data incorporation methods outlined in 3.4, we experiment with using different proportions of the synthetic data with the real data. In particular, we start by adding equal proportions of real and synthetic data (i.e., 5k samples each) and thereafter double the amount of synthetic data added until we hit the maximum amount of data available for synthetic data creation.
The results are presented in Table 1(detailed results in Table 2).Note that the synthetic data portion added in the experiment for each low resource language only contains data in that particular language. However, the real data consists of the same 5 related Indian languages.
5Note that XLM-Roberta model has 24 layers while IndicBERT v2 has 12 layers
None of these approaches conclusively outperform the baseline models (COMET-DA and IndicCOMET-DA). The Single-Stage approach shows modest improvement when equal proportions of real and synthetic data are used. However, the performance declines on adding more amount of synthetic data. Overall, the mixed results in these experiments question the effectiveness of using larger quantities of synthetic data for low resource language translation evaluation tasks. This highlights the need for further investigation in this area, presenting an avenue for future research.

heading: 5 Conclusions
text: Our work introduced an MQM dataset for four low resource languages consisting of 250 examples per language. Using this dataset, we analyzed the zero-shot performance of different types of existing metrics and observed that none of these existing methods showed good results in the case of low resource language. We explored different techniques to improve the performance, which includes finetuning on related language using Indic MT eval dataset 4.2, changing the base model to an Indicmodel 4.2.1 and using synthetic dataset of these low resource language 4.3. While some of these techniques provide small improvements, we find that there is still a long way to go for low-resource language evaluation.

heading: 6 Limitations
text: The size of our dataset being small makes it just about sufficient for testing purposes. The lack of a dev split for the data limits the possibilities of exploring certain other recipes for training. We hope this serves as a starting point though.

heading: 7 Ethical Consideration
text: For human annotations, language experts were provided with monthly salary based on their skill set and experience, under the norms of the government of our country. The annotations are collected on a publicly available dataset and will be released publicly for future use. All the datasets created as part of this work will be released under a CC-0 license6 and all the code and models will be released under an MIT license7.
6https://creativecommons.org/publicdomain/ zero/1.0
7https://opensource.org/licenses/MIT

heading: Acknowledgements
text: We would like to express our gratitude to the Ministry of Electronics and Information Technology (MeitY), Government of India, for setting up the ambitious Digital India Bhashini Mission with the goal of advancing Indian language technology. The annotators and language experts who worked on this project were supported by the generous grant given by Digital India Bhashini Mission to IIT Madras to serve as the Data Management Unit for the mission. We thank Shri Nandan Nilekani and Shrimati Rohini Nilekani for supporting our work through generous grants from EkStep Foundation and Nilekani Philanthropies. These grants were used for (i) supporting many of the students, research associates, and developers who worked on this project, (ii) fulfilling many of our computing needs, and (iii) recruiting project managers to oversee the massive pan-India data collection activity undertaken as a part of this work. We thank Pranjal Agadh Chitale for his helpful feedback and research discussions on this work.

heading: A GPT-4 as evaluator
text: We used GEMBA-MQM (Kocmi and Federmann, 2023), a GPT-based reference-free evaluation metric. This method employs a "three-shot prompting" technique, where GPT-4 is given three predetermined examples in en-de, en-cs, zh-en language pairs to help it understand the task of identifying
error-prone segments in the translated text. We experiment with replacing the English-focused examples in the prompt with examples from various Indian languages. We selected samples in en-hi, en-ta, en-gu directions by sampling from the Indic MT Eval dataset (Sai B et al., 2023). Table 1 shows results both on the vanilla GEMBA-MQM and our modified version named GEMBA-MQM(IL) which explicitly includes Indian language examples within the prompt.

heading: B Synthetic Data Creation
text: We first studied the MQM annotations in the related languages of Hindi, Marathi, Gujarati, Tamil, and Malayalam released by Sai B et al. (2023). We extracted the counts of various error types with their corresponding severity counts. We choose the error types and severities that can both be synthetically recreated and have a significant occurrence count in the distribution. To recreate the errors in the low resource languages considered in our work, we use the BPCC-seed dataset containing data in all these languages without any overlap with the FLORES test set. For each of the error types, we modify correct sentences in the following ways.
• Omission errors: We first determine whether the words are stop words or not depending on their frequency of occurrence in the BPCC corpus. We heuristically determine the top 100 words as the common words or stop words. We randomly drop an uncommon word in
each segment sampled for an omission error introduction.
• Addition errors: We randomly sample an uncommon word to be introduced at a random position in the segment. We found these errors to be less frequent and accordingly sampled fewer segments to include such errors.
• Mistranslation errors: We randomly select tokens to be replaced with a [’MASK’] token. We then sample perturbations using Muril model. To replicate errors of different severities, we sample tokens with reduced generation probabilities to represent more severe pseudo errors. For the generation of varied pseudo translations, we employ a random selection process wherein one token is chosen from the top k tokens with the highest generation probability. Specifically, we set k values at 2,3,5,8 and 10 for different levels of severities.
• Grammatical errors: We add, drop, or edit the common words to create fluency-based errors in the segments.

heading: C Training Details
text: For training, we follow a similar process as (Rei et al., 2020). We start by loading the encoder initialized with either COMET-DA, COMET-MQM, or IndicBERT weights. We divide our model parameters into two groups: the regressor parameter,
which involves the parameters of top feed-forward added for regression, and the encoder parameter, which comprises parameters of the pre-trained encoder. In the initial epoch, the encoder is frozen and only feed-forward is trained with a specific learning rate, after that entire model is trained using different learning rate. For detailed information about hyperparameters, please refer to table 3.
All our experiments used a single RTX 3090 Ti GPU, with a cumulative computational time of 8 hours. Different experiments in this paper used different learning rates (lr) based on hyperparameter tuning. For fine-tuning Indic-COMET-DA and Indic-COMET-MQM, we found 1.0e-06 and 1.5e06 learning rates to be the best respectively. While we fine-tuned indicBERT with a slightly higher learning rate of 1.0e-05.

heading: Future Work
text: The annotations are collected on a publicly available dataset and will be released publicly for future use. All the datasets created as part of this work will be released under a CC-0 license6 and all the code and models will be released under an MIT license7. 6https://creativecommons. org/publicdomain/ zero/1. 0
7https://opensource. org/licenses/MIT