IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages
Year: 2023

Abstract:
The rapid growth of machine translation (MT) systems necessitates meta-evaluations of evaluation metrics to enable selection of those that best reflect MT quality. Unfortunately, most meta-evaluation studies focus on European languages, the observations for which may not always apply to other languages. Indian languages, having over a billion speakers, are linguistically different from them, and to date, there are no such systematic studies focused solely on English to Indian language MT. This paper fills this gap through a Multidimensional Quality Metric (MQM) dataset consisting of 7000 fine-grained annotations, spanning 5 Indian languages and 7 MT systems. We evaluate 16 metrics and show that, pre-trained metrics like COMET have the highest correlations with annotator scores as opposed to n-gram metrics like BLEU. We further leverage our MQM annotations to develop an Indic-COMET metric and show that it outperforms COMET counterparts in both human scores correlations and robustness scores in Indian languages. Additionally, we show that the Indic-COMET can outperform COMET on some unseen Indian languages. We hope that our dataset and analysis will facilitate further research in Indic MT evaluation.

heading: 1 Introduction
text: Natural language generation (NLG) has seen rapid progress in the past few years due to advancements in the field of large language models (LLMs) (Lewis et al., 2020; Liu et al., 2020; Dabre et al., 2022; Scao et al., 2022). Although initial research had focused on high-resource languages, recently the focus has shifted to middle-resource and lowresource languages. In the context of machine translation (MT), there is increasing interest in building massively multilingual models supporting numerous translation directions. For example,
Costa-jussà et al. (2022) release a model which supports around 200 languages (40K directions). While this is commendable, to make MT truly inclusive, it is important that various design choices in the MT life-cycle are evaluated for low-resource languages and not simply transferred and adapted from English. One such important choice is of the correct evaluation metric to be used for evaluating MT systems.
A recent survey by Sai et al. (2022) has shown that over the last decade, many evaluation metrics have been proposed for MT. In parallel, several works (Callison-Burch et al., 2006; Sai et al., 2021; Mathur et al., 2020a; Tan et al., 2015; Fabbri et al., 2021) have shown the inadequacy of popular metrics, such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004). However, many languages are not represented in these works and most of the focus is on European languages. On the other hand, there is a growing body of work on machine translation focused on language groups such as Indian (Dabre et al., 2022), Indonesian (Cahyawijaya et al., 2021), and African (Reid et al., 2021). However, these works rely on English centric metrics due to lack of sufficient studies with tried and tested recommendations for their evaluation of the languages. While techniques like MQM (Multidimensional Quality Metric) are being used for collecting better quality human-evaluation data for English and a select few other languages (Freitag et al., 2021a), such multidimensional evaluations and analyses are not available for several language groups.
We narrow our focus to evaluation of one of these language groups, namely Indian languages which have more than a billion speakers worldwide. Indian languages are morphologically rich, especially Dravidian languages, which exhibit agglutination. Furthermore, they have relatively freeword order (Murthy et al., 2019; Kunchukuttan and
14210
Bhattacharyya, 2020) as compared to European languages which means that frequently used metrics such as BLEU may not always be reliable. This calls for an independent focused study on the evaluation of metrics for Indic languages in order to understand whether these conclusions drawn hold true for the Indian languages.
In this paper, we aim to bridge this gap by focusing on the evaluation of MT (from English) into 5 Indian languages from 2 different families and make significant contributions towards designing MT evaluation metrics for these languages. Our main contribution is in the form of the MQM dataset for Indian languages created by taking outputs generated by 7 popular MT systems and asking human annotators to judge the quality of the translations using the MQM style guidelines (Lommel et al., 2014). With the help of language experts who are experienced in translation, we generate an MQM dataset consisting of 7000 annotated sentences, 1400 per language.
We use the aforementioned dataset to establish correlations between the annotator scores and existing automatic metrics scores belonging to the following classes: (i) n-gram and character based such as BLEU, METEOR, chrF++, (ii) embeddings based such as Vector Extrema, BERTScore, (iii) pre-trained metrics like BLEURT-20, COMET. We observe that pre-trained metrics show the highest correlations with the annotator scores, with the COMET metric performing the best (§5.1). Additionally, we also observe that the metrics are not capable of capturing the fluency-based errors for Indian languages (§5.4). Finally, we use our data to train an Indic-COMET metric which not only shows stronger correlations with human judgement on Indian languages, but is also more robust to perturbations (§6). We hope that our dataset and metric, which are publicly available1, will help spur research in this field.

heading: 2 Related Work
text: Meta-evaluation studies: Evaluation metrics have been under intense scrutiny in order to establish their reliability. Mathur et al. (2020b) discuss that studying evaluation metrics needs to be a meticulous task by showing many potential issues and oversights that could lead to wrong conclusions. Other works focus on extending the resources for meta-evaluations (Sai et al., 2021; Karpinska et al.,
1https://github.com/AI4Bharat/IndicMT-Eval
2022) and different genres (van der Wees et al., 2018). While most of these works focus on English, there are works that evaluate the efficacy of metrics on other languages such as German, Chinese, Spanish, etc. (Rivera-Trigueros and Olvera-Lobo, 2021; Freitag et al., 2021b). On the other hand, we focus on Indian languages, which have not received much attention.
Collecting human annotations: Metaevaluation studies rely heavily on human-annotated translations of various systems. Since humans are better at providing relative ranking (i.e., comparing the qualities of 2 or more items) rather than providing absolute scores to quantify the quality of an item, WMT15-17 collected Relative Rankings (Bojar et al., 2015, 2016a, 2017). However, since they require a quadratic number of ratings, Direct Assessment (DA) scores, which are quality assessment scores over each output on a scale of 0-100, are easier and faster to collect (Kocmi et al., 2021). More recently, the Multidimensional Quality Metric (MQM) approach for collecting human judgments was adopted for Machine Translation by Freitag et al. (2021b). They obtained annotations from professional raters with MQM training, which Clark et al. (2021) recommend. On a related note, Klubicka et al. (2018) conduct human studies for Croatian, whereas Fabbri et al. (2021) followed systematic approaches to collect and provide multidimensional scores for other tasks such as summarization.

heading: 3 Indic-MT Eval Dataset
text: Following Freitag et al. (2021b), we collect MQM annotations for 5 Indian languages, i.e., Tamil (ta), Gujarati (gu), Hindi (hi), Marathi (mr), and Malayalam (ml). We sample 200 sentences from the FLORES-101 dataset (Goyal et al., 2022) and obtain the translation outputs from 7 machine translation systems (§3.1) for each of the 5 Indian languages.

heading: 3.1 MT Systems Considered
text: We use state-of-the-art models to obtain translation outputs in the 5 languages. These include EnglishXX translation outputs obtained from open-sourced pre-trained models like mBART (Liu et al., 2020), mT5 (Xue et al., 2021), IndicTrans (Ramesh et al., 2022), cvit (Philip et al., 2019), NLLB (Costajussà et al., 2022), as well as outputs obtained2
2Collected in February 2022
from Microsoft Azure Cognitive Services API3 and Google translation API4 (additional details in Appendix B.1). Note that for Gujarati, we find all mBART outputs to be unintelligible and filled with a mixture of characters from several languages. We hence re-allocate the budget corresponding to these sentences for Gujarati to collect annotations on the references instead. Similar to the findings of Clark et al. (2021), we observe that the references are not always perfect, and these sentences also have errors. However, we find that these errors are often of lower severity.

heading: 3.2 Methodology
text: We adopt the MQM-framework (Lommel et al., 2014) for collecting human annotations on the data at the segment level. In general, a segment may contain one or more sentences. Bilingual language experts, proficient in English and a native language, were employed as annotators for the task of identifying and marking errors in each segment. As shown in Figure 2, the source segment in English and the translated segment are presented to the annotators, along with provisions to mark up to 5 errors of various categories and severity (§3.4). If there are more than five errors, the annotators are asked to identify only the five most severe ones.
3Bing API 4Google API
In cases where there are more than five severe errors, or if it is not possible to reliably identify distinct errors because the translation is unrelated to the source, then the translation is marked as nontranslation, a special category error spanning the entire segment. Depending on the quality of the translation and the errors identified, the annotators were also asked to provide a score out of 25 for each translation after marking all the errors (if any) for that translation. More detailed guidelines are presented in Appendix A.

heading: 3.3 Quality Assurance
text: We first performed pilot studies on collecting data via crowd-sourced annotators who are native speakers of the languages we use in this study. In a pilot which directly asked for the final scores, similar to DA scores used in WMT in a few years (Bojar et al., 2016c,b), we found the scores to be highly subjective, similar to Clark et al. (2021). We also found that displaying the reference translations, which are not always perfect, was biasing the annotators to ignore some errors. Another pilot task involved MQM instead of DA scores in the same crowd-sourced setting. However, we found it difficult to achieve consistency in annotations with crowd-sourced raters. We tried the following strategies to improve the quality (i) We provided the same set of segments to 3 annotators per language and then organized a discussion among them to resolve disagreements. The idea was to eventually converge to a consistent marking scheme after a few initial sets of different markings, (Nema and Khapra, 2018). (ii) We collected annotations from 3 annotators per language and provided all the 3 annotations to a different fourth annotator to aggregate them. However, neither yielded fruitful results in terms of agreement with MQM annotations.
Finally, we employed language experts who have
experience in translation tasks and observe that we were able to achieve better consistency among annotators. We use the first 50 sentences (sampled randomly from various models and of various lengths) as a pilot to help the annotator get an idea of the variety and kind of translations in the dataset. Note that MQM-style annotations use a formula to automatically compute scores for each segment based on the errors identified. The score, s, for each segment with a set of identified errors, E, is given by s = 25−∑i∈E wi ∗ ei, where wi is the penalty associated with the severity of the error and ei is the penalty associated with the type of error. Appendix A provides more details on the penalties used for the different error types and severities, following Lommel et al. (2014).
In addition to the formula-based score, we also ask the annotator to provide an overall score after marking the errors for that segment. We then verify the correlations between the formula-based scores and the scores provided by the annotator and found them to be highly correlated (i.e., > 0.7 Kendall-tau correlation) for all languages. In order to compute the Inter Annotator agreement score, we sample 200 segments for each language and compute the Kendall-tau correlation between the scores given by two annotators. For all the languages, we observe high correlation scores of 0.61, 0.57, 0.55, 0.538, and 0.52 for Malayalam, Gujarati, Tamil, Hindi, and Marathi respectively.

heading: 3.4 Analysis
text: Distribution of Error types: Following the MQM guidelines and prior work on MQM (Freitag et al., 2021b), we have 13 categories of errors, including 4 sub-categories under fluency and 5 under accuracy, style error, source error, non-translation
(a special case to mark segments that are extremely poor translations or have more than 5 high severity errors) and an ‘other’ category for any error types that are not accounted for in the list of error types. The error types are listed in Appendix A. On all languages except Tamil, we found ‘Accuracy Mistranslation’ to have the highest error count among all error types. More generally, on average, the machine translation models today primarily err on accuracy-based errors and make fewer fluencybased mistakes as seen in Figure 1.
Severity of Errors: We plot all the fluency and accuracy errors graded by error severities for all languages in Figure 1. As depicted, there are 5 error severity types: Very High, High, Medium, Low, and Very Low. For all the Indo-Aryan languages (gu, hi and mr), the majority of the errors observed are accuracy-based errors. For Tamil, a Dravidian language, we find the accuracy errors and fluency errors in almost equal proportions. We find Malayalam, another Dravidian language, to have more accuracy errors than fluency errors, with majority medium-severity errors, as shown in Figure 1.
MT systems: Figure 3 shows the total number of errors per model (inclusive of all severities) for each language. We find that the recent MT models (NLLB, IndicTrans) have fewer errors compared to the relatively older models (CVIT). Table 9 in Appendix provides a more detailed picture which also inherently takes into account the severities of the errors. It shows the average score of each system computed as the mean of the human scores obtained on all the outputs from that system. We find that IndicTrans model, which focuses on Indian languages, has the highest scores on Hindi, Malayalam and Tamil. NLLB is the best performing model on Marathi and Bing API for Gujarati. Considering the average performance across all languages, the best performing models in descending order are IndicTrans, NLLB, Google API, Bing API, mT5, CVIT, and mBART.

heading: 4 Experimental Setup
text: In this section, we discuss the various evaluation metrics under consideration (§4.1) and evaluating strategies (§4.2) followed.

heading: 4.1 Evaluation Metrics Used for MT
text: We consider the most popular metrics being used in Barrault et al. (2021, 2020) along with other
variants to suit the languages under consideration. In total, we study 16 metrics belonging to different classes (Sai et al., 2022) of either word overlapbased, embedding-based, or trained metrics.
• In the word overlap-based category, we consider (i) BLEU (Papineni et al., 2002), (ii) SacreBLEU (Post, 2018), (iii) ROUGE (Lin, 2004), (iv) chrF++ (Popovic, 2017), (v) TER (Snover et al., 2006).
• For the embedding-based metrics, we use (i) Vector Extrema (VE) (Forgues and Pineau, 2014), (ii) Greedy Matching (GM) (Rus and Lintean, 2012), (iii) Embedding Averaging (EA) (Landauer and Dumais, 1997), (iv) LabSE (Feng et al., 2020) & (v) LASER (Artetxe and Schwenk, 2019) embeddings and (vi) BERTScore (Zhang et al., 2020).
• For computing BERTScore, in addition to the official implementation, which uses mBERT, we also consider other variants that use BERT models trained on Indian languages, namely IndicBERT (Kakwani et al., 2020) and MuRIL (Khanuja et al., 2021).
• The end-to-end trained metrics we consider are (i) PRISM (Thompson and Post, 2020),
(ii) BLEURT (Sellam et al., 2020) and (iii) COMET variants (Rei et al., 2020).

heading: 4.2 Meta Evaluation
text: For evaluating the evaluation metrics we measure how well the metrics correlate with human judgments on two granularities i.e.: segment-level and system-level. We use Pearson correlation (ρ) which measures the linear correlation between two sets of data and Kendall’s Tau (τ ) to measure the ordinal association between two quantities.

heading: 5 Results and Discussions
text: In this section, we present the segment-level correlations in §5.1 and system-level correlations in §5.2, followed by analyzing metrics in §5.3, §5.4.

heading: 5.1 Segment-level Evaluation
text: The correlation between MQM-based scores and metric scores, measured using Pearson and Kendalltau correlations on 1400 segments per language as shown in Table 1. We observe that out of the overlap-based metrics, chrF++ has the highest correlation across all languages, but overall overlapbased metrics are the worst performing which is in line with the findings of Kocmi et al. (2022). Among the embedding-based metrics, LabSE embeddings yields better correlations than any of the
other embedding-based approaches. The correlations improve further when we use BERTscore with embeddings obtained from different multilingual models. The results in this case are mixed, with MuRIL showing the best correlations on average. Overall, we observe that neural-networkbased, end-to-end trained metrics with exposure to Indian languages are the best-performing metrics on average. The trained metric PRISM, which has been trained on 39 languages, out of which the only Indian language is Bengali, performs very poorly on all the 5 Indian languages in our study, partially owing to the minimal Bengali data used for training. On the other hand, BLEURT-20, a metric finetuned on ratings from the WMT Metrics Shared Task and synthetic data from the WMT corpus, has fairly good correlations on all languages except Hindi. COMET-metric variants have the highest overall correlations for all the languages.

heading: 5.2 System-level Evaluation
text: Table 2 shows the Pearson and Kendall-tau correlations at the system-level following Louis and Nenkova (2013). Since Kendall-tau is based on pairwise score comparisons, it reflects a common ranking use case and is more reliable for systemlevel correlations. The metric rankings remain consistent across both granularities, with more vari-
ability observed in the segment-level task. Similar to the segment-level correlations, trained metrics show the highest correlations across all languages. COMET shows the highest correlations, followed by BLEURT-20. Although on the segment level COMET-QE was not at par with the COMET reference-based metrics, for system ranking the reference-free COMET-QE metrics show high correlations and are well suited for ranking system pairs. Although Kocmi et al. (2021) already observed this for other languages, with the help of our dataset and experiments we are able to pro-
vide empirical evidence to confirm this for Indian languages.

heading: 5.3 Spread of Metric Scores
text: While the correlations of metrics are good, we still find that the range of metric scores is skewed. That is, most of the metrics do not utilise their entire scoring range, and often provide scores in a narrow range. This skew in the spread hinders the interpretability of the scores provided by the metric. For example, SacreBLEU has a scoring range between 0 to 100. However, the scores are almost always in the lower half of the scoring range as seen in Figure 4 containing the spread of normalised scores of each metric5. This is not a case of an issue with the data being always poor as we can see in Figure 4 that the human scores for Tamil show a spread through-out the scale. On the other hand, the embedding-based metrics, which use cosine similarity, have a theoretical maximum of 1 and minimum of 0; however, the scores are concentrated at the higher end of the scale, rendering the individual scores uninterpretable despite decent correlations.

heading: 5.4 Correlations Conditioned on Error Type
text: Mathur et al. (2020b); Sai et al. (2021) show that correlations do not convey the true picture and it is important to perform in-depth analysis to understand the true ability of the metrics. Hence we perform the following experiment to examine the
5Some of the metrics, such as the trained metrics and editdistance-based metrics, are not bounded to a scoring range. We normalize such metrics using their maximum and minimum values in the current dataset.
performance of metrics on the two primary error categories in the MQM framework, i.e, fluency and accuracy. We select those annotated segments that contain only a single error type in order to clearly separate the two error types. This gives us two MQM data subsets, one containing only fluency errors and the other only accuracy errors. Since the dataset size could be different, we control for the size by sampling an equal number of segments from both sets. Figure 5 contains the correlation values for the various metrics. Splitting the dataset based on the error types shows a more nuanced picture. The majority of the metrics show a higher correlation with human scores when only accuracy errors are annotated. This implies that the metrics are able to capture the accuracy errors well but fail on fluency-based errors. We hope that future works on designing better evaluation metrics for Indian languages focus more on developing metrics that can capture fluency-based errors.

heading: 6 Indic COMET
text: Having analyzed various metrics, we fine-tune the best performing metric – COMET – using our MQM dataset (§6.1) and show that the new finetuned metric not only outperforms the COMET metric on the majority of the languages but also is more robust to perturbations (§6.2). Additionally, we also test the zero-shot evaluation ability of the Indic-COMET metric in §6.3.

heading: 6.1 Training
text: We build our metric with the architecture of COMET (Rei et al., 2020). We use the Estimator model, which uses XLM-RoBERTa (Conneau et al.,
2019) backbone to encode the source, hypothesis, and reference. We use the same training process and hyper-parameters as COMET for a fair comparison (additional details in Appendix B.2). Following Rei et al. (2021), we experiment with initializing the model with different checkpoints, namely, XLM-R, COMET-DA, and COMET-MQM, and fine-tune it on our MQM dataset.

heading: 6.2 Evaluation
text: Table 3 compares the correlation values of our fine-tuned Indic-COMET with the best-performing COMET baselines. Since no other evaluation datasets for Indian languages are available, we use our own MQM dataset for both training and testing. Hence to perform a throughout evaluation we perform a 3-fold cross-evaluation by splitting the dataset into 3 independent training and testing datasets and report the mean correlation values across the 5 languages in consideration. We observe that Indic-COMET fine-tuned from the COMET-MQM checkpoint shows higher correlations across all languages, compared to the other variants on average. Indic-COMET-MQM outperforms both the COMET baselines on 3 out of the 5 languages and shows higher correlations than COMET-MQM across all languages. The most notable gains are in Hindi. Inspired by recent works on meta-evaluation (Kocmi et al., 2022; Sai et al., 2021), we also analyze the robustness of metrics on challenge sets. We make use of the challenge set created by Amrhein et al. (2022) since it contains Indian languages. We use the subset of the dataset that only contains Indian languages and follow Amrhein et al. (2022) to report performance with Kendall’s tau-like correlations. Indic-COMETMQM has a correlation score of 0.306 and is more robust than the COMET counterpart which has a score of 0.272. Overall, we observe that fine-tuning the COMET metric on our MQM dataset not only
improves correlations with human scores but also increases the robustness to perturbations.

heading: 6.3 Zero-shot Evaluation
text: Since we evaluate only 5 Indian languages, out of the 22 official languages (and over a hundred major languages that are spoken in the country6), we investigate whether the metric has the potential to perform better in other Indian languages as well. In order to test this ability, we finetune on only 4 languages and test on the unseen one. We use the same evaluation setup as discussed in §6.2. Table 4 contains the comparison between the best performing Indic-COMET variant i.e.: Indic-COMETMQM and COMET baselines. We observe that IndicCOMET still outperforms both the COMET baselines on the majority of languages even though it is not trained on the specific Indian languages. It also shows higher correlations than COMET-MQM across all languages. This suggests that collecting annotations for some Indian languages is key for progress in Indic evaluation as it can benefit other low-resource languages too.

heading: 7 Conclusion
text: We present a large-scale MQM dataset consisting of 7000 fine-grained annotations, spanning 5 Indian languages and 7 MT systems, for evaluating machine translation metrics for Indian languages. With the help of this dataset, we show that the
6https://www.britannica.com/topic/Indian-languages
current pre-trained metrics outperform the overlapbased metrics (§5.1) in terms of correlations with the human scores. Additionally, we also perform an in-depth study (§5.4) to identify the drawbacks of the current metrics. We then use our dataset to train an Indic specific COMET metric that outperforms existing metrics in terms of both correlations and robustness scores (§6.2). We hope that our dataset and analysis will help promote further research in Indic MT evaluation.

heading: 8 Acknowledgements
text: We thank the Ministry of Electronics and Information Technology (MeitY), Government of India, for setting up the ambitious Digital India Bhashini Mission with the goal of advancing Indian language technology. The annotators and language experts who worked on this project were supported by the generous grant given by Digital India Bhashini Mission to IIT Madras to serve as the Data Management Unit for the mission. We are indebted to Shri Nandan Nilekani and Shrimati Rohini Nilekani for supporting our work through generous grants from EkStep Foundation and Nilekani Philanthropies. These grants were used for (i) supporting many of the students, research associates and developers who worked on this project, (ii) fulfilling many of our compute needs and (iii) recruiting project managers to oversee the massive pan-India data collection activity undertaken as a part of this work. We thank the Centre for Development and Advancement of Computing, Pune (CDAC Pune) for access to its Param Siddhi super-computer which was used for mining bitext pairs at scale. We thank the Google India Ph.D. Fellowship Program and the Prime Minister’s Fellowship Scheme for Doctoral Research for supporting Ananya Sai.

heading: 9 Limitations
text: The approach to collect our dataset is expensive and laborious. This along with the dependence on expert annotators makes the transfer of such an approach challenging for other low-resource languages. We however, find this a necessary endeavor to develop initial resources that can help provide a starting point to extend access to more languages and iteratively improve research, technologies and services across languages.

heading: 10 Ethical Considerations
text: For the human annotations on the dataset, the language experts were paid a competitive monthly salary to help with the task. The salary was determined based on the skill set and experience of the expert and adhered to the norms of the government of our country. The dataset has no harmful content. The annotations are collected on a publicly available dataset and will be released publicly for future use. All the datasets created as part of this work will be released under a CC-0 license7 and all the code and models will be release under an MIT license8.

heading: A MQM Guidelines to Annotators, Error types & Severities
text: The annotators assess translations at the segment level, where a segment may contain one or more sentences. Each translated segment is aligned with a corresponding source segment, and both the source and translated segments are displayed. Table 5 shows the error hierarchies for all the error types. Each category has severity levels ranging from very high to very low on a 5-point scale of (Very low, Low, Medium, High, and Very-high). Table 6 shows the descriptions of the end-points of the scale, as shown to the annotators. For computing scores for each segment based on the annotations, we use the following weights/penalties: very low: 1, low: 2, medium: 3, high: 4, very high: 5. Each of the sub-categories under Accuracy, Fluency, Terminology Inappropriate, Style have equal weightage since each of these are accompanied with a corresponding severity marking. Non-translation errors, by definition, elicit a score of 0. Sentences that are marked with a source error are discarded.
The following guidelines were provided to the annotators:
• Identify all errors within each translated segment, up to a maximum of five. If there are more than five errors, identify only the five most severe.
• To identify an error, highlight the relevant span of text using text colors, and select a
category/sub-category and severity level from the available options. (The span of text may be in the source segment if the error is a source error or an omission.)
• When identifying errors, be as fine-grained as possible. For example, if a sentence contains two words that are each mistranslated, two separate mistranslation errors should be recorded.
• If a single stretch of text contains multiple errors, (that is, if there are overlapping errors) one only needs to indicate the one that is most severe. If all have the same severity, choose the first matching category listed in the error typology (eg, Accuracy, then Fluency, then Terminology, etc).
• There are two special error categories: Source error and Non-translation. Source errors should be annotated separately, highlighting the relevant span in the source segment. A sentence that has a source error need not be scored but the error in the source segment is to be highlighted.
• If it is not possible to reliably identify distinct errors because the translation is too badly garbled or is unrelated to the source, then mark a single Non-translation error that spans the entire segment. There can be at most one Non-translation error per segment, which should span the entire segment. No other errors should be identified if Non-Translation is selected.
• Depending on the quality of the translation and the errors identified, provide a score out of 25 for each translation. Indicate the score in the final score column, after marking all the errors (if any) for that translation.

heading: B Additional details
text: B.1 MT systems Considered For the mBART we use the Huggingface Transformers (Wolf et al., 2020) for generating the outputs for the various languages. Specifically, we use the facebook/mbart-large-50-many -to-many-mmt model. For mT5 we finetune the pre-trained mTBASE model for the translation task using all existing sources of parallel data provided by Ramesh et al. (2022). We finetune one model
for every language pair. For IndicTrans and CVIT, we use the models released by Ramesh et al. (2022) and Philip et al. (2019) respectively.
B.2 Indic COMET Training
All experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.432 kgCO2eq/kWh. A cumulative of 10 hours of computation was performed on a single RTX A4000 GPU. Total emissions are estimated to be 0.6 kgCO2eq of which 0 percent were directly offset. Estimations were conducted using the MachineLearning Impact calculator presented in Lacoste et al. (2019).
For training, we follow the same process as Rei et al. (2020). We load the pretrained encoder and initialize it with either XLM-Roberta, COMETDA or COME-MQM weights. During training, we divide the model parameters into two groups: the encoder parameters, that include the encoder model and the regressor parameters, that include the parameters from the top feed-forward network. We apply gradual unfreezing and discriminative learning rates, meaning that the encoder model is frozen
for one epoch while the feed-forward is optimized with a learning rate. After the first epoch, the entire model is fine-tuned with a different learning rate. Since we are fine-tuning on a small dataset, we make use of early stopping with a patience of 3. The best saved checkpoint is decided using the
overall Kendall-tau correlation on the test set. The training hyper-parameters used are given in Table 7. Since we have a total of 7000 annotated segments, we perform a 3 fold cross validation split (500 training and 2000 testing) and ensure that the English sentences in the test set are not present during training. We report the mean correlation values for each language. The variance was observed to be less than 0.02. A similar experiment setup was followed for the zero shot evaluation of IndicCOMET, where additionally training segments belonging to a particular language was dropped from the training dataset.

heading: C Additional Results
text: Table 8 shows the maximum and minimum values of each metric on our dataset, across all languages. Note that while some of the metrics are bounded by a theoretical minimum and maximum, some others (especially the trained metrics) are not strictly restricted to a specific scoring range. It would be possible to see a lower minimum value or a higher maximum value on other datasets with such metrics.
Figure 6 shows metric scores for different human score intervals (0-5, 6-10, 11-20, 21-25). This helps analyse whether the metric scores are roughly in the same buckets or same range as human-scores without focusing on the fine-grained ratings that might not always be of significance. From the plots, we observe that high-performing metrics such as BERTScores and COMET-DA correlate positively with the metric scores as the human scores increase. However, poor-performing metrics on Indic languages such as PRISM (due to lack of training data for Indic languages) do not have correlated metric v/s human spreads even at a coarse-level.
Figure 7 depicts a scatter plot of metric scores on the y-axis against human scores on the x-axis. The scatter plots provide more insights than just
the correlation values (Mathur et al., 2020b). We note that the metrics falter by producing some false high and false-low scores. However, the metrics produce a higher density of decently correlated scores to produce a net positive correlation trend in most cases.
Table 9 shows the average scores per system considering the scores provided by the annotator on all the outputs from that system. We find that the best performing model changes across the 5 languages. For Hindi, Malayalam and Tamil, IndicTrans outputs are found to get higher scores on average. For Malayalam Bing API is a close-second and NLLB for Tamil. For Gujarati Bing-API is the best performing, with IndicTrans and NLLB performances being very close. In case of Marathi, NLLB outputs are better, followed by IndicTrans. Averaging further across all 5 languages, IndicTrans is found to be the highest scoring model.
Table 10 contains the correlation values for the
various metrics on the Fluency-only and Accuracyonly error subsets discussed in section 5.4. We observe that all the metrics on average correlate better with the human scores when only accuracy errors are annotated compared to having only fluency errors.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
9
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3,4,5,6
3 B1. Did you cite the creators of artifacts you used? 3,4,5,6
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 10

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3,4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3,4, Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4, Appendix B.2
C 3 Did you run computational experiments?

heading: 6.1, Appendix

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6.2

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix A

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 10

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix A
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 10

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 3.2

heading: Future Work
text: The annotations are collected on a publicly available dataset and will be released publicly for future use. All the datasets created as part of this work will be released under a CC-0 license7 and all the code and models will be release under an MIT license8