C-XNLI: Croatian Extension of XNLI Dataset
Year: 2023

Abstract:
Comprehensive multilingual evaluations have been encouraged by emerging cross-lingual benchmarks and constrained by existing parallel datasets. To partially mitigate this limitation, we extended the Cross-lingual Natural Language Inference (XNLI) corpus with Croatian. The development and test sets were translated by a professional translator, and we show that Croatian is consistent with other XNLI dubs. The train set is translated using Facebook’s 1.2B parameter m2m_100 model. We thoroughly analyze the Croatian train set and compare its quality with the existing machinetranslated German set. The comparison is based on 2000 manually scored sentences per language using a variant of the Direct Assessment (DA) score commonly used at the Conference on Machine Translation (WMT). Our findings reveal that a less-resourced language like Croatian is still lacking in translation quality of longer sentences compared to German. However, both sets have a substantial amount of poor quality translations, which should be considered in translation-based training or evaluation setups.

heading: 1 Introduction
text: Natural language processing has developed rapidly in recent years. Models are starting to achieve human-like performance, but most of these achievements are concentrated on only a small fraction of the world’s 7000+ languages. This is to be expected due to the nature of linguistic annotation, which is not only tedious, subjective, and costly, but also requires domain experts, which are in decline (Lauscher et al., 2020).
There are two main approaches commonly used to handle that problem from the models’ perspective. The first approach relies on cross-lingual transfer, where the model is pretrained to learn multilingual representations (Conneau et al., 2020; Pires et al., 2019), while the other approach relies
heavily on Machine Translation (MT) systems to translate the text from a low-resource language to a high-resource language (or vice versa). Both approaches can be easily evaluated on cross-lingual benchmarks such as XTREME (Hu et al., 2020) or XGLUE (Liang et al., 2020). They consist of crosslingual datasets grouped by task to allow comprehensive evaluation. Unfortunately, XTREME covers 40 languages and XGLUE only 19.
Since none of these benchmarks include Croatian language in any of their datasets, and Crosslingual Natural Language Inference (XNLI; Conneau et al., 2018) corpus is included in both, we decided to extend XNLI with Croatian (CXNLI). The task is to classify whether a premise contradicts, entails, or is neutral to the hypothesis. XNLI’s development and test sets are crowdsourced in English and human-translated into 14 languages, while MultiNLI’s (Williams et al., 2018) training set is used for training. It also consists of machine-translated sets required for the translatetrain and translate-test paradigms.
Our Croatian extension is created in the same manner as its XNLI parent. The development and test sets are translated by a professional translator. Since XNLI provides translate-train, translate-dev and translate-test sets, we opted for Facebook’s 1.2B parameter m2m_100 MT model (Fan et al., 2020) to create our own translations.
It has been shown that MT models still suffer from errors like mistranslations, non-translations and hallucinations (Freitag et al., 2021; Raunak et al., 2021), which motivated us to analyze the quality of our dataset. For this purpose, we sampled 2000 sentences per language in both Croatian and German, and evaluated the translations using a variant of the Direct Assessment (DA) score proposed in the Multilingual Quality Estimation dataset (MLQE; Fomicheva et al., 2022).
To summarize, our contributions are the following: (1) we create and analyze the Croatian exten-
2258
sion of XNLI and provide baseline models, (2) we create Quality Estimation (QE) datasets for Croatian and German to evaluate the quality of machinetranslated sentences from the translate-train sets, and (3) we quantify the textual overlap between hypothesis and premise and analyze its impact on baseline models.

heading: 2 Datasets

heading: 2.1 C-XNLI
text: In creating the dataset, we follow the same procedure as Conneau et al. (2018). We hired a native Croatian professional translator to translate the English development (2490 samples) and test (5010 samples) sets of the XNLI dataset into Croatian. Premises and hypotheses were given to the translator separately to ensure that the premises did not provide context for the hypotheses. The English training set, derived from MultiNLI and containing 392,702 samples, was translated into Croatian using a selected MT model. We considered a total of eight models and opted for Facebook’s multilingual m2m_100 model with 1.2B parameters because of its highest BLEU score (Papineni et al., 2002) on the FLORES dataset (Guzmán et al., 2019), as shown in Table 1. All of m2m_100 and mbart models are available on fairseq1 (Ott et al., 2019), whereas opus models are available on Helsinki-NLP2 (Tiedemann, 2020; Tiedemann and Thottingal, 2020) and are evaluated by MarianNMT (Junczys-Dowmunt et al., 2018).

heading: 2.2 DA Scores
text: To evaluate the quality of the system used to translate English to Croatian, we compare the generated translations with the available translations from
1https://github.com/facebookresearch/fairseq 2https://github.com/Helsinki-NLP
a high-resource language. We score a sample of Croatian and German translations from the train set and compare the results. The sentences were sampled using a semantic similarity-based metric that correlates with translation quality (Cer et al., 2017) to flatten the original distribution of scores and analyze samples of diverse quality. A cosine score between the multilingual sentence representations from both LASER (Artetxe and Schwenk, 2019) and SBERT (Reimers and Gurevych, 2019) were used to measure semantic similarity between the source and translated sentences. These models are commonly used at the Conference on Machine Translation (WMT) for QE task (Specia et al., 2021, 2020). The SBERT we used is a multilingual variant trained on the paraphrase dataset which has slightly better performance than the models trained on similarity tasks (Reimers and Gurevych, 2020).
By utilizing a histogram of cosine scores with a bin size of 0.05, we adopted a circular sampling approach to randomly select one premise from each bin until a total of 50 premises were obtained. Similarly, we followed the same procedure for hypotheses, alternating between SBERT and LASER cosine scores. Furthermore, we implemented an additional criterion to ensure the inclusion of all premises and hypotheses that share a common premise. This entire process was repeated until we reached a 1000 samples each, for both SBERT and LASER cosine scores (2000 in total).
We scored the samples using the procedure described by Fomicheva et al. (2022). Annotators were asked to rate the translation quality for each sentence on a scale 0–100. Sentences were initially annotated by three annotators. If the range of the most diverging scores exceeded 30 points, an additional annotator was asked to replace the most diverging one until convergence was achieved. The annotators’ raw scores were converted to z-scores3; the final score is the average of all scores after convergence. More information about annotators, and annotation procedure is presented in Appendix A.

heading: 3 Analyses and Results

heading: 3.1 C-XNLI and DA Scores
text: To demonstrate that our extension has similar properties to its parent XNLI, we perform the following analyses. We tokenize C-XNLI’s sentences with MOSES tokenizer and obtain the average number
3The normalization according to each individual annotator’s overall mean and standard deviation.
of tokens in premises (19.0) which is nearly double the number in hypotheses (9.3) – a ratio that is consistent with other XNLI languages (see Appendix C).
Another analysis Conneau et al. (2018) provide is the BLEU score of their MT systems translating to and from the target language. We have extended their results to include those for the Croatian language (Table 2). Our translations from English to Croatian (EN-XX in the table) have the fourth-best BLEU score. These findings are not too surprising since the MT we use is more recent. The distribution of DA scores for Croatian and German is shown in Figure 1. We can observe that Croatian, although is a lower-resourced language, it has a slightly higher translation quality, as the mean of Croatian DA scores is almost identical to a German one.
The correlations between the LASER and SBERT cosine scores and DA scores for both languages are shown in Table 3, with p < 0.05. The correlations for German are higher, and the LASER cosines tend to correlate less.
In Figure 2 we can see that the Croatian model is more likely to make a mistake on premises compared to the German model.

heading: 3.2 Overlaps
text: The analysis presented here extends Artetxe et al.’s (2020) work where authors demonstrate that the overlap between hypotheses and premises is an overlooked bias in the XNLI dataset, caused by access to premise during hypothesis generation in English, and no access to it during translation into other languages. They decrease the bias by back-translating data and improve their results. To demonstrate the existence of that bias, we take a more direct approach and define a metric that represents overlap – the proportion of copied text from premise to hypothesis. It is the number of character N -grams which occur in both hypothesis and premise, divided by the number of possible character N -grams in the hypothesis. In Table 4 we presented those overlaps using bi-grams, N = 2. We can observe that in the training set, the overlap is 5% to 20% higher compared to development and test sets. In order to investigate that even further, we asked our professional translator to translate 1% of our C-XNLI dataset: 100 sentences which consist of 25 premises and 75 of their hypotheses. We made sure that the premise was given alongside each hypothesis so that it provides context to it in order to measure the influence on the overlap since, in the translation effort, premises and hypotheses were given separately. Our representative sample contained similar genre distribution, overlap distribution, and similar development vs. test overlap ratio. Our results show that when using N = 2, biased sample has 8% increase in overlap, whereas for N = {3, 4, 5}, it increased by ∼ 17%.

heading: 3.3 XLM-R Setups
text: We tested cross-lingual transfer using zero-shot and translate-based setups. For each, we employ pretrained XLM-R Base model (Conneau et al., 2020), implemented in Transformers library (Wolf et al., 2020). In the zero-shot approach, we fine-tune our model on English samples. In the translate-train approach, we fine-tune on translations of a training set, whereas in translate-train-all, we fine-tune it on concatenated training translations. Evaluations are done in all languages. In the translate-test approach, we use the same model from our zero-shot approach and evaluate it on English translations of other languages. We experimented with various hyperparameter configurations and found appropriate ranges. Hyperparameter optimization is done for each setup, and details are presented in the Appendix B.
Results of baseline setups are shown in Table 5. To demonstrate the comparability of our training setup, we compare XLM-R’s reported accuracy with ours, which is only 0.6 points lower in the train-translate-all setup. The performance of the Croatian model is consistently among the TOP5 models. The reason for that might be in the high BLEU score shown in Table 2. Focusing on the best overall model – translate-train-all, we notice that adding Croatian did not drastically change the average performance and decreased it only for dis-
tant languages like Urdu and Swahili. Whereas for other languages, it increased or did not change significantly.
Finally, Table 5 also shows how the performance of models on the test set of each language correlates with the bi-gram overlaps in the train, development, and test sets of that particular language. There is a consistent high correlation between the overlap in all sets and models’ performance (p < 0.05). However, a lower correlation is seen in the development and test sets. This observation could be attributed to the fact that increasing the overlap of a particular language makes it more similar to the English set, in terms of overlap, thus improving the performance. However, as we showed in Subsection 3.2, the overlap in the development and test sets is artificially lower due to biased translation. Alternatively, high training overlaps might indicate that the model is learning to detect the occurrence of overlapping cues.

heading: 4 Conclusion
text: In this work, we extended XNLI to include the Croatian language. The development and test sets were translated by a professional translator. We have successfully demonstrated that the quality of the development and test sets is comparable to that of the other languages. To validate the machine-translated training set, we compare our
Croatian translations with those available for a high-resourced language – German. The comparison is based on 2000 manually scored sentences from German and Croatian train sets using a variant of DA scores normalized by z-score. Our results show that the Croatian MT model performs slightly better because it’s more up-to-date, even though it’s a lower-resourced language. We also found that the Croatian translation model performs poorly on longer sentences – premises.
Finally, we present an overlap metric to measure the textual overlap between the premise and hypothesis. We find that the training set has larger overlaps than the development and test sets. These overlaps resulted in a high correlation between the models’ scores, indicating that a model uses cues from the data that also correlate with overlaps.
We provide our datasets under the same license4
as the XNLI dataset, and also make the accompanying code available on GitHub5. We hope that by sharing our datasets, researchers will have the opportunity to gain further insights and expand their knowledge in the field of cross-lingual transfer.
Limitations
In each contribution of this work, we can isolate several potential limitations. In creating C-XNLI, the MT model for the formation of the train set was chosen based on the results from a single dataset. Additionally, an assumption that the model is plagued with typical issues that affect MT models was investigated on a small dataset. Although we are skeptical of the MT model’s performance and perform QE scoring of the small dataset by a group of annotators and analysis to ascertain its performance, we are only comparing Croatian machinetranslation results to results from a single language (German), assuming that results would hold for other high-resource languages. Also, for some MT evaluations, we use a single metric (BLEU) known to have many problems but only generally considered to correlate with human judgment.
Our hyperparameter optimizations are of limited scope. All hyperparameters are fixed, except the learning rate with four possible values we search over. Furthermore, we only used three seeds. We could not perfectly reproduce the results outlined in the paper of our baseline XLM-R Base model, partly due to a lack of elucidation in the original
4CC BY-NC 4.0 5https://github.com/lobadic/C-XNLI
paper and partly due to limited hyperparameter optimizations.
Finally, we do not elucidate further and experiment with the discovered correlation between the models’ performance and the overlap in datasets, and we leave it for future work.

heading: A Direct Assessment
text: A.1 Annotation
In order to increase the quality of our annotations, we firstly provided a set of 50 training samples, and only later provided the other samples to the annotators. The annotators are instructed to score each sentence on a scale 0–100 according to the perceived translation quality (Fomicheva et al., 2022). Specifically, the 0–10 range for incorrect translations; 11–29 for translations with a few correct keywords, but wrongly conveyed meaning; 30–50 for the ones containing major mistakes; 51–69 for translations that convey the meaning of the source, but contain grammatical errors; 70–90 for translations that preserve semantics of the source sentence; and 91–100 for correct translations.
Also, our Croatian annotators are Croatian native students majoring in Linguistics or pursuing a Translation degree. German annotators have the language competence of C1 or above. They were paid per hour. On the contrary, our professional translator was paid according to the regular translation rate in Croatia for a large corpus on a card basis (1800 characters including white spaces).
A.2 Scores Dataset Creation
When resolving final DA scores, if we ended up in a scenario where the outlier was on either side (e.g.
[0, 20, 40]), we randomly chose one. Furthermore, the process described by Fomicheva et al. (2022) is biased towards the first three annotations, meaning that if two of them are outliers, we’ll keep discarding annotations until a third outlier comes. In our process, it happened in ∼ 1% of cases.

heading: B Hyperparameters
text: Here we outline hyperparameters used for hyperparameter search of our XLM-R Base model on different XNLI training setups. Every model was trained on 3 epochs, and the best one (out of 3 epochs) was chosen based on evaluation results on the dev set. For all of our experiments we used 2 NVIDIA 3090 GPUs.
C XNLI Additional Analyses
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: 5 (first after the conclusion)
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: 0,1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 2,3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 1,2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 4
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 2,3
C 3 Did you run computational experiments? 3
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 2
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 2, appendix A.1
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 2

heading: Future Work
text: Finally, we do not elucidate further and experiment with the discovered correlation between the models’ performance and the overlap in datasets, and we leave it for future work