Title: XNLIeu: a dataset for cross-lingual NLI in Basque
Benchmark: XNLIeu
Summary: XNLIeu is an expanded version of the XNLI benchmark that includes Basque, created by machine-translating and then manually post-editing the original English data to support cross-lingual NLI research in low-resource languages. Experiments with various LLMs show that post-editing significantly improves performance and that the translate-train strategy is most effective, though its advantage lessens when applied to natively created datasets.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Natural language understanding with the focus on cross-lingual natural language inference (NLI).
Phenomenon defined by authors: Yes
Phenomenon definition: The Natural Language Inference (NLI) task consists in classifying pairs of sentences –a premise and a hypothesis– according to their semantic relation: entailment, when the meaning of the premise entails that of the hypothesis; contradiction, when both sentences have opposing truth conditions and can not co-occur at the same time; and neutral, when both sentences are not semantically related.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The task is to classify pairs of sentences—a premise and a hypothesis—into one of three categories based on their semantic relationship: entailment, contradiction, or neutral.
Task item definition: a premise, a hypothesis, and the related label (entailed, contradicts, or neutral)
Task source: Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 392, 702
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean and std
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The impact of machine translation vs. professional post-edition.
They justify the creation of a native Basque set to address biases and artefacts common in translation-based datasets.
Taxonomy: NLP / Understanding
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding