Title: Latxa: An Open Language Model and Evaluation Suite for Basque
Summary: The paper introduces 4 multiple-choice evaluation datasets for Basque: EusProfi-ciency, comprising 5,169 questions from official language proficiency exams; EusReading, comprising 352 reading comprehension questions; EusTrivia, comprising 1,715 trivia questions from 5 knowledge areas; and EusExams, comprising 16,774 questions from public examinations. 
Contribution: Another contribution of the paper is Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which they continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: language proficiency, knowledge and reasoning
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Purpose notes: The benchmark includes 4 different tasks
Task definition: There are 4 tasks: reading comprehension, language proficency, mcq questions on Basque language and culture, and mcq questions on Basque government 
Task item definition: an mcq question
Task source: Human exam questions (e.g. GRE questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: ~7.5k
Dataset metadata available: No
Sampling method: Unknown
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: accuracy, F1, standard deviation
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Multilinguality / 
New taxonomy: General Capability / Multilinguality