La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America
Authors: María Grandury, Javier Aula-Blasco, Júlia Falcão, Clémentine Fourrier, Miguel González Saiz, Gonzalo Martínez, Gonzalo Santamaria Gomez, Rodrigo Agerri, Nuria Aldama García, Luis Chiruzzo, Javier Conde, Helena Gomez Adorno, Marta Guerrero Nieto, Guido Ivetta, Natàlia López Fuertes, Flor Miriam Plaza-del-Arco, María-Teresa Martín-Valdivia, Helena Montoro Zamorano, Carmen Muñoz Sanz, Pedro Reviriego, Leire Rosado Plaza, Alejandro Vaca Serrano, Estrella Vallecillo-Rodríguez, Jorge Vallego, Irune Zubiaga
Accepted at: acl25

La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America

María Grandury1,2, Javier Aula-Blasco3, Júlia Falcão3, Clémentine Fourrier4, Miguel González2, Gonzalo Martínez5, Gonzalo Santamaría6
Rodrigo Agerri7, Nuria Aldama6, Luis Chiruzzo13, Javier Conde2, Helena Gómez12, Marta Guerrero6, Guido Ivetta11, Natalia López6, Flor Miriam Plaza-del-Arco9, María Teresa Martín-Valdivia10, Helena Montoro6, Carmen Muñoz6, Pedro Reviriego2, Leire Rosado6, Alejandro Vaca8, María Estrella Vallello-Rodríguez10, Jorge Vallego, Irune Zubiaga7

1SomosNLP, 2ETSIT, Universidad Politécnica de Madrid, 3Barcelona Supercomputing Center, 4Hugging Face, 5Universidad Carlos III de Madrid, 6Instituto de Ingeniería del Conocimiento, 7Centro HiTZ - Ixa, Universidad del País Vasco UPV/EHU, 8LenguajeNatural.AI, 9LIACS, Leiden University, 10Universidad de Jaén, 11Universidad Nacional de Córdoba, 12Universidad Nacional Autónoma de México, 13Universidad de la República, Uruguay

Correspondence: maria.grandury@somosnlp.org

Abstract

Leaderboards showcase the current capabilities and limitations of Large Language Models (LLMs). To motivate the development of LLMs that represent the linguistic and cultural diversity of the Spanish-speaking community, we present LA LEADERBOARD 1, the first open-source leaderboard to evaluate generative LLMs in languages and language varieties of Spain and Latin America. LA LEADERBOARD is a community-driven project that aims to establish an evaluation standard for everyone interested in developing LLMs for the Spanish-speaking community. This initial version combines 66 datasets in Basque, Catalan, Galician, and different Spanish varieties, showcasing the evaluation results of 50 models. To encourage community-driven development of leaderboards in other languages, we explain our methodology, including guidance on selecting the most suitable evaluation setup for each downstream task. In particular, we provide a rationale for using fewer few-shot examples than typically found in the literature, aiming to reduce environmental impact and facilitate access to reproducible results for a broader research community.

1 Introduction

The evaluation of multilingual Large Language Models (LLMs) is challenging. LLMs are expected to perform a large variety of tasks, from problem-solving to text summarization, all in multiple languages (Guo et al., 2023). In this context,

![Summary of the evaluation datasets included in LA LEADERBOARD.](page_1012_370_323_246.png)

Figure 1: Summary of the evaluation datasets included in LA LEADERBOARD. Disclaimer: A country does not represent a language; flags are used for simplicity.

leaderboards have emerged as one of the standard approaches for evaluating and comparing LLMs in a transparent manner. As we cannot improve what we cannot measure, it is important to develop leaderboards that enable a more comprehensive evaluation of LLMs across linguistic boundaries, contributing to the development of culturally aware AI systems that can serve diverse global linguistic communities.

Spanish is one of the most spoken languages worldwide, with more than 600 million speakers (Fernández and Mella, 2024). It is the predominant language in 21 countries, where it coexists with other languages. Many people use Spanish and the local language in their daily activities. Spain has four official languages: Spanish, Catalan, Basque, and Galician. While Catalan and Galician are Romance languages closely related to Spanish, Basque is one of the world’s few language isolates (Campbell, 2010). In Latin America (LATAM),
there are hundreds of indigenous languages, such as Guarani and Náhuatl, which have influenced local Spanish varieties (Lustig, 1996). From a sociolinguistic point of view, this creates a unique scenario for multilingual LLM evaluation. Moreover, knowing which LLMs perform best in these languages can have deep implications for multilingual communication (Strassel and Tracey, 2016).

Existing leaderboards predominantly focus on English or a small set of high-resource languages (Fourrier et al., 2024; Mialon et al., 2023; Pal et al., 2024; Contributors, 2023). While Spanish is often included in multilingual leaderboards, evaluation datasets are typically limited and translated, either by machines (Barth et al., 2024), failing to capture the linguistic richness of the language (Plaza et al., 2024) or by humans2, still failing to represent the target culture (Singh et al., 2024) Moreover, despite the growing presence of LLMs in multilingual settings, no leaderboard currently evaluates a combination of languages spoken in Spain and Latin America. This lack of representation limits the development of models that can truly serve these communities (Mager et al., 2018).

To address this gap, we introduce LA LEADERBOARD, the first open-source leaderboard designed to evaluate generative LLMs based on the needs of the Spanish-speaking community. Beyond the initial set of languages that includes Spanish and the official languages of Spain (Basque, Catalan, and Galician), LA LEADERBOARD is designed to evolve, gradually expanding to encompass more languages and linguistic varieties, ensuring it reflects the rich diversity of the global community. This new leaderboard consists of a diverse set of evaluation tasks (see Figure 1) designed in a way that reflects the nuances and actual usage of the target languages. It is a community-driven initiative launched by the #Somos600M Project (Grandury, 2024), aiming to foster the development of LLMs that better represent the linguistic and cultural diversity of the Spanish-speaking world. We share our approach to inspire other linguistic communities to create similar leaderboards.

The main contributions of this work are:

• We present the community-based methodology used to create the first open-source leaderboard for evaluating generative LLMs in Spanish and the official languages of Spain, with a scalable framework designed to include more languages and language varieties over time.

• We introduce a logical and resource-efficient approach to few-shot configurations, enabling accessible and reproducible evaluations for the wider community.

• We provide a comprehensive analysis of state-of-the-art (SOTA) LLMs, providing insights into their strengths and limitations in Spanish, Catalan, Basque, and Galician.

By addressing the linguistic and cultural diversity of Spain and LATAM, LA LEADERBOARD sets a new standard for multilingual LLM evaluation, which encourages the development of models that are not only linguistically competent but also culturally aware.

2 Related Work

Benchmarks Several benchmarks have been developed to evaluate the performance of LLMs in tasks like language understanding (Wang et al., 2019), general knowledge (Hendrycks et al., 2021a), reasoning (Sakaguchi et al., 2019), or mathematical problem solving (Hendrycks et al., 2021b). There are also efforts to develop holistic benchmarks or evaluation suites that provide a comprehensive evaluation of different capabilities of LLMs (Liang et al., 2023; Gao et al., 2021; Fourrier et al., 2023, 2024; Srivastava and et al., 2023).

Multilingual and multicultural benchmarks LLMs are now trained in multiple high-resource languages at the same time (Ali et al., 2024; Martins et al., 2024; Qwen Team, 2024; Jiang et al., 2023), which means that the benchmarks must reflect this linguistic diversity. A common approach is machine translating English tests (Holtermann et al., 2024; OpenAI, 2023). However, translation errors may add noise to the results, making them less reliable (Plaza et al., 2024). Furthermore, each language has its nuances, preferred styles, and cultural background, which unrevised machine translation may fail to capture (Plaza-del-Arco et al., 2020; Singh et al., 2024). Ideally, specific test sets should be originally written in the target language or manually adapted (Nangia et al., 2020) to capture the richness and cultural and linguistic subtleties associated with it. This is slowly becoming the trend as shown by the recently released language-specific (Mercorio et al., 2024; Quercia

2https://hf.co/datasets/openai/MMMLU
et al., 2024) and multilingual culturally aware (Romanou et al., 2024; Salazar et al., 2025; Myung et al., 2025; Romero et al., 2024) benchmarks.

Leaderboards  Benchmarks are the pieces of the LLM evaluation puzzle that provide valuable but fragmented information on their performance. Leaderboards and arenas use these evaluation sets to compare the performance of LLMs in a neutral, third-party manner through automatic evaluations (Mialon et al., 2023) or human judgments (Chiang et al., 2024). On some community-oriented leaderboards (Fourrier et al., 2024), anyone can submit their LLMs for evaluation, and the tools, tests, and results are open, allowing for reproducibility. This represents a good way to drive progress in LLM development by enabling people with limited compute to compare their models to the current SOTA.

Multilingual leaderboards  Leaderboards exhibit the same shortcomings as benchmarks when evaluating languages other than English. To address this problem, specific leaderboards are being developed in different languages such as Italian (Mercorio et al., 2024), Korean (Kim et al., 2024), Chinese (Contributors, 2023), Arabic (Elfilali et al., 2024) or Polish (Jassem et al., 2025).

Spanish leaderboards  Focusing on the Spanish language, the ODESIA leaderboard3 by UNED NLP features 14 bilingual Spanish-English datasets from discriminative shared tasks. The team evaluates 10 LLMs. While submissions are open, the evaluation datasets are private, avoiding task contamination (Salido et al., 2025) but making it impossible to reproduce the results locally. Regarding text generation, Spanish is represented in the Chatbot Arena4, which features a dedicated category, and in SCALE’s private leaderboard5. However, both exclusively evaluate a fixed set of models. The only existing leaderboard including a language from Spain or Latin America other than Spanish is CLUB6, developed by the BSC as part of the AINA Project, which combines 8 Catalan datasets and evaluates 5 BERT and RoBERTa-based models.

In this work, we present the methodology used to create a comprehensive, fully open-source leaderboard for languages and language varieties from Spain and Latin America that assesses different capabilities of generative models, including domain knowledge, information extraction, linguistic proficiency, and ethical aspects. LA LEADERBOARD aims to serve as a reference for the Spanish-speaking scientific community, fostering the development of more robust and culturally adequate LLMs.

3  LA LEADERBOARD

LA LEADERBOARD is a community-driven initiative that brings together 66 datasets in Spanish, Catalan, Basque, and Galician, covering diverse tasks and domains. Public since September 23, 2024, LA LEADERBOARD has received over 15,000 visits in four months and currently showcases evaluation results from 50 models.

3.1 Data Collection

Most of the datasets in LA LEADERBOARD were donated by 13 research groups. Initially, these contributions were received through a publicly shared Google Form (Appendix E) or direct outreach. In particular, 7 datasets were specifically created for LA LEADERBOARD (AQuAS, ClinTreatES, ClinDiagnosES, HumorQA, SpaLawEx, TELEIA, and RAGQuAS). We also included widely used open-source benchmarks such as Belebele.

LA LEADERBOARD keeps expanding with dataset contributions such as CONAN-EUS and VeritasQA. These new connections are bidirectional: we actively share this initiative in relevant conferences and reach out to research groups, while others contact us upon discovering LA LEADERBOARD . Beyond collecting existing datasets, we are also fostering collaborations to enhance the representation of languages and linguistic varieties across Latin America.

To thank research groups for their donations, we include in LA LEADERBOARD ’s interface the corresponding logo and dataset citation. Moreover, the dataset authors are acknowledged in this paper.

3.2 Task Construction

3.2.1 Datasets

Including diverse evaluation datasets is essential for building a comprehensive leaderboard. This section discusses the key axes that guided their selection. Table 1 enumerates the datasets organized by language and task type, while Table 2 shows the
upcoming datasets that have been recently donated and not yet evaluated. In Appendix A, we provide the citations and further details about the datasets, including origin and domain.

Languages LA LEADERBOARD contains 22 evaluation datasets in Spanish, including the varieties of Spain, Mexico, Argentina, Chile, and Uruguay. It also gathers datasets in all the official languages of Spain, with 18 datasets in Catalan, 17 in Basque, and 9 in Galician.

Origin We aim to evaluate models with high-quality datasets that reflect the cultural and linguistic idiosyncrasies of each language. For this reason, we only include datasets that have been annotated or revised by at least one native speaker of the language. We prioritize the inclusion of datasets originally created in the language they evaluate, which constitute 55% of the leaderboard. When this is not possible and translation is required, we prioritize datasets translated by human professionals. Not only does this prevent the loss of linguistic nuances that happens with machine translation (Plaza et al., 2024), but it also allows translators to adapt the text to the target culture (Nangia et al., 2020) and to identify errors in the source datasets and ensure that no extra hints regarding the answer are given in the input prompt (Bau-cells et al., 2025). In LA LEADERBOARD , 38% of the datasets have been manually translated from an existing English benchmark. We also acknowledge that, given the low-resource nature of some languages we cover, machine translation is more affordable than human translation. However, we only include such datasets if the automatic translation was comprehensively reviewed by a person proficient in the target language. Only 7% of the datasets in LA LEADERBOARD are manual reviews of machine-translated datasets.

Format The multiple-choice question-answering (MCQA) format is widely used for automatic evaluations due to its simplicity. Thus, MCQA is the format of 59% of the tasks included in LA LEADERBOARD . We acknowledge that the literature has identified some issues with MCQA tasks, such as models’ sensitivity to answer order (Pezeshkpour and Hruschka, 2024; Mina et al., 2025) or lack of task understanding (Khatun and Brown, 2024). Moreover, some suggest that this type of task does not reflect the actual models’ responses and capabilities (Li et al., 2024; Wang et al., 2024a). To address this issue, we also include text generation tasks, such as summarization, evaluated using NoticIA for Spanish, caBreu for Catalan, and Summarization-GL for Galician. We evaluate long-form question-answering in Spanish using the AQuAS and RagQuAS datasets. Finally, we assess counter-narrative generation with RefutES in Spanish and CONAN in Basque and Spanish.

Domains LA LEADERBOARD includes well-known generalist datasets aimed at evaluating a model’s capability to understand and complete a task, such as Belebele, WNLI, and XStoryCloze. We also include evaluation datasets focused on truthfulness assessment, such as VeritasQA and the Galician translation of TruthfulQA. There are, in addition, several domains represented in LA LEADERBOARD , such as the medical (e.g., ClinTreatES), legal (e.g., SpaLawEx), and press (e.g., caBreu, NoticIA). We also include ethics-oriented datasets, evaluating stereotype generation in Spanish and Catalan with CrowsPairs.

Tasks The types of tasks chosen for our leaderboard extend those usually included in well-known leaderboards (e.g., reasoning, natural language inference, question answering or summarization) to other task types for which high-quality datasets exist in our target languages (e.g., counter-narrative generation or linguistic acceptability). For consistent performance comparisons across languages, we prioritize tasks available in multiple languages.

3.2.2 Metrics

The MCQA tasks are evaluated by measuring the logarithmic probabilities (LogProbs) of models’ outputs among a restricted list of options. For text generation tasks, we compare the expected (gold-standard) and given responses using various metrics depending on the original authors’ implementation, including BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and Semantic Answer Similarity (SAS, Risch et al., 2021). Furthermore, following the recent trend of evaluating text generation tasks using LLMs, we are adapting an automated Judge-LLM metric from Zubiaga et al. (2024). Since SAS and LLM-based metrics are not currently supported in the evaluation suite we use, the LM Evaluation Harness (Gao et al., 2021), we implement them in our open-source fork7.

7https://github.com/somosnlp/lm-evaluation-harness
<table>
  <tr>
    <th>Task Type</th>
    <th>Spanish</th>
    <th>Catalan</th>
    <th>Basque</th>
    <th>Galician</th>
  </tr>
  <tr>
    <td>Common-sense reasoning</td>
    <td>copa_es<br>xstorycloze_es</td>
    <td>copa_ca<br>xstorycloze_ca</td>
    <td>xcopa_eu<br>xstorycloze_eu</td>
    <td>xstorycloze_gl</td>
  </tr>
  <tr>
    <td>Ethics</td>
    <td>crows_pairs_es</td>
    <td>crows_pairs_ca</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>Linguistic acceptability</td>
    <td>escola</td>
    <td>catcola</td>
    <td>–</td>
    <td>galcola</td>
  </tr>
  <tr>
    <td>Math</td>
    <td>mgsm_direct_es</td>
    <td>mgsm_direct_ca</td>
    <td>mgsm_direct_eu</td>
    <td>mgsm_direct_gl</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>wnli_es<br>xnli_es</td>
    <td>teca<br>wnli_ca<br>xnli_ca</td>
    <td>epec_koref_bin<br>qnli_eu, wiceu<br>wnli_eu, xnli_eu</td>
    <td>xnli_gl</td>
  </tr>
  <tr>
    <td>Paraphrasing</td>
    <td>paws_es<br>parafrases_sushi</td>
    <td>parafraseja<br>paws_ca</td>
    <td>–</td>
    <td>parafrases_gl<br>paws_gl</td>
  </tr>
  <tr>
    <td>Question answering</td>
    <td>aquas<br>clindiagnoses<br>clintreates<br>spalawex<br>teleia<br>ragquas<br>xquad_es</td>
    <td>arc_ca<br>catalandqa<br>coqcat<br>openbookqa_ca<br>piqa_ca<br>siqa_ca<br>xquad_ca</td>
    <td>bertaqa<br>bhtc_v2<br>eus_exams<br>eus_proficiency<br>eus_trivia<br>vaxx_stance</td>
    <td>openbookqa_gl</td>
  </tr>
  <tr>
    <td>Reading comprehension</td>
    <td>belebele_spa_Latn</td>
    <td>belebele_cat_Latn</td>
    <td>belebele_eus_Latn<br>eus_reading</td>
    <td>belebele_glg_Latn</td>
  </tr>
  <tr>
    <td>Summarization</td>
    <td>noticia<br>xlsum_es</td>
    <td>cabreu</td>
    <td>–</td>
    <td>summarization_gl</td>
  </tr>
  <tr>
    <td>Text classification</td>
    <td>humorqa<br>false_news_es<br>offendes</td>
    <td>catalonia_independence</td>
    <td>bec2016_eu</td>
    <td>–</td>
  </tr>
</table>

Table 1: Datasets of LA LEADERBOARD as of February 2025 organized by task type and language.

<table>
  <tr>
    <th>Task Type</th>
    <th>Dataset</th>
    <th>Languages</th>
  </tr>
  <tr>
    <td>Adaptation</td>
    <td><i>phrases</i></td>
    <td>Catalan, Spanish, Valencian</td>
  </tr>
  <tr>
    <td>Common-sense reasoning</td>
    <td><i>xstorycloze_gl</i></td>
    <td>Galician</td>
  </tr>
  <tr>
    <td>Counter-narrative generation</td>
    <td><i>conan_eus/mt_es refutes</i></td>
    <td>Basque, Spanish<br>Spanish</td>
  </tr>
  <tr>
    <td>Ethics</td>
    <td><i>h4rmony_eval</i></td>
    <td>Spanish</td>
  </tr>
  <tr>
    <td>Text classification</td>
    <td><i>haha</i></td>
    <td>Spanish</td>
  </tr>
  <tr>
    <td>Natural language inference</td>
    <td><i>americasnlp_nli meta4xnli</i></td>
    <td>Aymara, Asháninka, Bribri, Guarani, Náhuatl, Otomi, Quechua, Rarámuri, Shipibo-Konibo, Wixarika<br>Spanish</td>
  </tr>
  <tr>
    <td>Question answering</td>
    <td><i>paes_cl voces_originarias medexpqa quales</i></td>
    <td>Spanish<br>Aymara, Guarani, Tehuelche, Náhuatl, Quechua<br>Spanish<br>Spanish</td>
  </tr>
  <tr>
    <td>Translation</td>
    <td><i>flores americasnlp_mt tradu_latam</i></td>
    <td>Spanish, Catalan, Basque, Galician<br>Spanish, Aymara, Asháninka, Bribri, Guarani, Náhuatl, Otomi, Quechua, Rarámuri, Shipibo-Konibo, Wixarika<br>Spanish, Aymara, Guarani, Tehuelche, Náhuatl, Quechua</td>
  </tr>
  <tr>
    <td>Truthfulness</td>
    <td><i>truthfulqa veritasqa</i></td>
    <td>Spanish, Catalan, Basque, Galician<br>Spanish, Catalan, Galician</td>
  </tr>
</table>

Table 2: Datasets that have been recently donated to LA LEADERBOARD and are not yet included in the evaluation results, including benchmarks involving American Indigenous languages.
3.3 Code Bases

3.3.1 Backend
We acknowledge the cost of running evaluations and want to ensure that any researcher or developer can compare their models to the state-of-the-art and follow their evolution. This is why submitting a model for evaluation is open to the whole community. Once a model has been added to the evaluation queue, the last commit of the model is stored for reproducibility and to enable future comparisons of different versions. The results from the LM Evaluation Harness (Gao et al., 2021) are normalized according to the following formula:

\[
\text{normalized\_value} = \frac{\text{raw\_value} - \text{random\_baseline}}{\text{max\_value} - \text{random\_baseline}}
\]

where \( random\_baseline \) is 0 for generative tasks and \( 1/n \) for MCQA tasks with \( n \) choices.

3.3.2 Frontend
The implementation of LA LEADERBOARD is based on the HuggingFace leaderboard template8. The frontend is developed using Gradio (Abid et al., 2019) and presents the evaluation results categorized by language. To ensure transparency and reproducibility, we share the evaluation command and normalization formula. To bring the tool closer to the community, the information and submission guidelines are available in English and Spanish.

3.3.3 License
Since we want to motivate other communities to create their own, LA LEADERBOARD is published under the permissive Apache 2.0 license9.

3.4 Efficiency Considerations

3.4.1 Number of Few-Shot Examples
Recent literature reveals significant inconsistency in the number of examples (shots) used when evaluating large language models (LLMs). While early research demonstrated notable performance improvements with 3-5 in-context examples (Brown et al., 2020), current evaluation practices vary considerably across different models and benchmarks. For instance, the Open LLM Leaderboard employs 0-5 shots depending on the task, Mistral-7B generally follows this range with an exception of 8 shots for GSM8K (Cobbe et al., 2021), and Llama 3 and OLMo models focus primarily on zero-shot evaluation. In contrast, Gemini models use a broader range of 0-10 shots, including “variable-shot” configurations. This variation extends to language-specific models, with Salamandra10 and Latxa (Extnaniz et al., 2024) families using different shot configurations in their evaluations, typically ranging from 0 to 5 shots.

Given this myriad of options, when choosing the number of shots to use in LA LEADERBOARD , we take into consideration the following aspects:

A. Base vs. instruct models The number of shots should allow for a fair evaluation of the base models without helping instruct models too much. Also, the availability of structured datasets in specific evaluation formats—such as MCQA—is very low in mid- and low-resource languages. This means that models trained on English-heavy corpora are more likely to have encountered these structured formats in English than in other languages, potentially biasing their performance.

B. Cognitive bias Models suffer from cognitive bias depending on the order and options presented as few-shots (Zhao et al., 2021; Pezeshkpour and Hruschka, 2024; Mina et al., 2025). Thus, we ensure that, in MCQA tasks, all possible correct options are included in the in-context learning instances. For example, in an MCQA task with four possible answers, we evaluate on a 4-shot setting, with each shot showing one of the four options as correct, in random order. This is done unless it interferes with item A.

C. Context windows The context window limitations of language models vary significantly based on hardware constraints and architectural choices, affecting their ability to process long-form tasks such as summarization and reading comprehension. For example, while the Spanish government’s 40B-parameter ALIA model11 operates with a 4,096-token context window, Meta’s Llama 3.2 1B model can handle up to 128K tokens12. To ensure fair evaluation across models with different context window capacities, few-shot examples are employed with a maximum limit of 2,048 tokens, following the methodology established in previous research on LLM analysis (Biderman et al., 2023).

D. Prompt format The evaluation methodology employed task-specific prompts from the LM

8https://hf.co/spaces/demo-leaderboard-backend/leaderboard
9https://www.apache.org/licenses/LICENSE-2.0
10https://hf.co/BSC-LT/salamandra-7b-instruct
11https://hf.co/BSC-LT/ALIA-40b
12https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices
Evaluation Harness, with new prompts created for previously unimplemented tasks following established formats and validated by dataset authors. The number of few-shots varied based on prompt complexity: convoluted prompts (e.g., paraphrasing with PAWS and reasoning with COPA) used 3 in-context examples to allow models to understand the task while complying with items A and C (Brown et al., 2020); straightforward question-answering tasks employed 2-shot evaluation, while tasks with explicit, naturally structured prompts (like ClinDiagnosES and NoticIA) and those evaluating sentence continuation probability (e.g., XStoryCloze) were conducted using 0-shot evaluation.

3.4.2 Measuring Model Efficiency

The evaluation was performed using two NVIDIA H100 GPUs with Hopper architecture and 64 GB of HBM memory in the MareNostrum 5 High-Performance Computer13, maintaining identical configurations across instances to ensure consistent measurements. Performance metrics included task execution time and energy consumption, tracked using the Energy Aware Runtime (EAR) package14, with all tasks running at a batch size of 1.

Task execution duration, which includes token prediction time, response length, and tokenizer efficiency, was measured to assess model speed for time-sensitive applications. The duration of task execution is influenced by multiple factors beyond token prediction time, including the response length generated and the language-specific tokenization efficiency (Conde et al., 2024).

Energy consumption was recorded in kWh and converted to CO2 equivalents using the European Commission’s conversion ratio for Spain (0.158 kg CO2/kWh), as the evaluation was conducted in Barcelona (Lottick et al., 2019).

4 Evaluation Results and Analysis

We focus on evaluating models accessible to the broader community. We select 50 open-weights models from various families, primarily ranging from 1B to 9B parameters, while including larger quantized models. We consider multilingual models trained by large technological companies and startups like Meta-Llama (Grattafori et al., 2024) and Mistral (Jiang et al., 2023) as well as models such as EuroLLM (Martins et al., 2024) and Salamandra15, which have been designed by European and Spanish research groups specifically to process our target languages more efficiently and capture cultural nuances. We assess both the base and instruction-tuned versions when available (Appendix C). Each model-task pair was evaluated once, and the raw, pre-normalization results analyzed in this work are publicly available16.

<table>
  <tr>
    <th>Model</th>
    <th>Top10</th>
    <th>ES</th>
    <th>CA</th>
    <th>EU</th>
    <th>GL</th>
  </tr>
  <tr>
    <td>gemma-2-9b-it</td>
    <td>43</td>
    <td>61.69</td>
    <td>57.30</td>
    <td><b>54.13</b></td>
    <td>46.49</td>
  </tr>
  <tr>
    <td>gemma-2-9b</td>
    <td>43</td>
    <td>57.21</td>
    <td><b>59.60</b></td>
    <td>53.80</td>
    <td>48.58</td>
  </tr>
  <tr>
    <td>Meta-Llama-3.1-8B-IT</td>
    <td>41</td>
    <td>59.03</td>
    <td>57.01</td>
    <td>49.87</td>
    <td>45.07</td>
  </tr>
  <tr>
    <td>Qwen2.5-1B-IT-GPTQ-In4</td>
    <td>38</td>
    <td><b>64.06</b></td>
    <td>56.80</td>
    <td>49.23</td>
    <td><b>52.52</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-1B-IT-GPTQ-In8</td>
    <td>35</td>
    <td>60.59</td>
    <td>54.08</td>
    <td>49.05</td>
    <td>52.13</td>
  </tr>
  <tr>
    <td>EuroLLM-1.7B-IT</td>
    <td>30</td>
    <td>58.59</td>
    <td>53.80</td>
    <td>38.92</td>
    <td>46.56</td>
  </tr>
  <tr>
    <td>Meta-Llama-3.1-8B</td>
    <td>28</td>
    <td>55.62</td>
    <td>56.52</td>
    <td>46.90</td>
    <td>44.90</td>
  </tr>
  <tr>
    <td>salamandra-7b *</td>
    <td>26</td>
    <td>52.17</td>
    <td>54.13</td>
    <td>45.80</td>
    <td>39.88</td>
  </tr>
  <tr>
    <td>salamandra-7b-instruct *</td>
    <td>25</td>
    <td>51.41</td>
    <td>53.22</td>
    <td>46.19</td>
    <td>41.65</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>25</td>
    <td>58.79</td>
    <td>57.28</td>
    <td>42.51</td>
    <td>46.82</td>
  </tr>
  <tr>
    <td>aya-expans-8b</td>
    <td>20</td>
    <td>55.42</td>
    <td>53.99</td>
    <td>41.99</td>
    <td>43.38</td>
  </tr>
  <tr>
    <td>salamandra-2b *</td>
    <td>8</td>
    <td>46.60</td>
    <td>41.05</td>
    <td>39.28</td>
    <td>34.99</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>8</td>
    <td>52.60</td>
    <td>47.88</td>
    <td>38.13</td>
    <td>38.79</td>
  </tr>
  <tr>
    <td>gemma-2-2b</td>
    <td>5</td>
    <td>54.85</td>
    <td><b>50.27</b></td>
    <td>39.27</td>
    <td>39.79</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B-IT</td>
    <td>5</td>
    <td>54.38</td>
    <td>45.09</td>
    <td>39.04</td>
    <td>39.27</td>
  </tr>
  <tr>
    <td>salamandra-2b-instruct *</td>
    <td>3</td>
    <td>43.83</td>
    <td>43.28</td>
    <td>38.57</td>
    <td>34.53</td>
  </tr>
  <tr>
    <td>gemma-2-2b</td>
    <td>2</td>
    <td>51.59</td>
    <td><b>50.22</b></td>
    <td>39.81</td>
    <td>39.29</td>
  </tr>
  <tr>
    <td>lemacal-qwen2.1-5B-vd *</td>
    <td>2</td>
    <td>51.14</td>
    <td>40.97</td>
    <td>38.29</td>
    <td>36.16</td>
  </tr>
  <tr>
    <td>EuroLLM-1.7B-IT *</td>
    <td>2</td>
    <td>47.92</td>
    <td>45.28</td>
    <td>36.19</td>
    <td>34.67</td>
  </tr>
</table>

Table 3: Best performing models ranked by the number of tasks in which they achieved a top-10 placement (see “Top10” column), showing their average language performance. Target language–optimized models are indicated by an asterisk (*). Complete per-language top-10 frequencies can be found in Figure 4, and full average scores for all models are shown in Figure 3.

and Mistral (Jiang et al., 2023) as well as models such as EuroLLM (Martins et al., 2024) and Salamandra15, which have been designed by European and Spanish research groups specifically to process our target languages more efficiently and capture cultural nuances. We assess both the base and instruction-tuned versions when available (Appendix C). Each model-task pair was evaluated once, and the raw, pre-normalization results analyzed in this work are publicly available16.

4.1 Model performance

Table 3 shows that the models most consistently ranking in the top 10 across languages are Gemma-2-9B base and instructed, Llama-3.1-8B-IT, and the quantized versions of Qwen-2.5-IT 14B and 32B. When prioritizing transparency, EuroLLM-9B stands out, followed by Salamandra-7B. For resource-constrained settings, Salamandra-2B and Qwen-2.5-1.5B offer the best performance.

Performance per task As illustrated in Figure 2, the evaluation results are generally better for NLI tasks, including textual entailment and paraphrasing, and worse for language proficiency tests, with all four languages having similar performance on both tasks. Within the language proficiency tests, which combine reading comprehension, linguistic

13https://www.bsc.es/ca/marenostrom/marenostrom-5
14https://www.bsc.es/research-and-development/software-and-apps/software-list/ear-energy-management-framework-hpc
15https://hf.co/collections/BSC-LT/salamandra-66fc171485944df79469043a
16https://hf.co/datasets/la-leaderboard/results
acceptability, and summarization, results are particularly low for summarization tasks. In question answering and reasoning tasks, there is a larger inter-language difference, with Galician having significantly lower scores overall, while Basque obtains the best results for reasoning but the second worst for question answering. While commonsense reasoning results are generally good, math reasoning yields the lowest results, which could be related to a too strict metric (exact match). Further analysis is needed to understand whether these differences are due to the datasets used or indeed to the models’ performance. The poor results on language proficiency tests also warrant a detailed examination to understand their implications, as they may indicate fundamental limitations in the models’ knowledge across languages.

Performance per language  Regarding specific models, the same five (Gemma2, Llama3.1, and quantized Qwen2.5) consistently rank among the top ten performers. However, we find that some models stand out for specific language-task pairs. In Spanish, AYA-Expanse-8B joins the top three for QA, EuroLLM-9B for reasoning, and RigoChat-7B is in the top four for language proficiency. In Catalan, EuroLLM ranks among the top three for both QA and reasoning tasks. For Basque, Salamandra-7B-JT is the second-best QA model, while Latxa-7B and Salamandra-7B occupy the first and second positions in reasoning and the third and second positions in classification, respectively. In Galician, EuroLLM-9B is the best model for QA, followed by its instructed version, Salamandra-7B ranks third in reasoning, and AYA-Expanse-8B takes second place in NLI.

Performance by training data configuration  It is valuable to compare the different language configuration strategies in training. Salamandra and EuroLLM, by distributing tokens fairly among languages, achieve homogeneous performance and, in particular, reinforce Galician and Basque, but lose ground in Spanish and Catalan to models supported by high-resource corpora. Gemma2, which prioritizes volume over linguistic diversity, stands out for just the opposite; training on a large but linguistically imbalanced dataset allows the model to transfer patterns learned from English, maths, and code, and consistently rank among the top models overall. This shows that, sometimes, breadth of knowledge can compensate for the lack of language specific data.

The results confirm that the strategy of pre-training from scratch on a large multilingual corpus is the one that offers the most consistent and homogeneous coverage. Qwen-2.5, trained on 29 languages, and Llama-3.1, trained on 8, rank high in all languages. Moreover, systems that use continuous pre-training - adding large-scale data in a new language plus some English data to avoid catastrophic forgetting - achieve peak scores in the target language; e.g., Latxa holds the second-highest mark in Basque reasoning. Finally, models that just fine-tune on target language data without extending the pre-training (e.g., RigoChat or some Instruct variants) gain fluency and style, but the improvement in reasoning and QA tasks is modest, and models rarely enter the top 10. In summary, the earlier and deeper the model is exposed to the target languages, the higher its average scores; a posteriori strategies function as beneficial reinforcers, especially in less-resourced languages, but do not replace the power of extensive multilingual pre-training.

Performance by training compute  Regarding the computational budget and the hardware used, public data is scarce. Regarding large tech companies we can consider Llama-3.1-8B, which was trained for 1.5M GPU-h. By contrast, other evaluated models required three to four orders of magnitude fewer. As expected, the results confirm that teams with access to GPUs/TPUs and high-end accelerators (TPUv5p, H100 or A100 in large clusters) can apply the strategies that we highlighted in the previous section as the most successful - massive multilingual pre-training and continuous pre-training - reflecting in significantly higher average scores. However, the same results also nuance the linear relationship between investment and performance as Salamandra-7B, despite its large budget (600 k GPU-h), falls in the middle of the

![Bar chart showing results per type of task type and language.](page_120_120_646_320.png)

Figure 2: Results per type of task type and language.
table. Overall, the dashboard illustrates that computational abundance enables impactful techniques, but that dataset quality and fine-tuning remain decisive levers for converting that power into real performance.

Performance vs. size In general, our experiments show some correlation between performance and size, with models in the range of 1-2B parameters achieving better scores for their size. This is particularly true for Gemma-2-2B and Qwen-2.5-1.5B, both base and instructed models. Among the top 10 models, we find that all have between 8 and 9 billion parameters, except for the quantized versions from the Qwen family.

Performance per model type For the same VRAM requirement, quantized models with more parameters outperform full-precision models with fewer parameters. We don’t observe any correlation between performance and whether the model is pre-trained or instruction-tuned, as this varies depending on the model.

4.2 Energy consumption

The total computational resources amount to 660.87 hours of processing time and 582.84 kWh of energy consumption, resulting in 92.09 kg of CO₂ emissions. On average, each model consumes 9.25 kWh (median 6.88, SD 8.42), showing a wide variety in energy usage. The models that consume the most energy are Grommeanuer-7B-IT, Qwen-2.5-32B-IT-GPTQ-Int4, and Qwen-2.5-14B-IT-GPTQ-Int8, each exceeding 30 kWh. On the other hand, Salamandra-2b, FLOR-1.3B-IT, and LLama-3.2-1B-IT are the most energy-efficient, consuming less than 2.1 kWh each.

Energy consumption vs. size As expected, a strong correlation between model size and energy dissipation is observed, as the number of arithmetic operations required to predict a token is related to the number of parameters of the model. However, some outliers are observed, such as Qwen, which consumes significantly more energy across all its sizes compared to other models. Conversely, models like FLOR exhibit considerably lower energy consumption across their different sizes relative to other models of similar scale.

Energy consumption vs. task Similarly, as anticipated, text generation tasks such as summarization require more energy for evaluation. Since LLMs generate text token by token and their prediction speed remains constant (assuming the same hardware and stable conditions), the most energy-intensive tasks are expected to be those that require the generation of larger amounts of text.

Energy consumption vs. performance Our experiments show a strong correlation between the energy consumed at inference and the model performance. For one of the overall top models, Gemma-2-9B, its instruction-tuned version excels with a third of the energy consumed by the base version. We observe a trend of instructed models consuming less energy, due to the verbosity of base models.

5 Conclusions and Future Work

In this paper, we propose a methodology to create community-driven leaderboards, including key points to gather diverse datasets and the rationale behind a more efficient and accessible evaluation setup. In doing so, we hope to inspire the creation of more leaderboards that fulfil the needs of diverse linguistic communities.

In particular, we present LA LEADERBOARD , the first open-source leaderboard to evaluate LLMs in languages from Spain and Latin America. It is the result of a collaboration among 13 research groups. LA LEADERBOARD consists of 66 datasets in Spanish, Catalan, Basque, and Galician, and covers a wide range of task types and domains. The results of evaluating 50 LLMs suggest that the top-performing models in the target languages are Gemma-2-9B base and instruct, Llama-3.1-8B-IT, and the quantized versions of Qwen-2.5-IT 14B and 32B. However, for domain-specific applications, particular linguistic contexts, or deployment scenarios with computational constraints or transparency requirements, alternative models (e.g., Salamandra-7B and EuroLLM-9B) demonstrate competitive performance metrics.

Our planned next steps include evaluating the recently donated datasets, with a special focus on indigenous languages. We will also add larger open models and proprietary models. Moreover, we are organizing a hackathon to create a benchmark to measure cultural adequacy in each Spanish-speaking country. Finally, we welcome any person or organization interested in joining our effort. This way, we hope that LA LEADERBOARD will keep evolving to include more languages, language varieties, and use cases that motivate the development of LLMs that better serve our diverse community.
Limitations

Indigenous languages We acknowledge that indigenous languages from Latin America are not yet included among the evaluation results of LA LEADERBOARD. However, we have ongoing collaborations to include existing benchmarks and create new ones to keep extending LA LEADERBOARD to be as inclusive as possible and reflect the diversity of the Spanish-speaking community.

Spanish language varieties Currently, the leaderboard includes datasets in the Spanish varieties of Spain, Mexico, Argentina, Chile, and Uruguay. Although we don’t know the exact origin of all the samples from some third-party datasets, we estimate that less than 25% of all the Spanish datasets in the leaderboard come from LATAM. We plan to increase this percentage by collaborating with LATAM research groups in the creation of an open hackathon.

Large and proprietary models To improve the coverage of the state-of-the-art language models for the use cases included in LA LEADERBOARD, it would be interesting to evaluate larger language models as well as proprietary models.

Contamination Another pending task is to analyse potential contamination (Sainz et al., 2023) within our leaderboard. We have not addressed this yet because a high percentage of the datasets used are very recent and niche, making it unlikely that they have been incorporated into training data, unlike more established benchmarks such as MMLU (Hendrycks et al., 2021a; Wang et al., 2024b; Taghanaki et al., 2024) that serve as primary pillars in model evaluation in every model report. Nevertheless, we have started to evaluate contamination to ensure in the short-term future that we provide high-quality results.

For the datasets specifically created for LA LEADERBOARD, we advised the corresponding authors to release them gated to avoid being unintentionally included in training datasets by web scraping; AQuAS and RagQuAS are gated. The authors of TELEIA decided to release an adaptation of their dataset and keep the original private to be able to analyze contamination through time.

Ethical Considerations

Fair representation Since our objective is to establish an evaluation standard for Latin America and Spain, it is important to properly represent the linguistic and cultural diversity of the community in order to avoid the perpetuation, or even amplification, of stereotypes and inequalities.

Third-party datasets Some of the evaluation datasets included in LA LEADERBOARD were created by organizations other than our data contributors. As a result, we acknowledge the possibility that some of these datasets may have been developed using practices that could be considered unethical. These concerns range from potential legal violations to extractive data collection methods that may impact disadvantaged communities.

Environmental impact Evaluating 50 language models on 66 tasks required 660.87 hours of compute, translating to 92.09 kg of CO₂. However, we hope that by publishing a comprehensive evaluation of the available models, LA LEADERBOARD will contribute to reducing the total environmental impact of individual private evaluations.

Misuse of La Leaderboard We welcome model submissions from everyone. This could potentially lead to overuse, with people sending many different versions of the same model. We plan to mitigate this behaviour by following the spam mitigation strategies from the Open LLM Leaderboard (Fourrier et al., 2024).

Acknowledgments

LA LEADERBOARD is the result of the collaboration between many researchers, and we extend our heartfelt thanks to each of them.

Coordination María Grandury, Javier Aula-Blasco, Clémentine Fourrier, Marta Guerrero, Pedro Reviriego.

Dataset donation leads Javier Aula-Blasco, Omar Sansevierio, Alejandro Vaca, Marta Guerrero, Álvaro Barbero, Rodrigo Agerri, Javier Conde, Flor Miriam Plaza-del-Arco, María Teresa Martín-Valdivia, Helena Gómez, Luciana Benotti, Iker García-Ferrero, Jorge Vallego, Luis Chiruzzo.

New dataset creation and annotation Carmen Muñoz, Helena Montoro, Leire Rosado, Marta Guerrero, Nuria Aldama, Natàlia Fuertes, Alejandro Vaca, Marina Mayor-Rocher, Nina Melero,
Elena Merino-Gómez, Miguel González, Raquel Ferrando, Javier Conde, Gonzalo Martínez, Pedro Reviriego Daniel Vila, Álvaro Bartolomé, Ignacio Talavera.

Metric and task implementation  Gonzalo Santamaría, David Betancur, Anna Sallés, Irene Bau-cells, Javier Aula-Blasco, Julen Etxaniz, Alejandro Vaca, María Grandury, Gonzalo Martínez, Miguel González, Iker García-Ferrero, Guido Ivetta. María Estrella Vallecillo-Rodríguez, Irune Zubiaga

Leaderboard implementation  María Grandury, Clémantine Fourrier, Nathan Habib.

Model evaluation  Júlia Falcão, María Grandury, Miguel González, Gonzalo Martínez, Javier Aula-Blasco, Bram Vanroy.

Paper writing  All the co-authors, especially María Grandury, Rodrigo Agerri, Pedro Reviriego, Javier Aula-Blasco, Flor Miriam Plaza-del-Arco, Clémetine Fourrier, Gonzalo Martínez, Luis Chiruzzo, Alejandro Vaca, Carmen Muñoz, Marta Guerrero, Javier Conde, Guido Ivetta, Júlia Falcão, Álvaro Barbero, Manuel Mager.

Communication  Maria Sayavera, Jaume Lozano, Florent Daudens, Brigitte Tousignant, Rafael Muñoz, Valentín Cardeñoso.

Computational resources  Barcelona Supercomputing Center, Universidad Politécnica de Madrid, HuggingFace, The Flemish Supercomputer Center.

Funding  The work of the co-authors of this paper has been supported by multiple projects.

• The coordination and implementation of LA LEADERBOARD by María Grandury was carried out on a voluntary basis for six months, after which her work was supported by the Universidad Politécnica de Madrid (UPM).

• The work by the UPM team was supported by the Agencia Estatal de Investigación (AEI) (doi:10.13039/501100011033) under Grants FUN4DATE (PID2022-136684OB-C22) and SMARTY (PCI2024-153434) and by the European Commission through the Chips Act Joint Undertaking project SMARTY (Grant 101140087).

• The work by the Barcelona Supercomputing Center team and the datasets donated by ILENIA are funded by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia - Funded by EU – NextGenerationEU within the framework of the project ILE-NIA with references 2022/TL22/00215337, 2022/TL22/00215336, 2022/TL22/00215335 and 2022/TL22/00215334, and within the framework of the project Desarrollo Modelos ALIA.

• The work of LenguajeNatural.AI is self-funded and self-supported by the company’s resources.

• The contributions of Rodrigo Agerri and Irune Zubiaga are partially funded by the following MCIN/AEI/10.13039/501100011033 projects: (i) DeepKnowledge (PID2021-127777OB-C21) and ERDF A way of making Europe; (ii) DeepMinor (CNS2023-144375) and European Union NextGenerationEU/PRTR. Irune Zubiaga is supported by a UPV/EHU PIF 2025 grant.

• The contributions of Guido Ivetta are supported by Fundación Vía Libre and Universidad Nacional de Córdoba, Argentina.

• The work by the SINAI team has been partially supported by projects CONSENSO (PID2021-122263OB-C21), MODERATES (TED2021-130145B-I00), SocialTOX (PDC2022-133146-C21) funded by Plan Nacional I+D+i from the Spanish Government.

• The contributions of Jorge Vallego are supported by the H4rmory project and the Asociación Latinoamericana de Ecolingüística.

Community  This project is a community-driven effort that continues to evolve. We thank everyone who has provided valuable feedback via ACL Rolling Review (ARR), the SomosNLP Discord community, the interface’s Discussions tab, and in person at the KHIPU 2025 conference and the NAACL 2025 workshop "Language Models for Underserved Communities". We are grateful for the support, and invite the entire Ibero-American NLP community to join us.
References

Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. 2019. Gradio: Hassle-free sharing and testing of ml models in the wild. arXiv preprint arXiv:1906.02569.

Rodrigo Agerri, Roberto Centeno, María S. Espinosa, Joseba Fernandez de Landa, and Álvaro Rodrigo. 2021. VaxxStance@IberLEF 2021: Overview of the Task on Going Beyond Text in Cross-Lingual Stance Detection. Procesamiento del Lenguaje Natural, 67:173–181.

Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Ruttmann, Charvi Jain, Max Lübbing, Daniel Steinigen, Johannes Leveling, Katrin Klug, Jasper Schulze Buschhoff, Lena Jurkschat, Hammam Abdelwhab, Benny Jörg Stein, Karl-Heinz Sylla, Pavel Denisov, Nicolo’ Brandizzi, Qasid Saleem, Anirban Bhownick, Lennard Helmer, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Alex Jude, Lalith Manjunath, Samuel Weinbach, Carolin Penke, Oleg Filatov, Shima Asaadi, Fabio Barth, Rafet Sifa, Fabian Küch, Andreas Herten, René Jakel, Georg Rehm, Stefan Kesselheim, Joachim Köhler, and Nicolas Flores-Herr. 2024. Teuken-7b-base & teuken-7b-instruct: Towards european llms. Preprint, arXiv:2410.03730.

Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kyliček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2025. Smollm2: When smol goes big – data-centric training of a small language model. Preprint, arXiv:2502.02737.

Íñigo Alonso, Maite Oronoz, and Rodrigo Agerri. 2024. MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering. Artificial Intelligence in Medicine, 155:102938.

Jordi Armengol-Estepá, Casimiro Pio Carrino, Carlos Rodríguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Merelo, and Marta Villegas. 2021. Are multilingual models the best choice for moderately under-resourced languages? A comprehensive assessment for Catalan. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4933–4946, Online. Association for Computational Linguistics.

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.

Javier Aula-Blasco, Júlia Falcão, Susana Sotelo, Silvia Paniagua, Aitor Gonzalez-Agirre, and Marta Villegas. 2025. VeritasQA: A truthfulness benchmark aimed at multilingual transferability. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5463–5474, Abu Dhabi, UAE. Association for Computational Linguistics.

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics.

Fabio Barth, Manuel Brack, Maurice Kraus, Pedro Ortiz Suarez, Malte Ostendorf, Patrick Schramowski, and Georg Rehm. 2024. Occiglot euro llm leaderboard.

Irene Baucells, Javier Aula-Blasco, Iria de Dios-Flores, Silvia Paniagua Suárez, Naïara Perez, Anna Salles, Susana Sotelo Docio, Júlia Falcão, Jose Javier Saiz, Robiert Sepulveda Torres, Jeremy Barnes, Pablo Gamallo, Aitor Gonzalez-Agirre, German Rigau, and Marta Villegas. 2025. IberoBench: A benchmark for LLM evaluation in Iberian languages. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10491–10519, Abu Dhabi, UAE. Association for Computational Linguistics.

Nuria Bel, Marta Puntsola, and Valle Ruiz-Fernández. 2024a. EsCoLA: Spanish corpus of linguistic acceptability. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6268–6277, Torino, Italia. ELRA and ICCL.

Núria Bel, Marta Puntsola, and Valle Ruiz-Fernández. 2024b. CatCoLA: Catalan corpus of linguistic acceptability. Procesamiento del Lenguaje Natural, 73.

Jaione Bengoetxea, Yi-Ling Chung, Marco Guerini, and Rodrigo Agerri. 2024. Basque and Spanish Counter Narrative Generation: Data Creation and Evaluation. In Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING).

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Alfah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. Preprint, arXiv:2304.01373.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.

Lyle Campbell. 2010. Language isolates and their history, or, what’s weird, anyway? In Proceedings of the Annual Meeting of the Berkeley Linguistics Society, volume 36, pages 16–31. Linguistic Society of America.

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132.

Luis Chiruzzo, Santiago Castro, Santiago Góngora, Aiala Rosá, JA Meaney, and Rada Mihalcea. 2021. Overview of haha at iberlet 2021: Detecting, rating and analyzing humor in spanish. Procesamiento del Lenguaje Natural, 67:257–268.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.

Javier Conde, Miguel González, Pedro Reviriego, Zhen Gao, Shanshan Liu, and Fabrizio Lombardi. 2024. Speed and conversational large language models: Not all is about tokens per second. Computer, 57(8):74–80.

OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass.

Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrau, Gabriel Mejía Gonzalez, Prangthip Hansanti, John Hoffman, Searmary Jarret, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Sprout, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.

Severino Da Dalt, Joan Llop, Irene Baucells, Marc Pamies, Yishi Xu, Aitor Gonzalez-Agirre, and Marta Villegas. 2024. FLOR: On the effectiveness of language adaptation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7377–7388, Torino, Italia. ELRA and ICCL.

John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarkar Talupuru, Bharat Venkitesh, David Caizur, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosts, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, and Sara Hooker. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.

Iria de Dios-Flores, Juan Garcia Amoage, and Marcos Garcia. 2023. Dependency resolution at the syntax-semantics interface: psycholinguistic and computational insights on control dependencies. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 203–222, Toronto, Canada. Association for Computational Linguistics.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Leen Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shutting Pan, S. S. Li, Shuang Zhou, Shaqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu,
Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinran Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yu-jia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yuxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Shla, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zhihu Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipei Xu, Zhongyu Zhang, and Zhen Zhang. 2025. Deepseek-r1: Incen-tivizing reasoning capability in llms via reinforce-ment learning. Preprint, arXiv:2501.12948.

Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Meza-Ruiz, et al. 2021. Americasnli: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. arXiv preprint arXiv:2104.08726.

Ali Elfilali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. 2024. Open ara-bic llm leaderboard. https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard.

Julen Etxaniz, Oscar Sainz, Naiara Perez, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormazabal, Mikel Artetxe, and Aitor Soroa. 2024. Latxa: An open language model and evaluation suite for basque. Preprint, arXiv:2403.20266.

Francisco Moreno Fernández and Héctor Álvarez Mella. 2024. Demografía del español en el mundo 2024. In El español en el mundo. Anuario del Instituto Cervantes 2024, pages 30–97. Instituto Cervantes.

Blanca Calvo Figueras, Eneko Sagarrazu, Julen Etxaniz, Jeremy Barnes, Pablo Gamallo, Iria De Dios Flores, and Rodrigo Agerri. 2025. Truth Knows No Language: Evaluating Truthfulness Beyond English. In ACL.

Clémentine Fourrier, Nathan Habib, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. 2023. Lighteval: A lightweight framework for llm evaluation.

Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard.

Pablo Gamallo, Pablo Rodríguez, Iria de Dios-Flores, Susana Sotelo, Silvia Paniagua, Daniel Bardanca, José Ramon Pichel, and Marcos Garcia. 2024. Open generative large language models for galician. Preprint, arXiv:2406.13893.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.

Iker García-Ferrero and Begoña Altuna. 2024. Noticia: A clickbait article summarization dataset in spanish. Preprint, arXiv:2404.07611.

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86–92.

Aitor Gonzalez-Agirre, Montserrat Marimon, Carlos Rodriguez-Penagos, Javier Aula-Blasco, Irene Bau-cells, Carme Armentano-Oller, Jorge Palomar-Giner, Baybars Kulebi, and Marta Villegas. 2024. Building a data infrastructure for a mid-resource language: The case of Catalan. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2556–2566, Torino, Italia. ELRA and ICCL.

María Grandury. 2024. The #somos600m project: Generating NLP resources that represent the diversity of the languages from LATAM, the Caribbean, and Spain. In North American Chapter of the Association for Computational Linguistics Conference: LatinX in AI (LXAI) Research Workshop.

Aaron Grattafiori et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736.

Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubashir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. In International Conference on Learning Representations.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. Preprint, arXiv:2103.03874.

Maite Heredia, Julen Etxaniz, Muitze Zulaika, Xabier Saralegi, Jeremy Barnes, and Aitor Soroa. 2024. XN-LLeu: a dataset for cross-lingual NLI in Basque. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4177–4188. Mexico City, Mexico. Association for Computational Linguistics.

Carolyn Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. 2024. Evaluating the elementary multilingual capabilities of large language models with multil. arXiv preprint arXiv:2403.03814.

Instituto de Ingeniería del Conocimiento. 2024a. Abstractive Question-Answering in Spanish (AQuAS) dataset.

Instituto de Ingeniería del Conocimiento. 2024b. Retrieval-Augmented-Generation and Question-Answering in Spanish (RagQuAS) Dataset.

Instituto de Ingeniería del Conocimiento. 2025. Rigochat-7b-v2.

Krzysztof Jassem, Michał Ciesiółka, Filip Graliński, Piotr Jabłoński, Jakub Pokrywka, Marek Kubis, Monika Jabłońska, and Ryszard Staruch. 2025. Llmzszl: a comprehensive llm benchmark for polish. Preprint, arXiv:2501.02266.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Aisha Khatun and Daniel G. Brown. 2024. A study on large language models’ limitations in multiple-choice question answering. Preprint, arXiv:2401.07955.

Hyeonwoo Kim, Dahyun Kim, Jihoon Kim, Sukyung Lee, Yungi Kim, and Chanjun Park. 2024. Open koi-llm leaderboard2: Bridging foundational and practical evaluation for korean llms. arXiv preprint arXiv:2410.12445.

Javier De la Rosa, Eduardo G. Ponzferrada, Manu Romero, Paulo Villegas, Pablo González de Prado Salas, and María Grandury. 2022. Bertin: Efficient pre-training of a spanish language model using perplexity sampling. Procesamiento del Lenguaje Natural, 68(0):13–23.

Latam-GPT. 2025. paes (revision 2e7fda0).

LenguajeNatural.AI. 2024a. HumorQA.

LenguajeNatural.AI. 2024b. MedicalExpertES.

LenguajeNatural.AI. 2024c. SpaLawEx.

Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024. Can multiple-choice questions really be useful in detecting the abilities of llms? Preprint, arXiv:2403.17752.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekogull, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023. Holistic evaluation of language models. Transactions on Machine Learning Research. Featured Certification, Expert Certification.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Xi Victoria Lin, Todor Mihaylov, Mikael Artetxe, Tianlu Wang, Shuhui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Kadan Lottick, Silvia Susai, Sorelle A Friedler, and Jonathan P Wilson. 2019. Energy usage reports: Environmental awareness as part of algorithmic accountability. arXiv preprint arXiv:1911.08354.

Wolf Lustig. 1996. Mba’ëichapa oiko la guarani? guaraní y jopara en el paraguay. URL: www. staff.uni-mainz.de/lustig/guarani/art/yopara.pdf.–20 p.

Manuel Mager, Ximena Gutierrez-Vasques, Gerardo Sierra, and Ivan Meza. 2018. Challenges of language technologies for the indigenous languages of the americas. arXiv preprint arXiv:1806.04291.
Manuel Mager, Arturo Oncevay, Abteen Ebrahimi, John Ortega, Annette Rios Gonzales, Angela Fan, Ximena Gutierrez-Vasques, Luis Chiruzzo, Gustavo Giménez-Lugo, Ricardo Ramos, et al. 2021. Findings of the americasnlp 2021 shared task on open machine translation for indigenous languages of the americas. In Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, pages 202–217.

Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. Eurolm: Multilingual language models for europe. Preprint, arXiv:2409.16235.

Marina Mayor-Rocher, Nina Melero, Elena Merino-Gómez, Miguel González, Raquel Ferrando, Javier Conde, and Pedro Reviriego. 2025. Teleia: A spanish language dataset for evaluating artificial intelligence models. Data in Brief, page 111437.

Fabio Mercorio, Mario Mezzananzaica, Daniele Poterti, Antonio Serino, and Andrea Seveso. 2024. Disce aut deficere: Evaluating llms proficiency on the invalsi italian benchmark. arXiv preprint arXiv:2406.17535.

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. Preprint, arXiv:2311.12983.

Mario Mina, Valle Ruiz-Fernández, Júlia Falcão, Luis Vasquez-Reina, and Aitor Gonzalez-Agirre. 2025. Cognitive biases, task complexity, and result interpretability in large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1767–1784, Abu Dhabi, UAE. Association for Computational Linguistics.

Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinev Ali Ayele, Víctor Gutiérrez-Basulto, Yazmín Ibáñez-García, Hwaran Lee, Shamsuddene Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilehvar, Nedjma Ousidhoum, Jose Camacho-Collados, and Alice Oh. 2025. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. Preprint, arXiv:2406.09948.

Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. Gsws-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics.

OpenAI. 2023. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Ankit Pal, Pasquale Minervini, Andreas Geert Motzfeldt, and Beatrice Alex. 2024. openlife-scienceai/open_medical_llm_leaderboard.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311–318, USA. Association for Computational Linguistics.

Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics.

Irene Plaza, Nina Melero, Cristina del Pozo, Javier Conde, Pedro Reviriego, Marina Mayor-Rocher, and Marfa Grandury. 2024. Spanish and llm benchmarks: is mmlu lost in translation? arXiv preprint arXiv:2406.17789.

Flor Miriam Plaza-del-Arco, Arturo Montejo-Ráez, L. Alfonso Ureña-López, and María-Teresa Martín-Valdivia. 2021. OffendES: A new corpus in Spanish for offensive language research. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1096–1108, Held Online.

Flor Miriam Plaza-del-Arco, Carlo Strapparava, L. Alfonso Urena Lopez, and Maite Martin. 2020. EmoEvent: A multilingual emotion corpus based on different events. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1492–1498, Marseille, France. European Language Resources Association.

Posadas-Durán, Helena Gómez-Adorno, Sidorov, and Escobar. 2019. Detection of fake news in a new corpus for the spanish language. Journal of Intelligent & Fuzzy Systems, 36(5):4869–4876.

Amandine Quercia, Jamil Zaghiri, Christian Lovis, and Christophe Gaudet-Blavignac. 2024. Medfrenchmark, a small set for benchmarking generative llms in medical french. In Digital Health and Informatics Innovations for Sustainable Health Care Systems, pages 601–605. IOS Press.

Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. 2021. Semantic answer similarity for evaluating question answering models. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 149–157, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussonot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv e-prints, pages arXiv–2408.
Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalka Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A Haggag, Alfonso Amayuelas, et al. 2024. Include: Evaluating multilingual language understanding with regional knowledge. arXiv preprint arXiv:2411.19799.

David Romero, Chenyang Lyu, Haryo Akbarianti Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwu Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorir Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Biadgign Ademtew, Hernán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D’Haro, Marcelo Viridiano, Marcos Estecha-Garitagotia, Maria Camila Buitrago Cabrera, Mario Rodríguez-Cantelar, Mélanie Jouitteau, Mihail Mihaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazurda, Munkhjargal Gocho, Munkh-Erdene Otgonbold, Naome Etori, Olivier Niyomugisha, Paula Mónica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijaya, Santiago Góngora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuribayashi, Teresa Clifford, Thamyan Jayakumar, Tiago Timponi Torrent, Toqueer Ehsan, Vladimír Araújo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio, and Alham Fikri Aji. 2024. Cvqa: Culturally-diverse multilingual visual question answering benchmark. Preprint, arXiv:2406.05967.

Aiala Rosá, Luis Chiruzzo, Lucía Bouza, Alina Dragonetti, Santiago Castro, Mathias Etcheverry, Santiago Góngora, Santiago Goycochea, Juan Machado, Guillermo Moncecchi, et al. 2022. Overview of quales at iberlef 2022: Question answering learning from examples in spanish.

Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. Preprint, arXiv:1907.10641.

Israfel Salazar, Manuel Fernández Burda, Shayekh Bin Islam, Arshia Soltani Moakhar, Shivalka Singh, Fabian Farestam, Angelika Romanou, Danylo Boiko, Dipika Khullar, Mike Zhang, Dominik Krzeminski, Jekaterina Novikova, Luísa Shimabucoro, Joseph Marvin Imperial, Rishabh Maheshwary, Sharad Duwal, Alfonso Amayuelas, Swati Rajiwal, Jebish Purvey, Ahmed Ruby, Nicholas Popovic, Marek Suppa, Azmine Toushik Wasi, Ram Mohan Rao Kadiyala, Olga Tsymboi, Maksim Kostritsya, Bardia Soltani Moakhar, Gabriel da Costa Merlin, Otávio Ferracioli Coletti, Maral Jabbari Shivari, MohammadAmin farahani fard, Silvia Fernandez, María Grandury, Dmitry Abulkhanov, Drishti Sharma, Andre Guarnieri De Mitri, Leticia Bossatto Marchesi, Setayesh Heydari, Johan Obando-Ceron, Nazar Kohut, Beyza Ermis, Desmond Elliott, Enzo Ferrante, Sara Hooker, and Marzieh Fadaee. 2025. Kaleidoscope: In-language exams for massively multilingual vision evaluation. Preprint, arXiv:2504.07072.

Eva Sánchez Salido, Roser Morante, Julio Gonzalo, Guillermo Marco, Jorge Carrillo de Albornoz, Laura Plaza, Enrique Amigó, Andrés Fernández, Alejandro Benito-Santos, Adrián Ghajari Espinosa, and Victor Fresno. 2025. Bilingual evaluation of language models on general knowledge in university entrance exams with minimal contamination. Preprint, arXiv:2409.12746.

Elisa Sanchez-Bayona and Rodrigo Agerri. 2024. Meta4XXNL: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation. Preprint, arXiv:2404.07053.

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Sorosh Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Rudner, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations.

Shivalka Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngu, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. 2024. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. Preprint, arXiv:2412.03304.

Aarohi Srivastava and et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.

Stephanie Strassel and Jennifer Tracey. 2016. LORELEI language packs: Data, tools, and resources for technology development in low resource languages. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3273–3280, Portorož, Slovenia. European Language Resources Association (ELRA).
Saeid Asgari Taghanaki, Aliasaghr Khani, and Amir Khasahmadi. 2024. Mmlu-pro+.: Evaluating higher-order reasoning and shortcut learning in llms. arXiv preprint arXiv:2409.02257.

Gorka Urbizu, Ihaki San Vicente, Xabier Saralegi, Rodrigo Agerri, and Aitor Soroa. 2022. BasqueGLUE: A Natural Language Understanding Benchmark for Basque. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1603–1612.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amarpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537.

Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024a. “my answer is C”: First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7407–7416, Bangkok, Thailand. Association for Computational Linguistics.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyian Jiang, et al. 2024b. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574.

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.

Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. Preprint, arXiv:2102.09690.

Elena Zotova, Rodrigo Agerri, and German Rigau. 2021. Semi-automatic Generation of Multilingual Datasets for Stance Detection in Twitter. Expert Systems with Applications, 170:114547.

Irune Zubiaaga, Aitor Soroa, and Rodrigo Agerri. 2024. A LLM-based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9572–9585.
A  Evaluation Datasets

The datasets are used only for evaluation, aligning with their intended uses.

Spanish datasets
The Spanish datasets in LA LEADERBOARD are:
AQuAS (Instituto de Ingeniería del Conocimiento, 2024a), Belebele (Bandarkar et al., 2024), EsCoLA (Bel et al., 2024a), Fake News ES (Posadas-Durán et al., 2019), FLORES-200 (Costa-jussà et al., 2022), ClinTreatES and ClinDiagnosES (LenguajeNatural.AI, 2024b), HumorQA (LenguajeNatural.AI, 2024a), MGSM (Shi et al., 2023), MultiLingualCrowsPairs (Nangia et al., 2020), NoticIA (García-Ferrero and Altuna, 2024), OfendES (Plaza-del-Arco et al., 2021), RagQuAS (Instituto de Ingeniería del Conocimiento, 2024b), SpaLawEx (LenguajeNatural.AI, 2024c), TELEIA (Mayor-Rocher et al., 2025), WNLI (Gonzalez-Agirre et al., 2024; Baucells et al., 2025)17, XL-Sum (Hasan et al., 2021), XStoryCloze (Lin et al., 2022; Baucells et al., 2025), and XQuAD (Artetxe et al., 2020).

Catalan datasets
The Catalan datasets in LA LEADERBOARD are: caBREU, CatalanQA, COPA-ca, CoQCat, PAWS-ca, TE-ca, WNLI-ca and XNLI-ca (Gonzalez-Agirre et al., 2024), IberoBench (Baucells et al., 2025), CatCoLA (Bel et al., 2024b), FLORES-200 (Costa-jussà et al., 2022), MGSM (Shi et al., 2023), XStoryCloze (Lin et al., 2022), XQuAD-ca (Armengol-Estapé et al., 2021), XStoryCloze (Lin et al., 2022; Baucells et al., 2025), Parafraseja18, PAWS-X (Yang et al., 2019), and VeritasQA (Aula-Blasco et al., 2025).

Basque datasets
The Basque datasets in LA LEADERBOARD are: EusExams, EusReading, EusProficiency and EusTrivia from Etxaniz et al. (2024); BEC2016eu, BHTCv2, EpecKorrefBin, QNLIeu, WiCeu from BasqueGlue (Urbizu et al., 2022); QNLI-eu (Urbizu et al., 2022), VaxxStance (Agerri et al., 2021), XNLIeu (Heredia et al., 2024), FLORES-200 (Costa-jussà et al., 2022), MGSM (Shi et al., 2023), and XStoryCloze (Lin et al., 2022; Baucells et al., 2025).

Galician datasets
The Galician datasets in LA LEADERBOARD are: FLORES-200 (Costa-jussà et al., 2022), GalCoLA (de Dios-Flores et al., 2023), TruthfulQA-GL19, and XStoryCloze (Lin et al., 2022; Baucells et al., 2025)20.

Datasets created for La Leaderboard
The 7 datasets specifically created for the initial version of LA LEADERBOARD are AQuAS, ClinDiaqES, ClinTreatES, HumorQA, RagQuAS, SpaLawEx, and TELEIA. Their corresponding datasheets are included in Appendix F.

Newly donated datasets
The new datasets donated will be evaluated shortly. These include CONAN-EUS (Bengoetxea et al., 2024), RefutES21, TruthfulQA in Basque, Catalan, Galician and Spanish (Figueras et al., 2025), VeritasQA (Aula-Blasco et al., 2025), PAES Chile (Latam-GPT, 2025), meta4xnl (Sanchez-Bayona and Agerri, 2024), MedExpQA (Alonso et al., 2024), Catalonia Independence Corpus (CIC) in Catalan and Spanish (Zotova et al., 2021), HAHA humor detection and analysis in Spanish (Chiruzzo et al., 2021), QuALES for question-answering in Spanish in the COVID-19 domain (Rosá et al., 2022), AmericasNLP-MT (Mager et al., 2021), AmericasNLI (Ebrahimi et al., 2021), Tradu-LATAM, and VocesOriginarias evaluating indigenous languages.

Evaluation dataset details
The Tables 4 (Spanish), 5 (Catalan), 6 (Basque), and 7 (Galician) list these datasets, providing additional information about their task type, domain, and origin. We run the evaluations using our fork of the LM Evaluation Harness22, synced with the main repository on commit 6ecd520f3fb2b5d74c6f14e05f9d189521424719. The tables mentioned also include details about the evaluation configuration, providing the Harness task ID, metric, and number of shots.

17For Spanish, see https://hf.co/datasets/PlanTL-GOB-ES/wnli-es.
18https://hf.co/datasets/projecte-aina/Parafraseja
19https://hf.co/datasets/proxectonos/truthfulqa_gl
20For Galician, see https://hf.co/datasets/proxectonos/xstorycloze_gl.
21https://hf.co/datasets/SINAI/RefutES
22https://github.com/somosnlp/lm-evaluation-harness
<table>
  <tr>
    <th>Dataset</th>
    <th>Task</th>
    <th>Metric</th>
    <th>Domain</th>
    <th>Origin</th>
    <th>#Examples</th>
    <th>#Shots</th>
  </tr>
  <tr>
    <td>AQaAS</td>
    <td>Abstractive QA, Long Form QA</td>
    <td>sas_encoder</td>
    <td>Miscellaneous</td>
    <td>Original</td>
    <td>87</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Belebele Spa</td>
    <td>Reading Comprehension</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Human translation</td>
    <td>900</td>
    <td>2</td>
  </tr>
  <tr>
    <td>ClinDiagnOS</td>
    <td>Long Form QA</td>
    <td>sas_encoder</td>
    <td>Clinical</td>
    <td>Original</td>
    <td>62</td>
    <td>0</td>
  </tr>
  <tr>
    <td>ClinTriSES</td>
    <td>Long Form QA</td>
    <td>sas_encoder</td>
    <td>Clinical</td>
    <td>Original</td>
    <td>62</td>
    <td>0</td>
  </tr>
  <tr>
    <td>CoNLL_es</td>
    <td>Commonsense Reasoning</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Human translation</td>
    <td>500</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Crows Pairs Spanish</td>
    <td>Stereotype Detection</td>
    <td>pct_stereotype</td>
    <td>Ethnic, Hate speech</td>
    <td>Original</td>
    <td>1509</td>
    <td>0</td>
  </tr>
  <tr>
    <td>EsCoLa</td>
    <td>Linguistic Acceptability</td>
    <td>mce</td>
    <td>Language proficiency</td>
    <td>Original</td>
    <td>1060</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Fake News ES</td>
    <td>Fake News Detection</td>
    <td>acc</td>
    <td>Press</td>
    <td>Original</td>
    <td>572</td>
    <td>2</td>
  </tr>
  <tr>
    <td>HumorQA</td>
    <td>Humor Classification</td>
    <td>acc</td>
    <td>Language proficiency</td>
    <td>Original</td>
    <td>51</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MGSM_es</td>
    <td>Math Reasoning</td>
    <td>exact_match</td>
    <td>Math</td>
    <td>Human translation</td>
    <td>250</td>
    <td>2</td>
  </tr>
  <tr>
    <td>NotiCa</td>
    <td>Summarization</td>
    <td>rouge1</td>
    <td>Lang. prof., Press</td>
    <td>Original</td>
    <td>100</td>
    <td>0</td>
  </tr>
  <tr>
    <td>PAWS-ES</td>
    <td>Hate Speech Detection</td>
    <td>acc</td>
    <td>Hate speech</td>
    <td>Original</td>
    <td>15600</td>
    <td>2</td>
  </tr>
  <tr>
    <td>OpenBookQA_es</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>General knowledge</td>
    <td>Human translation</td>
    <td>500</td>
    <td>0</td>
  </tr>
  <tr>
    <td>PAWS-X_es</td>
    <td>Paraphrasing</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Original</td>
    <td>2000</td>
    <td>3</td>
  </tr>
  <tr>
    <td>RagQuAs</td>
    <td>Abstractive QA, Long Form QA</td>
    <td>sas_encoder</td>
    <td>Miscellaneous</td>
    <td>Original</td>
    <td>201</td>
    <td>1</td>
  </tr>
  <tr>
    <td>SpaLawEx</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>Legal</td>
    <td>Original</td>
    <td>119</td>
    <td>0</td>
  </tr>
  <tr>
    <td>TELEHA</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>Gen. knowl., Lang. prof.</td>
    <td>Original</td>
    <td>100</td>
    <td>2</td>
  </tr>
  <tr>
    <td>WNLI_ES</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Human translation</td>
    <td>146</td>
    <td>2</td>
  </tr>
  <tr>
    <td>XL-SuEs</td>
    <td>Summarization</td>
    <td>bleu</td>
    <td>Press</td>
    <td>Original</td>
    <td>4763</td>
    <td>1</td>
  </tr>
  <tr>
    <td>XNLJ_es</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Original</td>
    <td>5010</td>
    <td>3</td>
  </tr>
  <tr>
    <td>XQuAD_es</td>
    <td>Extractive QA</td>
    <td>f1</td>
    <td>Miscellaneous</td>
    <td>Original</td>
    <td>1190</td>
    <td>2</td>
  </tr>
  <tr>
    <td>xStoryCloze_es</td>
    <td>Commonsense Reasoning</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Human translation</td>
    <td>1510</td>
    <td>0</td>
  </tr>
</table>

Table 4: Details of the evaluation datasets in Spanish (ES).

<table>
  <tr>
    <th>Dataset</th>
    <th>Task</th>
    <th>Metric</th>
    <th>Domain</th>
    <th>Origin</th>
    <th>#Examples</th>
    <th>#Shots</th>
  </tr>
  <tr>
    <td>ARC_ca</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>Science</td>
    <td>Human translation</td>
    <td>869</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Belebele Cat</td>
    <td>Reading Comprehension</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Human translation</td>
    <td>900</td>
    <td>2</td>
  </tr>
  <tr>
    <td>caBREU</td>
    <td>Summarization</td>
    <td>bleu</td>
    <td>Press</td>
    <td>Original</td>
    <td>301</td>
    <td>1</td>
  </tr>
  <tr>
    <td>CatalanQA</td>
    <td>Extractive QA</td>
    <td>f1</td>
    <td>Miscellaneous</td>
    <td>Original</td>
    <td>2135</td>
    <td>2</td>
  </tr>
  <tr>
    <td>CatCoLa</td>
    <td>Linguistic Acceptability</td>
    <td>mce</td>
    <td>Language proficiency</td>
    <td>Original</td>
    <td>1020</td>
    <td>2</td>
  </tr>
  <tr>
    <td>CoNLL_ca</td>
    <td>Commonsense Reasoning</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Human translation</td>
    <td>500</td>
    <td>3</td>
  </tr>
  <tr>
    <td>CoQCa</td>
    <td>Extractive QA</td>
    <td>f1</td>
    <td>Miscellaneous</td>
    <td>Original</td>
    <td>8986</td>
    <td>1</td>
  </tr>
  <tr>
    <td>MGSM_ca</td>
    <td>Math Reasoning</td>
    <td>exact_match</td>
    <td>Math</td>
    <td>Human translation</td>
    <td>250</td>
    <td>2</td>
  </tr>
  <tr>
    <td>OpenBookQA_ca</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>General knowledge</td>
    <td>Human translation</td>
    <td>500</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Parafraseja</td>
    <td>Paraphrasing</td>
    <td>acc</td>
    <td>Language proficiency</td>
    <td>Original</td>
    <td>21984</td>
    <td>3</td>
  </tr>
  <tr>
    <td>PAWS_ca</td>
    <td>Paraphrasing</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Human translation</td>
    <td>2000</td>
    <td>3</td>
  </tr>
  <tr>
    <td>PiQA_ca</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>General knowledge</td>
    <td>Human translation</td>
    <td>1838</td>
    <td>2</td>
  </tr>
  <tr>
    <td>SIQA_ca</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>General knowledge</td>
    <td>Human translation</td>
    <td>1954</td>
    <td>2</td>
  </tr>
  <tr>
    <td>TF-Nara</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Original</td>
    <td>2117</td>
    <td>3</td>
  </tr>
  <tr>
    <td>WNLI_ca</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Human translation</td>
    <td>146</td>
    <td>2</td>
  </tr>
  <tr>
    <td>XNLJ_ca</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Human translation</td>
    <td>5010</td>
    <td>3</td>
  </tr>
  <tr>
    <td>XQuAD_ca</td>
    <td>Extractive QA</td>
    <td>f1</td>
    <td>Miscellaneous</td>
    <td>Human translation</td>
    <td>1190</td>
    <td>2</td>
  </tr>
  <tr>
    <td>xStoryCloze_ca</td>
    <td>Commonsense Reasoning</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Human translation</td>
    <td>1510</td>
    <td>0</td>
  </tr>
</table>

Table 5: Details of the evaluation datasets in Catalan (CA).

<table>
  <tr>
    <th>Dataset</th>
    <th>Task</th>
    <th>Metric</th>
    <th>Domain</th>
    <th>Origin</th>
    <th>#Examples</th>
    <th>#Shots</th>
  </tr>
  <tr>
    <td>BEC2016eu</td>
    <td>Sentiment Analysis</td>
    <td>f1</td>
    <td>Politics, Twitter</td>
    <td>Original</td>
    <td>1302</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Belebele Eus</td>
    <td>Reading Comprehension</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Human translation</td>
    <td>900</td>
    <td>2</td>
  </tr>
  <tr>
    <td>BertQA</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>Cultural Knowledge</td>
    <td>Original</td>
    <td>4766</td>
    <td>3</td>
  </tr>
  <tr>
    <td>BHTCv2-eu</td>
    <td>Topic Classification</td>
    <td>f1</td>
    <td>Press</td>
    <td>Original</td>
    <td>1854</td>
    <td>2</td>
  </tr>
  <tr>
    <td>EpecoqrefBin</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Press</td>
    <td>Original</td>
    <td>587</td>
    <td>3</td>
  </tr>
  <tr>
    <td>EusExams</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Original</td>
    <td>16000</td>
    <td>4</td>
  </tr>
  <tr>
    <td>EusProficiency</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>Language proficiency</td>
    <td>Original</td>
    <td>5169</td>
    <td>4</td>
  </tr>
  <tr>
    <td>EusReading</td>
    <td>Reading Comprehension</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Original</td>
    <td>352</td>
    <td>1</td>
  </tr>
  <tr>
    <td>EusTrivia</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>General knowledge</td>
    <td>Original</td>
    <td>1715</td>
    <td>4</td>
  </tr>
  <tr>
    <td>MGSM_eu</td>
    <td>Math Reasoning</td>
    <td>exact_match</td>
    <td>Math</td>
    <td>Human translation</td>
    <td>250</td>
    <td>2</td>
  </tr>
  <tr>
    <td>QNLeu</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Original</td>
    <td>238</td>
    <td>2</td>
  </tr>
  <tr>
    <td>VoxEspance</td>
    <td>Subtask A</td>
    <td>f1</td>
    <td>Politics, Twitter</td>
    <td>Original</td>
    <td>312</td>
    <td>3</td>
  </tr>
  <tr>
    <td>WiCeu</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Language proficiency</td>
    <td>Original</td>
    <td>1400</td>
    <td>2</td>
  </tr>
  <tr>
    <td>WNLI_eu</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Human translation</td>
    <td>146</td>
    <td>2</td>
  </tr>
  <tr>
    <td>XCOPA_eu</td>
    <td>Commonsense Reasoning</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Human translation</td>
    <td>500</td>
    <td>3</td>
  </tr>
  <tr>
    <td>XNLJ_eu</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Reviewed MT</td>
    <td>5010</td>
    <td>3</td>
  </tr>
  <tr>
    <td>xStoryCloze_eu</td>
    <td>Commonsense Reasoning</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Human translation</td>
    <td>1510</td>
    <td>0</td>
  </tr>
</table>

Table 6: Details for evaluation datasets in Basque (EU).

<table>
  <tr>
    <th>Dataset</th>
    <th>Task</th>
    <th>Metric</th>
    <th>Domain</th>
    <th>Origin</th>
    <th>#Examples</th>
    <th>#Shots</th>
  </tr>
  <tr>
    <td>BelebeleGig</td>
    <td>Reading Comprehension</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Reviewed MT</td>
    <td>900</td>
    <td>2</td>
  </tr>
  <tr>
    <td>GaiCoLa</td>
    <td>Linguistic Acceptability</td>
    <td>mce</td>
    <td>Language proficiency</td>
    <td>Original</td>
    <td>1710</td>
    <td>2</td>
  </tr>
  <tr>
    <td>MGSM_gi</td>
    <td>Math Reasoning</td>
    <td>exact_match</td>
    <td>Math</td>
    <td>Reviewed MT</td>
    <td>250</td>
    <td>2</td>
  </tr>
  <tr>
    <td>OpenBookQA_gi</td>
    <td>Multiple Choice QA</td>
    <td>acc</td>
    <td>General knowledge</td>
    <td>Reviewed MT</td>
    <td>500</td>
    <td>0</td>
  </tr>
  <tr>
    <td>ParafraseGL</td>
    <td>Paraphrasing</td>
    <td>acc</td>
    <td>Language proficiency</td>
    <td>Original</td>
    <td>294</td>
    <td>3</td>
  </tr>
  <tr>
    <td>PAWS_gi</td>
    <td>Paraphrasing</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Reviewed MT</td>
    <td>2000</td>
    <td>3</td>
  </tr>
  <tr>
    <td>SummarizationGL</td>
    <td>Summarization</td>
    <td>bleu</td>
    <td>Press</td>
    <td>Original</td>
    <td>8080</td>
    <td>1</td>
  </tr>
  <tr>
    <td>XNLJ_gi</td>
    <td>Natural Language Inference</td>
    <td>acc</td>
    <td>Lang. prof., Misc.</td>
    <td>Reviewed MT</td>
    <td>5010</td>
    <td>3</td>
  </tr>
  <tr>
    <td>xStoryCloze_gi</td>
    <td>Commonsense Reasoning</td>
    <td>acc</td>
    <td>Miscellaneous</td>
    <td>Human translation</td>
    <td>1510</td>
    <td>0</td>
  </tr>
</table>

Table 7: Details for evaluation datasets in Galician (GL).
B  Frontend Detailed Description

The implementation of LA LEADERBOARD is based on the HuggingFace leaderboard template.23 The frontend is developed using Gradio (Abid et al., 2019) and divided into four tabs:

• The landing tab, named "La Leaderboard", is divided into five sub-tabs, each containing tables with all the evaluated models and their corresponding average results. These sub-tabs include overall and language-specific results for Spanish, Catalan, Basque, and Galician. The results are aggregated by averaging the scores across all tasks for each language.

• For transparency and reproducibility purposes, the second tab, "Info", includes the command we use to evaluate the models and also the normalization formula. In the acknowledgements section, we list the institutions and every person who contributed to the project.

• The next tab describes all the "Tasks" included in LA LEADERBOARD .

• Finally, there is a tab where everyone can submit their model for evaluation.

The text of the information and submission tabs is available both in English and Spanish to bring the tool closer to the community.

In the footer, we can find the citation information for the software, all the included datasets, and the evaluation suite. Below are the fourteen logos from all the collaborating institutions. The entities in the acknowledgements are ordered chronologically by the date they joined the project to thank early adopters, whereas the logos in the footer are ordered by the number of datasets donated.

C  Models Evaluated

Table 8 details the 50 models evaluated, including the following families: Aitana24, BERTIN (la Rosa et al., 2022), Carballo (Gamallo et al., 2024), FLOR (Da Dalt et al., 2024), LeniaChat25, RigoChat (Instituto de Ingeniería del Conocimiento, 2025), Salamandra26, Occiglot27, EuroLLM (Martins et al., 2024), Aya (Dang et al., 2024), DeepSeek (DeepSeek-AI et al., 2025), Gemma (Riviere et al., 2024), Llama (Grattafiori et al., 2024), Mistral (Jiang et al., 2023), Phi (Li et al., 2023), SmoILM (Allal et al., 2025), and Qwen (Qwen Team, 2024).

D  Evaluation Results

This section detailed visualizations of the evaluation results, comparing models, languages, and tasks, considering metrics such as performance and energy efficiency.

Figure 3 presents the full list of average results of the models evaluated, overall and by language. Moreover, since a very bad or good score in a few tasks can lower or raise the average score for a model and distort the comparison, we show the results in terms of the number of tasks for which a model is in the top 10 in Figure 4. This provides an alternative view of the results, focusing on the number of tasks for which the performance of the model is good. Figure 5 provides a comprehensive view of all the evaluation results by language, size, and model type.

Figure 6 represents the total energy consumed by each model. On average, each model consumed 9.25 kWh (median = 6.88, SD = 8.42), showing a wide variety in energy usage. Figure 7 shows which tasks consume more energy, with the text generation tasks on the top of the list. Figure 8 presents a comparison between model size and energy consumption. Finally, Figure 9 shows the relation between performance and energy consumption.

23https://hf.co/spaces/demo-leaderboard-backend/leaderboard
24https://hf.co/gplsi/Aitana-6.3B
25https://hf.co/LenguajeNaturalAI/leniachat-gemma-2b-v0
26https://hf.co/collections/BSC-LT/salamandra-66fc171485944df79469043a
27https://hf.co/collections/occiglot/occiglot-eu5-7b-v01-65dbed02a6348b05269e01
<table>
  <tr>
    <th>Family</th>
    <th>Model ID</th>
    <th>Model Type</th>
    <th>Size (B)</th>
  </tr>
  <tr>
    <td>Aitana</td>
    <td>gplsi/Aitana-6.3B</td>
    <td>pretrained</td>
    <td>6.25</td>
  </tr>
  <tr>
    <td>BERTIN</td>
    <td>bertin-project/berlin-gpt-j-6B</td>
    <td>pretrained</td>
    <td>6.06</td>
  </tr>
  <tr>
    <td>BERTIN</td>
    <td>bertin-project/Gromenauer-7B</td>
    <td>pretrained</td>
    <td>7.24</td>
  </tr>
  <tr>
    <td>BERTIN</td>
    <td>bertin-project/Gromenauer-7B-Instruct</td>
    <td>instruction-tuned</td>
    <td>7.24</td>
  </tr>
  <tr>
    <td>Carballo</td>
    <td>proxectos/Carballo-bloom-1.3B</td>
    <td>pretrained</td>
    <td>1.31</td>
  </tr>
  <tr>
    <td>FLOR</td>
    <td>projecte-aina/FLOR-1.3B</td>
    <td>pretrained</td>
    <td>1.31</td>
  </tr>
  <tr>
    <td>FLOR</td>
    <td>projecte-aina/FLOR-1.3B-Instructed</td>
    <td>instruction-tuned</td>
    <td>1.31</td>
  </tr>
  <tr>
    <td>FLOR</td>
    <td>projecte-aina/FLOR-6.3B</td>
    <td>pretrained</td>
    <td>6.25</td>
  </tr>
  <tr>
    <td>FLOR</td>
    <td>projecte-aina/FLOR-6.3B-Instructed</td>
    <td>instruction-tuned</td>
    <td>6.25</td>
  </tr>
  <tr>
    <td>Latxa</td>
    <td>HITZ/latxa-7b-v1.2</td>
    <td>pretrained</td>
    <td>7.00</td>
  </tr>
  <tr>
    <td>LeniaChat</td>
    <td>LenguajeNaturalAI/leniachat-gemma-2b-v0</td>
    <td>instruction-tuned</td>
    <td>2.51</td>
  </tr>
  <tr>
    <td>LeniaChat</td>
    <td>LenguajeNaturalAI/leniachat-qwen2-1.5B-v0</td>
    <td>instruction-tuned</td>
    <td>1.54</td>
  </tr>
  <tr>
    <td>RigoChat</td>
    <td>IIC/RigoChat-7b-v2</td>
    <td>instruction-tuned</td>
    <td>7.62</td>
  </tr>
  <tr>
    <td>Salamandra</td>
    <td>BSC-LT/salamandra-2b</td>
    <td>pretrained</td>
    <td>2.25</td>
  </tr>
  <tr>
    <td>Salamandra</td>
    <td>BSC-LT/salamandra-2b-instruct</td>
    <td>instruction-tuned</td>
    <td>2.25</td>
  </tr>
  <tr>
    <td>Salamandra</td>
    <td>BSC-LT/salamandra-7b</td>
    <td>pretrained</td>
    <td>7.77</td>
  </tr>
  <tr>
    <td>Salamandra</td>
    <td>BSC-LT/salamandra-7b-instruct</td>
    <td>instruction-tuned</td>
    <td>7.77</td>
  </tr>
  <tr>
    <td>EuroLLM</td>
    <td>utter-project/EuroLLM-1.7B</td>
    <td>pretrained</td>
    <td>1.70</td>
  </tr>
  <tr>
    <td>EuroLLM</td>
    <td>utter-project/EuroLLM-1.7B-Instruct</td>
    <td>instruction-tuned</td>
    <td>1.70</td>
  </tr>
  <tr>
    <td>EuroLLM</td>
    <td>utter-project/EuroLLM-9B</td>
    <td>pretrained</td>
    <td>9.15</td>
  </tr>
  <tr>
    <td>EuroLLM</td>
    <td>utter-project/EuroLLM-9B-Instruct</td>
    <td>instruction-tuned</td>
    <td>9.15</td>
  </tr>
  <tr>
    <td>Occiglot</td>
    <td>occiglot/occiglot-7b-es-en</td>
    <td>pretrained</td>
    <td>7.24</td>
  </tr>
  <tr>
    <td>Occiglot</td>
    <td>occiglot/occiglot-7b-es-en-instruct</td>
    <td>instruction-tuned</td>
    <td>7.24</td>
  </tr>
  <tr>
    <td>Occiglot</td>
    <td>occiglot/occiglot-7b-eu5</td>
    <td>pretrained</td>
    <td>7.24</td>
  </tr>
  <tr>
    <td>Occiglot</td>
    <td>occiglot/occiglot-7b-eu5-instruct</td>
    <td>instruction-tuned</td>
    <td>7.24</td>
  </tr>
  <tr>
    <td>Aya</td>
    <td>CohereForAya-expanse-8b</td>
    <td>pretrained</td>
    <td>8.03</td>
  </tr>
  <tr>
    <td>DeepSeek</td>
    <td>deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B</td>
    <td>instruction-tuned</td>
    <td>1.78</td>
  </tr>
  <tr>
    <td>DeepSeek</td>
    <td>deepseek-ai/DeepSeek-R1-Distill-Qwen-7B</td>
    <td>instruction-tuned</td>
    <td>7.62</td>
  </tr>
  <tr>
    <td>DeepSeek</td>
    <td>unsloth/DeepSeek-R1-Distill-Qwen-14B-bnb-4bit</td>
    <td>instruction-tuned</td>
    <td>14.8 (8.37)</td>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>google/gemma-2-2b</td>
    <td>pretrained</td>
    <td>2.61</td>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>google/gemma-2-2b-it</td>
    <td>instruction-tuned</td>
    <td>2.61</td>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>google/gemma-2-9h</td>
    <td>pretrained</td>
    <td>9.24</td>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>google/gemma-2-9h-it</td>
    <td>instruction-tuned</td>
    <td>9.24</td>
  </tr>
  <tr>
    <td>Llama</td>
    <td>meta-llama/Llama-3.2-1B</td>
    <td>pretrained</td>
    <td>1.24</td>
  </tr>
  <tr>
    <td>Llama</td>
    <td>meta-llama/Llama-3.2-1B-Instruct</td>
    <td>instruction-tuned</td>
    <td>1.24</td>
  </tr>
  <tr>
    <td>Llama</td>
    <td>meta-llama/Meta-Llama-3.1-8B</td>
    <td>pretrained</td>
    <td>8.03</td>
  </tr>
  <tr>
    <td>Llama</td>
    <td>meta-llama/Meta-Llama-3.1-8B-Instruct</td>
    <td>instruction-tuned</td>
    <td>8.03</td>
  </tr>
  <tr>
    <td>Mistral</td>
    <td>mistralai/Mistral-7B-Instruct-v0.3</td>
    <td>instruction-tuned</td>
    <td>7.25</td>
  </tr>
  <tr>
    <td>Mistral</td>
    <td>mistralai/Mistral-7B-v0.3</td>
    <td>pretrained</td>
    <td>7.25</td>
  </tr>
  <tr>
    <td>Phi</td>
    <td>microsoft/phi-1_5</td>
    <td>pretrained</td>
    <td>1.42</td>
  </tr>
  <tr>
    <td>SmolLM</td>
    <td>HuggingFaceTB/SmolLM2-1.7B</td>
    <td>pretrained</td>
    <td>1.71</td>
  </tr>
  <tr>
    <td>SmolLM</td>
    <td>HuggingFaceTB/SmolLM2-1.7B-Instruct</td>
    <td>instruction-tuned</td>
    <td>1.71</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>Qwen/Qwen2.5-1.5B</td>
    <td>pretrained</td>
    <td>1.54</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>Qwen/Qwen2.5-1.5B-Instruct</td>
    <td>instruction-tuned</td>
    <td>1.54</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>Qwen/Qwen2.5-7B</td>
    <td>pretrained</td>
    <td>7.62</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>Qwen/Qwen2.5-7B-Instruct</td>
    <td>instruction-tuned</td>
    <td>7.62</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>Qwen/Qwen2.5-14B-Instruct-GPTQ-Int8</td>
    <td>instruction-tuned</td>
    <td>14.80 (4.99)</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>Qwen/Qwen2.5-32B-Instruct-GPTQ-Int4</td>
    <td>instruction-tuned</td>
    <td>32.80 (5.74)</td>
  </tr>
</table>

Table 8: Models evaluated in LA LEADERBOARD as of February 2025. The table is divided into sections starting at the top with models optimized for the languages of Spain, followed by models created by European projects, and finally models trained by international large technological companies and startups. The size is specified in billions of parameters, as appears in the SafeTensors information of the corresponding Hugging Face model page. For quantized models, the SafeTensors equivalent size of the model is added in parenthesis after the size of the base model.
<table>
  <tr>
    <th>Model Name</th>
    <th>Training Tokens (B)</th>
    <th>Languages Included</th>
    <th>Tokens (B) per Language</th>
    <th>Hardware</th>
    <th>Training Time (h)</th>
  </tr>
  <tr>
    <td>Aitana-6.3B</td>
    <td>2.6</td>
    <td>CA</td>
    <td>2.6</td>
    <td>4 A100</td>
    <td>80</td>
  </tr>
  <tr>
    <td>bertin-gpt-j-6B</td>
    <td>65</td>
    <td>ES</td>
    <td>65</td>
    <td>1 v3-8</td>
    <td>4320</td>
  </tr>
  <tr>
    <td>Gromenauer-7B</td>
    <td>14.5</td>
    <td>ES</td>
    <td>14.5</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Gromenauer-7B-Instruct</td>
    <td>14.5+</td>
    <td>ES, CA</td>
    <td>14.5+, -</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Carballo-bloom-1.3B</td>
    <td>2.1</td>
    <td>GL</td>
    <td>2.1</td>
    <td>5 A100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>FLOR-1.3B</td>
    <td>26</td>
    <td>ES, CA</td>
    <td>10.8, 10.9</td>
    <td>Cerebras CS-2</td>
    <td>-</td>
  </tr>
  <tr>
    <td>FLOR-1.3B-Instructed</td>
    <td>26+</td>
    <td>ES, CA</td>
    <td>10.8, 10.9+</td>
    <td>Cerebras CS-2</td>
    <td>-</td>
  </tr>
  <tr>
    <td>FLOR-6.3B</td>
    <td>140</td>
    <td>ES, CA</td>
    <td>46.7, 46.8</td>
    <td>16 Cereb. CS-2</td>
    <td>60</td>
  </tr>
  <tr>
    <td>FLOR-6.3B-Instructed</td>
    <td>140+</td>
    <td>ES, CA</td>
    <td>46.7, 46.8+</td>
    <td>16 Cereb. CS-2</td>
    <td>60+</td>
  </tr>
  <tr>
    <td>latxa-7b-v1.2</td>
    <td>4.2</td>
    <td>EU</td>
    <td>4.2</td>
    <td>4 A100</td>
    <td>953</td>
  </tr>
  <tr>
    <td>leniachat-gemma-2b-v0</td>
    <td>-</td>
    <td>ES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>leniachat-qwen2-1.5B-v0</td>
    <td>-</td>
    <td>ES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>RigoChat-7b-v2</td>
    <td>-</td>
    <td>ES</td>
    <td>-</td>
    <td>1 A100</td>
    <td>8.5</td>
  </tr>
  <tr>
    <td>salamandra-2b</td>
    <td>12875000</td>
    <td>ES, CA, GL, EU</td>
    <td>2075k, 253k, 39k, 30k</td>
    <td>256 Hopper</td>
    <td>221k</td>
  </tr>
  <tr>
    <td>salamandra-2b-instruct</td>
    <td>12875000+</td>
    <td>ES, CA, GL, EU</td>
    <td>2075k, 253k, 39k, 30k+</td>
    <td>256 Hopper</td>
    <td>221k+12</td>
  </tr>
  <tr>
    <td>salamandra-7b</td>
    <td>12875000</td>
    <td>ES, CA, GL, EU</td>
    <td>2075k, 253k, 39k, 30k</td>
    <td>512 Hopper</td>
    <td>602k</td>
  </tr>
  <tr>
    <td>salamandra-7b-instruct</td>
    <td>12875000+</td>
    <td>ES, CA, GL, EU</td>
    <td>2075k, 253k, 39k, 30k+</td>
    <td>512 Hopper</td>
    <td>602k+16</td>
  </tr>
  <tr>
    <td>EuroLLM-1.7B</td>
    <td>4000</td>
    <td>ES, CA, GL</td>
    <td>240, 12, 4</td>
    <td>256 H100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>EuroLLM-1.7B-Instruct</td>
    <td>4000+</td>
    <td>ES, CA</td>
    <td>240, 12, 4+</td>
    <td>256 H100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>EuroLLM-9B</td>
    <td>4000</td>
    <td>ES, CA, GL</td>
    <td>240, 12, 4</td>
    <td>400 H100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>EuroLLM-9B-Instruct</td>
    <td>4000+</td>
    <td>ES, CA</td>
    <td>240, 12, 4+</td>
    <td>400 H100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>occiglot-7b-es-en</td>
    <td>112</td>
    <td>ES</td>
    <td>58.2</td>
    <td>128 A100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>occiglot-7b-es-en-instruct</td>
    <td>112+0.16</td>
    <td>ES</td>
    <td>58.2+0.08</td>
    <td>+8 H100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>occiglot-7b-eu5</td>
    <td>293</td>
    <td>ES</td>
    <td>58.6</td>
    <td>128 A100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>occiglot-7b-eu5-instruct</td>
    <td>293+0.40</td>
    <td>ES</td>
    <td>58.6+0.08</td>
    <td>+8 H100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>aya-expanse-8b</td>
    <td>-</td>
    <td>ES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DeepSeek-R1-Dist-Qwen-1.5B</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DeepSeek-R1-Dist-Qwen-7B</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DeepSeek-R1-Dist-Qwen-14B-4bit</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>gemma-2-2b</td>
    <td>2000</td>
    <td>-</td>
    <td>-</td>
    <td>v5p</td>
    <td>-</td>
  </tr>
  <tr>
    <td>gemma-2-2b-it</td>
    <td>2000+</td>
    <td>-</td>
    <td>-</td>
    <td>v5p</td>
    <td>-</td>
  </tr>
  <tr>
    <td>gemma-2-9b</td>
    <td>8000</td>
    <td>-</td>
    <td>-</td>
    <td>v5p</td>
    <td>-</td>
  </tr>
  <tr>
    <td>gemma-2-9b-it</td>
    <td>8000+</td>
    <td>-</td>
    <td>-</td>
    <td>v5p</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Llama-3.2-1B</td>
    <td>9000</td>
    <td>ES</td>
    <td>-</td>
    <td>H100</td>
    <td>370k</td>
  </tr>
  <tr>
    <td>Llama-3.2-1B-Instruct</td>
    <td>9000+</td>
    <td>ES</td>
    <td>-</td>
    <td>H100</td>
    <td>370k+</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>15000</td>
    <td>ES</td>
    <td>-</td>
    <td>H100</td>
    <td>1460k</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>15000+</td>
    <td>ES</td>
    <td>-</td>
    <td>H100</td>
    <td>1460k+</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.3</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct-v0.3</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>phi-1.5</td>
    <td>150</td>
    <td>none</td>
    <td>0</td>
    <td>32 A100</td>
    <td>192</td>
  </tr>
  <tr>
    <td>SmoILM2-1.7B</td>
    <td>11000</td>
    <td>none</td>
    <td>0</td>
    <td>256 H100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SmoILM2-1.7B-Instruct</td>
    <td>11000+</td>
    <td>none</td>
    <td>0</td>
    <td>256 H100</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>18000</td>
    <td>ES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B-Instruct</td>
    <td>18000</td>
    <td>ES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>18000</td>
    <td>ES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>18000</td>
    <td>ES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Instruct-GPTQ-Int8</td>
    <td>18000</td>
    <td>ES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Instruct-GPTQ-Int4</td>
    <td>18000</td>
    <td>ES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 9: Public details about the training corpus and compute resources of the models evaluated on LA LEADERBOARD : total number of tokens in the training data, target languages included in the training data, number of tokens per language, hardware type, and hours of training time.
<table>
  <tr>
    <th></th>
    <th>AVG</th>
    <th>ES</th>
    <th>CA</th>
    <th>EU</th>
    <th>GL</th>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-32B-Instruct-GPTQ-int4 -</td>
    <td>55.6</td>
    <td>64.1</td>
    <td>56.8</td>
    <td>49.2</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td>google/gemma-2-9b-it</td>
    <td>54.9</td>
    <td>61.7</td>
    <td>57.3</td>
    <td>54.1</td>
    <td>46.5</td>
  </tr>
  <tr>
    <td>google/gemma-2-9b</td>
    <td>54.8</td>
    <td>57.2</td>
    <td>59.6</td>
    <td>53.8</td>
    <td>48.6</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-14B-Instruct-GPTQ-int8 -</td>
    <td>54.0</td>
    <td>60.6</td>
    <td>54.1</td>
    <td>49.0</td>
    <td>52.1</td>
  </tr>
  <tr>
    <td>meta-llama/Meta-Llama-3.1-BB-Instruct -</td>
    <td>52.7</td>
    <td>59.0</td>
    <td>57.0</td>
    <td>49.9</td>
    <td>45.1</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-7B -</td>
    <td>51.4</td>
    <td>58.8</td>
    <td>57.3</td>
    <td>42.5</td>
    <td>46.8</td>
  </tr>
  <tr>
    <td>meta-llama/Meta-Llama-3.1-BB -</td>
    <td>51.0</td>
    <td>55.6</td>
    <td>56.5</td>
    <td>46.9</td>
    <td>44.9</td>
  </tr>
  <tr>
    <td>utter-project/EuroLM-9B -</td>
    <td>49.4</td>
    <td>55.0</td>
    <td>57.3</td>
    <td>38.9</td>
    <td>46.4</td>
  </tr>
  <tr>
    <td>CohereForAlaya-expanse-Bb -</td>
    <td>48.7</td>
    <td>55.4</td>
    <td>54.0</td>
    <td>42.0</td>
    <td>43.4</td>
  </tr>
  <tr>
    <td>01-ai/Yi-1.5-9B -</td>
    <td>48.4</td>
    <td>54.5</td>
    <td>54.2</td>
    <td>40.4</td>
    <td>44.4</td>
  </tr>
  <tr>
    <td>occiglot/occiglot-7b-es-eu5 -</td>
    <td>48.3</td>
    <td>55.0</td>
    <td>53.7</td>
    <td>38.7</td>
    <td>45.6</td>
  </tr>
  <tr>
    <td>utter-project/EuroLM-9B-Instruct -</td>
    <td>48.2</td>
    <td>57.2</td>
    <td>53.0</td>
    <td>38.0</td>
    <td>44.5</td>
  </tr>
  <tr>
    <td>BSC-LT/salamandra-7b-instruct -</td>
    <td>48.1</td>
    <td>51.4</td>
    <td>53.2</td>
    <td>46.2</td>
    <td>41.6</td>
  </tr>
  <tr>
    <td>BSC-LT/salamandra-7b -</td>
    <td>48.0</td>
    <td>52.2</td>
    <td>54.1</td>
    <td>45.8</td>
    <td>39.9</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-7B-Instruct -</td>
    <td>47.5</td>
    <td>57.5</td>
    <td>48.2</td>
    <td>41.4</td>
    <td>43.1</td>
  </tr>
  <tr>
    <td>mistralai/Mistral-7B-Instruct-v0.3 -</td>
    <td>47.5</td>
    <td>56.3</td>
    <td>53.9</td>
    <td>38.8</td>
    <td>41.1</td>
  </tr>
  <tr>
    <td>mistralai/Mistral-7B-v0.3 -</td>
    <td>47.5</td>
    <td>54.6</td>
    <td>56.1</td>
    <td>38.6</td>
    <td>40.5</td>
  </tr>
  <tr>
    <td>bertin-project/Gromenauer-7B -</td>
    <td>47.3</td>
    <td>56.2</td>
    <td>55.2</td>
    <td>37.3</td>
    <td>40.5</td>
  </tr>
  <tr>
    <td>IIC/RigoChat-7b-v2 -</td>
    <td>47.3</td>
    <td>57.9</td>
    <td>46.7</td>
    <td>41.6</td>
    <td>42.9</td>
  </tr>
  <tr>
    <td>occiglot/occiglot-7b-es-en -</td>
    <td>47.2</td>
    <td>54.4</td>
    <td>55.0</td>
    <td>39.3</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>occiglot/occiglot-7b-es-eu5-instruct -</td>
    <td>46.8</td>
    <td>57.3</td>
    <td>48.7</td>
    <td>37.9</td>
    <td>43.2</td>
  </tr>
  <tr>
    <td>01-ai/Yi-1.5-9B-Chat -</td>
    <td>46.3</td>
    <td>53.4</td>
    <td>48.6</td>
    <td>40.4</td>
    <td>42.9</td>
  </tr>
  <tr>
    <td>google/gemma-2-9b-it</td>
    <td>45.8</td>
    <td>54.9</td>
    <td>48.3</td>
    <td>40.3</td>
    <td>39.8</td>
  </tr>
  <tr>
    <td>occiglot/occiglot-7b-es-en-instruct -</td>
    <td>45.3</td>
    <td>55.9</td>
    <td>47.8</td>
    <td>38.7</td>
    <td>39.0</td>
  </tr>
  <tr>
    <td>google/gemma-2-9b</td>
    <td>45.2</td>
    <td>51.6</td>
    <td>50.2</td>
    <td>39.8</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-1.5B-Instruct -</td>
    <td>44.4</td>
    <td>54.4</td>
    <td>45.1</td>
    <td>39.0</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2.5-1.5B -</td>
    <td>44.4</td>
    <td>52.6</td>
    <td>47.9</td>
    <td>38.1</td>
    <td>38.8</td>
  </tr>
  <tr>
    <td>bertin-project/Gromenauer-7B-Instruct -</td>
    <td>44.2</td>
    <td>56.6</td>
    <td>46.1</td>
    <td>37.5</td>
    <td>36.5</td>
  </tr>
  <tr>
    <td>HITZ/alata-7b-v1.2 -</td>
    <td>43.8</td>
    <td>48.9</td>
    <td>47.0</td>
    <td>45.0</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>projecte-aina/FLOR-6.3B -</td>
    <td>42.4</td>
    <td>47.5</td>
    <td>50.4</td>
    <td>37.1</td>
    <td>34.6</td>
  </tr>
  <tr>
    <td>LenguajeNaturalAI/leniachat-qwen2.1.5B-v0 -</td>
    <td>41.6</td>
    <td>51.1</td>
    <td>41.0</td>
    <td>38.3</td>
    <td>36.2</td>
  </tr>
  <tr>
    <td>gpli/Atana-6.3B -</td>
    <td>41.6</td>
    <td>45.9</td>
    <td>49.4</td>
    <td>37.1</td>
    <td>33.8</td>
  </tr>
  <tr>
    <td>projecte-aina/aigua-7b -</td>
    <td>41.2</td>
    <td>47.2</td>
    <td>44.5</td>
    <td>37.2</td>
    <td>35.9</td>
  </tr>
  <tr>
    <td>utter-project/EuroLM-1.7B-Instruct -</td>
    <td>41.0</td>
    <td>47.9</td>
    <td>45.3</td>
    <td>36.2</td>
    <td>34.7</td>
  </tr>
  <tr>
    <td>projecte-aina/FLOR-6.3B-instruct -</td>
    <td>40.8</td>
    <td>47.7</td>
    <td>45.0</td>
    <td>37.6</td>
    <td>33.0</td>
  </tr>
  <tr>
    <td>bertin-project/bertin-gpt-6B -</td>
    <td>40.6</td>
    <td>46.1</td>
    <td>44.3</td>
    <td>36.9</td>
    <td>35.1</td>
  </tr>
  <tr>
    <td>BSC-LT/salamandra-2b -</td>
    <td>40.5</td>
    <td>46.6</td>
    <td>41.0</td>
    <td>39.3</td>
    <td>35.0</td>
  </tr>
  <tr>
    <td>projecte-aina/FLOR-1.3B-instruct -</td>
    <td>40.3</td>
    <td>46.2</td>
    <td>44.7</td>
    <td>37.1</td>
    <td>33.2</td>
  </tr>
  <tr>
    <td>HuggingFaceTB/SmoILM2-1.7B -</td>
    <td>40.2</td>
    <td>47.6</td>
    <td>41.6</td>
    <td>36.8</td>
    <td>34.7</td>
  </tr>
  <tr>
    <td>BSC-LT/salamandra-2b-instruct -</td>
    <td>40.0</td>
    <td>43.8</td>
    <td>43.3</td>
    <td>38.6</td>
    <td>34.5</td>
  </tr>
  <tr>
    <td>utter-project/EuroLM-1.7B -</td>
    <td>39.9</td>
    <td>45.9</td>
    <td>42.7</td>
    <td>36.5</td>
    <td>34.7</td>
  </tr>
  <tr>
    <td>HuggingFaceTB/SmoILM2-1.7B-Instruct -</td>
    <td>39.5</td>
    <td>46.5</td>
    <td>41.2</td>
    <td>36.9</td>
    <td>33.2</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.2-1B -</td>
    <td>39.4</td>
    <td>46.1</td>
    <td>42.3</td>
    <td>36.0</td>
    <td>33.3</td>
  </tr>
  <tr>
    <td>projecte-aina/FLOR-1.3B -</td>
    <td>39.3</td>
    <td>46.3</td>
    <td>42.8</td>
    <td>36.0</td>
    <td>32.0</td>
  </tr>
  <tr>
    <td>LenguajeNaturalAI/leniachat-gemma-2b-v0 -</td>
    <td>38.8</td>
    <td>46.9</td>
    <td>39.0</td>
    <td>35.6</td>
    <td>33.8</td>
  </tr>
  <tr>
    <td>proxectonos/Carballio-bloom-1.3B -</td>
    <td>38.6</td>
    <td>45.1</td>
    <td>38.2</td>
    <td>36.9</td>
    <td>34.4</td>
  </tr>
  <tr>
    <td>meta-llama/Llama-3.2-1B-Instruct -</td>
    <td>38.6</td>
    <td>46.3</td>
    <td>38.9</td>
    <td>36.5</td>
    <td>32.9</td>
  </tr>
  <tr>
    <td>unsloth/DeepSeek-R1-Distill-Qwen-14B-bnb-4bit -</td>
    <td>38.6</td>
    <td>42.8</td>
    <td>34.9</td>
    <td>35.1</td>
    <td>33.6</td>
  </tr>
  <tr>
    <td>microsift/phili-1.5 -</td>
    <td>35.8</td>
    <td>42.6</td>
    <td>31.9</td>
    <td>37.4</td>
    <td>31.4</td>
  </tr>
  <tr>
    <td>deepseek-ai/DeepSeek-R1-Distill-Qwen-7B -</td>
    <td>35.1</td>
    <td>41.3</td>
    <td>30.6</td>
    <td>36.8</td>
    <td>31.8</td>
  </tr>
  <tr>
    <td>deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B -</td>
    <td>34.6</td>
    <td>40.9</td>
    <td>29.4</td>
    <td>37.0</td>
    <td>31.2</td>
  </tr>
</table>

Figure 3: Average results of the first set of models evaluated on LA LEADERBOARD , overall and by language.
Figure 4: Number of tasks in which a model is among the top 10 models, by language.
Figure 5: Results of the first set of models evaluated on La Leaderboard organized by language, size, and model type.
Figure 6: Distribution of results of models evaluated on LA LEADERBOARD organized by energy consumption.

Figure 7: Energy consumption for the tasks evaluated on LA LEADERBOARD. The top three tasks (summarization_gl, xlsum_es, and cabreu) correspond to text summarization tasks, which require the generation of many tokens. The next tasks correspond to QA tasks with thousands of questions.
Figure 8: Distribution of results of models evaluated on LA LEADERBOARD comparing energy consumption versus size. The plot shows some correlation between model size and energy consumed, with a Pearson correlation coefficient of 0.43.

Figure 9: Distribution of results of models evaluated on LA LEADERBOARD comparing energy consumption versus performance. The plot shows a high correlation between model performance and energy consumed, with a Pearson correlation coefficient of 0.50.
E  Data Collection Campaign

Below are the questions, translated into English, corresponding to the Google Form used in the open data collection campaign.

1. Email *

2. Data source (Select one option) *
   (a) The dataset is public
   (b) Instructions to recreate it are available
   (c) The dataset is private but access can be requested on a website
   (d) The dataset is currently private, but we want to open it as a donation
   (e) The dataset is private, but you should try contacting the organization that created it

3. Dataset link * This can be the dataset link, the instructions to recreate it, or the corresponding organization’s website if private.

4. If your dataset is not uploaded to Hugging Face, would you like us to take care of uploading it? (Select one option)
   (a) Yes, upload it to the SomosNLP organization
   (b) Yes, help me create my own organization and upload it
   (c) No, I prefer to create my own organization and upload it myself

5. Modality * (Select one option)
   (a) Text
   (b) Audio
   (c) Image (e.g., images with descriptions)

6. Language(s) * (Select all that apply)
   (a) Spanish
   (b) Other: _____

7. Country(ies) * Country(ies) of origin of the data and/or the people who annotated it. A region can also be specified if known. The more information, the better.

8. Tasks * (Select all that apply)
   (a) Language modeling (unannotated)
   (b) Question answering (QA)
   (c) Classification
   (d) Token classification (e.g., NER, PoS)
   (e) Translation
   (f) Summarization
   (g) Semantic similarity
   (h) Multimodal (e.g., text-to-image, audio-to-text)

9. Subtask For example, subtasks of "text classification" could be "sentiment analysis" or "hate speech detection."

10. Domain * (Select all that apply)
    (a) Legal
    (b) Clinical or biomedical
    (c) Academic or technical
    (d) Literature or music
    (e) Social media or forums
    (f) News or articles
    (g) Dialogues
    (h) General

11. Number of examples Enter the exact number of examples if known, otherwise provide a range.

12. License type *
    (a) Commercial
    (b) Non-commercial

13. License link

14. Link to the dataset documentation or any other relevant information: description, annotation and cleaning process, ethical considerations... *

15. Link to the script/repository on GitHub to download or process the dataset

16. Thank you very much for your contribution! To publicly acknowledge your contribution, you may share your name and/or affiliation to be displayed on the website. If this is a donation, we will contact you soon—thank you!

17. Name

18. Affiliation

19. How could we improve this campaign? Who would you recommend we contact? Anything else you’d like to tell us?
F Datasheets

We present the datasheets (Gebru et al., 2021) corresponding to each of the datasets specifically created for LA LEADERBOARD : AQuAS, Clin-DiagnosES, ClinTreatES, HumorQA, RagQuAS, SpaLawEx, TELEIA. Moreover, we propose an adaptation for leaderboards and fill it for LA LEADERBOARD.

La Leaderboard

Motivation for Leaderboard Creation

Why was the leaderboard created?
LA LEADERBOARD is the first open-source leaderboard to evaluate generative LLMs in languages of Spain and Latin America. By aiming to address the linguistic and cultural diversity of the Spanish-speaking community, LA LEADERBOARD aims to set a new standard for multilingual LLM evaluation. Our goal is to encourage the development of models that are not only linguistically competent but also culturally aware, ultimately driving progress in Natural Language Processing (NLP) for the benefit of our whole community.

Who funded the creation of the leaderboard?
LA LEADERBOARD is an initiative launched by an international open-source community and was promoted by volunteers. The funding of each of the individual datasets donated to LA LEADERBOARD will be disclosed after review.

Leaderboard Composition

What are the instances?
LA LEADERBOARD consists of 66 evaluation datasets. All the evaluation datasets in the leaderboard consist solely of text instances.

Are relationships between instances made explicit in the data
There are no known relationships between instances.

How many instances of each type are there?
Summing all the instances of the 66 evaluation datasets, the leaderboard consists of 149,782 examples.

Is everything included or does the data rely on external resources?
Everything is included in the datasets.

Are there recommended data splits or evaluation measures?
The splits used in LA LEADERBOARD are the corresponding test splits of each dataset.

Data Collection Process

How was the data collected?
The datasets were collected through an open data collection campaign.

Who was involved in the data collection process?
How were they compensated?
Professional researchers from academia and industry. The logo and names of the donators are included in the user interface, and the creators of the datasets are acknowledged in the paper.

Over what time-frame was the data collected?
During 2024.

Does the dataset contain all possible instances?
The evaluations are launched including all the available test instances for each donated dataset.

If the dataset is a sample, then what is the population?
Not applicable.

Is there information missing from the dataset and why?
No

Are there any known errors, sources of noise, or redundancies in the data?
No.

Leaderboard Distribution

How is the leaderboard distributed?
The leaderboard is available in the HuggingFace hub28.

When will the leaderboard be released/first distributed?
The leaderboard was released in September 2024.

What license (if any) is it distributed under?
The leaderboard is licensed under "Apache 2.0".

Are there any fees or access/export restrictions?
There are no fees or restrictions.

Leaderboard Maintenance

Who is supporting/hosting/maintaining the leaderboard?
The leaderboard is hosted at HuggingFace29, and the community can be contacted

28https://hf.co/spaces/la-leaderboard/
la-leaderboard
29https://hf.co/spaces/la-leaderboard/
la-leaderboard
through the "Discussions" tab in the interface or via email30.

Will the leaderboard be updated? How often and by whom? Yes, every time there is a new donation, the maintainer will update the leaderboard and communicate the update on the usual communication channels of the open-source community.

Is there a repository to link to any/all papers/systems that use this leaderboard? Yes, all the datasets and tools used by LA LEADERBOARD are referenced in the "Citation" section of the interface31.

Legal & Ethical Considerations

If the dataset relates to people or was generated by people, were they informed about the data collection? Not applicable.

If it relates to other ethically protected subjects, have appropriate obligations been met? Not applicable.

If it relates to people, were there any ethical review applications/reviews/approvals? Not applicable.

If it relates to people, were they told what the dataset would be used for and did they consent? Not applicable.

If it relates to people, could this dataset expose people to harm or legal action? Not applicable.

If it relates to people, does it unfairly advantage or disadvantage a particular social group? Not applicable.

If it relates to people, were they provided with privacy guarantees? Not applicable.

Does the dataset comply with the EU General Data Protection Regulation (GDPR)? Yes, it complies with GDPR.

Does the dataset contain information that might be considered sensitive or confidential? No.

Does the dataset contain information that might be considered inappropriate or offensive? No.

AQuAS

The Abstractive Question-Answering in Spanish (AQuAS) dataset (Instituto de Ingeniería del Conocimiento, 2024a), developed by Instituto de Ingeniería del Conocimiento, is a monolingual Spanish dataset designed for abstractive question-answering. It contains 107 examples covering a diverse range of topics, including finance, insurance, healthcare, music, and law. Each example consists of a context passage, a related question, and a human-crafted answer. The dataset is aimed at evaluating the ability of large language models (LLMs) to generate well-formed, coherent, and informative responses.

Motivation for Dataset Creation

Why was the dataset created? AQuAS was created to provide high-quality examples of pairs of questions and answers with a related context that can be used to evaluate the ability of large language models (LLMs) to generate well-formed, coherent, and informative responses (abstractive question answering).

What (other) tasks could the dataset be used for? There are no recommended uses for this dataset other than evaluation.

Who funded the creation of the dataset? If there is an associated grant, provide the grant number. The dataset was created and funded by the research institute.

Dataset Composition

What are the instances? Each instance is a pair of a question and an answer accompanied by the related context on which the answer has been based and the corresponding topic.

Are relationships between instances made explicit in the data There are no known relationships between instances.

How many instances of each type are there? The dataset consists of 107 examples.

What data does each instance consist of? The instances consist of text data and are labelled with the corresponding topic.

Is everything included or does the data rely on external resources? Everything is included in the dataset.

30maria.grandury@somosnlp.org
31https://hf.co/spaces/la-leaderboard/la-leaderboard
Are there recommended data splits or evaluation measures? Since the dataset is intended for testing, there is no recommended split.

Data Collection Process

How was the data collected? The data for the contexts was gathered from different sources on the web using software to crawl those sites. The rest of the dataset (question-answer pairs) was curated and created manually.

Who was involved in the data collection process? How were they compensated? The data was collected by computational linguists and data scientists from a research institute.

Over what time-frame was the data collected? The data was collected during 2023, when the dataset was created.

How was the data associated with each instance acquired? The question-answer pairs were created and revised by computational linguists.

Does the dataset contain all possible instances? The dataset is composed of selected instances of different datasets created by a research institute.

If the dataset is a sample, then what is the population? This dataset is a 24.5% sample of the original complete datasets. The instances were randomly selected from the original datasets.

Is there information missing from the dataset and why? There is no data missing.

Are there any known errors, sources of noise, or redundancies in the data? There are no known errors because the revision process ensured the data is as clean and error-free as possible.

Data Preprocessing

What preprocessing/cleaning was done? The text contained in the "context" part of each instance in the dataset has not undergone any preprocessing or changes. There was no need to apply any cleaning to the question-answer pairs because they were created manually by computational linguists following a rigorous methodology and were subjected to revision afterwards.

Was the “raw” data saved in addition to the preprocessed/clean data? No, the text in the dataset is the raw data.

Is the preprocessing software available? No preprocessing software was used.

Does this dataset collection/processing procedure achieve the motivation for creating the dataset stated in the first section of this datasheet? Yes, the collection procedure ensures the dataset is sufficiently varied so it can be used to evaluate a model on a wide range of topics. However, there are some potential limitations in the dataset which might slightly bias the data towards particular topics, because not all topics included have the exact same representation in the dataset, and obviously it was not possible to cover all topics in existence.

Dataset Distribution

How is the dataset distributed? The dataset is available in HuggingFace32.

When will the dataset be released/first distributed? The dataset was released in 2024.

What license (if any) is it distributed under? The dataset is licensed under CC BY-NC-SA 4.0.

Are there any fees or access/export restrictions? There are no fees or restrictions.

Dataset Maintenance

Who is supporting/hosting/maintaining the dataset? How does one contact the owner/curator/manager of the dataset?
The dataset is hosted at HuggingFace, and the research institute can be contacted through email contacto.iic@iic.uam.es.

Will the dataset be updated? How often and by whom? How will updates/revisions be documented and communicated? Is there an erratum?
It is not planned to update the dataset at the moment.

Is there a repository to link to any/all papers/systems that use this dataset? No.

Legal & Ethical Considerations

If the dataset relates to people or was generated by people, were they informed about the data collection? Not applicable. The data was collected from public web sources, and does not contain sensitive personal information.

32 https://hf.co/datasets/IIC/AQuAS
If it relates to other ethically protected subjects, have appropriate obligations been met?  Not applicable.

If it relates to people, were there any ethical review applications/reviews/approvals?  Not applicable.

If it relates to people, were they told what the dataset would be used for and did they consent?  Not applicable.

If it relates to people, could this dataset expose people to harm or legal action?  Not applicable.

If it relates to people, does it unfairly advantage or disadvantage a particular social group?  Not applicable.

If it relates to people, were they provided with privacy guarantees?  Not applicable.

Does the dataset comply with the EU General Data Protection Regulation (GDPR)?  The dataset complies with GDPR.

Does the dataset contain information that might be considered sensitive or confidential?  No.

Does the dataset contain information that might be considered inappropriate or offensive?  No.

ClinTreatES

The ClinTreatES (LenguajeNatural.AI, 2024b) dataset consists of clinical cases collected directly from doctors in various medical specialties (cardiology, traumatology, emergency, psychiatry, neurology, dermatology, ENT-laryngology, and anaesthesia) across European medical centers. It was developed through a joint collaboration between LenguajeNatural.AI and healthcare professionals. The dataset is intended for evaluating the ability of large language models (LLMs) to generate effective treatment plans based on provided clinical cases and diagnoses.

Motivation for Dataset Creation

Why was the dataset created?  ClinTreatES was created to evaluate LLMs’ capability to design appropriate treatments from real clinical cases and their corresponding diagnoses.

What (other) tasks could the dataset be used for?  In addition to treatment planning, the dataset may be used to study medical reasoning and decision-making; however, it is not recommended for diagnostic tasks.

Who funded the creation of the dataset?  The dataset was developed through a collaboration between an NLP startup and healthcare professionals.

Dataset Composition

What are the instances?  Each instance comprises a clinical case description and its associated diagnosis.

Are relationships between instances made explicit in the data?  No, there are no explicit relationships between instances.

How many instances of each type are there?  The dataset contains 62 examples.

What data does each instance consist of?  Each instance includes text data: a clinical case and its corresponding diagnosis, which serves as the basis for generating a treatment plan.

Is everything included or does the data rely on external resources?  The dataset is self-contained with no reliance on external resources.

Are there recommended data splits or evaluation measures?  No specific splits are recommended; the dataset is intended primarily for evaluation purposes.
Data Collection Process

How was the data collected? Data was collected directly from healthcare professionals across various specialities in European medical centers.

Who was involved in the data collection process? Medical professionals from cardiology, traumatology, emergency medicine, psychiatry, neurology, dermatology, ENT-laryngology, and anesthesia contributed to the dataset.

Over what time-frame was the data collected? The data was collected in 2024.

How was the data associated with each instance acquired? Clinical cases and their corresponding diagnoses were directly provided by the contributing healthcare professionals.

Does the dataset contain all possible instances? It is a curated collection and does not cover every possible clinical case.

If the dataset is a sample, then what is the population? The dataset represents a curated sample of clinical cases from European medical centers.

Is there information missing from the dataset and why? No, all relevant information is included.

Are there any known errors, sources of noise, or redundancies in the data? The data has been carefully curated and reviewed to minimize errors and noise.

Data Preprocessing

What preprocessing/cleaning was done? The clinical texts were formatted according to a standardized template; only minimal preprocessing was applied.

Was the “raw” data saved in addition to the preprocessed/clean data? Yes, the dataset contains the original clinical texts as provided by the contributors.

Is the preprocessing software available? No specific preprocessing software was used.

Does this dataset collection/processing procedure achieve the motivation for creating the dataset stated in the first section of this datasheet? Yes, the collection and curation process ensures the dataset is suitable for evaluating treatment design tasks by LLMs.

Dataset Distribution

How is the dataset distributed? The dataset is available on HuggingFace33.

When will the dataset be released/first distributed? The dataset was released in March 2024.

What license (if any) is it distributed under? It is distributed under the CC BY-NC-SA 4.0 license.

Are there any fees or access/export restrictions? There are no fees or restrictions.

Dataset Maintenance

Who is supporting/hosting/maintaining the dataset? The dataset is hosted on HuggingFace and maintained by the NLP startup.

Will the dataset be updated? How often and by whom? No updates are planned at this time.

Is there a repository to link to any/all papers/systems that use this dataset? The dataset is available on HuggingFace34.

Legal & Ethical Considerations

If the dataset relates to people, were they informed about the data collection? The clinical cases were provided by healthcare professionals; any personal details have been removed to ensure anonymity. They were anonymized by the healthcare professionals themselves, before transferring the data to the NLP startup.

If it relates to other ethically protected subjects, have appropriate obligations been met? Yes, all obligations have been met and ensured in the data collection process.

If it relates to people, were there any ethical review applications/reviews/approvals? Yes, healthcare professionals ensured the ethical review was complete.

If it relates to people, were they told what the dataset would be used for and did they consent? Yes, patients were told in advance about the objective of data collection and they provided their consent for this use.

33https://hf.co/datasets/LenguajeNaturalAI/ClinTreatES
34https://hf.co/datasets/LenguajeNaturalAI/ClinTreatES
If it relates to people, could this dataset expose people to harm or legal action?  No, as the data is anonymized by the healthcare professionals.

If it relates to people, does it unfairly advantage or disadvantage a particular social group?  No.

If it relates to people, were they provided with privacy guarantees?  Yes, all personal information has been removed by the healthcare professionals.

Does the dataset comply with the EU General Data Protection Regulation (GDPR)?  Yes, it complies with GDPR.

Does the dataset contain information that might be considered sensitive or confidential?  No, all potentially sensitive or confidential information has been removed.

Does the dataset contain information that might be considered inappropriate or offensive?  No.

ClinDiagnosES

The ClinDiagnosES (LenguajeNatural.AI, 2024b) dataset comprises clinical cases accompanied by corresponding diagnoses, collected directly from healthcare professionals across multiple specialties in Europe. It is intended for evaluating LLMs’ diagnostic reasoning abilities.

Motivation for Dataset Creation

Why was the dataset created?  ClinDiagnosES was created to assess the ability of LLMs to generate accurate diagnoses based on clinical case descriptions.

What (other) tasks could the dataset be used for?  Besides diagnostic evaluation, it can be used to study medical reasoning; however, it is not suitable for treatment planning tasks.

Who funded the creation of the dataset?  The dataset was developed through a collaboration between LenguajeNatural.AI and healthcare professionals.

Dataset Composition

What are the instances?  Each instance consists of a clinical case description along with its corresponding diagnosis.

Are relationships between instances made explicit in the data?  No, there are no explicit relationships between instances.

How many instances of each type are there?  The dataset contains 62 examples.

What data does each instance consist of?  Each instance includes text data representing a clinical case and its associated diagnosis.

Is everything included or does the data rely on external resources?  The dataset is self-contained.

Are there recommended data splits or evaluation measures?  No specific splits are recommended; it is intended for evaluation purposes.

Data Collection Process

How was the data collected?  Data was collected directly from healthcare professionals across various medical specialties.
Who was involved in the data collection process?
Healthcare professionals from fields such as cardiology, traumatology, emergency medicine, psychiatry, neurology, dermatology, ENT-laryngology, and anesthesia contributed.

Over what time-frame was the data collected?
The data was collected in 2024.

How was the data associated with each instance acquired? Each clinical case was accompanied by a diagnosis provided by a medical expert.

Does the dataset contain all possible instances?
It is a curated collection and does not encompass every possible clinical case.

If the dataset is a sample, then what is the population?
The dataset represents a curated sample of clinical cases from European medical centers.

Is there information missing from the dataset and why?
No, all necessary information is included.

Are there any known errors, sources of noise, or redundancies in the data?
The dataset has been reviewed to minimize errors and inconsistencies.

Data Preprocessing

What preprocessing/cleaning was done?
The clinical cases and diagnoses were formatted using a standardized template with minimal cleaning.

Was the “raw” data saved in addition to the preprocessed/clean data?
Yes, the raw clinical texts and diagnoses are preserved.

Is the preprocessing software available?
No specific preprocessing software was utilized.

Does this dataset collection/processing procedure achieve the motivation for creating the dataset stated in the first section of this datasheet?
Yes, the procedure ensures the dataset is suitable for evaluating diagnostic reasoning in LLMs.

Dataset Distribution

How is the dataset distributed?
The dataset is available on HuggingFace35.

When will the dataset be released/first distributed?
It was released in March 2024.

What license (if any) is it distributed under?
It is distributed under the CC BY-NC-SA 4.0 license.

Are there any fees or access/export restrictions?
There are no fees or restrictions.

Dataset Maintenance

Who is supporting/hosting/maintaining the dataset?
The dataset is hosted on HuggingFace and maintained by the NLP startup.

Will the dataset be updated? How often and by whom?
No updates are planned at this time.

Is there a repository to link to any/all papers/systems that use this dataset?
The dataset is available on HuggingFace36.

Legal & Ethical Considerations

If the dataset relates to people, were they informed about the data collection?
The clinical cases were provided by healthcare professionals; any personal details have been removed to ensure anonymity. They were anonymized by the healthcare professionals themselves, before transferring the data to the NLP startup.

If it relates to other ethically protected subjects, have appropriate obligations been met?
Yes, all obligations have been met and ensured in the data collection process.

If it relates to people, were there any ethical review applications/reviews/approvals?
Yes, healthcare professionals ensured the ethical review was complete.

If it relates to people, were they told what the dataset would be used for and did they consent?
Yes, patients were told in advance about the objective of data collection and they provided their consent for this use.

If it relates to people, could this dataset expose people to harm or legal action?
No, as the data is anonymized by the healthcare professionals.

If it relates to people, does it unfairly advantage or disadvantage a particular social group?
No.

If it relates to people, were they provided with privacy guarantees?
Yes, all personal information has been removed by the healthcare professionals.

35https://hf.co/datasets/LenguajeNaturalAI/ClinDiagnosES
36https://hf.co/datasets/LenguajeNaturalAI/ClinDiagnosES
Does the dataset comply with the EU General Data Protection Regulation (GDPR)? Yes, it complies with GDPR.

Does the dataset contain information that might be considered sensitive or confidential? No, all potentially sensitive or confidential information has been removed.

Does the dataset contain information that might be considered inappropriate or offensive? No.

HumorQA

The HumorQA dataset (LenguajeNatural.AI, 2024a), developed collaboratively by LenguajeNatural.AI and Human Profit Consulting, focuses on humor classification. It consists of jokes paired with labels indicating the joke type: C/E (Comparison/Exaggeration), JP (Wordplay), R3 (Rule of Three) and AI (Animating the Inanimate). The data set is based on a study involving 94 executives and is intended to evaluate the ability of LLMs to understand and classify humor.

Motivation for Dataset Creation

Why was the dataset created? HumorQA was created to assess the ability of LLMs to recognize and classify different types of humor.

What (other) tasks could the dataset be used for? It can also be used for research on sentiment analysis and humor recognition, although its primary purpose is humor classification.

Who funded the creation of the dataset? The dataset was developed through a collaboration between an NLP startup and a psychology consulting firm.

Dataset Composition

What are the instances? Each instance comprises a joke and its corresponding humor-type label.

Are relationships between instances made explicit in the data? No, there are no explicit relationships between instances.

How many instances of each type are there? The dataset contains 51 examples.

What data does each instance consist of? Each instance includes text data representing a joke and a label indicating its humor category.

Is everything included or does the data rely on external resources? The dataset is self-contained.

Are there recommended data splits or evaluation measures? No specific splits are recommended; it is intended for evaluation purposes.

Data Collection Process

How was the data collected? Jokes were collected and curated as part of a research study in-
volving humor workshops and interviews with 94 executives.

Who was involved in the data collection process? The data collection involved humor experts at Human Profit Consulting along with participating executives.

Over what time-frame was the data collected? The data was collected in 2024.

How was the data associated with each instance acquired? Jokes were labeled according to a pre-defined categorization based on the study’s methodology.

Does the dataset contain all possible instances? It is a curated sample representing various humor styles.

If the dataset is a sample, then what is the population? The sample represents humorous content identified in a study with executives from diverse sectors.

Is there information missing from the dataset and why? No, all relevant information is included.

Are there any known errors, sources of noise, or redundancies in the data? The dataset has been thoroughly reviewed; no significant errors or redundancies have been identified.

Data Preprocessing

What preprocessing/cleaning was done? The jokes and labels were formatted into a standardized template with minimal preprocessing.

Was the “raw” data saved in addition to the preprocessed/clean data? Yes, the original joke texts are preserved.

Is the preprocessing software available? No specific preprocessing software was used.

Does this dataset collection/processing procedure achieve the motivation for creating the dataset stated in the first section of this datasheet? Yes, the curation process supports the evaluation of humor classification by LLMs.

Dataset Distribution

How is the dataset distributed? The dataset is available on HuggingFace37.

When will the dataset be released/first distributed? It was released in March 2024.

What license (if any) is it distributed under? It is distributed under the CC BY-NC-SA 4.0 license.

Are there any fees or access/export restrictions? There are no fees or restrictions.

Dataset Maintenance

Who is supporting/hosting/maintaining the dataset? The dataset is hosted on HuggingFace by the NLP startup.

Will the dataset be updated? How often and by whom? No updates are planned at this time.

Is there a repository to link to any/all papers/systems that use this dataset? The dataset is available on HuggingFace38.

Legal & Ethical Considerations

If the dataset relates to people, were they informed about the data collection? The dataset is based on humorous content and research; it does not involve personal data.

If it relates to other ethically protected subjects, have appropriate obligations been met? Not applicable.

If it relates to people, were there any ethical review applications/reviews/approvals? Not applicable.

If it relates to people, were they told what the dataset would be used for and did they consent? Not applicable.

If it relates to people, could this dataset expose people to harm or legal action? No.

If it relates to people, does it unfairly advantage or disadvantage a particular social group? No.

If it relates to people, were they provided with privacy guarantees? Not applicable.

Does the dataset comply with the EU General Data Protection Regulation (GDPR)? Yes, it complies with GDPR.

Does the dataset contain information that might be considered sensitive or confidential? No.

Does the dataset contain information that might be considered inappropriate or offensive? No.

37 https://hf.co/datasets/LenguajeNaturalAI/HumorQA
38 https://hf.co/datasets/LenguajeNaturalAI/HumorQA
RagQuAS

The Retrieval-Augmented-Generation and Question-Answering in Spanish (RagQuAS) dataset (Instituto de Ingeniería del Conocimiento, 2024b), created by Instituto de Ingeniería del Conocimiento, is a high-quality monolingual Spanish dataset designed to evaluate models in retrieval-augmented generation (RAG) and question-answering tasks. It consists of 201 examples covering a wide range of knowledge domains, such as hobbies, linguistics, health, astronomy, and customer service. Each example includes a question, multiple context passages extracted from different documents, and a gold-standard answer. This dataset is particularly useful for assessing a model’s ability to retrieve relevant information from multiple sources and generate accurate, contextually appropriate responses.

Motivation for Dataset Creation

Why was the dataset created? (e.g., were there specific tasks in mind, or a specific gap that needed to be filled?) RagQuAS was created to provide high-quality examples of questions and answers with related contexts that can be used to evaluate models in retrieval-augmented generation (RAG) and question-answering tasks.

What (other) tasks could the dataset be used for? Are there obvious tasks for which it should not be used? There are no recommended uses for this dataset other than evaluation.

Who funded the creation of the dataset? If there is an associated grant, provide the grant number. The dataset was created and funded by Instituto de Ingeniería de Conocimiento.

Dataset Composition

What are the instances? (that is, examples; e.g., documents, images, people, countries) Are there multiple types of instances? (e.g., movies, users, ratings; people, interactions between them; nodes, edges) Each instance consists of several categories of text: the topic, a question, an indicator of the variant of the question (this represents questions with linguistic differences but pertaining to the same contexts than other questions), an answer, one to five contexts, one to five complete documents from where the contexts were extracted and the links to these documents.

Are relationships between instances made explicit in the data (e.g., social network links, user/movie ratings, etc.)? There are no known relationships between instances.

How many instances of each type are there? The dataset consists of 201 examples.

What data does each instance consist of? “Raw” data (e.g., unprocessed text or images)? Features/attributes? Is there a label/target associated with instances? If the instances are related to people, are subpopulations identified (e.g., by age, gender, etc.) and what is their distribution? The instances consist of text data and are labeled with the corresponding topic.

Is everything included or does the data rely on external resources? Everything is included in the dataset.

Are there recommended data splits or evaluation measures? Since the dataset is intended for testing, there is no recommended split.

Data Collection Process

How was the data collected? The data for the contexts was gathered from different sources manually with the help of generative models (to suggest web searches and results). The rest of the dataset was curated and created manually.

Who was involved in the data collection process? How were they compensated? The data was collected by computational linguists and data scientists from the research institute.

Over what time-frame was the data collected? The data was collected during 2023, when the dataset was created.

How was the data associated with each instance acquired? The question-answer pairs were created and revised by computational linguists.

Does the dataset contain all possible instances? The dataset is composed of selected instances of a dataset created by the research institute.

If the dataset is a sample, then what is the population? This dataset is a 24% sample of the original complete datasets. The instances were randomly selected from the original dataset.

Is there information missing from the dataset and why? There is no data missing.
Are there any known errors, sources of noise, or redundancies in the data? There are no known errors because the revision process ensured the data is as clean and error free as possible.

Data Preprocessing

What preprocessing/cleaning was done? The text contained in context and document part of each instance in the dataset has not undergone any pre-processing or changes. The questions were created manually by computational linguists following a rigorous methodology and were subjected to revision afterwards. The answers were carefully curated and revised by linguists from generated texts.

Was the “raw” data saved in addition to the preprocessed/clean data? No, the text in the dataset is the raw data.

Is the preprocessing software available? No preprocessing software was used.

Does this dataset collection/processing procedure achieve the motivation for creating the dataset stated in the first section of this datasheet? Yes, the methodology used when creating the dataset ensures it is sufficiently varied so it can be used to evaluate a model on a wide range of topics. However, there are some potential limitations in the dataset which might slightly bias the data towards particular topics, because not all topics included have the exact same representation in the dataset, and obviously it was not possible to cover all topics in existence.

Dataset Distribution

How is the dataset distributed? The dataset is available in HuggingFace39.

When will the dataset be released/first distributed? The dataset was released in 2024.

What license (if any) is it distributed under? Are there any copyrights on the data? The dataset is licensed under CC BY-NC-SA 4.0.

Are there any fees or access/export restrictions? There are no fees or restrictions.

Dataset Maintenance

Who is supporting/hosting/maintaining the dataset? How does one contact the owner/curator/manager of the dataset? The dataset is hosted at HuggingFace, and the research institute can be contacted through email contacto.iic@iic.uam.es.

Will the dataset be updated? How often and by whom? How will updates/revisions be documented and communicated? Is there an erratum? It is not planned to update the dataset at the moment.

Is there a repository to link to any/all papers/systems that use this dataset? No.

Legal & Ethical Considerations

If the dataset relates to people or was generated by people, were they informed about the data collection? Not applicable. The data was collected from public web sources, and does not contain sensitive personal information.

If it relates to other ethically protected subjects, have appropriate obligations been met? Not applicable.

If it relates to people, were there any ethical review applications/reviews/approvals? Not applicable.

If it relates to people, were they told what the dataset would be used for and did they consent? Not applicable.

If it relates to people, could this dataset expose people to harm or legal action? Not applicable.

If it relates to people, does it unfairly advantage or disadvantage a particular social group? Not applicable.

If it relates to people, were they provided with privacy guarantees? Not applicable.

Does the dataset comply with the EU General Data Protection Regulation (GDPR)? The dataset complies with GDPR.

Does the dataset contain information that might be considered sensitive or confidential? No.

Does the dataset contain information that might be considered inappropriate or offensive? No.

39https://hf.co/datasets/IIC/RagQuAS
SpaLawEx

The SpaLawEx dataset (LenguajeNatural.AI, 2024c) consists of multiple-choice legal questions extracted from Spanish Bar Examination papers of 2022 and 2023. Each instance includes a legal question along with four answer options (A, B, C, and D).

Motivation for Dataset Creation

Why was the dataset created? SpaLawEx was created to evaluate the legal reasoning and knowledge of LLMs within the context of Spanish law using multiple-choice questions.

What (other) tasks could the dataset be used for? In addition to benchmarking legal question answering systems, it may be used for legal education; it is not intended for non-legal tasks.

Who funded the creation of the dataset? The dataset was developed by an NLP startup, with contributions from legal experts.

Dataset Composition

What are the instances? Each instance is a multiple-choice legal question accompanied by four answer options.

Are relationships between instances made explicit in the data? No, there are no explicit relationships between instances.

How many instances of each type are there? The dataset contains 119 examples.

What data does each instance consist of? Each instance comprises text data, including a legal question and its four answer options (A, B, C, and D).

Is everything included or does the data rely on external resources? The dataset is self-contained, extracted from publicly available examination papers.

Are there recommended data splits or evaluation measures? No specific splits are recommended; the dataset is intended for evaluation purposes.

Data Collection Process

How was the data collected? Data were extracted from official Spanish Bar Examination papers from 2022 and 2023.

Who was involved in the data collection process? The extraction was performed by the developers at an NLP startup, with input from legal experts.

Over what time-frame was the data collected? The data was collected in 2024.

How was the data associated with each instance acquired? Questions and answer options were directly extracted from exam documents.

Does the dataset contain all possible instances? It is a comprehensive collection of questions from the specified examination periods. However, it is not exhaustive and it does not contain all possible instances.

If the dataset is a sample, then what is the population? It represents the pool of questions from the Spanish Bar Examinations of 2022 and 2023.

Is there information missing from the dataset and why? No, all relevant information is included.

Are there any known errors, sources of noise, or redundancies in the data? The dataset has been checked for accuracy; any minor extraction errors are not known to be significant.

Data Preprocessing

What preprocessing/cleaning was done? The exam questions and answer options were formatted into a standardized template with minimal cleaning.

Was the “raw” data saved in addition to the preprocessed/clean data? Yes, the original extracted text is preserved.

Is the preprocessing software available? No specific preprocessing software was used.

Does this dataset collection/processing procedure achieve the motivation for creating the dataset stated in the first section of this datasheet? Yes, the process ensures the dataset is suitable for evaluating legal reasoning in LLMs.

Dataset Distribution

How is the dataset distributed? The dataset is available on HuggingFace40.

When will the dataset be released/first distributed? It was released in March 2024.

40https://hf.co/datasets/LenguajeNaturalAI/SpaLawEx
What license (if any) is it distributed under?  It is distributed under the CC BY-NC-SA 4.0 license.

Are there any fees or access/export restrictions?  There are no fees or restrictions.

Dataset Maintenance
Who is supporting/hosting/maintaining the dataset?  The dataset is hosted on HuggingFace by the NLP startup.

Will the dataset be updated? How often and by whom?  No updates are planned at this time.

Is there a repository to link to any/all papers/systems that use this dataset?  No repository has been provided.

Legal & Ethical Considerations
If the dataset relates to people, were they informed about the data collection?  The dataset is derived from public examination materials and does not involve personal data.

If it relates to other ethically protected subjects, have appropriate obligations been met?  Not applicable.

If it relates to people, were there any ethical review applications/reviews/approvals?  Not applicable.

If it relates to people, were they told what the dataset would be used for and did they consent?  Not applicable.

If it relates to people, could this dataset expose people to harm or legal action?  No.

If it relates to people, does it unfairly advantage or disadvantage a particular social group?  No.

If it relates to people, were they provided with privacy guarantees?  Not applicable.

Does the dataset comply with the EU General Data Protection Regulation (GDPR)?  Yes, it complies with GDPR.

Does the dataset contain information that might be considered sensitive or confidential?  No.

Does the dataset contain information that might be considered inappropriate or offensive?  No.

TELEIA

The TELEIA (Mayor-Rocher et al., 2025) dataset is intended for the evaluation of Spanish language knowledge focusing on reading comprehension and grammatical competence. The dataset is designed as a set of multiple-choice questions that have the same format and level as those used in several Spanish evaluation tests for humans. The questions are divided into three blocks which resemble existing tests of Spanish for foreign learners and University access. In total, one hundred questions are included that have been prepared and revised by experts on Spanish language, and that have been validated by comparing the results with the original exams.

Motivation for Dataset Creation

Why was the dataset created?  The main motivation was to have a simple test to evaluate the competence of LLMs in Spanish, similar to tests used with humans.

What (other) tasks could the dataset be used for?  The test also checks reading comprehension and thus can be used to evaluate natural language understanding.

Who funded the creation of the dataset?  The development of the dataset was supported by the FUN4DATE (PID2022-136684OB-C22) project funded by the Spanish Agencia Estatal de Investigación (AEI) 10.13039/501100011033.

Dataset Composition

What are the instances?  The test is made of multiple-choice questions.

Are relationships between instances made explicit in the data  No.

How many instances of each type are there?  The dataset consists of 100 questions.

What data does each instance consist of?  Each question has a text presenting the question and four answer options, of which only one is correct.

Is everything included or does the data rely on external resources?  Everything is included in the dataset.

Are there recommended data splits or evaluation measures?  No.
Data Collection Process

How was the data collected? Questions were formulated and peer-reviewed by experts in Spanish.

Who was involved in the data collection process? Experts in Spanish who participated as researchers in our group.

Over what time-frame was the data collected? The questions were created during the spring of 2024.

How was the data associated with each instance acquired? Data was created by experts.

Does the dataset contain all possible instances? Questions are examples, and many other similar questions can be formulated.

If the dataset is a sample, then what is the population? Not applicable.

Is there information missing from the dataset and why? No.

Are there any known errors, sources of noise, or redundancies in the data? No.

Data Preprocessing

What preprocessing/cleaning was done? None.

Was the “raw” data saved in addition to the preprocessed/clean data? Not applicable.

Is the preprocessing software available? Not applicable.

Does this dataset collection/processing procedure achieve the motivation for creating the dataset stated in the first section of this datasheet? Yes.

Dataset Distribution

How is the dataset distributed? Websites.

When will the dataset be released/first distributed? Data is available since July 2024.

What license (if any) is it distributed under? No license or restrictions are applicable.

Are there any fees or access/export restrictions? No.

Dataset Maintenance

Who is supporting/hosting/maintaining the dataset? The dataset is hosted at Zenodo\footnote{https://zenodo.org/records/12571763} providing contact details for all the authors.

Will the dataset be updated? No updates are expected, but the repository supports versioning.

Is there a repository to link to any/all papers/systems that use this dataset? No.

Legal & Ethical Considerations

If the dataset relates to people (e.g., their attributes) or was generated by people, were they informed about the data collection? Not applicable.

If it relates to other ethically protected subjects, have appropriate obligations been met? Not applicable.

If it relates to people, were there any ethical review applications/reviews/approvals? Not applicable.

If it relates to people, were they told what the dataset would be used for and did they consent? Not applicable.

If it relates to people, could this dataset expose people to harm or legal action? Not applicable.

If it relates to people, does it unfairly advantage or disadvantage a particular social group? Not applicable.

If it relates to people, were they provided with privacy guarantees? Not applicable.

Does the dataset comply with the EU General Data Protection Regulation (GDPR)? Yes.

Does the dataset contain information that might be considered sensitive or confidential? No.

Does the dataset contain information that might be considered inappropriate or offensive? No.

Reviewer paper summaries: The paper introduces the La Leaderboard benchmark, aiming to test LM competence in four languages: Spanish, Galician, Catalan and Basque. The authors benchmark several SOTA LMs and collate a collection of existing as well as newly introduced datasets in the languages mentioned above.

This paper presents a benchmark (LA LEADERBOARD) for evaluating LLMs in languages from Spain and Latin America.
The benchmark contains many datasets (66) and the authors curated the balance between open-choice and generation tasks. The authors evaluate 50 models. The paper is a valuable contribution to assessing LLMs on Spanish datasets.

This paper introduced a leaderboard aiming to test LLMs’ ability to solve different types of problems in Spanish, Catala, Basque, and Galician. The benchmark includes 66 high-quality datasets in different tasks and domains. Except for the content of the benchmark, they also introduced the path from the beginning to the final community-driven project, which can provide some experience for others to build similar leaderboards for other languages. Besides, the authors also conducted an in-depth analysis of the performance across different languages, model sizes, tasks, etc.
Summary of strengths: 1. The benchmark does seem to offer substantial headroom for future improvement.
2. The authors contribute seven new datasets in the process of collating the benchmark.
3. The paper is well written and presented.

1) Comprehensive Benchmark.
2) Balanced task selection.
3) Detailed Analysis.
4) The majority of the datasets are in Spanish or manually translated.

1. It is evident that the author has invested a significant amount of effort in collecting high-quality data and testing models on these datasets, providing a solid foundation for future Spanish LLM work. Also, the approach to developing such a community-driven leaderboard for other languages is valuable.

2. The dataset covers a wide range, including multiple domains, various tasks, and several languages, which the authors present clearly.

3. The authors place great emphasis on the fairness of leaderboard comparisons and dedicate a substantial portion to describing the evaluation protocol used to ensure this fairness, which is crucial for comparing models.
Summary of weaknesses: 1. The benchmark doesn't seem to do a very good job at separating models, some of which have undergone vastly different styles or extents of pre-training on the languages under test.
2. The benchmark caters to a set of 4 languages, of which, several are already extremely well served by evaluation benchmarks.

1) Model categorization is not completely clear.
2) Limited discussion on data sources used for training the models.
3) The appendix is too long and contains relevant information that should be included in the main content.
4) The paper content is suitable for a journal paper instead of a conference/workshop paper.

1. The differences compared to previous works such as ODESIA and CLUB are not so significant. Especially, the statement in Lines 192-193, '... but making it impossible to reproduce the results.' is not correct, because everyone can submit their results to compute the benchmarking scores.

2. One of the contributions claimed in this paper is providing a comprehensive analysis. However, many experimental results are placed in the Appendix but not the main content. According to the ARR Reviewer Guidelines: '**Do I have to read the appendices?** The paper should be readable by itself, and any details important for understanding the key aspects of the work **should be in the paper rather than in appendices**.', the claimed contribution may not be supported.
Comments, suggestions and typos: N/A

1) I am not convinced of the main distinction between SOTA models and language-optimized models. The reason behind this choice needs to be better clarified. Indeed, it would have been interesting to investigate the amount and percentage of data in the languages of interest used for creating these models (of course, when this information is available).
2) It could also have helped to shed more light on the results since SOTA models performed better than language-optimized models.
3) The paper has a very long appendix that exceeds the length of the main paper: 8 pages and 38 pages of the appendix. The appendix contains essential content that should be included in the main content.

Typos
Please consider putting footnotes before punctuations (e.g. "Evaluation Harness,21" or "template.22")

## Presentation Improvement
- in Lines 241-247, it is your plan instead of your work in this paper. So, it should be moved to other sections such as the Section Future Work or Section Limitations.
Overall assessment scores: 3

3.5

3.5