Latxa: An Open Language Model and Evaluation Suite for Basque (2024)

Abstract: We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. Addressing the scarcity of high-quality benchmarks for Basque, we further introduce 4 multiple choice evaluation datasets: EusProficiency, comprising 5,169 questions from official language proficiency exams; EusReading, comprising 352 reading comprehension questions; EusTrivia, comprising 1,715 trivia questions from 5 knowledge areas; and EusExams, comprising 16,774 questions from public examinations. In our extensive evaluation, Latxa outperforms all previous open models we compare to by a large margin. In addition, it is competitive with GPT-4 Turbo in language proficiency and understanding, despite lagging behind in reading comprehension and knowledgeintensive tasks. Both the Latxa family of models, as well as our new pretraining corpora and evaluation datasets, are publicly available under open licenses.1 Our suite enables reproducible research on methods to build LLMs for lowresource languages.

heading: 1 Introduction
text: Motivated by their increasing training cost and commercial interest, the development of Large Language Models (LLMs) has been led by close initiatives like GPT (OpenAI et al., 2023), Claude (Wu et al., 2023) and Gemini (Team, 2023). In recent times, a more open ecosystem has emerged following the release of various competitive models like Llama 2 (Touvron et al., 2023) and Mistral (Jiang et al., 2024). However, despite early efforts to build open multilingual models (Lin et al., 2022; Scao et al., 2023), the most competitive ones
*Equal contribution. 1https://github.com/hitz-zentroa/latxa
are notoriously English-centric. As shown in Table 1, all these open models perform poorly in lowresource languages like Basque, with most results marginally surpassing random chance.
In this work, we present Latxa, an open family of LLMs for Basque that substantially outperforms all these previous models. Basque is an agglutinative language written in Latin script and with no known relatives, although a significant part of the vocabulary is shared with contact languages like Spanish and French. Basque is the 52th language in Common Crawl, with 0.035% of the total content –for reference, English is the 1st language with 46% of the content and Spanish is the 5th with 4.6%. Our work builds on various open resources and models that we further expand to Basque, highlighting the importance of an open ecosystem for the development of language technology for lowresource languages. In particular, our models are based on Llama 2, which we continue training in Basque using a new corpus with 4.3M documents from 4 existing and 3 new sources. In addition, we release 4 diverse and challenging multiple-choice benchmarks comprising a total of 23,282 questions, covering language proficiency, reading comprehension, trivia questions, and public examinations.
As shown in Table 1, Latxa performs substantially better than all existing open models, with the 70B variant outperforming the previous best open model (Yi 34B) by 18.95 points in average. In addition, it also outperforms the Llama 2 model it is based on by 25.18 points, and it is also superior to GPT-3.5 Turbo in all datasets we evaluate on. Interestingly, our best model also outperforms GPT-4 Turbo in language proficiency exams (EusProf), despite lagging behind in reading comprehension and knowledge-intensive tasks. This suggests that the capabilities that an LLM exhibits in a given language are not determined by its linguistic competence in this particular language, opening the doors to further improvements in low-resource LLMs as
14952
stronger English models become available. This paper makes the following contributions: (1) We release a high-quality corpus for Basque, comprising 4.3M documents and 4.2B tokens. The corpus combines the EusCrawl v1.1, Egunkaria, Booktegi, Wikipedia, CulturaX, Colossal OSCAR and HPLT v1 datasets (the first 3 being new), which we carefully deduplicate and filter. (2) We release the Latxa family of Basque LLMs, comprising 3 models with 7B, 13B and 70B parameters. (3) We release 4 new multiple-choice benchmarks for Basque: EusProficiency (official language proficiency exams), EusReading (reading comprehension), EusTrivia (trivia questions from 5 knowledge areas), and EusExams (public examinations). (4) We present extensive experiments comparing Latxa to previous open and closed models. (5) We show that it is possible to train significantly stronger LLMs for low-resource languages building on the existing ecosystem of open models and resources. In a similar spirit to other open LLMs, such as Pythia (Biderman et al., 2023), LLM360 (Liu et al., 2023) and OLMO (Groeneveld et al., 2024), we release all the necessary data, code, weights and documentation to run and evaluate our models, facilitating similar efforts for other low-resource languages.

heading: 2 Training Corpora
text: Our training corpus combines various existing datasets, as well as some new ones that we release with this work. We have prioritized quality over quantity when constructing our corpus, prioritizing
high-quality data sources and applying a thorough deduplication and filtering process. We next describe our data sources in §2.1, followed by our preprocessing pipeline in §2.2. Table 2 summarizes the statistics of the resulting dataset.

heading: 2.1 Data Sources
text: [new] EusCrawl v1.1. The original version of EusCrawl (Artetxe et al., 2022) was built using ad-hoc scrapers to extract text from 33 newswire websites, resulting in higher quality documents compared to general-purpose approaches. In this work, we release an updated version of EusCrawl, including new content up to November 2023. This increases the number of unique documents from 1.38 to 1.94 millions, for a total of 384 million words. [new] Egunkaria. Euskaldunon Egunkaria was a daily newspaper written fully in the Basque language. The corpus includes approximately 176k news articles, editorials, and various types of reviews from the years 2001 to 2006, totalling 39 million words. [new] Booktegi. The Booktegi platform hosts free content in Basque, such as books, interviews, and audio materials. The corpus comprises approximately 3 million words from 166 EPUB books covering essays, fiction, and poetry. Wikipedia. We download and process a Basque Wikipedia dump,2 obtaining nearly 550k documents and more than 54 million words. The plain text has been extracted with WikiExtractor.3
2The 20231101 dump corresponding to November 2023. 3https://github.com/attardi/wikiextractor
CulturaX. CulturaX (Nguyen et al., 2023) is a large multilingual dataset resulting from the combination and processing of mC4 (Xue et al., 2021) and the four OSCAR releases 2019, 21.09, 22.01, and 23.01 (Ortiz Suárez et al., 2019). These corpora originate, in turn, from 66 Common Crawl (CC) snapshots spanning from 2015 to 2022. Basque content constitutes 0.02% of CulturaX, encompassing nearly 1.60 million documents and 622 million words. Colossal OSCAR. The largest release of the OSCAR project (Ortiz Suárez et al., 2019) to date, Colossal OSCAR 1.0, is based on 10 CC snapshots. Here, we use the two snapshots not covered by CulturaX, namely, 06-07-22 and 05-06-23. Additionally, we have had access to an OSCAR-processed CC snapshot from April 2023. In total, we have obtained almost 650k documents in Basque from these datasets, totalling 282 million words. HPLT v1. The High Performance Language Technologies project (HPLT; Aulamo et al., 2023) compiled another massive, multilingual dataset from the Internet Archive and CC. In this work, we use the first release, which contains 2.29 million documents (1.55 billion words) in Basque. It must be noted that, unlike the aforementioned web-based sources, the HPLT dataset was released without any deduplication or filtering. Consequently, our preprocessing approach has been particularly aggressive with this dataset (see §2.2).

heading: 2.2 Preprocessing
text: We used the Dolma toolkit (Soldaini et al., 2024) and Corpus Cleaner v2 (CCv2; Palomar-Giner et al., 2024) to normalize, deduplicate and filter the datasets. Since the majority of our data sources were intentionally selected, organized, and/or curated by their respective authors, our main focus has been on removing outliers and cross-dataset
duplicates. This process is briefly outlined below, with further details available in Appendix A. The final size of the processed corpus is shown in Table 2. In total, it amounts to 1.22B words and 4.17B Llama 2 tokens. Each dataset is shuffled and then split separately into testing (1%), development (1%) and training (98%) example sets.
Normalization. CCv2 is first used to fix document encoding and whitespace normalization.
Deduplication. Cross-dataset document repetitions are identified and removed at both the URL and document content levels. Specifically, we conduct near-deduplication with Bloom filters (Bloom, 1970) as implemented in Dolma. To maximise corpus quality, we prioritized content from wellcurated sources (Wikipedia, EusCrawl, Egunkaria and Booktegi) then from massive but comparatively cleaner sources (CulturaX and Colossal OSCAR) over HPLT (see Figure 2(a) in Appendix A). The latter undergoes additional deduplication at the paragraph level.
Filtering. Documents unlikely to contain quality content are identified and removed in two stages. First, a combination of heuristics from Gopher (Rae et al., 2022) and C4 (Raffel et al., 2020), with adaptations tailored to the Basque language is applied (e.g., regarding average word length). We also perform language identification with CLD2 through Dolma, which has predominantly impacted HPLT, with approximately one-third of this corpus being discarded at this stage. Finally, the corpora are processed with CCv2, which assigns an aggregated quality score per document based on a comprehensive set of taggers. Once again, HPLT has been affected most, resulting in a further 25% reduction in document counts for this dataset.

heading: 3 Latxa Models
text: We train 7B, 13B and 70B models following a continued pretraining approach. To that end, we use Llama 2 as the base model (Touvron et al., 2023), and continue training it using the corpus described in §2. To mitigate catastrophic forgetting from the original model4, we also include English data in the continued pretraining stage. For that purpose, we use 500k random documents from The Pile (Gao et al., 2020), totaling 0.9B tokens.

heading: 3.1 Pretraining Details
text: The training of Latxa has been conducted using the GPT-Neox (Andonian et al., 2023) library. As infrastructure, we have leveraged the CINECA HPC Leonardo computing cluster located in Italy, which is powered by 3,456 nodes each containing 4x custom A100 64GB GPUs. The models are trained for 10k steps with a sequence length of 4,096 tokens and an effective batch size of 1M tokens, resulting in a total of 10B tokens. We use a cosine learning rate schedule, with a warm-up of 500 steps and decaying down to 3% of the peak learning rate. We set up the peak learning rate to be 1 × 10−4. All other hyperparameters follow Touvron et al. (2023). Figure 1 shows the validation perplexity during training.

heading: 3.2 Carbon Emissions
text: Pretraining LLMs requires compute-expensive experiments, carrying a significant carbon footprint. The 7B, 13B and 70B models were trained on 32, 64 and 256 GPUs, respectively. We report the compute hours and power consumption involved in our experiments in Table 3. The carbon emitted was es-
4Our preliminary experiments showed that adding English data was critical for strong few-shot performance. For instance, an early version of our 7B model without English data obtained 23.67 points on BeleBele (Bandarkar et al., 2023), 13 points less than the corresponding version with English data.
timated using a GPU power consumption of 440W and a carbon efficiency of 0.297kg/kWh (carbon efficiency on Italy on February 9, 2024, according to ElectricityMaps5).

heading: 4 New Evaluation Datasets
text: To overcome the scarcity of Basque benchmarks that are suitable for evaluating base language models, we collect new evaluation data from various online games and tests. We have decided to take this approach instead of translating existing datasets to avoid translation artifacts (Artetxe et al., 2020). Most importantly, this allows to have localized datasets that test the models’ knowledge about topics that are most relevant for Basque speakers. These tasks cover language proficiency (EusProficiency), reading comprehension (EusReading), trivia questions (EusTrivia), and exams of advanced professional level (EusExams). All the datasets consist of multiple-choice questions, making them suitable for few-shot learning akin to MMLU (Hendrycks et al., 2021) in English. We next describe each dataset in more detail, while Table 4 summarizes their statistics. For examples of each task, see Table 9 in Appendix C.
EusProficiency. EusProficiency comprises 5,169 exercises on different topics from past EGA exams, the official C1-level certificate of proficiency in Basque. We have collected the atarikoa exercises from EGA exams through the years 1998 to 2008. Atarikoa is the first qualifying test of EGA, which measures different aspects of language competency, such as reading comprehension, grammar, vocabulary, spelling, and writing. Each test generally has 85 multiple-choice questions, with 4 choices and a single correct answer. Currently, there is no comparable dataset available, nor could one be obtained by translating existing analogous datasets from other languages.
5https://www.electricitymaps.com/
EusReading. EusReading consists of 352 reading comprehension exercises (irakurmena) sourced from the same set of past EGA exams. Each test generally has 10 multiple-choice questions, with 4 choices and a single correct answer. These exercises are more challenging than Belebele (Bandarkar et al., 2023) due to the complexity and length of the input texts (see Table 4). As a result, EusReading is useful to measure long context understanding of models.
EusTrivia. EusTrivia consists of 1,715 trivia questions from multiple online sources. 56.3% of the questions are elementary level (grades 3-6), while the rest are considered challenging. A significant portion of the questions focuses specifically on the Basque Country, its language, and culture. Each multiple-choice question contains two, three or four choices (3.84 on average) and a single correct answer. Five areas of knowledge are covered:
• Humanities and Natural Sciences (27.8%): This category encompasses questions about history, geography, biology, ecology and other social and natural sciences. • Leisure and Art (24.5%): This category includes questions on sports and athletes, performative and plastic arts and artists, architecture, cultural events, and related topics. • Music (16.0%): Here are grouped all the questions about music and musicians, both classical and contemporary. • Language and Literature (17.1%): This category is concerned with all kinds of literature productions and writers, as well as metalinguistic questions (e.g., definitions, synonyms, and word usage). • Mathematics and ICT (14.5%): This category covers mathematical problems and questions about ICT, as well as questions about people known for their contributions to these fields of knowledge.
EusExams. EusExams is a collection of tests designed to prepare individuals for Public Service examinations conducted by several Basque institutions, including the public health system Osakidetza, the Basque Government, the City Councils of Bilbao and Gasteiz, and the University of the Basque Country (UPV/EHU). Within each of these groups, there are different exams for public positions, such as administrative and assistant roles. Each multiple-choice question contains 2 to
4 choices (3.90 on average) and one correct answer. The dataset is mostly parallel with 16k questions in Basque and 18k in Spanish, from which we only consider the Basque subset. It could be said to be similar to MMLU’s professional-level questions, but focusing on knowledge relevant to the Basque community, with questions related to local public services and law.

heading: 5 Experimental Setting
text: To assess the quality of Latxa models, we thoroughly evaluate them on a suite of diverse and challenging tasks against state-of-the-art models. In this section, we present the baseline models (§5.1) and tasks (§5.2), and describe the evaluation framework (§5.3). Results are analyzed and discussed in the next section (§6).

heading: 5.1 Baseline Models
text: We test the Llama 2 models (Touvron et al., 2023) to show the difference after our continued pretraining. We also evaluated other multilingual base models that do not intentionally include Basque in pretraining, namely, Mistral-7B (Jiang et al., 2023), Mixtral-8x7B (Jiang et al., 2024), and 01.AI’s Yi34B (AI et al., 2024).
Furthermore, we evaluate some of the leading open multilingual language models for Basque available to date, including BLOOM-7B (Scao et al., 2023), XGLM-7B (Lin et al., 2022), and mGPT-13B (Shliazhko et al., 2024). These models cover a broader range of languages than more re-
cent models, but are trained on fewer tokens and exhibit generally weaker performance.
Finally, we tested the latest GPT-3.5 Turbo (gpt-3.5-turbo-0125) and GPT-4 Turbo (gpt-4-1106-preview for BasqueGLUE tasks and gpt-4-0125-preview for the rest), as they are the leading commercial models for Basque.

heading: 5.2 Evaluation Datasets
text: In addition to the new evaluation datasets introduced in §4, the models have been evaluated on the following benchmarks:
• Belebele (Bandarkar et al., 2023): A multiplechoice reading comprehension dataset spanning 122 language variants. • XStoryCloze (Lin et al., 2022): A professionally translated version of the StoryCloze dataset (Mostafazadeh et al., 2017) to 10 nonEnglish languages. StoryCloze is a commonsense reasoning dataset that consists in choosing the correct ending to a four-sentence story. • BasqueGLUE (Urbizu et al., 2022): A collection of 6 NLU datasets for Basque: sentiment analysis (BEC), stance detection (VaxxStance), topic classification (BTHCv2), coreference detection (EpecKorrefBin), questionanswering NLI (QNLIeu), and word-incontext (WiCeu).
Collectively, these datasets allow us to evaluate the performance of the models on a wide range of competences including world knowledge, linguistic knowledge, reading comprehension, and common sense reasoning.
Following previous work (Brown et al., 2020; Touvron et al., 2023), we check for n-gram overlaps between these evaluation datasets and Latxa’s training corpus, and find no evidence of wholesale or annotation contamination (Chowdhery et al., 2023; Sainz et al., 2023). Further information on our contamination study can be consulted in Appendix B.

heading: 5.3 Evaluation Framework
text: The models are evaluated using the LM Evaluation Harness library (Biderman et al., 2024) by Eleuther AI. To that end, we have implemented the new evaluation datasets following similar multiplechoice datasets that are already included in the library, such as Belebele. The specific prompts and examples for each task are reported in Table 9 in Appendix C. BasqueGLUE has also been im-
plemented as a generative evaluation dataset (see Table 10).6
We use 5 in-context examples for all tasks but two: following common practice, XStoryCloze is evaluated in a 0-shot setting, and EusReading is evaluated in a 1-shot fashion, as more examples would not fit into the context of most models. In all cases, we compute the log probabilities of all candidates, and pick the one with the highest score as the models’ final answer.
For GPT models, we have implemented the evaluation using the OpenAI API. We have kept the evaluation as similar as possible to allow a fair comparison with our models. As getting log probabilities of candidate XStoryCloze continuations from the API is not possible, we have decided not to evaluate GPT in that task. For few-shot tasks, we use the same prompts and provide few-shot examples as user and assistant messages. In addition, we use a system prompt in English to specify the set of candidate answers per task (see Appendix C).

heading: 6 Results
text: We report our main results in Table 1, while Table 5 and Table 6 report fine-grained results on the different subsets of EusTrivia and BasqueGLUE, respectively. In what follows, we summarize our main findings:
Effectiveness of continued pretraining. Latxa obtains the best results in each compute class, outperforming all previous open models by a large margin. As the only exception, Mistral 7B is better
6Where possible, we based our prompts on existing analogous datasets: QNLI for QNLIeu, WiC for WiCeu, and WSC for EpecKorrefBin.
than Latxa 7B on Belebele and EusReading, but Latxa 7B wins in all the other 5 datasets. Our best model obtains 61.08 points on average, outperforming the previous best open model by 18.95 points. In addition, it outperforms the Llama 2 model it is based on by 25.18 points, confirming the effectiveness of continued pretraining to build language models for low-resource languages.
Open vs. closed models. With the exception of EusProficiency, the best results are obtained by GPT-4 Turbo, a closed commercial system. The difference between GPT-4 Turbo and the previous best open model (Yi) is abysmal. For instance, GPT-4 Turbo is 30.55 points better than Yi on EusTrivia, while the latter is only 16.02 points better than random chance. This can partly be attributed to previous open initiatives being primarily English-centric. While Latxa substantially narrows this gap, we believe that future research on open models should pay more attention to low-resource languages.
English-centric vs. multilingual models. The 3 multilingual models we evaluate (XGLM, BLOOM and mGPT) do better than all English-centric models on XStoryCloze, which requires linguistic competence in Basque and is evaluated in a zero-shot fashion. However, English-centric open models do generally better on other tasks, presumably due to their better in-context learning capabilities and general knowledge captured. Consistent with our pre-
vious point, this suggests that existing open models are either English-centric and struggle in lowresource languages like Basque, or are multilingual and significantly lag behind the English-centric models in language-agnostic capabilities.
Impact of scale. We find that larger Latxa models obtain substantially better results: the 70B model is 10.99 points better than the 13B model on average, which is 9.14 points better than the 7B model. This transcends conventional scaling laws, which establish that, when pretraining models from scratch in low-resource scenarios, the performance is bottlenecked by the amount of training data rather than the model size (Kaplan et al., 2020). However, our results show a different picture for continued pretraining, where bigger and stronger base models result in better performance despite the limited pretraining data in the target language. This suggests that even better results could be obtained through continued pretraining as stronger English-centric models become available, which is encouraging for low-resource languages.
General vs. language-specific knowledge. We find evidence that Latxa is particularly strong in tasks that test for proficiency in the Basque language. In particular, Latxa obtains the best results on Basque language proficiency exams (EusProficiency), despite lagging behind GPT-4 Turbo in the rest of the tasks. Similarly, Latxa outperforms
GPT-4 Turbo on the Language & Literature subset of EusTrivia, even if GPT-4 Turbo is superior in the rest of the categories (Table 5). This suggests that Latxa is more proficient than GPT-4 Turbo in Basque, but the latter does better in most tasks due to its stronger general capabilities. Another evidence of this is that Latxa is particularly weak in the Maths & ICT subset of EusTrivia, where it even lags behind Yi, while GPT-4 Turbo is particularly strong in this category. This is likely because most problems in this category can be understood with basic knowledge of Basque, but solving them may require more complex mathematical reasoning. This suggests that the general capabilities of language models are highly language-agnostic which, in line with our previous finding, suggests that stronger English-centric models can lead to stronger models for low-resource languages by using the same continued pretraining recipe.
Classical NLP tasks. Table 6 reports finegrained results on BasqueGLUE, which comprises various classical NLP tasks like topic classification and coreference detection. In addition to our usual set of decoder-only models evaluated in a few-shot fashion, we report results for BERTeus (Agerri et al., 2020) and ElhBERTeu (Urbizu et al., 2023), which are encoder-only models that were fine-tuned specifically on these tasks. The best results are obtained by these specialized encoderonly models, which shows that the traditional pretraining/fine-tuning paradigm with BERT-style models is still competitive for classical NLP tasks. The only exception is EpecKorrefBin, where both GPT-4 Turbo and Latxa 70B perform substantially better than the fine-tuned encoder-only models. In future work, we would like to explore fine-tuning Latxa and other decoder-only models on these tasks.

heading: 7 Related Work
text: The amount of documents per language in Common Crawl7 is helpful to organize the literature. There is no agreed-upon definition of low-resource language, so we set five arbitrary buckets of languages following a logarithmic distribution: high bucket with just English (rank 1, 46% of latest crawl), high-medium languages around Spanish (rank 5, 4.6%), medium around Danish (rank 24, 0.46%), low resource around Nepalese (rank 46,
7CC-MAIN-2024-22 crawl at https://commoncrawl. github.io/cc-crawl-statistics/plots/languages
0.044), and very-low around Somali (rank 81, 0.0046%). Basque would be low-resource (rank 52, 0.035%).
While there were some early efforts to build open multilingual language models like XGLM (Lin et al., 2022), BLOOM (Scao et al., 2023), mGPT (Shliazhko et al., 2024) and the translation oriented MADLAD-400 (Kudugunta et al., 2023), their performance significantly lags behind more recent English-centric models like Llama 2 (Touvron et al., 2023) or Mistral (Jiang et al., 2023). Concurrent to our work, Üstün et al. (2024) present Aya, a fine-tuned mT5 encoder-decoder (Xue et al., 2021), which has been instruction-tuned and supports 101 languages.
In the case of high-medium-resource languages, different teams have focused on building models from scratch. Ekgren et al. (2022) built a Swedish 20B model which was favorably evaluated on perplexity, followed by (Ekgren et al., 2024), which builds a 40B model for five Nordic languages, including low-resource Danish and Faroese, English and code. It was evaluated on perplexity for the three richest languages. Faysse et al. (2024) present a French-English bilingual model trained with the same number of tokens for each language, although the amount of text for French (376M documents) is half the English text. The authors stress the fact that the tokenizer should not be biased towards any of the two languages. The results on French (and English) LLM evaluation benchmarks show that their largest model (1.2B parameters) underperforms both 3B Llama 2 and Mistral models.
Regarding medium-resource, Luukkonen et al. (2023) focus on Finnish. Using a 19B word corpus they trained several models from scratch, ranging from 186M to 13B parameters. As an alternative, they also continued pretraining a multilingual model, the 176B BLOOM. The evaluation on FINbench, a version of Big-Bench (Srivastava et al., 2023), shows the 13B model underperforms the 7B model, while the continued pretraining model obtains the best results by a large margin. In followup work, Luukkonen et al. (2024) train a 40B bilingual model from scratch with a larger Finnish corpus, and obtain the best results in FIN-bench. They do not compare the results to commercial models. The results suggest that monolingual models trained from scratch saturate at relatively small sizes, and multilingual or continued pretraining would be the best option for these languages.
Focusing on very-low-resource languages,
Yong et al. (2023) compare different approaches to extend the 176B BLOOM model from 46 languages to new languages unseen at training, including Guarani (rank 116, 0.0006%) and seven larger languages for which they sample 100K documents at most. Model sizes range from 0.56B to 7.1B parameters. They report good results for adapters on zero-shot benchmarks. In concurrent work, Lin et al. (2024) present MaLA-500, a continued pretrained model using LORA based on Llama 2 that supports 500 languages. MaLA seems to improve Llama 2 on low to very-low resource languages, but degrades in some medium languages already covered in Llama 2. Evaluation is based on a variant of perplexity and text classification.
Previous work on low-resource languages has been done in the context of multilingual models (see above). In all cases, evaluation is based on perplexity and/or some readily available datasets, and does not include native benchmarks designed to evaluate base models.

heading: 8 Conclusion and Future Work
text: In this work, we present a new open framework for the development and evaluation of LLMs in Basque. The framework includes Latxa, a set of state-of-the-art generative LLMs that have been built by continuing to pretrain the Llama 2 7B, 13B and 70B models in Basque. The pretraining dataset is the largest public dataset available to date and includes data from carefully curated sources, as well as content derived from automatically filtered versions of Common Crawl. After preprocessing and deduplication, the released corpora comprise 1.22B words and 4.17B tokens. We also present 4 new evaluation datasets, collectively the largest evaluation benchmark for Basque that allows assessing the knowledge of the models about the Basque language and culture.
The Latxa models outperform all previous open models and GPT-3.5 Turbo, but they still lag behind GPT-4 Turbo in most benchmarks. Interestingly, Latxa 70B outperforms GPT-4 Turbo on EusProficiency and Language & Literature EusTrivia questions, suggesting that the capabilities of LLMs in a particular language are not determined by their linguistic competence in this language. This, along with the effectiveness of scale, suggests that applying the exact same continued pretraining recipe could lead to better Basque models as stronger English-only models become available.
In the future, we plan to extend the training dataset by gathering quality content from diverse Basque sources such as publishers or media, as well as building evaluation datasets to assess aspects such as truthfulness or hallucinations. We also plan to further tune Latxa to follow instructions, which should improve the overall capabilities of our models.
Limitations
To alleviate the potentially disturbing or harmful content, Latxa has been trained on carefully selected and processed data which comes mainly from local media, national/regional newspapers, encyclopedias and blogs. Still, the model is based on Llama 2 models and can potentially carry the same biases, risks and limitations.
Latxa models are pretrained LLMs without any task-specific or instruction fine-tuning. The model was not fine-tuned to follow instructions or to work as a chat assistant, therefore, this kind of usage is not tested nor recommended. That is, the model can either be prompted to perform a specific task or further fine-tuned for specific use cases.

heading: Acknowledgements
text: This work has been partially supported by the Basque Government (Research group funding IT1805-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and of Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2023E01-013. Julen Etxaniz and Oscar Sainz hold a PhD grant from the Basque Government (PRE_2023_2_0060 and PRE_2023_2_0137, respectively).

heading: A Data Preprocessing Details
text: The mix of raw documents —7.39 million documents, 2.96 billion words— was deduplicated with Dolma (Soldaini et al., 2024) to discard both intraand cross-dataset document repetitions. The number of words discarded per dataset at this stage is illustrated in Figure 2(a), where datasets are shown in order of preference (Egunkaria and Booktegi are omitted due to their size). In the case of EusCrawl v1.1, the duplicate documents arise mainly from an overlap between EusCrawl v1 and the updated content. For Colossal OSCAR, more than 60% of the documents are already present in the preferred datasets (i.e., EusCrawl, Egunkaria, Booktegi, EuWiki or CulturaX). HPLT was further deduplicated at the paragraph level due to its quality. After deduplication, the pretraining corpus amounts to 5.80 million documents and 1.45 billion words.
The deduplicated corpus was further filtered based on document-level features. Specifically, we applied Dolma’s implementation of a set of heuristics from Gopher (Rae et al., 2022) and C4 (Raffel et al., 2020), and the Corpus Cleaner v2 (CCv2; Palomar-Giner et al., 2024). The document-level features are as follows:
• eu: percentage of the text that is written in Basque according to CLD2 • # words: number of words • word len: mean word length • bullet: fraction of lines starting with ‘*’ or ‘-’ • ellipsis: fraction of lines ending with ‘. . . ’ • lorem ipsum: whether it occurs in the text • brackets: whether ‘{’ occurs in the text • symbols: ‘#’ and ‘. . . ’ to word ratio • alpha: fraction of words with at least one al-
phabetic character • CCv2: aggregated score given by CCv2
The threshold applied to each feature and its impact on the datasets is shown in Table 7. We adapted Dolma’s default thresholds regarding document and word length to better fit our data and the Basque language based on observed distributions of our corpora (see examples in Figure 2). Note that the filters are not mutually exclusive, that is, the same document might be flagged for removal by several filters.
As a result of this process, the least curated corpora, HPLT and Colossal OSCAR, were further reduced by 61% and 5% respectively, in terms of words. The 22.37% EuWiki documents flagged as
being too short correspond to Wikipedia’s redirect pages, which ultimately did not affect the word count significantly. The final size of the pretraining corpus is 4.30 million documents and 1.22 billion words, which equates to 4.17B Llama 2 tokens.

heading: B Dataset Contamination
text: Following previous work on data contamination (Brown et al., 2020; Touvron et al., 2023), we check for token n-gram overlaps between test items and training data. To that end, we index training documents in Elasticsearch, applying the standard tokenizer to lowercase text, and removing stopwords (built-in Basque stopwords and all auxiliary verbs).
Given that test items vary greatly in length from one benchmark to another, we avoid establishing an arbitrary n-gram length threshold above which to consider a test item contaminated. Instead, we report statistics based on the longest n-grams matched, spanning our search from n-grams equal to each item’s total length to just one word. For each n-gram size n we only considered test items of equal or bigger size when assessing contamination.
Results are summarized in Table 8, which reports the contamination percentages (cont %) across each benchmark with respect to a specific quartile of test questions or context lengths (n). Higher contamination values are to be expected at lower quartiles, as shorter n-grams (typically 1 to 5 words) tend to involve frequent word combinations and are thus more likely to overlap with the training data. We observe that contamination percentages tend to decrease to near zero after the first quartile, with the following exceptions.
Notably, QNLI has substantial overlap even at higher n-gram lengths. This is explained by the fact that QNLI contexts were taken from Wikipedia. In line with the analysis of Chowdhery et al. (2023), we do not consider these items to be contaminated, as the questions and answers have not been found alongside the contexts in the training data.
In the case of EusProficiency, it must be noted it comprises particularly short test items, the median length of a question being 4 words. Upon manual analysis, we did not observe any annotation contamination.
As for EusExams, this evaluation benchmark consists of Public Service examinations and thus contains many references to national and regional laws, directives and plans, services, administration offices, etc. Such mentions can also be found on the
web, as this information is of public interest. Again, we did not observe annotation contamination.

heading: C Task Examples and Prompts
text: Tables 9 and 10 contain, respectively, the prompt templates and examples of our new datasets (i.e., EusProficiency, EusReading, EusTrivia, and EusExams) and BasqueGLUE tasks. “System” refers to system prompts and applies only to GPT evaluations. Additionally, the number of shots and metrics used to measure the results are also specified per task.

heading: D Model Card
text: We report Latxa’s model card in Table 11.

heading: E Detailed EusExams Results
text: We provide detailed results for EusExams by category in Table 12. Results are consistent across categories, our models outperform every model in the same size category by a large margin. Latxa 13B outperforms GPT-3.5 Turbo in most categories, but Latxa 70B is far from GPT-4 Turbo performance. This is expected as all categories in these exams require advanced knowledge. Health System is the most challenging category, followed by City Council. Public Office and University tests are easier for most models. For specific results of each test check Table 13.

heading: Future Work
text: In the future, we plan to extend the training dataset by gathering quality content from diverse Basque sources such as publishers or media, as well as building evaluation datasets to assess aspects such as truthfulness or hallucinations. We also plan to further tune Latxa to follow instructions, which should improve the overall capabilities of our models. Limitations
To alleviate the potentially disturbing or harmful content, Latxa has been trained on carefully selected and processed data which comes mainly from local media, national/regional newspapers, encyclopedias and blogs. Still, the model is based on Llama 2 models and can potentially carry the same biases, risks and limitations. Latxa models are pretrained LLMs without any task-specific or instruction fine-tuning. The model was not fine-tuned to follow instructions or to work as a chat assistant, therefore, this kind of usage is not tested nor recommended. That is, the model can either be prompted to perform a specific task or further fine-tuned for specific use cases