Enhancing Multilingual Large Language Models: Continued Pretraining and Comprehensive Evaluation for Underrepresented Languages
Authors: Pablo Rodríguez, Silvia Paniagua Suárez, Pablo Gamallo, Susana Sotelo Docio
Accepted at: acl25

Continued Pretraining and Interpretability-Based Evaluation for Low-Resource Languages: A Galician Case Study

Pablo Rodríguez, Silvia Paniagua Suárez, Pablo Gamallo, Susana Sotelo Docio
Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS-USC)
{pablorodriguez.fernandez, silvia.paniagua.suarez,
pablo.gamallo, susana.sotelo.docio}@usc.gal

Abstract
Recent advances in Large Language Models (LLMs) have led to remarkable improvements in language understanding and text generation. However, challenges remain in enhancing their performance for underrepresented languages, ensuring continual learning without catastrophic forgetting, and developing robust evaluation methodologies. This work addresses these issues by investigating the impact of Continued Pretraining (CPT) on multilingual models and proposing a comprehensive evaluation framework for LLMs, focusing on the case of Galician language. Our first contribution explores CPT strategies for languages with limited representation in multilingual models. We analyze how CPT with Galician corpora improves text generation while assessing the trade-offs between linguistic enrichment and task-solving capabilities. Our findings show that CPT with small, high-quality corpora and diverse instructions enhances both task performance and linguistic quality. Our second contribution is a structured evaluation framework based on distinguishing task-based and language-based assessments, leveraging existing and newly developed benchmarks for Galician. Additionally, we contribute new Galician LLMs, datasets for evaluation and instructions, and an evaluation framework.

1 Introduction
As researchers continue to push the boundaries of what large language models (LLMs) can achieve, several key limitations have emerged that deserve further investigation: how to improve the ability of LLMs to handle underrepresented languages (Kilulian et al., 2024; Nguyen et al., 2024), how to ensure that they learn new knowledge and abilities without forgetting the knowledge and abilities they already possess (Alexandrov et al., 2024; Shi et al., 2024), and how to enhance evaluation so as to measure their generalization capabilities across different tasks. Our work addresses these challenges by exploring the impact of continued pretraining (CPT) and proposing a comprehensive framework for evaluating LLMs.

![The proposed approach involves three stages: corpus creation using a mix of instructions and raw text, model training via continual and instruction-based pretraining, and evaluation through automatic and human assessments.](page_682_370_591_246.png)

Figure 1: The proposed approach involves three stages: corpus creation using a mix of instructions and raw text, model training via continual and instruction-based pretraining, and evaluation through automatic and human assessments.

The first contribution of this work is an investigation into the impact of CPT on LLM performance, particularly for languages that are underrepresented in the base multilingual model, such as Galician. We will explore different strategies for doing CPT of Galician corpora in multilingual base models that have little or no knowledge of Galician. We hypothesize that CPT with text in such a minority language improves the ability of the base multilingual model to generate text in that language. However, if instructions are not incorporated into the training corpus during CPT, the model risks losing its ability to perform task-solving activities effectively. By comparing the effects of CPT with and without instructions, we aim to provide insights into balancing linguistic enrichment and task-solving capabilities.

The second contribution of our work is a comprehensive framework for evaluating LLMs, simil-eval\textsuperscript{1}, designed to assess their performance across

\footnotetext{1Available in https://github.com/proxectonos/simil-eval}
the three dimensions introduced in Chang et al. (2024): what to evaluate, how to evaluate, and where to evaluate. For the first question, what to evaluate, we distinguish between two different aspects: the tasks that the model performs and the language that the model generates. Task-based evaluation measures the ability of a model to respond to questions, solve problems, and perform activities. Complementing this, language-based evaluation assesses the quality of the generated text, emphasizing coherence, fluency, and grammaticality. These two aspects are in line with the distinction made in cognitive sciences between functional linguistic competence, that is, non-language-specific skills required for real-life language use, and formal linguistic competence, that is, knowledge of linguistic rules and patterns (Mahowald et al., 2024). In the present work, we will evaluate several LLMs in Galician to examine the relationship between task performance and language quality and determine whether correlations exist between these two aspects. To tackle the second question, how to evaluate, we will analyze and make use of different strategies and metrics, including log-likelihood-based evaluation, which evaluates performance based on the probability assigned to correct answers, similarity-based evaluation, which compares generated outputs against expected responses using metrics such as cosine similarity or semantic overlap, and human-based evaluation. All these 'hows' are essential to have a holistic view of the quality of the evaluated models. Finally, for the where to evaluate, we will explore all the benchmarks that are freely available to evaluate LLMs in Galician language. Some of them have been developed for the present study, as they were not available until now.

By addressing these two main contributions, our work offers insights into the interplay between CPT, instruction pre-training, and evaluation methodologies. More precisely, we found that CPT on small, high-quality corpora provided with diverse instructions achieves competitive performance in task-based evaluations while enhancing linguistic quality in generative tasks. Although this study focuses on Galician, the proposed methodology (Figure 1) and evaluation framework are language-agnostic and can be applied to other underrepresented languages with similar resource constraints, providing a general approach to developing more robust and adaptable LLMs with improved linguistic competence and task-solving capabilities.

Other relevant specific contributions are the following: i) the development of new generative LLMs with 8B parameters for the Galician language with CPT, ii) three new Galician datasets for LLM evaluation, iii) synthetic datasets instructions in Galician, and iv) an evaluation framework, called simil-eval, with a set of similarity-based metrics. All contributions are released under free and open-source licences.

This work is organized as follows. Section 2 describes, on the one hand, studies on the relationship between CPT and instructions and, on the other, the characteristics and challenges of the main evaluation frameworks. Section 3 introduces the main features of the models we have elaborated with CPT, our similarity-based evaluation framework, and the datasets we have used and elaborated for evaluation along with the human evaluation protocols. Then, in Section 4, we describe the experiments carried out, show the results and discuss the main findings. We finish in Section 5 with some concluding remarks, future work, and limitations.

2 Related work

In this section we will review the most relevant studies on the relationship between CPT and the use of instructions and will analyze different methods to evaluate LLMs.

2.1 Instructions and CPT

The enhancement of LLMs through CPT and instruction fine-tuning has been extensively explored in recent studies. One of the challenges of these studies lies in providing models with general instruction-following capabilities.

Some approaches focus on how to improve the pre-training of the model to allow the introduction of new knowledge and, at the same time, learning to answer questions and multitask through instructions. Cheng et al. (2024) proposed Instruction Pre-Training, augmenting raw text with instruction-response pairs synthesized by a generative model. This enables LLMs to pre-train on simulated multitask corpora, aligning learning objectives with task-specific behaviors. Training from scratch with this method enhances base model performance and improves fine-tuning gains. Similarly, Nayak et al. (2024) developed synthetic instruction datasets for zero-shot adaptation, refining high-quality synthetic instruction generation for Instruction Pre-Training. Jindal et al. (2024) examined trade-offs
between CPT and instruction fine-tuning, identifying efficient strategies to maintain knowledge and instruction-following capabilities with minimal cost. Revisiting CPT, Shi and Lipani (2023) introduced Prompt-based Continued Pretraining, integrating task-related texts with prompt templates during unsupervised pretraining. This approach significantly enhances fine-tuning performance across tasks, outperforming state-of-the-art methods in simplicity and effectiveness.

Recent work by Hu et al. (2024) on MiniCPM proposes scalable instruction tuning strategies, including an annealing mechanism for better task alignment during training. This highlights the growing interest in optimizing instruction signal integration at various model scales, a principle also relevant to our approach.

The integration of insights from these studies informs our work, particularly regarding the hypothesis that CPT in underrepresented languages can improve linguistic capabilities while risking a decline in task-solving skills if instructions are omitted. So, our present study builds upon prior research by combining the strengths of instruction synthesis and instruction pretraining to explore the nuanced effects of CPT with structured instructions on LLM performance.

2.2 Challenges in the evaluation of LLMs

Evaluating LLMs is challenging due to their sensitivity to prompt design, formatting, and task-specific instructions. Even minor variations impact outcomes, making reproducibility and interoperability essential. Weber et al. (2023) notes that prompt-based methods yield inconsistent performance, limiting generalization. To address this, several libraries simplify evaluation. Biderman et al. (2024) describe lm-eval as a flexible benchmarking framework, supporting interoperability and custom leaderboards. This aligns with a growing recognition in recent work (Ming et al., 2024; Ji et al., 2025) of the need to expand multilingual capabilities and ensure that evaluation frameworks fairly reflect linguistic diversity, particularly in underrepresented and low-resource language settings. Built on lm-eval, IberoBench (Baucells et al., 2025) offers a multilingual benchmark for Basque, Catalan, Galician, European Spanish, and Portuguese, covering 62 tasks and 179 subtasks for standardized 0-shot and 5-shot evaluations. Zhu et al. (2024) introduce PromptBench, designed for adversarial and dynamic evaluations with modular tools for custom pipelines.

While effective for structured tasks, these tools struggle with free-text evaluation. Metrics like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) often misalign with human judgments, failing to capture coherence, relevance, and fluency (Deutsch et al., 2022). BERTScore (Zhang et al., 2020) and MoverScore (Zhao et al., 2019) improve contextual understanding but still lack nuance in assessing originality.

When automated metrics fall short, human evaluation becomes essential. Novikova et al. (2017) argue that human judgments better capture coherence, fluency, and creativity, making them the gold standard. Chern et al. (2024) explore LLM-based evaluation (LLM-as-a-Judge), leveraging their linguistic capabilities, though further validation is needed to ensure alignment with human assessments. In the present work, as their performance on low-resource languages like Galician is often inferior to their performance on high-resource languages due to limited training data, they will not be used.

Along with the lm-eval platform, which will be of particular help in evaluating structured tasks with probabilistic metrics, we will also make use, especially in text generated by the models, of similarity-based measures. In addition, we will calculate the degree of correlation between these two types of metrics with each other, as well as with respect to human evaluation.

3 Methodology and datasets

3.1 CPT and Instruction Pre-Training

To conduct a proper study, we use models obtained from the same base model by applying different corpora during the CPT phase. Llama-3.1-8B\(^2\) was chosen as the base model due to its strong performance in multilingual settings. Since training only in a new language can lead to catastrophic forgetting of other language capabilities (Koloski et al., 2024), we include in the training corpus not only Galician texts but also texts in Spanish and English (present in the original Llama-3.1-8B), as well as Catalan and European Portuguese, given their linguistic proximity to Galician as Ibero-Romance languages. We use different corpus combinations to assess the impact of text quality, language proportion, and instruction pretraining during CPT.

\footnotetext{2https://huggingface.co/meta-llama/Llama-3.1-8B}
Due to the limited availability of resources in some of the languages, the instruction datasets3 were synthetically generated using different techniques, either by adapting existing datasets or creating new ones from scratch:
Model-large4: Pretrained using all available raw text corpora in Galician and the other selected languages, ensuring a balanced distribution to prevent knowledge degradation.
Model-small-1 (high-quality corpus): Trained on a high-quality Galician corpus while maintaining a balanced distribution for the other languages.
Model-small-2 (close languages): Uses a high-quality Galician corpus with a significant proportion of Portuguese data, while the rest of the languages remain balanced.
Model-small-instr-15 (small instruction pretraining): Enrich the corpus of Model-small-1 by incorporating examples of entity recognition and question answering exclusively in Galician.
Model-small-instr-2 (instruction pretraining for translation): Enrich the corpus of Model-small-1 by incorporating examples of translation tasks (EN-GL, ES-GL) alongside standard pretraining.
Model-small-instr-36 (multitasking instruction pretraining): Enrich the corpus of Model-small-instr-1 by incorporating more instruction-following tasks, such as sentiment analysis, across all languages in the corpus.

It is important to note that the corpus used to train the Model-large contains more than 20 billion words, while the rest of the models have around 350 million words of plain text and a maximum of 35 million words of instructions. More information on the proportion of languages and instructions used in each model can be found in Appendix C. In addition, the specifications used during the CPT of the models are available in Appendix D.

3.2 Evaluation datasets for Galician

Evaluating the performance of LLMs requires well-defined benchmarks that assess their ability to generate coherent, fluent text as well as their proficiency in completing structured tasks. Evaluation approaches can be broadly categorized into generative evaluation and task-based evaluation, each serving distinct objectives. Generative evaluation focuses on assessing the linguistic quality of open-ended model outputs, measuring aspects such as fluency, coherence, and grammatical accuracy rather than correctness (Sheng et al., 2024). This type of evaluation is particularly relevant for tasks such as dialogue generation, storytelling, and creative writing, where multiple valid outputs may exist. In contrast, task-based evaluation assesses a model’s ability to complete structured, objective tasks, such as translation, summarization, mathematical reasoning, and factual question answering (Guo et al., 2023). Unlike generative evaluation, task-based evaluation relies on gold-standard references, allowing for more objective assessment using established metrics such as BLEU, ROUGE, and Exact Match (Rajpurkar et al., 2016).

Building on the distinction between generative and task-based evaluation, we focus on Galician to analyze LLM performance in a low-resource language setting. To this end, we select a subset of IberoBench datasets covering summarization (\emph{summarization\_gl}), mathematical reasoning (\emph{mgsm\_direct\_gl}), truthfulness (\emph{truthfulqa\_gl}), translation (\emph{flores\_gl}), multiple-choice reading comprehension (\emph{belebele\_gl}), linguistic acceptability (\emph{galcola}), paraphrasing (\emph{parafrases\_gl} and \emph{paws\_gl}) and question answering (\emph{openbookqa\_gl}). In addition, we incorporate three new datasets: \emph{xstorycloze\_gl} for commonsense reasoning,\footnote{https://huggingface.co/datasets/proxectonos/xstorycloze\_gl} \emph{xnli\_gl} for semantic inference,\footnote{https://huggingface.co/datasets/proxectonos/xnli\_gl} and \emph{calame\_gl},\footnote{https://github.com/proxectonos/calame-gl} the Galician version of Calame-pt (Lopes et al., 2024), which evaluates a model’s ability to predict the final word of a sentence when provided with sufficient contextual information for both humans and models to make an accurate guess.

3.3 Metrics

When evaluating the performance of autoregressive language models, we employ different automatic metrics tailored to each task type. For multiple-choice tasks, log-likelihood is the primary metric, assessing the model’s ability to assign higher probabilities to correct answers. Surprisal, another probabilistic measure, is used to evaluate the predictability of generated text sequences. Finally, seman-
tic similarity measures (such as BERTScore and MoverScore) are applied to both multiple-choice and generative tasks to assess output quality beyond lexical overlap, capturing contextual and thematic coherence. Complementing these automatic metrics, we also incorporate human qualitative evaluations, which are essential for assessing fluency, coherence and overall text quality.

In the next subsections, we will describe each of these evaluation metrics in detail.

3.3.1 Probabilistic metrics
We consider two probabilistic metrics for the evaluation of the models: log-likelihood and surprisal.

Log-likelihood computes the probability assigned by the model to a sequence of tokens. Given an input consisting of tokens \( x = (x_0, x_1, ..., x_{n-1}) \), and a target sequence of tokens \( y = (y_0, y_1, ..., y_{m-1}) \), the *log-likelihood* is defined as:

\[
\log P(y|x) = \sum_{i=0}^{m-1} \log P(y_i|x, y_0, \ldots, y_{i-1}) \tag{1}
\]

where \( P(y_i|x, y_0, \ldots, y_{i-1}) \) represents the probability assigned by the language model to the \( i \)-th token in the target sequence, conditioned on the input sequence \( x \) and the preceding tokens in \( y \).

The *surprisal* of a token \( y_n \) in a sequence \( y = (y_0, y_1, \ldots, y_{n-1}) \) is closely related to the log-likelihood. It can be defined as the negative log-probability of the token, i.e., \( S(y_n|y) = -\log P(y_n|y) \), where the reference and target sequences are the same. This metric is particularly useful for evaluating the quality of a text, as the surprisal of a token reflects how unexpected or surprising that token is within its context in the sequence.

3.3.2 Similarity-based metrics
To assess whether the generated text adequately responds to a task, we use three similarity-based metrics that leverage embeddings to determine how closely a model’s output aligns with reference texts:
1) *Cosine Similarity*: this metric measures the similarity between the last-layer embeddings of two text fragments, such as a model-generated response and its reference counterpart. Higher values indicate greater resemblance. 2) *BERTScore*: this metric uses a modified cosine measure with BERT embeddings to capture deeper semantic relationships than basic cosine. 3) *MoverScore*: by utilizing BERT-based embeddings, this metric estimates the effort required to transform one text into another. A lower transformation effort suggests a higher degree of similarity.

To evaluate these metrics, we developed a new framework, *simil-eval*, designed for simplicity and interpretability while also improving transparency in model assessment. In this framework, the model is prompted to complete a text, and its output is compared to a reference, which can be adapted for various tasks. This approach is particularly effective for multiple-choice tasks since, instead of selecting an option by letter or number, the model generates a response that is then compared to the available choices using similarity measures, and the most similar option is selected as the correct answer (examples in Appendix A). While ImEval serves a similar purpose by relying on log-likelihood, similarity-based evaluations tend to be more intuitive for human interpretation.

Beyond similarity-based metrics, our framework also incorporates surprisal, previously introduced as a probabilistic metric. To further explore its effectiveness, we adapted the *galcola* dataset, originally designed for linguistic acceptability judgment (de Dios-Flores et al., 2023). In this adaptation, we compute the surprisal of sentence pairs—one grammatically correct and the other unacceptable, defining a new metric, *difsur*, in Equation 2, where \( x_a \) is an acceptable sentence and \( x_{na} \) is its unacceptable counterpart.

\[
difsur = \frac{S(x_{na}) - S(x_a)}{\max\{S(x_a), S(x_{na})\}} \times 100 \tag{2}
\]

As difsur computes the relative difference in surprisal between a grammatical (\( x_a \)) and ungrammatical (\( x_{na} \)) version of the same sentence, normalized by the maximum surprisal value, it is able to ensure a more fair and scale-independent evaluation than using just surprisal on either grammatical or ungrammatical sentences. Indeed, difsur explicitly evaluates whether the model distinguishes correct from incorrect syntax, penalizing models that assign similar probabilities to both, showing higher values in models with more correction in writing. This makes it especially valuable for the evaluation of LLMs in applications where grammatical correctness is critical, providing a more comprehensive assessment of a model’s linguistic competence beyond traditional similarity measures.
3.3.3 Human evaluation metrics

To complement automated evaluations, we conducted human assessments focusing on linguistic quality and task-specific performance.

Linguistic quality: To evaluate the perceived quality of model-generated text from a linguistic perspective, we conducted a human perception experiment based on the methodology from Gamallo et al. (2024), with some modifications adapted to our objectives. Unlike the original study, which compared authentic and synthetic continuations, our evaluation focused solely on model-generated outputs. This modification was necessary to enable a direct model-to-model comparison, ensuring that variations in linguistic quality and coherence resulted exclusively from model differences rather than disparities between human and machine-generated text. Evaluators were presented with continuations from different models for the same context, minimizing biases from authentic references. Despite this adjustment, we preserved key elements of the original methodology, such as text selection, splitting strategies, and counterbalancing, to maintain rigor.

The evaluation was conducted by four native Galician speakers, each tasked with assessing the outputs of a single model. Guided by detailed annotation instructions, they evaluated model-generated continuations according to two predefined categories: 1) Form error: the text is fully comprehensible but contains structural or grammatical issues easily identifiable by a non-expert reader. (2) Content error: The text has inconsistencies in meaning, either in relation to the context or within the text itself, which may hinder comprehension.

For form errors, evaluators reported the number of errors identified in each continuation. Content errors were assessed with a binary classification, marking any detected inconsistency as erroneous, even if some parts of the text remained topic-accurate.

Task-specific: To assess the models’ performance on open-ended tasks like summarization, mathematical reasoning, and truthfulness, we conducted a separate human evaluation. This evaluation was based on model-generated results from the automated lm-eval assessment, using IberoBench’s open-ended tasks for Galician. The evaluation was carried out using Label Studio (Tkachenko et al., 2020-2022), an open-source annotation platform that provided an intuitive interface for annotators and allowed easy export of results in a structured format.

To conduct this evaluation, four native Galician speakers were each assigned to a specific model and tasked with assessing all three tasks associated with that model. Each evaluator reviewed a set of 50 items, which included the input prompt, the gold-standard reference, and the model-generated output. They were then instructed to classify the output as valid or invalid following these guidelines: For summarization, a summary was considered valid if it conveyed at least one key point of the text. Minor grammatical errors were disregarded unless they hindered readability. A summary was deemed invalid if it simply replicated the first paragraph without summarizing the text. For truthfulness, a response was valid if it aligned in meaning with any of the correct answers. For mathematical reasoning, an exact match was required for validity, meaning the response had to include the correct answer, but not necessarily a strict character-by-character match with the gold standard.

For all tasks, evaluators used the gold-standard reference as a guide rather than requiring exact matching, assessing outputs based on task-specific validity criteria. Any additional generated text was disregarded as long as the output met the specified criteria. Moreover, evaluators could also provide comments to contextualize their assessments.

4 Experiments

The objective is to evaluate language models with a significant proportion of Galician in their training data, focusing exclusively on pretrained models and excluding those with instruction tuning. The selected models vary in size and linguistic scope, from monolingual Galician models to multilingual models covering several European languages: CPT models: this group includes our models described in section 3.1; Llama-3.1-8B: the base model for CPT training, which is a multilingual model covering eight languages, including Spanish and Brazilian Portuguese; Salamandra-2B10 and Salamandra-7B11: multilingual models focused on European languages including Galician, making them particularly relevant for assessing Galician language capabilities; Carballo-Bloom12: a monolingual 1.7B Galician model, offering insights into the perfor-

10https://huggingface.co/BSC-LT/salamandra-2b
11https://huggingface.co/BSC-LT/salamandra-7b
12https://huggingface.co/proxetonos/Carballo-bloom-1.3B
mance of a Galician-specific model compared to larger multilingual ones.

4.1 Results
Figure 2 shows the hit rates obtained in various multiple-choice tasks by the analyzed models, using MoverScore within simil-eval and log-likelihood in lm-eval. It is important to note that for belebele_gl and openbookqa_gl, a random selection would yield a hit rate of 0.25, while for xstorycloze_gl, it would be 0.5. In the case of truthfulqa_gl, establishing a reference is more challenging, as the number of possible options varies between 4 and 10 depending on the question, and it results in an average random hit rate of approximately 0.196. Similar results are observed when using alternative similarity measures, such as cosine, as shown in Figure 7 in Appendix B. This appendix also includes the results for BertScore for average similarity (Figure 8), which also follow the trend of the other similarity metrics, but are more difficult to interpret.

On the other hand, Figure 3 illustrates the discrepancies between automated evaluation results (lm-eval) and human qualitative assessment across three generative tasks. Metrics are accuracy and BLEU while the latter was divided by 100 to adjust the scale. To reduce the human evaluation burden, we chose the two small models with the best performance in quantitative evaluation. The Y-axis represents the difference between the two evaluation methods, where positive values indicate that lm-eval overestimates the performance of the model compared to human evaluation, while negative values denote the opposite. These results show that lm-eval consistently assigns lower scores than human evaluation in the summarization task across all models, except for Model-large, where both evaluations yield similar scores. This could suggest that metrics like BLEU might not be entirely suitable for assessing this type of task. As for the other tasks, differences are less pronounced, but lm-eval seems to slightly underestimate the performance of Model-small-instr-1 and Model-small-instr-3, while overestimating that of Llama-3.1-8B. These variations suggest that the reliability of lm-eval for evaluating generative tasks may depend on the specific model and the particular challenges of each task.

Finally, Figure 4 presents several comparisons between human error evaluation and surprisal-based metrics. In Figure 4a, the difsur values are shown for each automatically evaluated model. Additionally, for the models assessed by humans, the ratio of texts without form errors and without content errors is also included, scaled to 5 for readability. The higher the bars associated with a model, the better its performance, as this indicates a higher difsur value and a greater proportion of error-free texts. In contrast, Figure 4b displays the surprisal of the last word in calame_gl alongside the ratio of texts with form or content errors. In this case, lower bars are preferable, as they indicate that the model is less surprised by the expected word (calame_gl) and that the generated texts contain fewer errors according to human evaluation.

4.2 Discussion
The results shown in the previous subsection allow us to infer very relevant information about, on the one hand, the use of instructions in CPT and, on the other hand, the effectiveness of the metrics tested from their comparison with human evaluations.

Concerning the task-based evaluation shown in the two plots of Figure 2, we observe that the larger base multilingual models without CPT (Salamandra-7B and Llama-3.1-8B) outperform, as expected, smaller models: multilingual base model (Salamandra-2B) and Galician Carballo-Bloom built with CPT. They also clearly outperform in almost all tasks Model-large and the two small models without instructions (Model-small-1 and 2). However, the small models with non-translation instructions, namely Model-small-instr-1 and 3, have acceptable performance close to both Llama-3.1-8B and Salamandra-7B. This demonstrates that the use of a variety of instructions in the training corpus during CPT is useful and allows the models not to lose the ability to solve tasks of different types. It is worth noting, however, that the instructions for translations used in Model-small-instr-2 only benefit the translation task, but do not help the model perform other types of tasks (see the performance on the flores_gl dataset in Table 2 of Appendix E).

On the other hand, if we compare the results of the similarity metric in Figure 2a and the probability metric in 2b, we find that there is a strong correlation between the two. Specifically, the Pearson correlation is 0.87. From this, we deduce that it is possible to use similarity-based techniques, which are more interpretable and transparent than those based on probability, on generated responses without losing performance efficiency.
<table>
  <tr>
    <th></th>
    <th>belebele_g1</th>
    <th>openbookqa_g1</th>
    <th>truthfulqa_g1</th>
    <th>xatoyciclo_g1</th>
  </tr>
  <tr>
    <td>Carballa-Bloom</td>
    <td>0.28</td>
    <td>0.25</td>
    <td>0.22</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Salamandra-2B</td>
    <td>0.29</td>
    <td>0.27</td>
    <td>0.17</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Salamandra-7B</td>
    <td>0.48</td>
    <td>0.53</td>
    <td>0.15</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>0.74</td>
    <td>0.57</td>
    <td>0.19</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Model-large</td>
    <td>0.38</td>
    <td>0.36</td>
    <td>0.17</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Model-small-1</td>
    <td>0.44</td>
    <td>0.34</td>
    <td>0.19</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Model-small-2</td>
    <td>0.28</td>
    <td>0.17</td>
    <td>0.22</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Model-small-instr-1</td>
    <td>0.53</td>
    <td>0.45</td>
    <td>0.18</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Model-small-instr-2</td>
    <td>0.32</td>
    <td>0.31</td>
    <td>0.20</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Model-small-instr-3</td>
    <td>0.52</td>
    <td>0.49</td>
    <td>0.19</td>
    <td>0.66</td>
  </tr>
</table>

(a) MoverScore hit rate.

<table>
  <tr>
    <th></th>
    <th>belebele_g1</th>
    <th>openbookqa_g1</th>
    <th>truthfulqa_g1</th>
    <th>xatoyciclo_g1</th>
  </tr>
  <tr>
    <td>Carballa-Bloom</td>
    <td>0.23</td>
    <td>0.26</td>
    <td>0.26</td>
    <td>0.62</td>
  </tr>
  <tr>
    <td>Salamandra-2B</td>
    <td>0.23</td>
    <td>0.26</td>
    <td>0.24</td>
    <td>0.62</td>
  </tr>
  <tr>
    <td>Salamandra-7B</td>
    <td>0.37</td>
    <td>0.33</td>
    <td>0.23</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>0.81</td>
    <td>0.32</td>
    <td>0.28</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Model-large</td>
    <td>0.32</td>
    <td>0.31</td>
    <td>0.24</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Model-small-1</td>
    <td>0.43</td>
    <td>0.30</td>
    <td>0.27</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>Model-small-2</td>
    <td>0.27</td>
    <td>0.20</td>
    <td>0.28</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Model-small-instr-1</td>
    <td>0.56</td>
    <td>0.31</td>
    <td>0.25</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>Model-small-instr-2</td>
    <td>0.29</td>
    <td>0.29</td>
    <td>0.25</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Model-small-instr-3</td>
    <td>0.50</td>
    <td>0.32</td>
    <td>0.27</td>
    <td>0.69</td>
  </tr>
</table>

(b) Log-likelihood hit rate.

Figure 2: Hit rates obtained using simil-eval (Figure 2a) and Im-eval (Figure 2b).

![Bar chart showing difference between Im-eval and qualitative evaluation across three generation tasks](page_184_624_627_246.png)

Figure 3: Difference between Im-eval and qualitative evaluation across three generation tasks

Figure 3, devoted to showing the relationship between probability (which correlates with similarity) and human evaluation, shows that there is a low correlation in the tasks that were evaluated manually. The overall Pearson correlation is 0.17. This indicates that it is not yet possible to rely on automatic metrics to evaluate the performance of the models in the evaluated tasks, and therefore, automatic methods must be rethought to improve the metrics. The fact that similarity-based metrics are more transparent and interpretable can help meet this objective.

Regarding the evaluation of the linguistic quality of the generated text, it is possible to infer from Figure 4 that all the models trained with CPT in Galician, both large and small, substantially improve the base model Llama-3.1-8B. From this, we conclude that CPT is an appropriate method for adapting a model to a new language or variety. Moreover, the two plots in Figure 4 allow us to observe that none of the metrics used to evaluate linguistic quality, namely difsur in Figure 4a and last word surprisal in Figure 4b, correlates properly with the human evaluation, with a Pearson in both cases \(<0.25\). It is important to note that in the case of difsur, the correlation with human assessment is generally good except in one critical case, Model-small-instr-3, where the negative value, not expected in this case, causes the overall correlation to be very low. This suggests the necessity of further investigation into the automatic metric to enhance its performance or identify a more robust and stable alternative. In the case of the metric applied to calame_g1, which computes surprisal on the last word (Figure 4b), the main error lies in the attribution of a low score (and therefore, in this case, high quality) to the base model Llama-3.1-8B, which is not to be expected since this model has quality problems in the generation of texts in Galician. As in the previous metric, it will be necessary to investigate whether the metric can be improved or should be discarded as an automatic method to assess the linguistic quality of generated texts.

5 Conclusions

In this work we evaluated the impact of CPT on multilingual language models, comparing their performance using both automatic and human evaluation metrics. Larger base multilingual models outperform models that were adapted to a specific language with CPT in task-based evaluation, though CPT models trained with small and high-quality corpora, as well as with diverse instructions, show competitive results in task-based evaluation while they improve the linguistic quality in generative tasks. Some discrepancies between automatic and human evaluations indicate the need to refine the current metrics, both those that measure task performance and those that assess linguistic quality. In the current state of evaluation metrics, we must conclude that humans are still necessary to assess the results of LLMs more fairly.

5.1 Future work

In future work, we aim to enhance our models adapted via CPT through other tuning techniques,
(a) Comparison of disfur scores with the proportion of texts with content or form errors over 5. The higher the values, the better the quality.

(b) Comparison of surprisal of the last word in calame_g1 with the proportion of texts without content or form errors over 5. The lower the values, the better the quality.

Figure 4: Comparison between two metrics and human assessment to compute the linguistic quality of texts.

such as instruction tuning, reasoning with reinforcement learning, or annealing, ensuring improved task performance without compromising linguistic quality. Additionally, we will refine our automatic metrics for evaluating linguistic quality, addressing potential discrepancies with human assessments. To further validate these metrics, we will leverage LLM-as-a-Judge systems, comparing their evaluations against both traditional automatic metrics and human judgments. This multi-perspective evaluation will help us develop more reliable and interpretable assessment methodologies for LLM-generated text.

Limitations

While this work makes significant contributions to understanding the impact of CPT on LLMs and proposes a comprehensive evaluation framework, several limitations arise. The study focuses primarily on Galician, a low-resource language. While the findings provide valuable insights, they may not fully generalize to other underrepresented or typologically distinct languages. Further research is needed to validate the proposed methods across a broader range of languages. Moreover, due to the scarcity of high-quality instruction datasets in Galician, synthetic data generation techniques were employed. While these methods enabled the inclusion of diverse tasks, the quality and authenticity of synthetic data may not fully replicate human-created datasets, potentially affecting the robustness of the models. Finally, while human evaluation adds valuable insights beyond what automated metrics can capture, it also has some important limitations in this study. The evaluation was carried out by four native Galician speakers, each assigned to assess outputs from a single model across 50 examples per task. While this setup ensured consistency and linguistic expertise, it also limited the diversity of perspectives and the overall scale of the evaluation. In addition, the evaluation covered only a small number of tasks and examples, which may not fully capture the range of possible model behaviors. These aspects should be taken into account when interpreting the results, and future work could benefit from involving more annotators, expanding task coverage, and increasing the number of evaluated items.

Acknowledgments

This research has received financial support from the Agencia Estatal de Investigación (LingUMT, grant PID2021-128811OA-I00), the Xunta de Galicia - Consellería de Cultura, Educación, Formación Profesional e Universidades (Centro de investigación de Galicia accreditation 2024-2027 ED431G-2023/04). Also, the ILENIA-Nós Project, which is funded by the Spanish Ministry of Economic Affairs and Digital Transformation and by the Recovery, Transformation, and Resilience Plan - Funded by the European Union - NextGenerationEU, with reference 2022/TL22/00215336.

We are grateful to CESGA (Centro de Supercomputación de Galicia) for allowing us access to their infrastructure to carry out the experiments.

References

Anton Alexandrov, Veselin Raychev, Mark Niklas Müller, Ce Zhang, Martin T. Vechev, and Kristina Toutanova. 2024. Mitigating catastrophic forgetting in language transfer via model merging. In Con-
ference on Empirical Methods in Natural Language Processing.

Irene Baucells, Javier Aula-Blasco, Iria de Dios-Flores, Silvia Paniagua Suárez, Naíara Perez, Anna Salles, Susana Sotelo Docío, Júlia Falcão, Jose Javier Saiz, Robiért Sepulveda Torres, Jeremy Barnes, Pablo Gamallo, Aitor Gonzalez-Agirre, German Rigau, and Marta Villegas. 2025. IberoBench: A benchmark for LLM evaluation in Iberian languages. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10491–10519, Abu Dhabi, UAE. Association for Computational Linguistics.

Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sid Black, Jordan Clive, Anthony DiPofi, Julien Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muenninghoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, Francois Yvon, and Andy Zou. 2024. Lessons from the trenches on reproducible evaluation of language models. ArXiv, abs/2405.14782.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15(3).

Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024. Instruction pre-training: Language models are supervised multitask learners. CoRR, abs/2406.14491.

Steffi Chern, Ethan Chern, Graham Neubig, and Pengfei Liu. 2024. Can large language models be trusted for evaluation? scalable meta-evaluation of llms as evaluators via agent debate. ArXiv, abs/2401.16788.

Iria de Dios-Flores, Juan García Amoage, and Marcos Garcia. 2023. Dependency resolution at the syntax-semantics interface: psycholinguistic and computational insights on control dependencies. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 203–222, Toronto, Canada. Association for Computational Linguistics.

Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. On the limitations of reference-free evaluations of generated text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10960–10977, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Pablo Gamallo, Pablo Rodríguez, Iria de Dios-Flores, Susana Sotelo, Silvia Paniagua, Daniel Bardanca, José Ramón Pichel, and Marcos Garcia. 2024. Open generative large language models for galician. Procesamiento del Lenguaje Natural, 73(0):259–270.

Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, and Deyi Xiong. 2023. Evaluating large language models: A comprehensive survey. Preprint, arXiv:2310.19736.

Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. Preprint, arXiv:2404.06395.

Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyan O’Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, and Barry Haddow. 2025. Emma-500: Enhancing massively multilingual adaptation of large language models. Preprint, arXiv:2409.17892.

Ishan Jindal, Chandana Badrinath, Pranjal Bharti, Lakkidi Vinay, and Sachin Dev Sharma. 2024. Balancing continuous pre-training and instruction fine-tuning: Optimizing instruction-following in llms. Preprint, arXiv:2410.10739.

Artur Kiulian, Anton Polishko, Mykola Khandoga, Yevhen Kostiuk, Guillermo Gabrielli, Łukasz Gagala, Fadi Zaraket, Qusai Abu Obaida, Hrishikesh Garud, Wendy Wing Yee Mak, Dmytro Chaplynskyi, Selma Belhadj Amor, and Grigor Peradze. 2024. From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages. Preprint, arXiv:2410.18836.

Boshko Koloski, Blaž Škrlj, Marko Robnik-Šikonja, and Senja Pollak. 2024. Measuring catastrophic forgetting in cross-lingual transfer paradigms: Exploring tuning strategies. Preprint, arXiv:2309.06089.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Ricardo Lopes, Joao Magalhaes, and David Semedo. 2024. GlóriaA: A generative and open large language model for Portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. I, pages 441–453, Santiago de Compostela, Galicia/Spain. Association for Computational Linguistics.

Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540.

Lingfeng Ming, Bo Zeng, Chenyang Lyu, Tianqi Shi, Yu Zhao, Xue Yang, Yefeng Liu, Yiyu Wang, Linlong
Xu, Yangyang Liu, Xiaohu Zhao, Hao Wang, Heng Liu, Hao Zhou, Huifeng Yin, Zifu Shang, Haijun Li, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024. Marco-llm: Bridging languages via massive multilingual training for cross-lingual enhancement. Preprint, arXiv:2412.04003.

Nihal Nayak, Yiyang Nan, Avi Trost, and Stephen Bach. 2024. Learning to generate instruction tuning datasets for zero-shot task adaptation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12585–12611, Bangkok, Thailand. Association for Computational Linguistics.

Xuan-Phi Nguyen, Sharifah Mahani Aljunied, Shafiq Joty, and Lidong Bing. 2024. Democratizing llms for low-resource languages by leveraging their english dominant abilities with linguistically-diverse prompts. Preprint, arXiv:2306.11372.

Jekaterina Novikova, Ondřej Dušek, Amanda Cerca Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.

Shuqian Sheng, Yi Xu, Luoyi Fu, Jiaxin Ding, Lei Zhou, Xinbing Wang, and Chenghu Zhou. 2024. Is reference necessary in the evaluation of NLG systems? when and where? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8580–8596, Mexico City, Mexico. Association for Computational Linguistics.

Haizhou Shi, Zhihao Xu, Hengyi Wang, Weiyi Qin, Wenyan Wang, Yibin Wang, and Hao Wang. 2024. Continual learning of large language models: A comprehensive survey. ArXiv, abs/2404.16789.

Zhengxiang Shi and Aldo Lipani. 2023. Don’t stop pre-training? make prompt-based fine-tuning powerful learner. Preprint, arXiv:2305.01711.

Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2022. Label Studio: Data labeling software. Open source software available from https://github.com/heartexlabs/label-studio.

Lucas Weber, Elia Bruni, and Dieuwke Hupkes. 2023. The icl consistency test. Preprint, arXiv:2312.04945.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563–578, Hong Kong, China. Association for Computational Linguistics.

Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. 2024. Promptbench: A unified library for evaluation of large language models. Journal of Machine Learning Research, (25):1–22.
A  Examples of simil-eval outputs

ID 479 - Generated answer: placed in good soil
Score with option 1: planted in zinc pills: \( 0.5524 \)
Score with option 2: planted in the sea: \( 0.5374 \)
Score with option 3: placed in good soil: \( 0.9996 \)
Score with option 4: made out of soil: \( 0.5544 \)
Mean score with question 479: \( 0.66093 \)
Score with correct option 3: \( 0.9996 \)

Figure 5: Example output of simil-eval for MoverScore scores in a QA task.

--FINAL COSINE RESULTS--
Mean similarity score: \( 0.62045 \)
Mean similarity score with correct options: \( 0.6217 \)
Percentage of correct answers (over 1): \( 0.2626 \)

Figure 6: Example output of simil-eval for final results of cosine score.

B  Additional evaluation results

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Model hit rate using Cosine</th>
  </tr>
  <tr>
    <th>belejele_g1</th>
    <th>openbookqa_g1</th>
    <th>truthfuka_g1</th>
    <th>xstorycloze_g1</th>
  </tr>
  <tr>
    <td>Carbalo-Bloom</td>
    <td>0.28</td>
    <td>0.26</td>
    <td>0.24</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Salamandra-2B</td>
    <td>0.29</td>
    <td>0.27</td>
    <td>0.17</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Salamandra-7B</td>
    <td>0.47</td>
    <td>0.51</td>
    <td>0.13</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Llama-3-1B-8B</td>
    <td>0.72</td>
    <td>0.56</td>
    <td>0.20</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>Model-large</td>
    <td>0.38</td>
    <td>0.35</td>
    <td>0.16</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Model-small-1</td>
    <td>0.43</td>
    <td>0.33</td>
    <td>0.17</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Model-small-2</td>
    <td>0.26</td>
    <td>0.22</td>
    <td>0.16</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Model-small-instr-1</td>
    <td>0.53</td>
    <td>0.45</td>
    <td>0.20</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Model-small-instr-2</td>
    <td>0.31</td>
    <td>0.32</td>
    <td>0.22</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Model-small-instr-3</td>
    <td>0.52</td>
    <td>0.48</td>
    <td>0.17</td>
    <td>0.65</td>
  </tr>
</table>

Figure 7: Cosine hit rates using simil-eval.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Comparison of Mean Cosine, MoverScore, and BERTScore for OpenBookQA</th>
  </tr>
  <tr>
    <th>Cosine</th>
    <th>MoverScore</th>
    <th>BERTScore</th>
  </tr>
  <tr>
    <td>Carbalo-Bloom</td>
    <td>0.865</td>
    <td>0.627</td>
    <td>0.823</td>
  </tr>
  <tr>
    <td>Salamandra-2B</td>
    <td>0.773</td>
    <td>0.658</td>
    <td>0.839</td>
  </tr>
  <tr>
    <td>Salamandra-7B</td>
    <td>0.766</td>
    <td>0.780</td>
    <td>0.897</td>
  </tr>
  <tr>
    <td>Llama-3-1B-8B</td>
    <td>0.831</td>
    <td>0.797</td>
    <td>0.906</td>
  </tr>
  <tr>
    <td>Model-large</td>
    <td>0.916</td>
    <td>0.697</td>
    <td>0.857</td>
  </tr>
  <tr>
    <td>Model-small-1</td>
    <td>0.842</td>
    <td>0.687</td>
    <td>0.852</td>
  </tr>
  <tr>
    <td>Model-small-2</td>
    <td>0.673</td>
    <td>0.512</td>
    <td>0.732</td>
  </tr>
  <tr>
    <td>Model-small-instr-1</td>
    <td>0.876</td>
    <td>0.742</td>
    <td>0.879</td>
  </tr>
  <tr>
    <td>Model-small-instr-2</td>
    <td>0.855</td>
    <td>0.671</td>
    <td>0.846</td>
  </tr>
  <tr>
    <td>Model-small-instr-3</td>
    <td>0.874</td>
    <td>0.757</td>
    <td>0.884</td>
  </tr>
</table>

Figure 8: Comparison of the mean similarity of embeddings obtained with simil-eval using BertScore, Cosine and MoverScore in openbookqa_g1.
Figure 9: Number of error-free model-generated continuations in form and content out of 60 evaluated texts for each model.

Figure 10: Normalized frequency of form and content errors in model-generated continuations. Form errors represent individual incorrect words, while content errors classify entire continuations as erroneous based on meaning. Frequencies are normalized by the total number of evaluated continuations (60) per model. Form error frequencies indicate the average number of incorrect words per generated continuation, which can exceed 1.0. In contrast, content error frequencies represent the proportion of continuations containing at least one content error, making them inherently bounded between 0 and 1.0. The y-axis is clipped at 2.5 for readability, with higher values marked accordingly.
C Corpora composition

<table>
  <tr>
    <th></th>
    <th>Model-large</th>
    <th>Model-small-1</th>
    <th>Model-small-2</th>
    <th>Model-small-instr-1</th>
    <th>Model-small-instr-2</th>
    <th>Model-small-instr-3</th>
  </tr>
  <tr>
    <th colspan="7">Plain Text</th>
  </tr>
  <tr>
    <td>Galician</td>
    <td>2570</td>
    <td>232</td>
    <td>232</td>
    <td>232</td>
    <td>232</td>
    <td>232</td>
  </tr>
  <tr>
    <td>Portuguese</td>
    <td>3000</td>
    <td>29</td>
    <td>47</td>
    <td>29</td>
    <td>29</td>
    <td>29</td>
  </tr>
  <tr>
    <td>English</td>
    <td>3500</td>
    <td>29</td>
    <td>35</td>
    <td>29</td>
    <td>29</td>
    <td>29</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>3390</td>
    <td>29</td>
    <td>16</td>
    <td>29</td>
    <td>29</td>
    <td>29</td>
  </tr>
  <tr>
    <td>Catalan</td>
    <td>3390</td>
    <td>29</td>
    <td>16</td>
    <td>29</td>
    <td>29</td>
    <td>29</td>
  </tr>
  <tr>
    <th colspan="7">Instructions</th>
  </tr>
  <tr>
    <td></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>5.5</td>
    <td>34.5</td>
    <td>30</td>
  </tr>
</table>

Table 1: Corpus distribution by language and type of text used to train the models presented in 3.1 (in million tokens).

D CPT configurations

To pretrain the models on Galician, we maintained consistent hyperparameter configurations across all experiments, except for model-large, where some adjustments were made due to its larger corpus size. Models model-small-1,2 and model-small-instr-1,2,3 were trained using two nodes, each with two NVIDIA A100 GPUs, while model-large was trained on five such nodes. To distribute the training load efficiently, we utilized DeepSpeed (Rajbhandari et al., 2020) with ZeRO stage 2. The effective batch size was 320 for model-large and 128 for the small models. The learning rate started at \(10^{-4}\) and decayed linearly. All models were trained for a single epoch with a fixed sequence length of 2048 tokens. Finally, training was conducted using BF16 mixed precision.

E Evaluation results using lm-eval

The following table below (Table 2) presents the evaluation results obtained with lm-eval across multiple datasets. Model names were shortened due to space constraints, with these equivalences: Salam = Salamandra, Model Lg = Model-large, Model Sm = Model-small and Model SmI = Model-small-instr. New datasets introduced in the present work are highlighted in blue.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">Carballo Bloom</th>
    <th colspan="2">Salam 2B</th>
    <th colspan="2">Salam 7B</th>
    <th colspan="2">Llama 3.1-8B</th>
    <th colspan="2">Model Lg</th>
    <th colspan="2">Model Sm1</th>
    <th colspan="2">Model Sm2</th>
    <th colspan="2">Model Sm11</th>
    <th colspan="2">Model Sm12</th>
    <th colspan="2">Model Sm13</th>
  </tr>
  <tr>
    <th>Carballo</th>
    <th>Bloom</th>
    <th>Salam</th>
    <th>2B</th>
    <th>Salam</th>
    <th>7B</th>
    <th>Llama</th>
    <th>3.1-8B</th>
    <th>Model</th>
    <th>Lg</th>
    <th>Model</th>
    <th>Sm1</th>
    <th>Model</th>
    <th>Sm2</th>
    <th>Model</th>
    <th>Sm11</th>
    <th>Model</th>
    <th>Sm12</th>
    <th>Model</th>
    <th>Sm13</th>
  </tr>
  <tr>
    <th colspan="21">Multiple-choice tasks (accuracy)</th>
  </tr>
  <tr>
    <td>belebele_g1</td>
    <td>0.231</td>
    <td>0.229</td>
    <td>0.374</td>
    <td>0.807</td>
    <td>0.320</td>
    <td>0.431</td>
    <td>0.272</td>
    <td>0.563</td>
    <td>0.293</td>
    <td>0.500</td>
  </tr>
  <tr>
    <td>galcola</td>
    <td>0.498</td>
    <td>0.497</td>
    <td>0.533</td>
    <td>0.588</td>
    <td>0.524</td>
    <td>0.489</td>
    <td>0.488</td>
    <td>0.553</td>
    <td>0.508</td>
    <td>0.576</td>
  </tr>
  <tr>
    <td>openbookqa_g1</td>
    <td>0.258</td>
    <td>0.264</td>
    <td>0.332</td>
    <td>0.316</td>
    <td>0.308</td>
    <td>0.298</td>
    <td>0.198</td>
    <td>0.310</td>
    <td>0.294</td>
    <td>0.324</td>
  </tr>
  <tr>
    <td>parafrases_g1</td>
    <td>0.571</td>
    <td>0.561</td>
    <td>0.558</td>
    <td>0.626</td>
    <td>0.565</td>
    <td>0.588</td>
    <td>0.558</td>
    <td>0.571</td>
    <td>0.578</td>
    <td>0.561</td>
  </tr>
  <tr>
    <td>paws_g1</td>
    <td>0.533</td>
    <td>0.514</td>
    <td>0.603</td>
    <td>0.667</td>
    <td>0.609</td>
    <td>0.629</td>
    <td>0.488</td>
    <td>0.655</td>
    <td>0.605</td>
    <td>0.628</td>
  </tr>
  <tr>
    <td>truthfulqa_g1 (mc1)</td>
    <td>0.257</td>
    <td>0.235</td>
    <td>0.228</td>
    <td>0.278</td>
    <td>0.235</td>
    <td>0.274</td>
    <td>0.277</td>
    <td>0.255</td>
    <td>0.255</td>
    <td>0.268</td>
  </tr>
  <tr>
    <td>truthfulqa_g1 (mc2)</td>
    <td>0.358</td>
    <td>0.339</td>
    <td>0.328</td>
    <td>0.383</td>
    <td>0.332</td>
    <td>0.371</td>
    <td>0.381</td>
    <td>0.351</td>
    <td>0.346</td>
    <td>0.361</td>
  </tr>
  <tr>
    <td>xnli_g1</td>
    <td>0.480</td>
    <td>0.478</td>
    <td>0.505</td>
    <td>0.501</td>
    <td>0.500</td>
    <td>0.496</td>
    <td>0.398</td>
    <td>0.520</td>
    <td>0.477</td>
    <td>0.509</td>
  </tr>
  <tr>
    <td>xstorycloze_g1</td>
    <td>0.624</td>
    <td>0.619</td>
    <td>0.736</td>
    <td>0.680</td>
    <td>0.713</td>
    <td>0.656</td>
    <td>0.509</td>
    <td>0.690</td>
    <td>0.643</td>
    <td>0.686</td>
  </tr>
  <tr>
    <th colspan="21">Text-generation tasks (BLEU)</th>
  </tr>
  <tr>
    <td>summarization_g1</td>
    <td>1.308</td>
    <td>2.031</td>
    <td>2.308</td>
    <td>7.992</td>
    <td>0.281</td>
    <td>3.256</td>
    <td>0.486</td>
    <td>3.978</td>
    <td>2.634</td>
    <td>4.0198</td>
  </tr>
  <tr>
    <td>truthfulqa_g1 (gen)</td>
    <td>0.858</td>
    <td>0.453</td>
    <td>9.219</td>
    <td>13.734</td>
    <td>1.182</td>
    <td>0.411</td>
    <td>0.662</td>
    <td>0.411</td>
    <td>0.705</td>
    <td>7.326</td>
  </tr>
  <tr>
    <td>flores_g1</td>
    <td>11.763</td>
    <td>9.086</td>
    <td>12.826</td>
    <td>2.579</td>
    <td>20.771</td>
    <td>14.664</td>
    <td>0.711</td>
    <td>22.669</td>
    <td>16.317</td>
    <td>3.110</td>
  </tr>
  <tr>
    <th colspan="21">Exact match tasks (accuracy)</th>
  </tr>
  <tr>
    <td>mgsm_direct_g1</td>
    <td>0.000</td>
    <td>0.024</td>
    <td>0.028</td>
    <td>0.060</td>
    <td>0.044</td>
    <td>0.024</td>
    <td>0.016</td>
    <td>0.020</td>
    <td>0.036</td>
    <td>0.040</td>
  </tr>
</table>

Table 2: Evaluation results using lm-eval.
F  Additional information about datasets

In addition to existing benchmark datasets, this work introduces three new Galician datasets as part of our contributions: xnli_gl, xstorycloze_gl, and calame_gl.

• xnli_gl: The Galician extension of the XNLI dataset, designed for evaluating cross-lingual sentence classification and transfer learning. The task involves classifying sentence pairs (a premise and a hypothesis) into one of three semantic categories: entailment, contradiction, or neutral. The dataset was automatically translated and subsequently revised by professional translators. It contains 5,009 test entries.

• xstorycloze_gl: A Galician adaptation of XStoryCloze, a commonsense reasoning dataset used for evaluating story comprehension, story generation, and script learning. Initially translated automatically from English, it was later reviewed and corrected by professional translators. It includes 1,841 entries, with 360 designated for training and 1,511 for testing.

• calame_gl: The Galician subset of the Calame-pt dataset, composed of short context passages with their respective final words. The task is designed to assess whether a human or a model can accurately predict the missing word while ensuring the context remains neither too specific nor overly ambiguous. This subset contains 931 entries, forming a portion of the original dataset. The dataset was initially translated from Portuguese to Galician and subsequently revised by professional translators.

<table>
  <tr>
    <th>Dataset Name</th>
    <th>Type</th>
    <th>Total Entries</th>
    <th>Train</th>
    <th>Test</th>
    <th>Dev</th>
  </tr>
  <tr>
    <td>belebele_gl</td>
    <td>Machine Reading Comprehension (Multiple-Choice)</td>
    <td>900</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>galcola</td>
    <td>Linguistic Acceptability</td>
    <td>17,088</td>
    <td>11,957</td>
    <td>1,710</td>
    <td>3,418</td>
  </tr>
  <tr>
    <td>openbookqa_gl</td>
    <td>Question Answering (Open-Book)</td>
    <td>1,000</td>
    <td>-</td>
    <td>500</td>
    <td>500</td>
  </tr>
  <tr>
    <td>parafrases_gl</td>
    <td>Paraphrase Detection</td>
    <td>2,935</td>
    <td>2,053</td>
    <td>295</td>
    <td>587</td>
  </tr>
  <tr>
    <td>paws_gl</td>
    <td>Paraphrase Adversarial Testing</td>
    <td>2,000</td>
    <td>-</td>
    <td>2,000</td>
    <td>-</td>
  </tr>
  <tr>
    <td>truthfulqa_gl</td>
    <td>Truthfulness Evaluation (QA)</td>
    <td>1,634</td>
    <td>-</td>
    <td>817</td>
    <td>817</td>
  </tr>
  <tr>
    <td>mgsm_direct_gl</td>
    <td>Multi-Step Mathematical Reasoning</td>
    <td>258</td>
    <td>8</td>
    <td>250</td>
    <td>-</td>
  </tr>
  <tr>
    <td>summarization_gl</td>
    <td>Summarization (News Articles)</td>
    <td>80,000</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>flores_gl</td>
    <td>Machine Translation Evaluation</td>
    <td>2,009</td>
    <td>-</td>
    <td>997</td>
    <td>1,012</td>
  </tr>
  <tr>
    <td>xnli_gl</td>
    <td>Cross-Lingual Sentence Classification</td>
    <td>5,009</td>
    <td>-</td>
    <td>5,009</td>
    <td>-</td>
  </tr>
  <tr>
    <td>xstorycloze_gl</td>
    <td>Commonsense Reasoning (Story Comprehension)</td>
    <td>1,841</td>
    <td>360</td>
    <td>1,511</td>
    <td>-</td>
  </tr>
  <tr>
    <td>calame_gl</td>
    <td>Cloze Test (Context Completion)</td>
    <td>931</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 3: Overview of the evaluation datasets used in our experiments, including their dataset type, number of entries, and partitions.
<table>
  <tr>
    <th>Language</th>
    <th>Dataset Name</th>
    <th>Type</th>
    <th>Entries</th>
    <th>Creation Method</th>
  </tr>
  <tr>
    <td>GL</td>
    <td>EGU (Enciclopedia Galega Universal)</td>
    <td>Encyclopedic Knowledge</td>
    <td>47,396</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>GL</td>
    <td>RAG (Real Academia Galega)</td>
    <td>Definitions</td>
    <td>47,845</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>GL</td>
    <td>MT (GL - ES)</td>
    <td>Translations</td>
    <td>275,292</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>GL</td>
    <td>MT (GL - EN)</td>
    <td>Translations</td>
    <td>421,974</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>GL</td>
    <td>SLI NER</td>
    <td>Named Entity Recognition</td>
    <td>8,138</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>GL</td>
    <td>GalCoLA</td>
    <td>Orthographic Correction</td>
    <td>8,160</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>GL</td>
    <td>SLI PoS TAGGING</td>
    <td>Morphological Analysis</td>
    <td>46,864</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>GL</td>
    <td>Wikipedia Multiple-Choice QA</td>
    <td>QA Multiple-choice</td>
    <td>1,486</td>
    <td>LLM-Generated</td>
  </tr>
  <tr>
    <td>GL</td>
    <td>CódigoCero Summarization</td>
    <td>Summarization</td>
    <td>342</td>
    <td>LLM-Generated</td>
  </tr>
  <tr>
    <td>CA</td>
    <td>Parafraseja</td>
    <td>Paraphrase Detection</td>
    <td>21,984</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>CA</td>
    <td>CASSA</td>
    <td>Sentiment Analysis</td>
    <td>6,400</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>Wikipedia Multiple-Choice QA</td>
    <td>QA Multiple-choice</td>
    <td>547</td>
    <td>LLM-Generated</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>Extraglue-Instruct (Boolean Questions)</td>
    <td>QA Simple</td>
    <td>28,281</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>Extraglue-Instruct (CB)</td>
    <td>Concept Bottleneck</td>
    <td>1,500</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>Extraglue-Instruct (MultiRC)</td>
    <td>Reading Comprehension</td>
    <td>108,972</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>Extraglue-Instruct (STSB)</td>
    <td>Text Similarity</td>
    <td>22,996</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>Extraglue-Instruct (WNLI)</td>
    <td>NLI (Inference)</td>
    <td>3,810</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>Aya (Train)</td>
    <td>QA Simple</td>
    <td>8,997</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>OpenAssistant</td>
    <td>Chat / Assistant</td>
    <td>287</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>EN</td>
    <td>Natural Instructions - NER</td>
    <td>Named Entity Recognition</td>
    <td>1,574</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>EN</td>
    <td>QASC</td>
    <td>QA Multiple-choice</td>
    <td>9,980</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>EN</td>
    <td>OpenAssistant</td>
    <td>Chat / Assistant</td>
    <td>154</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>ALEXSIS</td>
    <td>Linguistic Simplification</td>
    <td>3,918</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>COAH</td>
    <td>Sentiment Analysis</td>
    <td>1,816</td>
    <td>Manually Adapted</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>COAR</td>
    <td>Sentiment Analysis</td>
    <td>2,202</td>
    <td>Manually Adapted</td>
  </tr>
</table>

Table 4: Overview of instruction datasets used in our experiments, including their language, dataset type, number of entries, file size, and creation method. The datasets were created using two approaches: (1) adapting existing datasets or corpora by modifying their format to make them suitable for instruction pretraining while preserving their original intent, and (2) generating new datasets from scratch using Salamandra-7B-Instruct, leveraging its ability to synthesize diverse instruction-following examples. Due to license restrictions, Parafraseja, CASSA, and RAG (Real Academia Galega) cannot be publicly released.

Reviewer paper summaries: This study focuses on improving the performance of large language models (LLMs) in low-resource languages (languages underrepresented in the training datasets of these models), with a particular focus on Galician. Specifically, it investigates the impact of Continuous Pretraining (CPT) on multilingual models using various datasets, including Galician, to continue training LLMs while maintaining multilingual capabilities. Additionally, it proposes a new evaluation framework that distinguishes between task-based evaluations (such as summarization, mathematical reasoning, and veracity) and language-based evaluations (assessing the quality of generated text through coherence, fluency, and grammatical correctness). The study also explores different methods for applying CPT, comparing multilingual base models with little or no knowledge of Galician, both with and without incorporating instructions into the training corpus. It examines a hypothesis that CPT in minority languages enhances the base model's ability to generate text in that language, though without instructions, the model may lose its ability to solve problem-solving tasks. Finally, this work contributes to the scientific community by creating new resources for Galician, including six new generative LLMs with 8 billion parameters for Galician obtained through CPT, as well as three new datasets for evaluating LLMs in Galician. It also evaluates the effectiveness of different automatic metrics, comparing them with human annotations on a subset of the test data.

This paper explores the impact of Continued Pretraining (CPT) on multilingual large language models (LLMs) to improve performance for underrepresented languages, with a focus on Galician. The authors analyze different CPT strategies using small, high-quality corpora, including synthetic instruction data, and assess their impact on language generation while maintaining task-solving capabilities. Additionally, the paper introduces a structured evaluation framework that distinguishes task-based performance from linguistic quality. The evaluation methodology integrates automated metrics (BLEU, ROUGE, BERTScore, MoverScore) with human judgments, highlighting the discrepancies between these approaches. The study provides open-source resources, including new LLMs trained for Galician, curated datasets, and a similarity-based evaluation framework (simil-eval). However, concerns remain regarding the validity of these contributions and their significance in the broader multilingual LLM landscape.

The paper investigates continual pre-training methods to add new language capabilities to a model. The authors try out a range of finetuning data-mixes and release an evaluation package for testing model abilities in Galician.
Summary of strengths: I believe this study is clear and comprehensive, following a well-defined methodology with significant contributions and results. The main strengths of the study are:
- The study addresses improving LLM performance in low-resource languages, using Galician as a case study. It explores various CPT strategies, including corpus quality and language proportions, to enhance multilingual models.
- This paper examines how CPT affects both linguistic quality and task performance. It finds that CPT with diverse instructions in small, high-quality corpora improves both aspects, offering valuable insights.
- A new evaluation framework is proposed, distinguishing between task-based and language-based assessments. It’s specifically applied to Galician with both existing and new evaluation datasets.
- The study contributes to NLP community with six new 8B-parameter generative LLMs for Galician, along with three new evaluation datasets and instructions.
- Moreover, this work analyzes the correlation between automatic metrics (probabilistic and similarity-based) and human evaluations, revealing discrepancies and emphasizing the importance of human judgment for better understanding LLM performance, especially for low-resource languages.

- **Work in the low resource language domain:** I have encountered less work on Galician languages, the paper explores an underexplored research avenue. 
- **Open-Source Contributions:** By releasing newly trained models, evaluation datasets, and an assessment framework, the paper contributes to reproducibility and further research in multilingual NLP. 
- **Empirical Validation:** The study is supported by experimental results, although the interpretation and generalization of these results raise concerns.

The authors show that: 

1. Finetuning on high-quality language-specific corpora improves models' abilities in that language
2. Adding instructional data in the fine-tuning mix improves model abilities further
Summary of weaknesses: Some of the weaknesses found in the paper are:
- The study focuses on Galician, but the authors acknowledge that their findings may not generalize to other languages that are typologically different or have varying resource levels.
- Using synthetic data for pretraining with Galician instructions could introduce biases compared to human-created data and might not be of high quality.
- Human evaluations are limited in size and diversity due to their high cost, which could affect the reliability of conclusions based on these evaluations and the validation of automatic metrics.
- There is a concerning low correlation between automatic metrics (both probabilistic and similarity-based) and human qualitative evaluations, raising doubts about the reliability of automatic metrics in assessing generative tasks and linguistic quality in Galician.
- While CPT improves linguistic quality, larger multilingual base models without CPT often outperform smaller adapted models in task-based performance, questioning the overall impact of CPT on task-solving ability.

- **Limited Generalizability:** While Galician serves as a compelling case study, it remains unclear how well these findings extend to other underrepresented languages, particularly those with different linguistic typologies. 
- **Synthetic Data Concerns:** The reliance on synthetically generated instruction datasets introduces potential biases and artifacts that may not fully replicate real-world linguistic phenomena. 
- **Evaluation Discrepancies:** The observed inconsistencies between automated metrics and human judgments suggest that further refinement is needed for evaluation techniques. The paper does not sufficiently validate whether simil-eval and difsur are substantial contributions beyond existing evaluation methodologies. 
- **Limited Human Evaluation Scope:** The human assessment portion is constrained in sample size, raising concerns about the robustness of qualitative conclusions. 
- **Weak Substantiation of Evaluation Metrics:** The simil-eval and difsur frameworks are not convincingly shown to outperform or significantly add value beyond existing evaluation techniques. Without more rigorous comparative analysis, their introduction appears incremental rather than groundbreaking. 
- **Sparse Discussion on Computational Costs:** The paper does not thoroughly address the computational feasibility of CPT approaches, particularly for researchers with limited resources.

I have some major issues with the paper.

1. The authors have not uncovered anything new. The findings of using high-quality fine-tuning data to increase models' language range are well known, with several large efforts covering many more languages and more extensively evaluated [1, 2]. Oddly, these papers are not mentioned, cited or benchmarked against.

2. Similarly, the findings of utilizing instruction-heavy or instruction-like data in the last training phases have been well studied under the term of annealing. This is now common practice in most major model releases, and again, seminal papers in this area have not been cited [3].

3. I'm not sure how transferable these findings are given that the continual finetuneing langauge is very close to languages the chosen base models are already trained on.

References:

1. Lingfeng Ming, Bo Zeng, Chenyang Lyu, Tianqi Shi, Yu Zhao, Xue Yang, Yefeng Liu, Yiyu Wang, Linlong Xu, Yangyang Liu, Xiaohu Zhao, Hao Wang, Heng Liu, Hao Zhou, Huifeng Yin, Zifu Shang, Haijun Li, Longyue Wang, Weihua Luo, Kaifu Zhang:
Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement. CoRR abs/2412.04003 (2024)

2. Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, Barry Haddow:
EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models. CoRR abs/2409.17892 (2024)

3. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kai Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun:
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. CoRR abs/2404.06395 (2024)
Comments, suggestions and typos: The paper is clear and follows a coherent methodology. However, it could benefit from some improvements:
- I would include a flowchart of the methodology followed to make the different stages of the process clearer.
- In line 357, there is an equation, but it does not have the proper equation format.
- In section 3.3.2 (lines 381 and 385), the definitions of BERTScore and MoverScore are given. I believe that both metrics and their definitions are swapped. MoverScore captures deeper semantic relationships.
- In figure 3, the colors are not very distinguishable.
- In graphs like those in figure 2, I would add numerical labels so the values can be seen more precisely.

- **Clarify Corpus Composition:** Provide a more detailed explanation of the language proportions in the training corpora and the rationale behind selecting Spanish, English as languages rather than other scripts like Hindi or Chinese (which also have abundance of data). More clarity is needed on this front. 
- **Section 3.3 Needs Streamlining:** The discussion on surprisal and probabilistic metrics (3.3.1) is somewhat unnecessary and could be more concisely presented. The paper spends a lot of time expanding on fundamental and well-known topics like BERTScore, Surprisal, Log Likelihood, etc. 
- **Expand Error Analysis:** The paper should elaborate on the reasons behind metric discrepancies (e.g., why BLEU underperforms in summarization tasks while human evaluators provide higher scores). 
- **Scalability Discussion:** Address the scalability of the proposed methodology to other languages beyond Galician, considering both linguistic and resource constraints. 
- **Ablation Study on Instruction Impact:** A more detailed breakdown of how different types of instruction datasets affect performance would strengthen the findings on instruction tuning in CPT. 
- **Justification for Evaluation Metrics:** Provide a clearer justification for why simil-eval and difsur are necessary advancements over existing metrics and include a direct comparison with alternative evaluation techniques. 
- **Overall**: I believe that this paper would be very apt for a workshop focused on low-resource languages after a rewrite.

An ablation on the synthetic data generation process would have been useful, especially on unconditional vs existing text-conditioned synthetic data generation a. la. Nemotron [1]

References:

1. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro:
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. CoRR abs/2412.02595 (2024)
Overall assessment scores: 4

3

2.5