Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP
Authors: Kurt Micallef, Claudia Borg
Accepted at: acl25

MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP

Kurt Micallef
kurt.micallef@um.edu.mt

Claudia Borg
claudia.borg@um.edu.mt

Department of Artificial Intelligence, University of Malta

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across various Natural Language Processing (NLP) tasks, largely due to their generalisability and ability to perform tasks without additional training. However, their effectiveness for low-resource languages remains limited. In this study, we evaluate the performance of 55 publicly available LLMs on Maltese, a low-resource language, using a newly introduced benchmark covering 11 discriminative and generative tasks. Our experiments highlight that many models perform poorly, particularly on generative tasks, and that smaller fine-tuned models often perform better across all tasks. From our multidimensional analysis, we investigate various factors impacting performance. We conclude that prior exposure to Maltese during pre-training and instruction-tuning emerges as the most important factor. We also examine the trade-offs between fine-tuning and prompting, highlighting that while fine-tuning requires a higher initial cost, it yields better performance and lower inference costs. Through this work, we aim to highlight the need for more inclusive language technologies and recommend that researchers working with low-resource languages consider more “traditional” language modelling approaches.

1 Introduction

Large Language Models (LLMs) have seen a huge rise in use due to their strong performance and remarkable generalisability across a wide range of diverse tasks (OpenAI et al., 2024; Grattafori et al., 2024; Gemma Team et al., 2024). Their appeal is evident from their ability to perform tasks without additional training, with models able to infer from a few examples (few-shot or in-context learning) or an instruction (zero-shot) (Raffel et al., 2020; Brown et al., 2020). Furthermore, the availability of a plethora of multilingual models makes this technology more accessible for many languages, due to the inherent cross-lingual transfer capabilities. Despite this, many low-resource languages still face significant challenges in achieving strong performance with these models (Ahuja et al., 2023; Asai et al., 2024).

Prior research on older multilingual models, such as mBERT, highlighted the challenges faced by low-resource languages, particularly when a language is absent from pre-training (Chau et al., 2020; Muller et al., 2021). However, modern LLMs often have an additional training phase, designed to improve their generalisability: instruction-tuning. This raises new questions about the extent to which instruction-tuning mitigates or exacerbates performance gaps for low-resource languages.

In this work, we aim to address this for Maltese, an official EU language that ranks the lowest in the Digital Language Equality score (Rosner and Borg, 2023). While our primary objective is to understand which LLM properties influence downstream task performance, we compare this to more traditional fine-tuning approach with relatively smaller models. The main contributions of this work are:

1. A new evaluation benchmark composed of a variety of 11 discriminative and generative Maltese NLP tasks, to facilitate the evaluation and development of language technology.

2. A comprehensive experimental setup on 55 LLMs, for which we analyse which LLM properties are most important for better downstream task performance.

3. Several fine-tuned models of a relatively smaller size for each of these tasks, often surpassing all LLMs included in this study.

Our evaluation code and results are made publicly available.1 We also make the best performing

1https://huggingface.co/spaces/MLRS/MELABench
fine-tuned models publicly available.\footnote{BERTu: https://huggingface.co/collections/MLRS/bertu-683ac54c1b6ab3ae715cb43d; mT5-Small: https://huggingface.co/collections/MLRS/mt5-small-683eeecd001179a722c98298b.} Through our evaluation, we explore the following research questions:

1. How well do LLMs perform compared to smaller fine-tuned models?
2. What factors contribute to a model’s performance on downstream tasks?
3. How viable is it to train smaller but task-specific models as opposed to prompting larger but generic models?

2 Experimental Setup

2.1 Evaluation Benchmark
We conducted a survey of publicly available Maltese datasets, which allows us to benchmark the performance of various models on Maltese. We make a distinction with the type of task depending on whether the output is discrete (discriminative) or a text in natural language (generative). In total, we collected 11 datasets, shown in Table 1, with an even mixture of discriminative and generative tasks.

2.2 Models
To answer our primary research question, we use a variety of generative models. We consider 55 different language models whose weights are publicly available, covering various properties which we consider important for our analysis. These are model size (300M – 15B), language coverage (18 – 511, where known), and whether the model is pre-trained (PT) or instruction-tuned (IT). Moreover, we identify whether the model has seen Maltese during pre-training (PT), during instruction-tuning (IT), or never (NO). In the case of commercially released models, this information is not available and is categorised as unknown (NK). These details are summarised in Table 2. Additionally, in Appendix D we present an evaluation for ChatGPT 4o in a limited experimental setup.

2.3 Evaluation
We use the Language Model Evaluation Harness (Gao et al., 2024) to conduct the prompting experiments. For each task, we define a template in which we structure the input in textual format together with an instruction, as well as formatting the target in textual format. For generative tasks, the output is simply given as is, but for discriminative tasks, discrete label(s) are mapped into textual format as necessary. Our main experiments are conducted with English instructions, but we also include a set of experiments with Maltese instructions which we manually translate. See Appendix A for further details regarding the prompt templates used.

In our setup, we conduct two main experiments: zero-shot and one-shot. In the zero-shot case, the model is given only the input and the instruction, and it is expected to produce the corresponding output. In the one-shot case, we additionally prepend this with the input and output of a sample from the given task, formatted with the same template. In both cases, inference is carried out on the final instance, where the output is not provided to the model. Any examples used for in-context learning (one-shot) are taken from the training set, when this is available, or the validation set otherwise. Since no training or validation set is available for Belebele, this task is omitted from one-shot experiments.

In terms of automated evaluation metrics, we report the following. We use macro-averaged F1 for Sentiment Analysis, SIB-200, Taxi1500, Maltese News Categories, and MultiEURLEX. For Belebele, we report the accuracy. We report ChrF scores for OPUS-100, Flores-200, and WebNLG, and Rouge-L for EUR-Lex-Sum and Maltese News Headlines. Additionally, we also provide BLEU scores for OPUS-100 and Flores-200, and Rouge-L scores for WebNLG, and ChrF scores for EUR-Lex-Sum and Maltese News Headlines in Appendix C.

When evaluating the output, the appropriate metrics are calculated on the generated output for generative tasks. For discriminative tasks, this is not as straightforward since the expected output is discrete. Hence, the output is extracted by comparing the log-likelihood of generating each label. The label with the highest log-likelihood is chosen for single-label classification tasks (Sentiment, SIB-200, Taxi1500, and Belebele). For multi-label classification tasks (News Categories and MultiEURLEX), we extract the predicted labels based on the number of gold labels.

2.4 Fine-Tuned Models
We want to compare LLMs to the performance of smaller fine-tuned models, which also serve as baselines. The models are trained on the training
<table>
  <tr>
    <th>Type</th>
    <th>Name</th>
    <th>Task</th>
    <th>[train]</th>
    <th>[validation]</th>
    <th>[test]</th>
  </tr>
  <tr>
    <td rowspan="6">discriminative</td>
    <td>Sentiment (Martínez-García et al., 2021)</td>
    <td>Sentiment Analysis</td>
    <td>595</td>
    <td>85</td>
    <td>433</td>
  </tr>
  <tr>
    <td>SIB-200 (Adelani et al., 2024)</td>
    <td>Topic Classification</td>
    <td>701</td>
    <td>99</td>
    <td>204</td>
  </tr>
  <tr>
    <td>Taxi1500 (Ma et al., 2024)</td>
    <td>Topic Classification</td>
    <td>860</td>
    <td>106</td>
    <td>111</td>
  </tr>
  <tr>
    <td>News Categories (Chaudhary et al., 2024)</td>
    <td>Topic Classification (Multi-Label)</td>
    <td>10,784</td>
    <td>2,293</td>
    <td>2,297</td>
  </tr>
  <tr>
    <td>MultiEURLEX (Chalkidis et al., 2021)</td>
    <td>Topic Classification (Multi-Label)</td>
    <td>17,521</td>
    <td>5,000</td>
    <td>5,000</td>
  </tr>
  <tr>
    <td>Belebele (Bandarkar et al., 2024)</td>
    <td>Machine Reading Comprehension</td>
    <td>0</td>
    <td>0</td>
    <td>900</td>
  </tr>
  <tr>
    <td rowspan="5">generative</td>
    <td>OPUS-100 Fixed (Abela et al., 2024)</td>
    <td>Machine Translation (EN→MT)</td>
    <td>1,000,000</td>
    <td>2,000</td>
    <td>2,000</td>
  </tr>
  <tr>
    <td>Flores-200 (NLLB Team et al., 2022)</td>
    <td>Machine Translation (EN→MT)</td>
    <td>0</td>
    <td>997</td>
    <td>1,012</td>
  </tr>
  <tr>
    <td>WebNLG (Cripwell et al., 2023)</td>
    <td>Data-to-Text</td>
    <td>*13,211</td>
    <td>1,665</td>
    <td>1,778</td>
  </tr>
  <tr>
    <td>EUR-Lex-Sum (Aumiller et al., 2022)</td>
    <td>Abstractive Summarisation</td>
    <td>940</td>
    <td>187</td>
    <td>188</td>
  </tr>
  <tr>
    <td>News Headlines (Chaudhary et al., 2024)</td>
    <td>Abstractive Summarisation</td>
    <td>17,782</td>
    <td>3,810</td>
    <td>3,811</td>
  </tr>
</table>

Table 1: Dataset Summary
*Indicates noisy data obtained through machine translation.

<table>
  <tr>
    <th>Name</th>
    <th>Parameter Count</th>
    <th>Languages</th>
    <th>Training overall/Maltese</th>
  </tr>
  <tr>
    <td>PolyLM (Wei et al., 2023)</td>
    <td>1.7B, 13B</td>
    <td>18</td>
    <td>PT/NO</td>
  </tr>
  <tr>
    <td>XGLM (Lin et al., 2022)</td>
    <td>564M, 1.7B, 2.9B, 4.5B, 7.5B</td>
    <td>30</td>
    <td>PT/NO</td>
  </tr>
  <tr>
    <td>mGPT (Shliazhko et al., 2024)</td>
    <td>1.3B, 13B</td>
    <td>61</td>
    <td>PT/NO</td>
  </tr>
  <tr>
    <td>BLOOM (BigScience Workshop et al., 2023)</td>
    <td>560M, 2B, 3B, 8B</td>
    <td>46</td>
    <td>IT/NO</td>
  </tr>
  <tr>
    <td>Aya-23 (Aryabumi et al., 2024)</td>
    <td>8B</td>
    <td>23</td>
    <td>IT/NO</td>
  </tr>
  <tr>
    <td>BLOOMZ (Muenninghoff et al., 2023)</td>
    <td>560M, 2B, 3B, 8B</td>
    <td>46</td>
    <td>IT/NO</td>
  </tr>
  <tr>
    <td>BX-LLaMA (Li et al., 2023)</td>
    <td>7B, 13B</td>
    <td>*52</td>
    <td>IT/NO</td>
  </tr>
  <tr>
    <td>BX-BLOOM (Li et al., 2023)</td>
    <td>7B</td>
    <td>*77</td>
    <td>IT/NO</td>
  </tr>
  <tr>
    <td>Salamandra (Gonzalez-Agirre et al., 2025)</td>
    <td>2B, 7B</td>
    <td>35</td>
    <td>PT/PT</td>
  </tr>
  <tr>
    <td>EuroLLM (Martins et al., 2025)</td>
    <td>1.7B, 9B</td>
    <td>35</td>
    <td>PT/PT</td>
  </tr>
  <tr>
    <td>mT5 (Xue et al., 2021)</td>
    <td>300M, 582M, 1.23B, 3.74B, 13B</td>
    <td>101</td>
    <td>PT/PT</td>
  </tr>
  <tr>
    <td>MaLA-500 (Lin et al., 2024)</td>
    <td>8.6B</td>
    <td>511</td>
    <td>PT/PT</td>
  </tr>
  <tr>
    <td>Teukon Instruct Research v0.4 (Ali et al., 2024)</td>
    <td>7B</td>
    <td>*24</td>
    <td>IT/PT</td>
  </tr>
  <tr>
    <td>Salamandra Instruct (Gonzalez-Agirre et al., 2025)</td>
    <td>2B, 7B</td>
    <td>*35</td>
    <td>IT/PT</td>
  </tr>
  <tr>
    <td>mT0 (Muenninghoff et al., 2023)</td>
    <td>300M, 582M, 1.23B, 3.74B, 13B</td>
    <td>*120</td>
    <td>IT/PT</td>
  </tr>
  <tr>
    <td>EuroLLM Instruct (Martins et al., 2025)</td>
    <td>1.7B, 9B</td>
    <td>35</td>
    <td>IT/IT</td>
  </tr>
  <tr>
    <td>Aya-101 (Ustün et al., 2024)</td>
    <td>13B</td>
    <td>101</td>
    <td>IT/IT</td>
  </tr>
  <tr>
    <td>Gemma 2 (Gemma Team et al., 2024)</td>
    <td>2B, 9B</td>
    <td>?</td>
    <td>PT/NK</td>
  </tr>
  <tr>
    <td>Llama 2 (Touvron et al., 2023)</td>
    <td>7B, 13B</td>
    <td>?</td>
    <td>PT/NK</td>
  </tr>
  <tr>
    <td>Llama 3 (Grattahiori et al., 2024)</td>
    <td>8B</td>
    <td>?</td>
    <td>PT/NK</td>
  </tr>
  <tr>
    <td>Ministral Instruct 2410 (Mistral AI Team, 2024)</td>
    <td>8B</td>
    <td>?</td>
    <td>IT/NK</td>
  </tr>
  <tr>
    <td>Gemma 2 Instruct (Gemma Team et al., 2024)</td>
    <td>2B, 9B</td>
    <td>?</td>
    <td>IT/NK</td>
  </tr>
  <tr>
    <td>Llama 2 Chat (Touvron et al., 2023)</td>
    <td>7B, 13B</td>
    <td>?</td>
    <td>IT/NK</td>
  </tr>
  <tr>
    <td>Llama 3 Instruct (Grattahiori et al., 2024)</td>
    <td>8B</td>
    <td>?</td>
    <td>IT/NK</td>
  </tr>
</table>

Table 2: Language Model Summary
*Since the set of languages used during PT and IT is not the same, the union of both sets is represented.
? = For models with closed-source data, the set of languages used during training is unknown.

set by performing parameter updates on the model. For each dataset, we train a separate model. Since no training set is available for Belebele and Flores-200, no baselines are fine-tuned for these tasks.

We consider BERT-based PT models, for which we add a linear classification head on top of the PT model. These are BERTu (Micallef et al., 2022) – a monolingual Maltese 126M parameter model – and mBERT (Devlin et al., 2019) – a multilingual model with 179M parameters. However, these are only applied to discriminative tasks since they are encoder-only models.

Therefore, we also fine-tune mT5-Small (Xue et al., 2021) – a multilingual encoder-decoder PT model with 300M parameters. Similar to the prompted models, we convert every input and output into textual format. However, we simply train on the textual input-output pairs and do not apply any prompt templates. Moreover, we do not include any task prefix which were used to fine-tune the original models (Raffel et al., 2020; Xue et al., 2021).3 Evaluation metrics for fine-tuning mT5 are otherwise computed similarly to prompted models. More details on our fine-tuning setup are included in Appendix B.

3This decision was made because the model is fine-tuned separately on each task, and the prefix did not have much influence on performance during our initial tests.
Figure 1: Zero-shot prompting performance of individual models aggregated across tasks.

3 Overall Trends

The starting point of our analysis is to understand the performance of all individual models across tasks by performing zero-shot prompting. Thus, we aggregate the scores of each model averaged across discriminative and generative tasks.4 For each model, we then calculate an overall score across all tasks by averaging these two scores. The results are shown in Figure 1.

Overall, Aya-101 is the best-performing model, followed closely by mT0-XXL. We attribute this primarily to the models’ exposure to Maltese data, which we further investigate in Section 4.1. The smaller mT0 models are the next best-performing models overall, along with Salamandra Instruct 7B, Gemma 2 Instruct 9B, and Llama 3 Instruct 8B.

For generative tasks, Aya-101 performs better than any other model, often by a significant margin. This is followed by mT0-XXL, EuroLLM Instruct 1.7B, Teuken Research Instruct 7B, Salamandra Instruct 7B, and Llama 3 Instruct 8B. In the case of discriminative tasks, on average, mT0-XXL and Gemma 2 Instruct 9B perform better than Aya-101.

We take a closer look at the score distribution for each task in Figure 2. More models are competitive with the fine-tuned baselines on discriminative tasks than generative tasks. Performance on generative tasks hovers near 0 for many models. A more in-depth analysis of the individual models on each task (see Figure 9) reveals that for generative tasks, the top performers are Aya-101, mT0-XXL, and Teuken Instruct Research, and to a lesser degree, EuroLLM 1.7B and Llama 3 Instruct 8B. These models often act as outliers from the rest of the models. The insights highlight that generating text data is much more challenging than understanding it and that many models fail to capture the linguistic nuances of a low-resource language like Maltese.

Looking at each task in Figure 2, we observe that models generally struggle with Taxi1500 and MultiEURLEX among discriminative tasks and EUR-Lex-Sum among generative tasks. This could be attributed to the specific domain of these tasks: the Bible for Taxi1500 and European Union documents for MultiEURLEX and EUR-Lex-Sum. The latter two tasks have input sequences that are also generally longer, which hampers performance due to the model’s limited context length. Moreover, we note that MultiEURLEX is the only discriminative task where models perform quite on par with one another, particularly since it is a multi-label classification task on a massive scale.

When compared to the baselines, all prompted models perform worse, with the exception of the Sentiment Analysis task, for which some models outperform mBERT. Among the baselines, mT5 generally performs better than mBERT, except for MultiEURLEX, potentially due to the task being ill-suited for generative models, as discussed earlier. Overall, BERTu performs the best in discriminative tasks due to its Maltese pre-training, except for Sentiment Analysis for which we observe a perfect score for mT5. While we are uncertain why the model does so well, we posit that this is due to the task being quite simplistic, since it is the only task where prompted models outperform some of the baselines. Moreover, out of all the tasks, this dataset is the most likely to contain code-switching, which might make it easier for multilingual models to pick up on certain linguistic signals.

3.1 Model Training

We now examine the relationship between zero-shot prompting performance and overall model training (PT vs IT). We group models by their training and visualise the average performance for each task in Figure 4.

It is very evident that IT models perform better than PT models on all tasks. The performance difference between model types is quite small for some tasks, such as MultiEURLEX and EUR-Lex-
Figure 2: Zero-shot prompting performance distribution of models per task.

Figure 3: One-shot prompting performance distribution of models per task.

Figure 4: Zero-shot prompting performance distribution per task, with models grouped by different training types.

Sum, which further reinforces our argument that these datasets are challenging for the prompted models. The performance disparity on some tasks is proportionately larger, which could be due to the difficulty of the task and/or previous instruction-tuning on the same task.

3.2 Number of Shots

We want to understand the potential impact of in-context learning. We perform one-shot prompting and compare it to the previous zero-shot results.

Figure 3 shows the distribution of models on each of the tasks in relation to the baseline models
with one-shot prompting. Overall, the performance gap between fine-tuned models and prompted models is reduced. In Sentiment Analysis, the best-performing prompted models perform almost as well as a fine-tuned BERTu and almost as good as a fine-tuned mT5-Small on Maltese News Headlines. We also observe that some models perform better than fine-tuned mBERT on SIB-200 with one-shot compared to zero-shot. However, for most of the other tasks, the gap between fine-tuned and prompted models remains significant.

To better interpret the changes between zero-shot and one-shot, we calculate the performance difference aggregated across PT and IT models for each task. This is calculated by subtracting the zero-shot performance from the corresponding one-shot performance of every prompted model.5 Figure 5 shows the performance difference, with a positive score indicating better one-shot results on average.

Similar to Zhang et al. (2024), we observe consistent performance improvements with one-shot across all generative tasks, but mixed results in discriminative tasks, with slight degradations for Taxi1500, News Categories, and MultiEURLEX. IT models also give significant improvements in Sentiment Analysis, SIB-200, OPUS-100, and Maltese News Headlines. The performance on MultiEURLEX and EUR-Lex-Sum is largely the same regardless of the number of shots. We attribute this to the longer inputs which are being truncated to a limited sequence length.

5Similar to Zhang et al. (2024), we observe negative effects for mT0 and BLOOMZ due to their zero-shot instruction-tuning, and hence we omit them for this analysis.

Figure 5: Performance difference of zero-shot prompting and one-shot prompting averaged across models with different training types.

4 Effect of Maltese Exposure

We now examine the effect that exposing the model to Maltese has on its performance. Apart from a model’s overall training (PT or IT), models are further grouped into different categories based on their explicit training on Maltese data, indicated by: NO, PT, and IT, referring to no exposure to Maltese data, exposure during pre-training, and exposure during instruction-tuning, respectively. For these analyses, we exclude models for which this information cannot be inferred from publicly available metadata (NK).

4.1 Maltese Training

We first examine the impact on model performance given its training on Maltese at different stages. In the zero-shot experiments (Figure 6a), we observe that models exposed to Maltese during either IT or PT achieve better results, with the best overall results observed in the IT/IT category. For discriminative tasks, IT/PT models perform on par with IT/IT models on News Categories and MultiEURLEX. On generative tasks, IT/IT generally gives better results than IT/PT.

Similar to the observation made in Section 3.1, IT results in better overall scores than PT models. We also note that Maltese PT is generally helpful, especially when comparing PT/PT against PT/NO.

For the one-shot experiments (Figure 6b), we note that IT/IT models outperform any other type of model more consistently and significantly, with the exception of the MultiEURLEX task. PT/PT models are also generally less performant than PT/NO models. This highlights that while instruction-tuning on a target language is beneficial, models sometimes need in-context examples to access their Maltese knowledge.

In Appendix E, we also present further analyses exploring the relationship between performance and other dimensions such as a model’s size and the number of languages on which it was trained. We initially observed a slight correlation between performance and these variables. However, model training on Maltese remains the main confounding variable, having a larger impact on performance. When models trained on Maltese are excluded from these analyses, we observe that this correlation diminishes or is reversed.
Figure 6: Prompting performance distribution per task, with models grouped by different training types and Maltese training.

4.2 Maltese Prompts

We now examine the impact of providing models with more Maltese text, not only in the form of in-context examples, but also by manually translating the instruction from English to Maltese.6 Therefore, we repeat all previous prompting experiments using Maltese prompt templates. Each score with English prompting is then subtracted from these new scores with Maltese prompting to get the performance difference, and the overall results are shown in Figure 7.

We observe mixed results in both zero-shot and one-shot, but performance is generally worse with Maltese prompts. However, models are negatively impacted in most discriminative tasks, regardless of the model’s exposure to Maltese during its training. With one-shot, the negative difference is even more pronounced, highlighting that in-context learning examples are better suited to prime the model to Maltese as opposed to language instructions. PT/NO models seem to get significant improvements in Sentiment Analysis and Flores-200 with Maltese prompts in zero-shot, but this drastically diminishes in one-shot.

IT/IT models exhibit some of the worst degradations with Maltese prompts, particularly in zero-shot, even though these are the models that were exposed to Maltese the most. However, although all models in this category – EuroLLM Instruct and Aya-101 – are exposed to Maltese examples during their IT, the actual instructions are still in English.

6More detail regarding our prompt translation process is included in Appendix A.
<table>
  <tr>
    <th>Model</th>
    <th>Training FLOPs</th>
    <th>Inference FLOPs</th>
  </tr>
  <tr>
    <td>BERTu</td>
    <td>1.54e16</td>
    <td>3.28e10</td>
  </tr>
  <tr>
    <td>mBERT</td>
    <td>2.50e16</td>
    <td>4.06e10</td>
  </tr>
  <tr>
    <td>mT5-Small</td>
    <td>7.19e15</td>
    <td>7.14e09</td>
  </tr>
  <tr>
    <td>mT0-XXL</td>
    <td>0</td>
    <td>5.96e12</td>
  </tr>
  <tr>
    <td>Aya-101</td>
    <td>0</td>
    <td>5.84e12</td>
  </tr>
  <tr>
    <td>Llama 3 Instruct 8B</td>
    <td>0</td>
    <td>5.06e13</td>
  </tr>
</table>

Table 3: Computational cost estimates in terms of Floating Point Operations (FLOPs).

![Performance difference of prompting with English and Maltese instructions averaged across models with different training types and Maltese training.](page_120_120_624_480.png)

(a) Zero-shot

(b) One-Shot

Figure 7: Performance difference of prompting with English and Maltese instructions averaged across models with different training types and Maltese training.

This highlights the discrepancy between model performance and usability, since speaker populations of low-resource languages like Maltese would have to resort to providing English instructions in their interactions with these models.

5 Efficiency

The computational efficiency of fine-tuning and prompting is also an important factor to consider. We select the following prompted models based on the best overall performance and the model architecture variety: Aya-101, mT0-XXL, and Llama 3 Instruct 8B. We consider all fine-tuned models: BERTu, mBERT, and mT5-Small.

To estimate the computational requirements, we follow Liu et al. (2022) and compute the Floating-Point Operations Per Second (FLOPs, Kaplan et al., 2020) required for a single instance. We take the median sequence length of an instance for each dataset and use it to calculate the inference FLOPs for a given model. For fine-tuned models, we use only the raw input sequence in textual format. For prompted models, we also include the accompanying instruction. For the purpose of this analysis, we only consider zero-shot prompting with English instructions. We also compute the training FLOPs for fine-tuned models by calculating the median sequence length on every training dataset, but also take into account the batch size and the total number of steps\footnote{This is calculated by averaging the total number of steps (including early stopping patience) across all runs.} during training. We then take the average number of FLOPs across all tasks. Since baseline models were not fine-tuned for Belebele and Flores-200, we do not include any calculations for these tasks in this analysis. Table 3 shows the resulting calculations.

As expected, the inference cost of fine-tuned models is magnitudes smaller than that of prompted models due to the smaller model sizes. We also note that for prompted models, as the number of few-shot examples increases, so does the inference cost. The cost for fine-tuning models is also magnitudes larger than applying inference on a single instance. However, as we apply inference on more examples, this initial upfront cost diminishes.

If we define efficiency as a function of the overall performance, the cost, and the number of inference samples as follows:

\[
\frac{\text{performance}}{\text{cost}_{\text{training}} + \text{cost}_{\text{inference}} \times \text{samples}}
\]

then, as the number of samples increases, the efficiency of prompting larger models drastically decreases, as shown in Figure 8. Furthermore, the sheer size of the prompted models also necessitates more expensive hardware to store models on disk and load them into memory.

6 Related Work

Due to the lack of research on generative Maltese NLP, our research is primarily related to various multilingual benchmarking and evaluation works, although none of them include Maltese. Ahuja et al. (2023) evaluate LLM performance on a newly developed benchmark covering 70 languages. They
find that low-resource languages are negatively affected by ill-fitted tokenisers and limited pre-training data used by the model. Zhang et al. (2024) study the impact of few-shot prompting across 56 languages, finding that the few-shot performance does not always improve over zero-shot, depending on the model, task, and language. Asai et al. (2024) similarly analyse few-shot prompting performance but in a cross-lingual setup, finding that while few-shot generally helps, models perform particularly poorly on lower-resourced languages. Zhang et al. (2023) analyse the performance on code-switched languages. In their analysis, they find that both an increased number of shots and model size (when considering the same model family) generally improve performance.

Most of these works also fine-tune smaller BERT-based and/or mT5 models, similarly finding that they outperform prompted LLMs (Ahuja et al., 2023; Asai et al., 2024; Zhang et al., 2023). Likewise, Lai et al. (2023) find that ChatGPT underperforms supervised SOTA models on most tasks.

Our research is complementary to these works as we look closer at the performance of a single language. We also significantly scale the number of language models which allows us to study multiple factors affecting model performance.

7 Conclusion

In this paper, we present a comprehensive evaluation of LLMs for Maltese, a low-resource language. We introduce a new benchmark covering a total of 11 discriminative and generative tasks. We carried out an evaluation of 55 publicly available models under zero-shot and one-shot prompting, revealing significant performance variations. Our study highlights the need to improve the state of low-resource languages such as Maltese.

In analysing our results, we uncover several key trends. Prompted models consistently lag behind smaller fine-tuned models, particularly in generative tasks where their performance is significantly lower. Crucially, the level of exposure to Maltese in a model’s training has the largest bearing on performance, especially in instruction-tuning.

Additionally, our efficiency analysis highlights the trade-offs between fine-tuning and prompting. While fine-tuning incurs higher initial costs, the inference cost is significantly lower. The initial cost incurred to fine-tune smaller models quickly pays off with more inference instances, apart from the higher performance on downstream tasks.

These findings underscore the pressing need for more inclusive language models that better support low-resource languages at every stage of the training pipeline. While LLMs offer strong generalisability, their limited performance on low-resource languages like Maltese reduces their usability in these scenarios. For researchers with limited computational resources, fine-tuning smaller models presents a viable alternative to prompting larger models, despite the trade-off in generalisability. Ultimately, our study calls for a more balanced approach to model development, ensuring that low-resource languages like Maltese receive adequate representation in the evolving landscape of LLMs.

8 Limitations

Models  Although we consider a wide variety of models, the coverage of models trained on Maltese was very limited. Moreover, due to the large number of models considered as well as computational constraints, we only choose models with no more than 15 billion parameters. In addition, for our main analysis, we do not consider any commercial models, not only due to the prohibitive costs to conduct this evaluation, but also due to our constrained evaluation on discriminative tasks. However, we present a limited comparative evaluation on ChatGPT in Appendix D, showing mixed results on the tasks tested compared to the fine-tuned baselines.

Datasets  While our benchmark is certainly an improvement on the state of NLP for a low-resource language, certain issues impact our evaluation. Firstly, our results are confounded by the amount of data in some tasks, which is why we strived to
show per-task results as much as possible. Despite the small training data sizes, we show that fine-tuned models still outperform the prompted large language models. Secondly, the variance of tasks is also limited, as we have Topic Classification for the majority of our discriminative tasks, and Machine Translation dominates our generative tasks. The latter is often a large source of instruction-tuning data for multilingual instruction-tuning, and often the only data considered when Maltese was used for instruction-tuning (Üstün et al., 2024). Thirdly, the domains of these datasets are quite narrow as they are mostly composed of data derived from news articles, EU legislative documents, and Wikimedia. All in all, we hope that our work raises awareness on the importance of developing newer and more diverse datasets for low-resource languages.

Prompt Engineering  Various works have shown that different models can be optimised with different prompts (Zhao et al., 2021; Jiang et al., 2020; Shin et al., 2020; inter alia). We sidestep this by mostly using prompt templates from previous works (see Appendix A for more details). We highlight that we did consider prompting in Maltese (Section 4.2), which is often understudied in the context of multilingual evaluation (Ahuja et al., 2023; Asai et al., 2024). We did not experiment with a larger number of shots to keep our experiments sustainable.

Unconsidered Model Properties  We tried to analyse many possible dimensions but still had to limit our search space. Despite looking at models exposed to Maltese training, we largely treated this as a categorical variable. However, the raw amount of tokens, as well as the proportion in relation to the rest of the training data, would have a large bearing on performance. In a similar vein, we did not consider the different scales and quality of the datasets used for training different models. We also do not factor for different kinds of training processes used during instruction-tuning such as Reinforcement Learning with Human Feedback (Ziegler et al., 2020) and Direct Preference Optimisation (Rafailov et al., 2023). As we make this data available, we encourage future work to analyse different dimensions not considered in this work.

Dataset Contamination  It is likely that models have been exposed to language data that is not included in the figures listed in Table 2 (Blevins and Zettlemoyer, 2022; Muennighoff et al., 2023), but we do not account for it. Since our benchmarks are based on publicly available datasets, it is likely that increased performance in some models is due to data contamination during training. While this can be accidental, instruction-tuned models may have used certain datasets deliberately. Hence, higher scores would be attributed to the model’s training on that task, rather than its capabilities to generalise to unseen tasks. Dataset contamination is also an active area of research (Blevins and Zettlemoyer, 2022; Ballocu et al., 2024), and we therefore treat models as black-box systems in this regard.

9 Ethics Statement

We inherit any biases that may be present in the data and language models that we use. The new generative models fine-tuned on Maltese data could be used to produce text that inherit these biases. However, given their relatively low performance on generative tasks and the fact that we train these using publicly available resources, we do not foresee any major risks.

Acknowledgements

We acknowledge support from the LT-Bridge Project (GA 952194) and DFKI for access to the Virtual Laboratory. We further acknowledge funding by Malta Enterprise Research and Development Scheme.

References

Kurt Abela, Kurt Micallef, Marc Tanti, and Claudia Borg. 2024. Tokenisation in machine translation does matter: The impact of different tokenisation approaches for Maltese. In Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024), pages 109–120, Bangkok, Thailand. Association for Computational Linguistics.

David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiu Annie Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 226–245, St. Julian’s, Malta. Association for Computational Linguistics.

Kabir Ahuja, Harshita Diddee, Rishav Hada, Milli-cent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, et al. 2023. MEGA: Multilingual evaluation
of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics.

Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Ruttmann, Charvi Jain, Max Lübbering, Daniel Steinigen, Johannes Leveling, et al. 2024. Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs. Preprint, arXiv:2410.03730.

Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeleine Smith, Jon Ander Campos, Yi Chern Tan, et al. 2024. Aya 23: Open weight releases to further multilingual progress. Preprint, arXiv:2405.15032.

Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2024. BUFFET: Benchmarking large language models for few-shot cross-lingual transfer. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1771–1800, Mexico City, Mexico. Association for Computational Linguistics.

Dennis Aumiller, Ashish Chouhan, and Michael Gertz. 2022. EUR-lex-sum: A multi- and cross-lingual dataset for long-form summarization in the legal domain. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7626–7639, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. 2024. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 67–93, St. Julian’s, Malta. Association for Computational Linguistics.

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics.

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, et al. 2023. BLOOM: A 176B-parameter open-access multilingual language model. Preprint, arXiv:2211.05100.

Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics.

Amit Kumar Chaudhary, Kurt Micallef, and Claudia Borg. 2024. Topic classification and headline generation for Maltese using a public news corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16274–16281, Torino, Italia. ELRA and ICCL.

Liam Cripwell, Anya Belz, Claire Gardent, Albert Gatt, Claudia Borg, Marthese Borg, John Judge, Michela Lorandi, Anna Nikiforovskaya, and William Soto Martinez. 2023. The 2023 WebNLG shared task on low resource languages, overview and evaluation results (WebNLG 2023). In Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pages 55–66, Prague, Czech Republic. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, et al. 2024. A framework for few-shot language model evaluation.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118.

Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, Daniel Tamayo, José Javier Saiz, Ferran Espuña, Jaume Prats, Javier Aula-Blasco, et al. 2025. Salamandra technical report. Preprint, arXiv:2502.08489.

Aaron Grattaiorii, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783.

Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.

Viet Dac Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13171–13189, Singapore. Association for Computational Linguistics.

Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-X: Multilingual replicable instruction-following models with low-rank adaptation. Preprint, arXiv:2305.15011.

Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, and Hinrich Schütze. 2024. MaLA-500: Massive language adaptation of large language models. Preprint, arXiv:2401.13303.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Smig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems.

Chunlan Ma, Ayyoob ImaniGooghari, Haotian Ye, Renhao Pei, Ehsaneddin Asgari, and Hinrich Schütze. 2024. Taxi1500: A multilingual dataset for text classification in 1500 languages. Preprint, arXiv:2305.08487.

Antonio Martínez-García, Toni Badia, and Jeremy Barnes. 2021. Evaluating morphological typology in zero-shot cross-lingual transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3136–3153, Online. Association for Computational Linguistics.

Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pomhial, Amin Farajian, Manuel Faysses, Mateusz Klimaszewski, et al. 2025. Eurollm: Multilingual language models for europe. Procedia Computer Science, 255:53–62. Proceedings of the Second EuroHPC user day.

Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas, and Claudia Borg. 2022. Pre-training data quality and quantity for a low-resource language: New corpus and BERT models for Maltese. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 90–101, Hybrid. Association for Computational Linguistics.

Mistral AI Team. 2024. Un Minstral, des Ministraux. https://mistral.ai/news/ministraux/. Accessed: 2024-12-20.

Niklas Muenninghoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.

Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamel Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448–462, Online. Association for Computational Linguistics.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, et al. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al. 2024. GPT-4 technical report. Preprint, arXiv:2303.08774.
Rafael Raffailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.

Michael Rosner and Claudia Borg. 2023. Language Report Maltese. In Georg Rehm and Andy Way, editors, European Language Equality: A Strategic Agenda for Digital Language Equality, pages 183–186. Springer International Publishing, Cham.

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Online. Association for Computational Linguistics.

Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana Shavrina. 2024. mGPT: Few-shot learners go multilingual. Transactions of the Association for Computational Linguistics, 12:58–79.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.

Ahmet Üstün, Virat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thailand. Association for Computational Linguistics.

Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023. PolyLM: An open source polyglot large language model. Preprint, arXiv:2307.06018.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.

Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Je-sujoba Alabi, Xiaoyu Shen, Dietrich Klakow, and Marius Mosbach. 2024. The impact of demonstrations on multilingual in-context learning: A multidimensional analysis. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7342–7371, Bangkok, Thailand. Association for Computational Linguistics.

Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Fikri Aji. 2023. Multilingual large language models are not (yet) code-switchers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12567–12582, Singapore. Association for Computational Linguistics.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR.

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. Preprint, arXiv:1909.08593.
A  Prompt Templates

Table 4 shows the prompts that were used to evaluate every model on each task. When available, we use a template suggested by the original dataset paper. Otherwise, we adapt a template from a related task which was available from the Language Model Evaluation Harness repository (Gao et al., 2024). For generative tasks, we ensure that the instruction explicitly mentions that the text should be generated in Maltese.

To generate the Maltese prompts, we reuse the English templates and translate them into Maltese. This is done by first passing the instruction through Google Translate and then performing post-editing with a Maltese native speaker.

B  Fine-Tuning Details

For fine-tuned models, we consider BERTu (Miccallef et al., 2022), mBERT (Devlin et al., 2019), and mT5-Small (Xue et al., 2021). Our training scripts are implemented using the transformers library (Wolf et al., 2020).

We train all models for at most 200 epochs but use early stopping on the main metric (as defined in Section 2.3) with a patience of 20 epochs. Due to the significantly larger scale of the data for OPUS-100, we only train for a maximum of 10 epochs instead.

For the BERT-based models, we use a learning rate of 1e-4 with an AdamW optimiser, an inverse square-root learning rate scheduler, a warmup of 1 epoch, and a weight decay of 0.01. We use a batch size of 16 for Sentiment Analysis, SIB-200, and Taxi1500 and a batch size of 32 for the other discriminative tasks. A dropout of 0.1 is used for Taxi1500 and MultiEURLEX, and 0.5 for the other discriminative tasks.

When fine-tuning mT5, we mostly follow the original fine-tuning recipe (Raffel et al., 2020; Xue et al., 2021) with a constant learning rate of 1e-3 with an Adafactor optimiser and a batch size of 32. BERT-based models are fine-tuned 5 separate times with different random seeds, and we report the mean performance across these runs. mT5 is only fine-tuned once per task.

C  All Results

In this section, we present individual results for each model and task considered. Figure 9 shows the zero-shot performance with English instructions and Figure 10 shows the one-shot performance with English instructions.

We also present the individual performance with all metrics of each model with English prompts in Tables 5, 6, 7, and 8. Results for fine-tuned models are shown in Tables 9 and 10.

D  Closed-Source Model Results

We include experiments with ChatGPT 4o (OpenAI et al., 2024) as a comparison to our main experiments. However, since this is a closed-source model accessible only through an API, our experiments with this model are limited. Firstly, since we do not have access to log-likelihoods, it is not possible for us to conduct discriminative experiments in a comparative manner, so we skip these tasks. Secondly, we also skip EUR-Lex-Sum due to the large context lengths needed for this task, which exceed our quota. Thirdly, for the remaining four tasks, we only prompt with 100 test samples for each task to limit our costs.

The results are shown in Table 11. Comparing these figures to the results obtained by our fine-tuned mT5 baseline (Table 10), ChatGPT 4o performs significantly worse on OPUS-100, significantly better on WebNLG and on par on Maltese News Headlines.

E  Analysing Other Model Properties

E.1  Model Size

We look at the performance as the model size grows in terms of the number of parameters. To analyse this, we fit linear regression models for PT and IT models on performance results aggregated by task type. We only do this for zero-shot results.

As shown in Figure 11a, a general improvement is observed in performance with model size increase. In general, IT models with larger sizes give better performances than PT models. In fact, a smaller performance gap is observed between PT and IT models with fewer than 10B parameters, especially in few-shot, where PT models overall perform better than IT models on generative tasks.

However, we note that among the largest IT models are the Aya-101 and mT0 models, which are trained on Maltese. If we exclude models which we know are trained on Maltese, then our previous observations do not hold as shown in Figure 11b. In fact, we see a negative impact on performance as model size grows for IT models, albeit with a larger confidence interval.
<table>
  <tr>
    <th>Task</th>
    <th>English Prompt Template</th>
    <th>Maltese Prompt Template</th>
  </tr>
  <tr>
    <td>Sentiment Analysis</td>
    <td>{text} Is the sentiment positive or negative?</td>
    <td>{text} Is-sentiment huwa pozittiv jew negativ?</td>
  </tr>
  <tr>
    <td>SIB-200</td>
    <td>The topic of the news “{text}” is</td>
    <td>Is-suġġett tal-ahbarjiet “{text}” huwa</td>
  </tr>
  <tr>
    <td>Taxi1500</td>
    <td>The topic of the verse is “{text}” is</td>
    <td>Is-suġġett tal-vers “{text}” huwa</td>
  </tr>
  <tr>
    <td>Maltese News Categories</td>
    <td>{text}</td>
    <td>{text}</td>
  </tr>
  <tr>
    <td></td>
    <td>What are the topic(s) of this news article?</td>
    <td>X’inhu(ma) s-suġġett(i) ta’ dan l-artiklu tal-ahbarjiet?</td>
  </tr>
  <tr>
    <td>MultiEURLEX</td>
    <td>{text}</td>
    <td>{text}</td>
  </tr>
  <tr>
    <td></td>
    <td>What are the topics of this text?</td>
    <td>X’inhum a s-suġġetti ta’ dan it-test?</td>
  </tr>
  <tr>
    <td>Belebele</td>
    <td>Given the following passage, query, and answer choices, output the letter corresponding to the correct answer.<br>###<br>Passage:<br>{text}<br>###<br>Query:<br>{question}<br>###<br>Choices:<br>(A) {answer1}<br>(B) {answer2}<br>(C) {answer3}<br>(D) {answer4}<br>###<br>Answer:</td>
    <td>Permezz tas-silta, mistoqsija, u għażliet ta’ twegibiet li ġejjin, aghti l-ittra li tikkorrispondi għat-twegiba t-tajba.<br>###<br>Passagg:<br>{text}<br>###<br>Mistoqsija:<br>{question}<br>###<br>Choices:<br>(A) {answer1}<br>(B) {answer2}<br>(C) {answer3}<br>(D) {answer4}<br>###<br>Twegiba:</td>
  </tr>
  <tr>
    <td>OPUS-100 Flores-200</td>
    <td>{source_sentence}</td>
    <td>{source_sentence}</td>
  </tr>
  <tr>
    <td></td>
    <td>The previous text is in {source_language}. Here is a translation to {target_language}:</td>
    <td>It-test preċedenti huwa bl-{source_language}. Din hija traduzzjoni ghall-{target_language}:</td>
  </tr>
  <tr>
    <td>WebNLG</td>
    <td>Verbalize in Maltese the following triples separated by a comma: {triples | join(', ')}<br></td>
    <td>Ivverbalizza bil-Maltu t-tripli li ġejjin separati b'virgola: {triples | join(', ')}<br></td>
  </tr>
  <tr>
    <td>EUR-Lex-Sum</td>
    <td>{text}</td>
    <td>{text}</td>
  </tr>
  <tr>
    <td></td>
    <td>Write a summary in Maltese for the text above:</td>
    <td>Ikteb sommarju bil-Malti għat-test t'hawn fuq:</td>
  </tr>
  <tr>
    <td>Maltese News Headlines</td>
    <td>{text}</td>
    <td>{text}</td>
  </tr>
  <tr>
    <td></td>
    <td>Write a headline in Maltese for the news article above:</td>
    <td>Ikteb titolu bil-Malti ghall-artiklu tal-ahbarjiet t'hawn fuq:</td>
  </tr>
</table>

Table 4: Prompt Templates used for each task.
Figure 9: Zero-shot prompting performance of individual models on each task. Horizontal lines represent models fine-tuned specifically on the task.

E.2 Model Multilinguality

We also analysed a model’s performance against the number of languages it was exposed to during its training. Similar to Section 4, we exclude models with unknown Maltese training (NK). We also exclude MaLA-500 from this analysis, as the high degree of languages skews our plots. Other than that, we plot aggregated zero-shot performance results against model multilinguality and fit separate linear regression models for PT and IT models.

In Figure 12a we observe a positive influence with the number of languages a model is exposed to for IT models. For PT models there is also a positive effect on generative tasks, although smaller than that for IT models. On the other hand, there is a negative impact as the number of languages
Figure 10: One-shot prompting performance of individual models on each task. Horizontal lines represent models fine-tuned specifically on the task.

increases for discriminative tasks.

Despite this, highly multilingual models which have seen more than 100 languages, are all models which have been exposed to Maltese during some part of their training. When excluding these models and refitting logistic models on the remaining data we see that the previously observed improvements are drastically reduced.
<table>
  <tr>
    <th>Model</th>
    <th>Sentiment Analysis<br>F1</th>
    <th>SIB-200<br>F1</th>
    <th>Taxi1500<br>F1</th>
    <th>News Categories<br>F1</th>
    <th>MultiEURLEX<br>F1</th>
    <th>Belebele<br>Accuracy</th>
  </tr>
  <tr>
    <td>PolyLM1.7B</td>
    <td>46.3</td>
    <td>3.4</td>
    <td>12.0</td>
    <td>5.4</td>
    <td>12.1</td>
    <td>23.0</td>
  </tr>
  <tr>
    <td>PolyLM13.0B</td>
    <td>26.0</td>
    <td>3.8</td>
    <td>5.0</td>
    <td>20.4</td>
    <td>12.6</td>
    <td>22.2</td>
  </tr>
  <tr>
    <td>XGLM0_564B</td>
    <td>29.4</td>
    <td>17.1</td>
    <td>7.5</td>
    <td>7.1</td>
    <td>15.1</td>
    <td>23.1</td>
  </tr>
  <tr>
    <td>XGLM1_7B</td>
    <td>40.9</td>
    <td>16.6</td>
    <td>5.5</td>
    <td>18.9</td>
    <td>12.7</td>
    <td>23.0</td>
  </tr>
  <tr>
    <td>XGLM2_9B</td>
    <td>32.7</td>
    <td>8.9</td>
    <td>9.7</td>
    <td>21.0</td>
    <td>14.3</td>
    <td>23.4</td>
  </tr>
  <tr>
    <td>XGLM4_5B</td>
    <td>26.6</td>
    <td>5.8</td>
    <td>11.4</td>
    <td>22.1</td>
    <td>18.2</td>
    <td>24.7</td>
  </tr>
  <tr>
    <td>XGLM7_5B</td>
    <td>29.9</td>
    <td>10.2</td>
    <td>6.4</td>
    <td>19.3</td>
    <td>14.9</td>
    <td>22.9</td>
  </tr>
  <tr>
    <td>mGPT1_3B</td>
    <td>33.0</td>
    <td>10.1</td>
    <td>12.5</td>
    <td>12.0</td>
    <td>7.9</td>
    <td>22.7</td>
  </tr>
  <tr>
    <td>mGPT13.0B</td>
    <td>38.5</td>
    <td>15.7</td>
    <td>11.7</td>
    <td>16.0</td>
    <td>7.5</td>
    <td>25.7</td>
  </tr>
  <tr>
    <td>BLOOM0_56B</td>
    <td>26.6</td>
    <td>11.1</td>
    <td>9.6</td>
    <td>8.9</td>
    <td>16.4</td>
    <td>23.1</td>
  </tr>
  <tr>
    <td>BLOOM2_0B</td>
    <td>56.1</td>
    <td>15.9</td>
    <td>10.0</td>
    <td>10.7</td>
    <td>13.7</td>
    <td>22.2</td>
  </tr>
  <tr>
    <td>BLOOM3_0B</td>
    <td>32.9</td>
    <td>30.6</td>
    <td>5.3</td>
    <td>15.6</td>
    <td>14.4</td>
    <td>24.8</td>
  </tr>
  <tr>
    <td>BLOOM4_0B</td>
    <td>26.0</td>
    <td>36.6</td>
    <td>15.3</td>
    <td>20.0</td>
    <td>12.8</td>
    <td>24.1</td>
  </tr>
  <tr>
    <td>Aya-23_0B</td>
    <td>53.1</td>
    <td>16.9</td>
    <td>8.9</td>
    <td>4.5</td>
    <td>12.2</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td>BLOOMZ0_56B</td>
    <td>49.9</td>
    <td>32.6</td>
    <td>8.9</td>
    <td>20.8</td>
    <td>14.9</td>
    <td>27.4</td>
  </tr>
  <tr>
    <td>BLOOMZ2_0B</td>
    <td>50.4</td>
    <td>36.0</td>
    <td>9.8</td>
    <td>23.3</td>
    <td>15.1</td>
    <td>21.0</td>
  </tr>
  <tr>
    <td>BLOOMZ3_0B</td>
    <td>62.8</td>
    <td>42.9</td>
    <td>16.3</td>
    <td>27.5</td>
    <td>14.9</td>
    <td>32.9</td>
  </tr>
  <tr>
    <td>BLOOMZ4_0B</td>
    <td>56.4</td>
    <td>48.8</td>
    <td>21.2</td>
    <td>27.4</td>
    <td>14.2</td>
    <td>28.9</td>
  </tr>
  <tr>
    <td>BX-LLaMA1_0B</td>
    <td>26.0</td>
    <td>8.0</td>
    <td>11.6</td>
    <td>6.9</td>
    <td>7.4</td>
    <td>25.7</td>
  </tr>
  <tr>
    <td>BX-LLaMA13.0B</td>
    <td>36.7</td>
    <td>9.1</td>
    <td>5.7</td>
    <td>7.0</td>
    <td>11.3</td>
    <td>22.8</td>
  </tr>
  <tr>
    <td>BX-BLOOM8.0B</td>
    <td>26.0</td>
    <td>14.8</td>
    <td>12.1</td>
    <td>15.2</td>
    <td>10.8</td>
    <td>21.7</td>
  </tr>
  <tr>
    <td>Salamandra2_0B</td>
    <td>59.0</td>
    <td>9.8</td>
    <td>13.6</td>
    <td>18.3</td>
    <td>8.6</td>
    <td>21.6</td>
  </tr>
  <tr>
    <td>Salamandra7_0B</td>
    <td>27.0</td>
    <td>26.4</td>
    <td>15.4</td>
    <td>27.7</td>
    <td>10.5</td>
    <td>35.7</td>
  </tr>
  <tr>
    <td>EuroLLM1_7B</td>
    <td>28.6</td>
    <td>27.5</td>
    <td>4.2</td>
    <td>17.4</td>
    <td>11.9</td>
    <td>23.1</td>
  </tr>
  <tr>
    <td>EuroLLM9.0B</td>
    <td>58.5</td>
    <td>38.7</td>
    <td>18.1</td>
    <td>6.1</td>
    <td>5.9</td>
    <td>51.2</td>
  </tr>
  <tr>
    <td>mT5_0_3B</td>
    <td>26.0</td>
    <td>2.4</td>
    <td>3.7</td>
    <td>6.2</td>
    <td>8.7</td>
    <td>21.9</td>
  </tr>
  <tr>
    <td>mT5_0.582B</td>
    <td>31.6</td>
    <td>2.8</td>
    <td>4.9</td>
    <td>2.9</td>
    <td>7.3</td>
    <td>27.9</td>
  </tr>
  <tr>
    <td>mT5_1.23B</td>
    <td>28.2</td>
    <td>6.7</td>
    <td>4.9</td>
    <td>2.1</td>
    <td>11.0</td>
    <td>22.9</td>
  </tr>
  <tr>
    <td>mT5_1.74B</td>
    <td>27.0</td>
    <td>4.3</td>
    <td>3.7</td>
    <td>2.1</td>
    <td>9.9</td>
    <td>21.9</td>
  </tr>
  <tr>
    <td>mT5_1.80B</td>
    <td>26.0</td>
    <td>14.6</td>
    <td>4.9</td>
    <td>4.1</td>
    <td>14.0</td>
    <td>29.3</td>
  </tr>
  <tr>
    <td>MaLA-50010.0B</td>
    <td>26.0</td>
    <td>21.4</td>
    <td>7.4</td>
    <td>2.3</td>
    <td>3.6</td>
    <td>27.9</td>
  </tr>
  <tr>
    <td>Teuken Instruct Research7.0B</td>
    <td>32.1</td>
    <td>5.7</td>
    <td>15.2</td>
    <td>5.7</td>
    <td>6.2</td>
    <td>45.0</td>
  </tr>
  <tr>
    <td>Salamandra Instruct2.0B</td>
    <td>75.4</td>
    <td>20.0</td>
    <td>10.7</td>
    <td>31.0</td>
    <td>10.3</td>
    <td>24.9</td>
  </tr>
  <tr>
    <td>Salamandra Instruct7.0B</td>
    <td>73.4</td>
    <td>53.2</td>
    <td>15.1</td>
    <td>26.9</td>
    <td>11.7</td>
    <td>69.0</td>
  </tr>
  <tr>
    <td>mTO1_0B</td>
    <td>55.5</td>
    <td>33.6</td>
    <td>4.3</td>
    <td>25.8</td>
    <td>13.8</td>
    <td>24.4</td>
  </tr>
  <tr>
    <td>mTO1.582B</td>
    <td>66.3</td>
    <td>43.1</td>
    <td>5.9</td>
    <td>33.8</td>
    <td>13.4</td>
    <td>28.9</td>
  </tr>
  <tr>
    <td>mTO1.74B</td>
    <td>75.9</td>
    <td>45.8</td>
    <td>13.6</td>
    <td>29.8</td>
    <td>17.6</td>
    <td>33.7</td>
  </tr>
  <tr>
    <td>mTO1.80B</td>
    <td>72.3</td>
    <td>47.5</td>
    <td>9.6</td>
    <td>32.5</td>
    <td>16.8</td>
    <td>65.9</td>
  </tr>
  <tr>
    <td>mTO1.81B</td>
    <td>78.5</td>
    <td>54.2</td>
    <td>21.7</td>
    <td>36.1</td>
    <td>18.3</td>
    <td>81.7</td>
  </tr>
  <tr>
    <td>EuroLLM Instruct1.7B</td>
    <td>28.0</td>
    <td>14.0</td>
    <td>4.2</td>
    <td>26.3</td>
    <td>13.7</td>
    <td>24.1</td>
  </tr>
  <tr>
    <td>EuroLLM Instruct9.0B</td>
    <td>68.1</td>
    <td>41.2</td>
    <td>23.0</td>
    <td>12.6</td>
    <td>7.7</td>
    <td>69.6</td>
  </tr>
  <tr>
    <td>Aya-10112.9B</td>
    <td>78.1</td>
    <td>50.2</td>
    <td>29.5</td>
    <td>31.5</td>
    <td>18.5</td>
    <td>76.6</td>
  </tr>
  <tr>
    <td>Gemma 2_0B</td>
    <td>26.0</td>
    <td>27.0</td>
    <td>11.2</td>
    <td>7.9</td>
    <td>10.2</td>
    <td>32.7</td>
  </tr>
  <tr>
    <td>Gemma 2_9B</td>
    <td>26.0</td>
    <td>59.2</td>
    <td>25.5</td>
    <td>20.7</td>
    <td>13.2</td>
    <td>74.6</td>
  </tr>
  <tr>
    <td>Llama 2_7OB</td>
    <td>43.8</td>
    <td>25.8</td>
    <td>17.9</td>
    <td>21.0</td>
    <td>10.7</td>
    <td>26.9</td>
  </tr>
  <tr>
    <td>Llama 2_13OB</td>
    <td>32.9</td>
    <td>5.1</td>
    <td>10.6</td>
    <td>15.0</td>
    <td>16.4</td>
    <td>31.8</td>
  </tr>
  <tr>
    <td>Llama 3_0B</td>
    <td>26.0</td>
    <td>29.5</td>
    <td>15.7</td>
    <td>26.8</td>
    <td>16.2</td>
    <td>43.7</td>
  </tr>
  <tr>
    <td>Ministral Instruct8.0B</td>
    <td>47.4</td>
    <td>34.4</td>
    <td>6.2</td>
    <td>11.3</td>
    <td>13.4</td>
    <td>40.6</td>
  </tr>
  <tr>
    <td>Gemma 2 Instruct2.0B</td>
    <td>27.0</td>
    <td>39.4</td>
    <td>18.2</td>
    <td>15.8</td>
    <td>14.4</td>
    <td>47.6</td>
  </tr>
  <tr>
    <td>Gemma 2 Instruct0.0B</td>
    <td>72.1</td>
    <td>60.9</td>
    <td>34.2</td>
    <td>22.3</td>
    <td>16.4</td>
    <td>83.9</td>
  </tr>
  <tr>
    <td>Llama 2 Chat7.0B</td>
    <td>31.8</td>
    <td>34.0</td>
    <td>11.9</td>
    <td>19.9</td>
    <td>10.3</td>
    <td>31.6</td>
  </tr>
  <tr>
    <td>Llama 2 Chat13.0B</td>
    <td>28.0</td>
    <td>30.9</td>
    <td>19.6</td>
    <td>13.1</td>
    <td>16.6</td>
    <td>34.0</td>
  </tr>
  <tr>
    <td>Llama 3 Instruct8.0B</td>
    <td>37.3</td>
    <td>43.6</td>
    <td>18.4</td>
    <td>22.0</td>
    <td>13.2</td>
    <td>51.9</td>
  </tr>
</table>

Table 5: Results on discriminative tasks for models prompted with English zero-shot instructions.
<table>
  <tr>
    <th>Model</th>
    <th colspan="2">OPUS-100 BLEU</th>
    <th colspan="2">Flores-200 BLEU</th>
    <th colspan="2">WebNLG ChRF</th>
    <th colspan="2">EUR-Lex-Sum ChRF</th>
    <th colspan="2">News Headlines ChRF</th>
  </tr>
  <tr>
    <th></th>
    <th>BLEU</th>
    <th>ChrF</th>
    <th>BLEU</th>
    <th>ChrF</th>
    <th>ChRF</th>
    <th>Rouge-L</th>
    <th>ChRF</th>
    <th>Rouge-L</th>
    <th>ChRF</th>
    <th>Rouge-L</th>
  </tr>
  <tr><td>PolyLM<sub>1.7B</sub></td><td>0.0</td><td>0.6</td><td>0.1</td><td>7.7</td><td>8.2</td><td>3.7</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>PolyLM<sub>13.0B</sub></td><td>0.0</td><td>0.4</td><td>0.1</td><td>3.7</td><td>11.5</td><td>6.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>XGLM<sub>0.564B</sub></td><td>0.0</td><td>0.1</td><td>0.0</td><td>9.5</td><td>4.3</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>XGLM<sub>1.7B</sub></td><td>0.0</td><td>0.1</td><td>0.0</td><td>10.1</td><td>4.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>XGLM<sub>2.9B</sub></td><td>0.0</td><td>0.1</td><td>0.0</td><td>10.6</td><td>4.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.4</td><td>0.1</td></tr>
  <tr><td>XGLM<sub>4.5B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>10.6</td><td>4.8</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.1</td><td>0.0</td></tr>
  <tr><td>XGLM<sub>7.5B</sub></td><td>0.0</td><td>0.1</td><td>0.0</td><td>10.2</td><td>4.8</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.2</td><td>0.1</td></tr>
  <tr><td>mGPT<sub>1.3B</sub></td><td>0.2</td><td>7.8</td><td>0.3</td><td>11.9</td><td>7.7</td><td>3.0</td><td>0.2</td><td>0.6</td><td>0.4</td><td>0.1</td></tr>
  <tr><td>mGPT<sub>13.0B</sub></td><td>0.3</td><td>8.5</td><td>0.2</td><td>11.0</td><td>9.3</td><td>4.3</td><td>0.1</td><td>0.1</td><td>5.0</td><td>1.9</td></tr>
  <tr><td>BLOOM<sub>0.56B</sub></td><td>0.2</td><td>5.5</td><td>0.3</td><td>3.9</td><td>9.1</td><td>4.6</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>BLOOM<sub>2.0B</sub></td><td>0.1</td><td>1.1</td><td>0.0</td><td>0.1</td><td>10.5</td><td>5.4</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>BLOOM<sub>3.0B</sub></td><td>0.0</td><td>0.4</td><td>0.0</td><td>0.3</td><td>10.4</td><td>5.4</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>BLOOM<sub>8.0B</sub></td><td>0.0</td><td>0.2</td><td>0.0</td><td>0.2</td><td>10.6</td><td>5.3</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Aya-23k<sub>0B</sub></td><td>0.1</td><td>2.6</td><td>0.0</td><td>0.5</td><td>11.2</td><td>4.9</td><td>0.0</td><td>0.0</td><td>11.5</td><td>6.2</td></tr>
  <tr><td>BLOOMZ<sub>0.56B</sub></td><td>0.5</td><td>5.3</td><td>0.2</td><td>6.8</td><td>7.1</td><td>5.0</td><td>0.7</td><td>1.0</td><td>12.9</td><td>6.6</td></tr>
  <tr><td>BLOOMZ<sub>2.0B</sub></td><td>1.2</td><td>10.9</td><td>0.2</td><td>7.7</td><td>4.7</td><td>3.4</td><td>2.0</td><td>2.9</td><td>14.3</td><td>8.3</td></tr>
  <tr><td>BLOOMZ<sub>3.0B</sub></td><td>0.7</td><td>7.8</td><td>0.5</td><td>10.3</td><td>11.5</td><td>10.2</td><td>0.4</td><td>0.5</td><td>17.5</td><td>11.4</td></tr>
  <tr><td>BLOOMZ<sub>8.0B</sub></td><td>1.4</td><td>11.0</td><td>1.0</td><td>13.7</td><td>5.3</td><td>4.6</td><td>0.8</td><td>1.1</td><td>16.8</td><td>10.7</td></tr>
  <tr><td>BX-LLaMA<sub>7.0B</sub></td><td>0.0</td><td>2.0</td><td>0.2</td><td>2.4</td><td>2.6</td><td>0.8</td><td>1.2</td><td>0.0</td><td>1.3</td><td>0.1</td></tr>
  <tr><td>BX-LLaMA<sub>13.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.6</td><td>10.9</td><td>5.5</td><td>0.1</td><td>0.1</td><td>0.8</td><td>0.1</td></tr>
  <tr><td>BX-BLOOM<sub>8.0B</sub></td><td>0.0</td><td>0.3</td><td>0.0</td><td>0.2</td><td>10.3</td><td>5.1</td><td>0.0</td><td>0.0</td><td>5.9</td><td>2.1</td></tr>
  <tr><td>Salamandra<sub>2.0B</sub></td><td>1.4</td><td>17.3</td><td>3.7</td><td>25.7</td><td>2.9</td><td>2.5</td><td>0.0</td><td>0.0</td><td>10.9</td><td>5.8</td></tr>
  <tr><td>Salamandra<sub>7.0B</sub></td><td>2.0</td><td>17.1</td><td>7.9</td><td>33.0</td><td>2.4</td><td>1.6</td><td>0.0</td><td>0.0</td><td>0.3</td><td>0.2</td></tr>
  <tr><td>EuroLLM<sub>1.7B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>10.9</td><td>5.5</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>EuroLLM<sub>9.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>10.8</td><td>4.5</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>mT5<sub>0.5B</sub></td><td>0.1</td><td>1.5</td><td>0.0</td><td>1.5</td><td>1.4</td><td>0.6</td><td>0.2</td><td>0.5</td><td>6.3</td><td>4.9</td></tr>
  <tr><td>mT5<sub>0.82B</sub></td><td>0.2</td><td>3.9</td><td>0.4</td><td>4.1</td><td>4.8</td><td>2.9</td><td>0.3</td><td>0.9</td><td>8.7</td><td>7.4</td></tr>
  <tr><td>mT3<sub>1.2B</sub></td><td>0.3</td><td>9.4</td><td>0.1</td><td>9.9</td><td>11.1</td><td>4.3</td><td>3.2</td><td>4.4</td><td>8.8</td><td>3.4</td></tr>
  <tr><td>mT3<sub>1.4B</sub></td><td>0.2</td><td>8.9</td><td>0.0</td><td>9.0</td><td>6.3</td><td>2.4</td><td>4.3</td><td>4.4</td><td>7.4</td><td>2.6</td></tr>
  <tr><td>mT3<sub>1.0B</sub></td><td>0.2</td><td>8.0</td><td>0.1</td><td>9.4</td><td>8.4</td><td>3.0</td><td>4.3</td><td>4.3</td><td>6.8</td><td>2.3</td></tr>
  <tr><td>MaLA-500<sub>10.0B</sub></td><td>0.0</td><td>6.7</td><td>0.0</td><td>7.3</td><td>9.7</td><td>3.8</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Teuken Instruct Research<sub>7.0B</sub></td><td>0.9</td><td>6.4</td><td>12.0</td><td>46.4</td><td>9.7</td><td>5.1</td><td>2.5</td><td>2.5</td><td>31.2</td><td>23.0</td></tr>
  <tr><td>Salamandra Instruct<sub>2.0B</sub></td><td>2.7</td><td>23.4</td><td>3.5</td><td>34.1</td><td>2.0</td><td>1.8</td><td>0.0</td><td>0.0</td><td>1.1</td><td>0.6</td></tr>
  <tr><td>Salamandra Instruct<sub>7.0B</sub></td><td>9.1</td><td>39.3</td><td>5.5</td><td>41.6</td><td>2.1</td><td>2.6</td><td>0.2</td><td>0.1</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>mTO<sub>0.3B</sub></td><td>4.0</td><td>17.5</td><td>2.2</td><td>20.0</td><td>14.3</td><td>15.3</td><td>3.9</td><td>1.3</td><td>3.3</td><td>2.9</td></tr>
  <tr><td>mTO<sub>0.82B</sub></td><td>0.7</td><td>4.8</td><td>0.2</td><td>5.3</td><td>21.1</td><td>21.3</td><td>2.6</td><td>0.4</td><td>4.2</td><td>4.9</td></tr>
  <tr><td>mTO<sub>1.2B</sub></td><td>0.6</td><td>6.3</td><td>0.2</td><td>6.6</td><td>24.8</td><td>24.5</td><td>5.4</td><td>2.4</td><td>6.3</td><td>6.9</td></tr>
  <tr><td>mTO<sub>1.4B</sub></td><td>1.7</td><td>10.0</td><td>0.5</td><td>8.6</td><td>30.3</td><td>29.6</td><td>4.7</td><td>1.1</td><td>9.7</td><td>13.2</td></tr>
  <tr><td>mTO<sub>1.0B</sub></td><td>7.8</td><td>28.3</td><td>3.3</td><td>25.0</td><td>25.7</td><td>26.2</td><td>3.8</td><td>3.2</td><td>27.4</td><td>26.8</td></tr>
  <tr><td>EuroLLM Instruct<sub>1.7B</sub></td><td>9.2</td><td>37.0</td><td>15.7</td><td>51.1</td><td>11.7</td><td>5.7</td><td>0.0</td><td>0.0</td><td>16.4</td><td>9.4</td></tr>
  <tr><td>EuroLLM Instruct<sub>9.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>11.2</td><td>4.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Aya-101<sub>12.9B</sub></td><td>26.4</td><td>56.6</td><td>19.5</td><td>52.3</td><td>36.0</td><td>32.3</td><td>8.2</td><td>7.7</td><td>30.6</td><td>27.6</td></tr>
  <tr><td>Gemma<sub>2.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>10.0</td><td>4.3</td><td>0.2</td><td>0.2</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Gemma<sub>2.9B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>9.8</td><td>3.9</td><td>0.2</td><td>0.1</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Llama 2<sub>7.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>11.0</td><td>4.5</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Llama 2<sub>13.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>11.3</td><td>4.7</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Llama 3<sub>8.0B</sub></td><td>3.3</td><td>12.5</td><td>5.3</td><td>32.8</td><td>11.3</td><td>6.0</td><td>0.0</td><td>0.0</td><td>0.8</td><td>0.3</td></tr>
  <tr><td>Ministral Instruct<sub>0.8B</sub></td><td>1.2</td><td>6.7</td><td>1.2</td><td>16.9</td><td>10.5</td><td>5.0</td><td>0.2</td><td>0.2</td><td>0.1</td><td>0.0</td></tr>
  <tr><td>Gemma 2 Instruct<sub>2.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>11.6</td><td>6.5</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Gemma 2 Instruct<sub>9.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>8.0</td><td>6.3</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Llama 2 Chat<sub>7.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>12.3</td><td>5.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Llama 2 Chat<sub>13.0B</sub></td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>10.0</td><td>6.8</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Llama 3 Instructs<sub>0.8B</sub></td><td>0.0</td><td>2.4</td><td>8.8</td><td>39.9</td><td>12.6</td><td>6.5</td><td>1.0</td><td>1.1</td><td>31.7</td><td>24.1</td></tr>
</table>

Table 6: Results on generative tasks for models prompted with English zero-shot instructions.
<table>
  <tr>
    <th>Model</th>
    <th>Sentiment Analysis F1</th>
    <th>SIB-200 F1</th>
    <th>Taxi1500 F1</th>
    <th>News Categories F1</th>
    <th>MultiEURLEX F1</th>
  </tr>
  <tr>
    <td>PolyLM1.7B</td>
    <td>52.1</td>
    <td>19.3</td>
    <td>6.3</td>
    <td>5.1</td>
    <td>10.4</td>
  </tr>
  <tr>
    <td>PolyLM13.0B</td>
    <td>26.0</td>
    <td>21.6</td>
    <td>6.3</td>
    <td>12.3</td>
    <td>12.3</td>
  </tr>
  <tr>
    <td>XGLM1.564B</td>
    <td>48.0</td>
    <td>2.2</td>
    <td>6.3</td>
    <td>10.9</td>
    <td>14.5</td>
  </tr>
  <tr>
    <td>XGLM1.7B</td>
    <td>50.1</td>
    <td>15.8</td>
    <td>6.3</td>
    <td>10.4</td>
    <td>12.5</td>
  </tr>
  <tr>
    <td>XGLM2.9B</td>
    <td>46.5</td>
    <td>25.8</td>
    <td>6.3</td>
    <td>13.5</td>
    <td>13.3</td>
  </tr>
  <tr>
    <td>XGLM4.5B</td>
    <td>56.4</td>
    <td>14.2</td>
    <td>6.4</td>
    <td>10.6</td>
    <td>15.4</td>
  </tr>
  <tr>
    <td>XGLM7.5B</td>
    <td>32.8</td>
    <td>26.1</td>
    <td>6.3</td>
    <td>9.2</td>
    <td>14.7</td>
  </tr>
  <tr>
    <td>mGPT1.3B</td>
    <td>39.4</td>
    <td>2.2</td>
    <td>6.3</td>
    <td>3.7</td>
    <td>7.3</td>
  </tr>
  <tr>
    <td>mGPT13.0B</td>
    <td>41.8</td>
    <td>47.7</td>
    <td>6.3</td>
    <td>5.1</td>
    <td>7.2</td>
  </tr>
  <tr>
    <td>BLOOM0.56B</td>
    <td>50.0</td>
    <td>4.4</td>
    <td>6.3</td>
    <td>4.2</td>
    <td>15.9</td>
  </tr>
  <tr>
    <td>BLOOM2.0B</td>
    <td>41.1</td>
    <td>2.2</td>
    <td>6.3</td>
    <td>5.5</td>
    <td>14.2</td>
  </tr>
  <tr>
    <td>BLOOM3.0B</td>
    <td>41.1</td>
    <td>14.1</td>
    <td>6.3</td>
    <td>7.2</td>
    <td>14.0</td>
  </tr>
  <tr>
    <td>BLOOM8.0B</td>
    <td>54.5</td>
    <td>41.6</td>
    <td>6.3</td>
    <td>7.7</td>
    <td>13.3</td>
  </tr>
  <tr>
    <td>Aya-23.0B</td>
    <td>68.3</td>
    <td>34.6</td>
    <td>6.3</td>
    <td>22.8</td>
    <td>14.5</td>
  </tr>
  <tr>
    <td>BLOOMZ0.56B</td>
    <td>51.1</td>
    <td>34.1</td>
    <td>6.3</td>
    <td>7.5</td>
    <td>14.5</td>
  </tr>
  <tr>
    <td>BLOOMZ2.0B</td>
    <td>60.6</td>
    <td>23.4</td>
    <td>6.3</td>
    <td>17.8</td>
    <td>14.7</td>
  </tr>
  <tr>
    <td>BLOOMZ3.0B</td>
    <td>49.3</td>
    <td>33.0</td>
    <td>6.3</td>
    <td>9.6</td>
    <td>15.1</td>
  </tr>
  <tr>
    <td>BLOOMZ8.0B</td>
    <td>57.5</td>
    <td>45.1</td>
    <td>6.3</td>
    <td>12.3</td>
    <td>13.3</td>
  </tr>
  <tr>
    <td>BX-LLaMA7.0B</td>
    <td>26.0</td>
    <td>7.0</td>
    <td>9.0</td>
    <td>6.0</td>
    <td>6.5</td>
  </tr>
  <tr>
    <td>BX-LLaMA13.0B</td>
    <td>26.0</td>
    <td>2.2</td>
    <td>6.3</td>
    <td>3.6</td>
    <td>10.1</td>
  </tr>
  <tr>
    <td>BX-BLOOM8.0B</td>
    <td>53.2</td>
    <td>23.8</td>
    <td>6.3</td>
    <td>4.4</td>
    <td>7.5</td>
  </tr>
  <tr>
    <td>Salamandra3.0B</td>
    <td>26.0</td>
    <td>41.1</td>
    <td>6.3</td>
    <td>14.6</td>
    <td>9.2</td>
  </tr>
  <tr>
    <td>Salamandra7.0B</td>
    <td>79.3</td>
    <td>48.1</td>
    <td>8.3</td>
    <td>25.9</td>
    <td>10.8</td>
  </tr>
  <tr>
    <td>EuroLLM1.7B</td>
    <td>52.2</td>
    <td>20.9</td>
    <td>6.3</td>
    <td>18.5</td>
    <td>11.6</td>
  </tr>
  <tr>
    <td>EuroLLM9.0B</td>
    <td>77.4</td>
    <td>65.1</td>
    <td>11.7</td>
    <td>8.8</td>
    <td>10.1</td>
  </tr>
  <tr>
    <td>mT50.3B</td>
    <td>26.0</td>
    <td>2.4</td>
    <td>3.7</td>
    <td>5.8</td>
    <td>8.3</td>
  </tr>
  <tr>
    <td>mT50.582B</td>
    <td>35.5</td>
    <td>2.8</td>
    <td>10.3</td>
    <td>4.1</td>
    <td>7.2</td>
  </tr>
  <tr>
    <td>mT51.23B</td>
    <td>49.6</td>
    <td>2.2</td>
    <td>6.3</td>
    <td>2.1</td>
    <td>11.6</td>
  </tr>
  <tr>
    <td>mT51.74B</td>
    <td>39.4</td>
    <td>2.2</td>
    <td>6.3</td>
    <td>1.2</td>
    <td>9.9</td>
  </tr>
  <tr>
    <td>mT51.30B</td>
    <td>48.0</td>
    <td>2.2</td>
    <td>6.4</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MaLA-50010.0B</td>
    <td>26.0</td>
    <td>39.3</td>
    <td>6.3</td>
    <td>3.2</td>
    <td>4.1</td>
  </tr>
  <tr>
    <td>Teuken Instruct Research7.0B</td>
    <td>76.4</td>
    <td>56.9</td>
    <td>22.8</td>
    <td>17.5</td>
    <td>6.8</td>
  </tr>
  <tr>
    <td>Salamandra Instruct2.0B</td>
    <td>76.1</td>
    <td>52.4</td>
    <td>6.3</td>
    <td>21.3</td>
    <td>8.7</td>
  </tr>
  <tr>
    <td>Salamandra Instruct7.0B</td>
    <td>65.1</td>
    <td>64.2</td>
    <td>17.4</td>
    <td>24.5</td>
    <td>12.0</td>
  </tr>
  <tr>
    <td>mTO0.3B</td>
    <td>54.4</td>
    <td>15.2</td>
    <td>6.3</td>
    <td>18.9</td>
    <td>13.7</td>
  </tr>
  <tr>
    <td>mTO0.582B</td>
    <td>35.1</td>
    <td>19.5</td>
    <td>6.3</td>
    <td>24.3</td>
    <td>10.8</td>
  </tr>
  <tr>
    <td>mTO1.23B</td>
    <td>77.0</td>
    <td>21.1</td>
    <td>6.3</td>
    <td>18.7</td>
    <td>15.9</td>
  </tr>
  <tr>
    <td>mTO1.74B</td>
    <td>28.0</td>
    <td>30.2</td>
    <td>6.3</td>
    <td>15.4</td>
    <td>14.5</td>
  </tr>
  <tr>
    <td>mTO1.30B</td>
    <td>61.1</td>
    <td>46.4</td>
    <td>6.3</td>
    <td>20.3</td>
    <td>15.7</td>
  </tr>
  <tr>
    <td>EuroLLM Instruct1.7B</td>
    <td>69.0</td>
    <td>36.5</td>
    <td>6.3</td>
    <td>24.5</td>
    <td>10.2</td>
  </tr>
  <tr>
    <td>Aya-10112.9B</td>
    <td>86.5</td>
    <td>52.3</td>
    <td>19.7</td>
    <td>23.5</td>
    <td>15.2</td>
  </tr>
  <tr>
    <td>Gemma 2z.0B</td>
    <td>30.5</td>
    <td>36.1</td>
    <td>6.3</td>
    <td>21.6</td>
    <td>12.1</td>
  </tr>
  <tr>
    <td>Llama 2z.0B</td>
    <td>36.3</td>
    <td>31.3</td>
    <td>6.3</td>
    <td>24.1</td>
    <td>14.8</td>
  </tr>
  <tr>
    <td>Llama 2z13.0B</td>
    <td>60.2</td>
    <td>47.4</td>
    <td>10.0</td>
    <td>9.7</td>
    <td>13.6</td>
  </tr>
  <tr>
    <td>Llama 3z.0B</td>
    <td>80.6</td>
    <td>42.1</td>
    <td>6.3</td>
    <td>23.0</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td>Ministral Instructs.0B</td>
    <td>70.9</td>
    <td>43.7</td>
    <td>6.3</td>
    <td>9.0</td>
    <td>16.4</td>
  </tr>
  <tr>
    <td>Gemma 2 Instructz.0B</td>
    <td>70.2</td>
    <td>58.3</td>
    <td>10.1</td>
    <td>7.6</td>
    <td>12.9</td>
  </tr>
  <tr>
    <td>Gemma 2 Instruct9.0B</td>
    <td>85.0</td>
    <td>74.3</td>
    <td>31.9</td>
    <td>22.6</td>
    <td>8.1</td>
  </tr>
  <tr>
    <td>Llama 2 Chatz.0B</td>
    <td>66.2</td>
    <td>44.3</td>
    <td>8.7</td>
    <td>24.5</td>
    <td>9.5</td>
  </tr>
  <tr>
    <td>Llama 2 Chat13.0B</td>
    <td>72.2</td>
    <td>62.6</td>
    <td>16.9</td>
    <td>16.5</td>
    <td>16.0</td>
  </tr>
  <tr>
    <td>Llama 3 Instructs.0B</td>
    <td>79.9</td>
    <td>56.7</td>
    <td>8.3</td>
    <td>18.8</td>
    <td>9.7</td>
  </tr>
</table>

Table 7: Results on discriminative tasks for models prompted with English one-shot instructions.
<table>
  <tr>
    <th>Model</th>
    <th colspan="2">OPUS-100</th>
    <th colspan="2">Flores-200</th>
    <th colspan="2">WebNLG</th>
    <th colspan="2">EUR-Lex-Sum</th>
    <th colspan="2">News Headlines</th>
  </tr>
  <tr>
    <th></th>
    <th>BLEU</th>
    <th>ChrF</th>
    <th>BLEU</th>
    <th>ChrF</th>
    <th>ChrF</th>
    <th>Rouge-L</th>
    <th>ChrF</th>
    <th>Rouge-L</th>
    <th>ChrF</th>
    <th>Rouge-L</th>
  </tr>
  <tr><td>PolyLM<sub>1.7B</sub></td><td>0.4</td><td>11.6</td><td>0.3</td><td>16.5</td><td>10.7</td><td>8.9</td><td>0.0</td><td>0.0</td><td>11.9</td><td>4.2</td></tr>
  <tr><td>PolyLM<sub>13.0B</sub></td><td>6.5</td><td>19.1</td><td>1.8</td><td>19.8</td><td>10.4</td><td>6.2</td><td>0.0</td><td>0.0</td><td>19.9</td><td>10.7</td></tr>
  <tr><td>XGLM<sub>0.564B</sub></td><td>2.8</td><td>19.0</td><td>0.6</td><td>17.3</td><td>11.7</td><td>6.4</td><td>0.0</td><td>0.0</td><td>20.5</td><td>9.3</td></tr>
  <tr><td>XGLM<sub>1.7B</sub></td><td>3.5</td><td>15.3</td><td>1.0</td><td>17.6</td><td>13.4</td><td>10.2</td><td>0.0</td><td>0.0</td><td>21.5</td><td>9.5</td></tr>
  <tr><td>XGLM<sub>2.9B</sub></td><td>3.4</td><td>13.5</td><td>1.1</td><td>13.8</td><td>12.8</td><td>8.6</td><td>0.0</td><td>0.0</td><td>21.6</td><td>9.8</td></tr>
  <tr><td>XGLM<sub>4.5B</sub></td><td>3.8</td><td>15.7</td><td>1.1</td><td>15.6</td><td>13.9</td><td>9.9</td><td>0.0</td><td>0.0</td><td>21.5</td><td>9.2</td></tr>
  <tr><td>XGLM<sub>7.5B</sub></td><td>3.2</td><td>12.7</td><td>1.1</td><td>16.4</td><td>12.9</td><td>10.7</td><td>0.0</td><td>0.0</td><td>20.8</td><td>9.1</td></tr>
  <tr><td>mGPT<sub>1.3B</sub></td><td>1.6</td><td>11.6</td><td>1.0</td><td>16.9</td><td>10.0</td><td>6.8</td><td>0.2</td><td>0.6</td><td>1.4</td><td>0.7</td></tr>
  <tr><td>mGPT<sub>13.0B</sub></td><td>1.0</td><td>9.2</td><td>0.4</td><td>11.6</td><td>9.8</td><td>6.1</td><td>0.1</td><td>0.1</td><td>1.0</td><td>0.2</td></tr>
  <tr><td>BLOOM<sub>0.56B</sub></td><td>1.0</td><td>9.8</td><td>0.4</td><td>13.2</td><td>11.2</td><td>8.3</td><td>0.0</td><td>0.0</td><td>11.0</td><td>3.8</td></tr>
  <tr><td>BLOOM<sub>2.0B</sub></td><td>0.4</td><td>9.9</td><td>0.2</td><td>14.5</td><td>11.6</td><td>7.9</td><td>0.0</td><td>0.0</td><td>17.8</td><td>7.4</td></tr>
  <tr><td>BLOOM<sub>3.0B</sub></td><td>1.8</td><td>10.7</td><td>0.6</td><td>14.9</td><td>12.5</td><td>10.6</td><td>0.0</td><td>0.0</td><td>18.1</td><td>7.7</td></tr>
  <tr><td>BLOOM<sub>8.0B</sub></td><td>1.6</td><td>11.4</td><td>0.9</td><td>15.1</td><td>12.7</td><td>11.2</td><td>0.0</td><td>0.0</td><td>21.0</td><td>10.4</td></tr>
  <tr><td>Aya-23<sub>8.0B</sub></td><td>2.3</td><td>13.1</td><td>0.5</td><td>11.6</td><td>12.3</td><td>8.2</td><td>0.0</td><td>0.0</td><td>28.2</td><td>19.8</td></tr>
  <tr><td>BLOOMZ<sub>0.56B</sub></td><td>0.2</td><td>4.6</td><td>0.0</td><td>5.0</td><td>10.7</td><td>9.7</td><td>0.7</td><td>1.0</td><td>11.6</td><td>4.1</td></tr>
  <tr><td>BLOOMZ<sub>2.0B</sub></td><td>1.0</td><td>8.8</td><td>0.2</td><td>9.1</td><td>12.6</td><td>12.0</td><td>2.0</td><td>2.9</td><td>13.1</td><td>4.8</td></tr>
  <tr><td>BLOOMZ<sub>3.0B</sub></td><td>0.3</td><td>5.9</td><td>0.2</td><td>7.7</td><td>17.9</td><td>17.5</td><td>0.4</td><td>0.5</td><td>13.2</td><td>4.9</td></tr>
  <tr><td>BLOOMZ<sub>8.0B</sub></td><td>0.5</td><td>7.6</td><td>0.3</td><td>9.3</td><td>12.6</td><td>12.8</td><td>0.8</td><td>1.1</td><td>12.4</td><td>4.5</td></tr>
  <tr><td>BX-LLaMA<sub>7.0B</sub></td><td>0.0</td><td>1.6</td><td>0.0</td><td>1.4</td><td>1.9</td><td>0.4</td><td>1.2</td><td>0.0</td><td>0.8</td><td>0.0</td></tr>
  <tr><td>BX-LLaMA<sub>13.0B</sub></td><td>1.0</td><td>12.0</td><td>0.7</td><td>16.0</td><td>11.7</td><td>7.8</td><td>0.1</td><td>0.1</td><td>8.2</td><td>3.2</td></tr>
  <tr><td>BX-BLOOM<sub>4.0B</sub></td><td>0.6</td><td>10.3</td><td>0.6</td><td>16.5</td><td>11.0</td><td>9.3</td><td>0.0</td><td>0.0</td><td>5.2</td><td>3.8</td></tr>
  <tr><td>Salamandra<sub>2.0B</sub></td><td>3.7</td><td>22.3</td><td>7.1</td><td>34.3</td><td>8.2</td><td>6.5</td><td>0.0</td><td>0.0</td><td>20.2</td><td>12.8</td></tr>
  <tr><td>Salamandra<sub>7.0B</sub></td><td>6.9</td><td>29.0</td><td>12.6</td><td>42.7</td><td>14.9</td><td>13.7</td><td>0.0</td><td>0.0</td><td>22.8</td><td>17.9</td></tr>
  <tr><td>EuroLLM<sub>1.7B</sub></td><td>16.4</td><td>37.6</td><td>20.9</td><td>53.0</td><td>12.4</td><td>9.9</td><td>0.0</td><td>0.0</td><td>19.4</td><td>10.5</td></tr>
  <tr><td>EuroLLM<sub>9.0B</sub></td><td>23.6</td><td>47.1</td><td>34.0</td><td>63.7</td><td>14.4</td><td>14.1</td><td>0.0</td><td>0.0</td><td>4.4</td><td>3.0</td></tr>
  <tr><td>mT5<sub>0.3B</sub></td><td>0.1</td><td>1.8</td><td>0.0</td><td>2.0</td><td>1.5</td><td>0.6</td><td>0.2</td><td>0.5</td><td>5.7</td><td>4.1</td></tr>
  <tr><td>mT5<sub>0.82B</sub></td><td>0.4</td><td>4.7</td><td>0.1</td><td>5.9</td><td>6.1</td><td>7.0</td><td>0.3</td><td>0.9</td><td>5.9</td><td>5.4</td></tr>
  <tr><td>mT5<sub>1.2B</sub></td><td>0.2</td><td>9.0</td><td>0.1</td><td>12.4</td><td>10.6</td><td>3.4</td><td>3.2</td><td>4.4</td><td>6.2</td><td>1.7</td></tr>
  <tr><td>mT5<sub>1.7B</sub></td><td>0.3</td><td>11.0</td><td>0.2</td><td>15.1</td><td>8.9</td><td>3.0</td><td>4.3</td><td>4.4</td><td>5.7</td><td>1.6</td></tr>
  <tr><td>mT5<sub>1.8B</sub></td><td>0.2</td><td>8.2</td><td>0.1</td><td>11.0</td><td>9.0</td><td>3.1</td><td>4.3</td><td>4.4</td><td>8.7</td><td>3.4</td></tr>
  <tr><td>MaLA-500<sub>10.0B</sub></td><td>0.0</td><td>11.2</td><td>0.1</td><td>13.7</td><td>9.8</td><td>5.5</td><td>0.0</td><td>0.0</td><td>13.2</td><td>4.8</td></tr>
  <tr><td>Teuken Instruct Research<sub>7.0B</sub></td><td>18.8</td><td>45.8</td><td>18.8</td><td>54.7</td><td>13.6</td><td>14.1</td><td>2.5</td><td>2.4</td><td>31.0</td><td>24.1</td></tr>
  <tr><td>Salamandra Instruct<sub>2.0B</sub></td><td>4.2</td><td>28.1</td><td>3.4</td><td>35.2</td><td>3.3</td><td>1.9</td><td>0.0</td><td>0.0</td><td>18.9</td><td>12.7</td></tr>
  <tr><td>Salamandra Instruct<sub>7.0B</sub></td><td>8.5</td><td>43.4</td><td>6.2</td><td>44.3</td><td>13.2</td><td>10.3</td><td>0.2</td><td>0.1</td><td>25.7</td><td>18.9</td></tr>
  <tr><td>mTO<sub>0.3B</sub></td><td>2.5</td><td>16.8</td><td>0.6</td><td>14.5</td><td>5.0</td><td>3.9</td><td>3.9</td><td>1.3</td><td>0.6</td><td>0.1</td></tr>
  <tr><td>mTO<sub>0.82B</sub></td><td>0.4</td><td>5.7</td><td>0.0</td><td>3.9</td><td>13.3</td><td>9.2</td><td>2.6</td><td>0.4</td><td>8.4</td><td>2.3</td></tr>
  <tr><td>mTO<sub>1.2B</sub></td><td>1.1</td><td>10.5</td><td>0.2</td><td>10.2</td><td>13.8</td><td>10.8</td><td>5.4</td><td>2.4</td><td>1.8</td><td>1.8</td></tr>
  <tr><td>mTO<sub>1.7B</sub></td><td>1.4</td><td>10.3</td><td>0.1</td><td>7.1</td><td>18.1</td><td>17.1</td><td>4.7</td><td>1.1</td><td>12.3</td><td>4.5</td></tr>
  <tr><td>mTO<sub>1.8B</sub></td><td>9.0</td><td>31.3</td><td>4.8</td><td>29.6</td><td>10.8</td><td>10.8</td><td>3.8</td><td>3.2</td><td>11.6</td><td>6.5</td></tr>
  <tr><td>EuroLLM Instruct<sub>1.7B</sub></td><td>13.1</td><td>38.3</td><td>21.0</td><td>54.0</td><td>13.3</td><td>10.5</td><td>0.0</td><td>0.0</td><td>22.8</td><td>14.3</td></tr>
  <tr><td>Aya-101<sub>12.0B</sub></td><td>26.8</td><td>57.8</td><td>21.2</td><td>54.3</td><td>41.2</td><td>36.4</td><td>8.2</td><td>7.7</td><td>30.0</td><td>28.0</td></tr>
  <tr><td>Gemma 2<sub>0.8B</sub></td><td>3.3</td><td>14.3</td><td>2.4</td><td>23.7</td><td>13.0</td><td>13.1</td><td>0.2</td><td>0.2</td><td>19.2</td><td>12.6</td></tr>
  <tr><td>Llama 2<sub>7.0B</sub></td><td>4.6</td><td>17.0</td><td>1.5</td><td>19.3</td><td>12.6</td><td>10.3</td><td>0.0</td><td>0.0</td><td>23.0</td><td>16.0</td></tr>
  <tr><td>Llama 2<sub>13.0B</sub></td><td>6.3</td><td>23.1</td><td>2.0</td><td>23.5</td><td>16.7</td><td>15.9</td><td>0.0</td><td>0.0</td><td>26.7</td><td>19.4</td></tr>
  <tr><td>Llama 3<sub>8.0B</sub></td><td>14.9</td><td>39.7</td><td>10.1</td><td>41.8</td><td>17.4</td><td>17.8</td><td>0.0</td><td>0.0</td><td>33.0</td><td>25.2</td></tr>
  <tr><td>Minstral Instruct<sub>8.0B</sub></td><td>3.9</td><td>17.2</td><td>1.4</td><td>18.8</td><td>14.4</td><td>11.9</td><td>0.8</td><td>1.5</td><td>30.9</td><td>21.3</td></tr>
  <tr><td>Gemma 2 Instruct<sub>2.0B</sub></td><td>3.1</td><td>13.6</td><td>1.7</td><td>19.5</td><td>12.2</td><td>8.9</td><td>0.0</td><td>0.0</td><td>9.8</td><td>6.5</td></tr>
  <tr><td>Gemma 2 Instruct<sub>8.0B</sub></td><td>14.4</td><td>36.7</td><td>18.4</td><td>52.4</td><td>22.8</td><td>23.3</td><td>0.0</td><td>0.0</td><td>32.3</td><td>27.0</td></tr>
  <tr><td>Llama 2 Chat<sub>7.0B</sub></td><td>3.8</td><td>16.3</td><td>2.5</td><td>20.0</td><td>14.0</td><td>13.4</td><td>0.0</td><td>0.0</td><td>25.7</td><td>18.4</td></tr>
  <tr><td>Llama 2 Chat<sub>13.0B</sub></td><td>7.9</td><td>25.8</td><td>4.0</td><td>27.6</td><td>15.2</td><td>15.5</td><td>0.0</td><td>0.0</td><td>29.4</td><td>20.8</td></tr>
  <tr><td>Llama 3 Instruct<sub>8.0B</sub></td><td>17.6</td><td>44.7</td><td>10.9</td><td>43.7</td><td>30.3</td><td>26.4</td><td>1.0</td><td>1.1</td><td>33.6</td><td>26.6</td></tr>
</table>

Table 8: Results on generative tasks for models prompted with English one-shot instructions.

<table>
  <tr>
    <th>Model</th>
    <th>Sentiment Macro-F1</th>
    <th>SIB-200 Macro-F1</th>
    <th>Taxi1500 Macro-F1</th>
    <th>News Categories Macro-F1</th>
    <th>MultiEURLEX Macro-F1</th>
  </tr>
  <tr>
    <td>BERTu</td>
    <td>83.0</td>
    <td>84.9</td>
    <td>77.5</td>
    <td>58.3</td>
    <td>67.1</td>
  </tr>
  <tr>
    <td>mBERT</td>
    <td>64.7</td>
    <td>75.3</td>
    <td>47.7</td>
    <td>53.1</td>
    <td>60.7</td>
  </tr>
  <tr>
    <td>XLM-R</td>
    <td>59.6</td>
    <td>68.5</td>
    <td>36.5</td>
    <td>50.6</td>
    <td>60.5</td>
  </tr>
  <tr>
    <td>Glot500</td>
    <td>74.6</td>
    <td>82.3</td>
    <td>64.0</td>
    <td>57.2</td>
    <td>62.2</td>
  </tr>
  <tr>
    <td>mT5</td>
    <td>100.0</td>
    <td>76.8</td>
    <td>42.2</td>
    <td>52.5</td>
    <td>31.2</td>
  </tr>
</table>

Table 9: Fine-tuned model results on discriminative tasks.
<table>
  <tr>
    <th>Model</th>
    <th colspan="2">OPUS-100<br>BLEU<br>ChrF</th>
    <th colspan="2">WebNLG<br>ChrF<br>Rouge-L</th>
    <th colspan="2">EUR-Lex-Sum<br>ChrF<br>Rouge-L</th>
    <th colspan="2">News Headlines<br>ChrF<br>Rouge-L</th>
  </tr>
  <tr>
    <td></td>
    <td>BLEU</td>
    <td>ChrF</td>
    <td>ChrF</td>
    <td>Rouge-L</td>
    <td>ChrF</td>
    <td>Rouge-L</td>
    <td>ChrF</td>
    <td>Rouge-L</td>
  </tr>
  <tr>
    <td>mT5 (fine-tuned)</td>
    <td>51.8</td>
    <td>75.9</td>
    <td>31.6</td>
    <td>28.0</td>
    <td>51.5</td>
    <td>42.5</td>
    <td>32.2</td>
    <td>28.1</td>
  </tr>
</table>

Table 10: Fine-tuned model results on generative tasks.

<table>
  <tr>
    <th>Prompt</th>
    <th>Shots</th>
    <th colspan="2">OPUS-100<br>BLEU<br>ChrF</th>
    <th colspan="2">Flores-200<br>BLEU<br>ChrF</th>
    <th colspan="2">WebNLG<br>ChrF<br>Rouge-L</th>
    <th colspan="2">News Headlines<br>ChrF<br>Rouge-L</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>BLEU</td>
    <td>ChrF</td>
    <td>BLEU</td>
    <td>ChrF</td>
    <td>ChrF</td>
    <td>Rouge-L</td>
    <td>ChrF</td>
    <td>Rouge-L</td>
  </tr>
  <tr>
    <td>English</td>
    <td>Zero</td>
    <td>38.1</td>
    <td>69.7</td>
    <td>44.1</td>
    <td>74.4</td>
    <td>61.8</td>
    <td>58.1</td>
    <td>32.1</td>
    <td>26.6</td>
  </tr>
  <tr>
    <td>Maltese</td>
    <td>Zero</td>
    <td>34.2</td>
    <td>64.0</td>
    <td>43.5</td>
    <td>72.4</td>
    <td>56.0</td>
    <td>53.6</td>
    <td>32.2</td>
    <td>27.2</td>
  </tr>
  <tr>
    <td>English</td>
    <td>One</td>
    <td>36.8</td>
    <td>67.3</td>
    <td>46.2</td>
    <td>74.5</td>
    <td>61.9</td>
    <td>57.4</td>
    <td>33.8</td>
    <td>25.9</td>
  </tr>
  <tr>
    <td>Maltese</td>
    <td>One</td>
    <td>35.8</td>
    <td>65.6</td>
    <td>46.2</td>
    <td>74.5</td>
    <td>61.8</td>
    <td>58.3</td>
    <td>31.9</td>
    <td>24.7</td>
  </tr>
</table>

Table 11: ChatGPT Results.

(a) Including PT/PT, IT/PT, and IT/IT models

(b) Excluding PT/PT, IT/PT, and IT/IT models

Figure 11: Zero-shot aggregated performance against model size.
(a) Including PT/PT, IT/PT, and IT/IT models
(b) Excluding PT/PT, IT/PT, and IT/IT models

Figure 12: Zero-shot aggregated performance against model multilinguality.

Reviewer paper summaries: This work thoroughly evaluates the performance of 47 publicly available large language models (LLMs) in Maltese, a low-resource language. To do this, the authors introduce a new benchmark that covers 11 discriminative and generative natural language processing (NLP) tasks. The experiments reveal that many LLMs perform poorly, particularly in generative tasks, and that smaller fine-tuned models often outperform LLMs across all tasks. The study concludes that prior exposure to Maltese during pretraining and fine-tuning with instructions is the most important factor influencing performance. Additionally, it analyzes the balance between fine-tuning and prompting, highlighting that while fine-tuning requires a higher initial cost, it offers better performance and lower inference costs. The research emphasizes the need for more inclusive linguistic technologies for low-resource languages and recommends considering more traditional language modeling approaches.
Summary of strengths: here are several important reasons why this article would be of interest to readers and justifies its publication:
- It presents a new evaluation dataset covering 11 discriminative and generative NLP tasks for Maltese, a low-resource language. This contribution is crucial as it facilitates the evaluation and development of language technology for this language.
- The paper provides a thorough evaluation of a wide range of LLMs (exactly 47 models) in a low-resource language, which is significant given the limited attention low-resource languages typically receive in the context of LLMs.
- The experiments reveal that many LLMs perform poorly in Maltese, especially in generative tasks, and that their performance is often worse than smaller models fine-tuned specifically for these tasks.
- The work analyzes how prior exposure to Maltese during pretraining (PT) and fine-tuning with instructions (IT) significantly affects model performance, with results suggesting that this exposure is a key factor.
- It examines the pros and cons of using prompting (zero-shot and one-shot) compared to fine-tuning smaller models, highlighting that while fine-tuning has a higher initial cost, it generally provides better performance and lower inference costs.
- The paper includes an analysis of the computational efficiency of prompting and fine-tuning, providing valuable insights for researchers with limited resources.
Summary of weaknesses: Although this article presents valuable contributions, there are some concerns that could lead to prioritizing other high-quality works for publication:
- The article itself acknowledges that the coverage of Maltese-trained models was very limited, which could restrict the generalization of conclusions about the importance of specific exposure to Maltese during training.
- Due to computational limitations, the study only considered models with no more than 15 billion parameters. Larger and more recent LLMs might exhibit different behavior in Maltese.
- The study did not include commercial models due to evaluation costs and challenges. Since many users and researchers use commercial models, a study including them could offer a more complete and practical perspective.
- While the introduction of a new benchmark for Maltese is a contribution, the authors mention several limitations in the datasets used, including the small amount of data in some tasks, the limited variety of tasks (mainly topic classification and machine translation), and the narrow domains of the data.
- The paper mentions that prompt optimization for each model was not thoroughly explored, relying mainly on existing or translated templates.
- The analysis of the impact of Maltese training was mainly treated as a categorical variable, without considering the amount or proportion of Maltese tokens in the training data, nor the quality of the datasets or instruction-tuning processes (like RLHF or DPO).
- The authors acknowledge the possibility that the models may have been exposed to the benchmark datasets during their training, which could inflate the scores obtained.
Comments, suggestions and typos: This work is a very comprehensive study that covers a large number of models and resources focused on Maltese. Still, I have some suggestions:
- In line 38, "a few examples (in context learning)" appears. I think the strategy known as "in-context learning" is better known in the scientific community as "Few-Shot Learning" or FSL.
- In line 119, it says "but for generative task," and I think it should be "discriminative task."
- In line 142, it says "report the following" followed by a period. I think it should be replaced with a colon and the items listed in the text, such as: "Report the following: (1) For Sentiment...."
- In line 167, it says "each dataset we train a separate model." Why not try to train a single model with different tasks, allowing the model to learn about different tasks at once?
- In table 1, the text in the "Type" column is not centered.
- In lines 213-215, it says "Model performance is much more competitive on discriminative task than on generative," but the results are more similar in generative tasks.
- Sometimes the text repeats the same expressions too frequently. For example, "much more" in lines 213 and 216, or "even though" in lines 378 and 380.
- When analyzing the number of shots between the experiments (Section 3.2), I think the conclusions would be clearer if there were other experiments changing the number of shots (including zero and one, I suggest adding 2, 3, or 4).
- Figures with bar charts should include numerical labels or a table where readers can clearly observe the differences between each experiment.
- Between lines 423-424, an equation appears without proper equation formatting.
Overall assessment scores: 3.5